Comware

On a 3500yl L3 switch, we plan to connect (ESX) server interfaces to one VLAN at 1 Gb/s and workstation interfaces to a second VLAN at 100 Mb/s.

To optimize for inter-server performance would mean setting jumbo frame MTU size on the server VLAN, and normal size MTU on the user VLAN. However, this will force the switch to fragment server-workstation traffic, presumably with a performance hit.

What is best-practice for deciding MTU size here?

If your workstations are set to 1500B, TCP will negotiate with the server at the connection setup for the frame size.

But if you set the workstations and the server to 9000B and the router can only handle 1500B traffic you will have major slowdowns.

In case anyone else is evaluating this question, here are some test results:
http://www.boche.net/blog/index.php/2011/01/24/jumbo-frames-comparison-testing-with-ip-storage-and-vmotion/

Evan

If I read the docs correctly, MTU size on a 100 Mb/s interface on that switch is limited to 1500 bytes.

Do you have a ref that suggests otherwise?

I believe you will want to quantify the percentage of packets that will actually utilize the larger size frame. At some point, if your files are not large enough, the larger packet size may actually increases latencies.

Here is some expert advice that may be beneficial.
http://groups.google.com/group/virtualizationbunker/browse_thread/thread/a8b37b82f8ded344