Teaming NICs in ESX - configuration

This article is continuation of Teaming NICs in ESX - how to choose a switch?

VMware ESX Server's default one-way load balancing of outgoing traffic is quite sufficient feature for applications (VMs) sending major amount of data to multiple network clients while only receiving small size transfers. That's a common case for web servers or mail clients.

There are applications however, which also need the pipe from network clients to ESX - in the opposite direction - to be broad enough. For them you'll need to configure both physical and virtual switch.

What effect to expect?

Since the two-way balancing policy being described in this article is IP-based, the ESX server and your physical switch will balance all trafic through different ports for each IP address pair (client-server). It means you'll get no throughput increase for a single client connected through one connection to a server.

For multiple connections however, ESX load balancing with switch support described in this article ensures there will be all interfaces evenly distributed in both directions. Such setting can bring a positive effect on common VM services like file serving, email access, database access etc.

There is a separate set of people seeking for ESX port trunking - users of iSCSI SAN storage. I'll probably create a separate article about balancing iSCSI on ESX, just leave me a comment to remind me to do so if you are eager for it ;-). Generally said, the effect on iSCSI traffic depends on which vendor's array you use. For now, I can say you can reach aggregated throughput with LeftHand and iStor arrays.

Configure the ESX virtual switch - the GUI way

Configuring the ESX side teaming is as easy as placing some (four, in our example) physical NIC ports in a single virtual Switch and setting a balancing policy to that switch.

1. Assign few NIC ports to a virtual switch. You can do that in Configuration tab, then Networking, select a vSwitch you wish to setup, and Properties/Network Adapters.Click the Add button and select NICs you wish to append to the vSwitch. Confirm all the opened windows.
2. Set the load balancing policy. At the vSwitch Properties choose the vSwitch, click Edit button.Choose NIC Teaming tab, set Load Balancing field to "Route based on ip hash". There are other policies available, but no other seem to be working together with the switch-level trunking. Close all windows.

Configure the ESX virtual switch - the CLI way

All the ESX balancing steps can be done through the Service Console too. It's worth to know the command line way in case a lost network connection to ESX is the only effect of your performance tunning ;-). In such case, the winners have a DRAC or ILO or some other remote management tool to gain the console access.

1. Start with an overview of physical NICs available.
   

   [root@esxcoma root]# esxcfg-nics -l
   Name    PCI      Driver      Link Speed    Duplex MTU    Description
   vmnic0  05:00.00 bnx2        Up   1000Mbps Full   1500   Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
   vmnic1  09:00.00 bnx2        Up   1000Mbps Full   1500   Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
   vmnic3  0f:03.00 e1000       Up   1000Mbps Full   1500   Intel Corporation 8254NXX Gigabit Ethernet Controller
   vmnic4  0f:03.01 e1000       Up   1000Mbps Full   1500   Intel Corporation 8254NXX Gigabit Ethernet Controller
   vmnic2  0c:00.00 cxgb3       Up   10000MbpsFull   1500   Chelsio Communications Inc T310 Single Port 10Gb Ethernet Controller
   

2. List the existing virtual switches and their uplinks.

[root@esxcoma root]# esxcfg-vswitch -l
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       64          8           64                1500    vmnic0

PortGroup Name      VLAN ID  Used Ports  Uplinks
VM Network          0        4           vmnic0
Service Console     0        1           vmnic0

3. Add the otherNICs to existing vSwitch with command esxcfg-vswitch. For example, to add vmnic1 to switch vSwitch0 you do:
   

   [root@esxcoma root]# esxcfg-vswitch -L vmnic1 vSwitch0
   

4. Refresh the ESX networking stack.
   

   [root@esxcoma root]# vimsh -n -e "/internalsvc/refresh_network"

5. Set the balancing policy.
   

   [root@esxcoma root]# vimsh -n -e "hostsvc/net/vswitch_setpolicy --nicteaming-policy loadbalance_ip vSwitch0"

Configure the physical switch - HP ProCurve 2900

It's perhaps impossible to write a general tutorial for all sorts of switches. However, it's quite easy to translate this article to your vendor's speech when you know what to look for. So, if you are not using Cisco, then look in your guide for the term trunking or non-protocol trunk. For Cisco, look for EtherChannel and you might also find the guide at blog.scottlowe.org useful.
I can share with you how I configured HP ProCurve 2900-24G:

1. Log into the switch console.
2. Change to config mode:
   

   ProCurve Switch 2900-24G# configure
   ProCurve Switch 2900-24G(config)#

3. Display any existing trunks:
   

   ProCurve Switch 2900-24G(config)# show trunk
   
   Load Balancing
   
   Port  Name                             Type       Group Type
   ---- + -------------------------------- --------- + ----- -----
   
   ProCurve Switch 2900-24G(config)#

4. Configure a static trunk for ports 15, 16, 17 and 18:
   

   ProCurve Switch 2900-24G(config)# trunk ethernet 15-18 trk1 trunk

5. ...and verify the trunk has been created:
   

   ProCurve Switch 2900-24G(config)# show trunks
   
   Load Balancing
   
   Port  Name                             Type       Group Type
   ---- + -------------------------------- --------- + ----- -----
   15                                     100/1000T  Trk1  Trunk
   16                                     100/1000T  Trk1  Trunk
   17                                     100/1000T  Trk1  Trunk
   18                                     100/1000T  Trk1  Trunk
   
   ProCurve Switch 2900-24G(config)#
   

Plotify the result

Without being sponsored by HP I need to say I'm really happy with their web GUI feature displaying current port load of all switch ports. It's a great tool to ensure the load balancing works.

So, to conclude our work look at the following picture displaying iSCSI traffic between LeftHand SAN/IQ (active ports on left) and a VM running on ESX (active ports on right):