Wednesday, January 12, 2005

Sharing Network Adapters and Virtual Networks

I blog this so I don't forget it later....
Here is the link:
Sharing Network Adapters and Virtual Networks
And, in case they change the KB, here is the text:

Sharing Network Adapters and Virtual Networks

In many ESX Server configurations, there is a clear distinction between networking resources used by the virtual machines and those used by the service console. This may be important for security reasons, for example — isolating the management network from the network used by applications in the virtual machines.

However, there may be times when you want to share resources, including physical network adapters and virtual networks.

This technical note provides instructions on sharing in both directions — making the virtual machines' resources available to the service console and allowing virtual machines to share the network adapter used by the service console.

This sharing is made possible by the vmxnet_console driver, which is installed with the service console.

Caution: We recommend that only advanced users make these configuration changes. The steps below are easier for someone who is familiar with administering a Linux system.

Note: If you accidentally bring down the local loopback interface while you are reconfiguring network devices, the VMware Management Interface does not function properly. To bring it back up, use the command ifconfig lo up.

Allowing the Service Console to Use the Virtual Machines' Devices

All network adapters used by virtual machines (that is, assigned to the VMkernel) and virtual networks can be made accessible to the service console. Virtual networks — identified as vmnet_ on the Edit Configuration page of the VMware Management Interface — provide high-speed connections among virtual machines on the same physical server.

To give the service console access to VMkernel network adapters and virtual networks, you must install the vmxnet_console module. When you install it, you provide a list of VMkernel network adapters and virtual networks that the vmxnet_console module should attach to. For example, if the VMkernel had an adapter named vmnic1 and a virtual network named vmnet_0 and you wanted to provide access to them from the service console, you would use the following command to install the vmxnet_console module.

insmod vmxnet_console devName="vmnic1;vmnet_0"

The devName parameter is a comma-separated list of names of VMkernel network adapters and virtual networks.

When you install the module, it adds the appropriate number of eth devices on the service console in the order that you list the VMkernel network adapter and virtual network names after the devName parameter. In the example above, if the service console already had a network adapter named eth0, when you load vmxnet_console with vmnic1 and vmnet_0; vmnic1 is seen as eth1 on the service console and vmnet_0 is seen as eth2.

Once the eth devices are created on the service console, you can bring the interfaces up in the normal manner. For example, if you want the service console to use IP address for the network accessed via the vmnic1 adapter, use the following command:

ifconfig eth1 up

If you want an easy way to see which eth devices are added by the insmod command, you can add the tagName parameter to the insmod command, as shown in this example:

insmod vmxnet_console devName="vmnic1;vmnet_0" tagName=

In this case the vmxnet_console module adds the names of each of the eth devices that it created to /var/log/messages. Each message begins with the string .

To figure out the names of the devices that were added, use this command:

grep /var/log/messages

Starting Shared VMkernel Network Adapters and Virtual Networks when the Service Console Boots

There are two ways you can configure the service console to start VMkernel network adapters when the service console boots. The simpler case involves sharing a network adapter other than eth0. Sharing eth0 is more complicated and is described later.

Continuing with the example from the previous section, you can append the following lines to /etc/rc.d/rc.local:

insmod vmxnet_console devName="vmnic1;vmnet_0"
ifconfig eth1 up
ifconfig eth2 up

Another method is to set up the files /etc/sysconfig/network-scripts/ifcfg-eth1 and /etc/sysconfig/network-scripts/ifcfg-eth2 with the appropriate network information. And be sure the ONBOOT= line is ONBOOT=yes. The ifcfg-eth1 file for this example would be:


In this case, the lines you add to /etc/rc.d/rc.local would be:

insmod vmxnet_console devName="vmnic1;vmnet_0"
ifup eth1
ifup eth2

Sharing the Service Console's Network Adapter with Virtual Machines

Caution: If you intend to share the adapter that is eth0 on the service console, be careful as you implement the following steps. In order to configure ESX Server initially, you need to have a network connection. Once the initial configuration is set, you make several changes. At one point in the process, there is no network connection to the service console, and you must work directly at the server.

When you first install and configure ESX Server, the VMkernel is not loaded, so the service console needs to control the network adapter that is eth0. When you configure ESX Server, assign the adapter that is eth0 to the service console.

Once you have completely configured ESX Server properly and rebooted the machine, the VMkernel is loaded. At that point, you need to take the following steps:

Edit /etc/modules.conf and comment out the line that refers to alias eth0.
If the original line is
alias eth0 e100
edit it to be
# alias eth0 e100

This disables eth0 on the service console when it boots.

Use the VMware Management Interface to reconfigure the server. Log in as root and go to http:///pcidivy, then click the Edit link for the configuration you want to change. Find the table row that lists the Ethernet controller assigned to the console and click the radio button in the Virtual Machine column to reassign it.
Click Save Configuration, then reboot the machine when prompted.

When the machine reboots, no network adapter is assigned to the service console, so you must do this step at the server.
Add the appropriate lines to /etc/rc.d/rc.local. For example, if eth0 is the only network adapter that you intend to share between the VMkernel and the service console, and if it is named vmnic0 in the VMkernel, you add the lines:

insmod vmxnet_console devName="vmnic0"
ifup eth0

If you are unsure what name the VMkernel has assigned to the network adapter that formerly was eth0 in the service console, you can determine its name using the findnic program (see The VMkernel Network Card Locator).

Note: The adapter you wish to share may be assigned to an adapter bond. If so, you need to specify the bond name, of the form bond, instead of the adapter name. See Finding Bonds and Adapters in the Service Console for instructions on how to find a bond name.

The next time you reboot the system, the network adapter is shared by the service console and the virtual machines.
To begin sharing the network adapter without rebooting the system, you can manually issue the same commands you added to /etc/rc.d/rc.local:

insmod vmxnet_console devName="vmnic0"
ifup eth0

Using Virtual Switches

ESX Server allows you to create abstracted network devices called virtual ethernet switches. Each virtual switch is a network hub that can be used by virtual machines. A virtual switch can route traffic internally between virtual machines, or link to external networks.

Virtual switches can be used to combine the bandwidth of multiple network adapters and balance communications traffic among them. They can also be configured to maintain persistent network connections despite link failures for individual adapters.

A virtual switch models a physical ethernet switch. A virtual switch contains 32 logical ports. You can connect one network adapter of a virtual machine to each port.

Each virtual switch can also have one or more port groups assigned to it. For more information on port groups, see Creating Port Groups.

Choosing a Network Label

ESX Server uses network labels to represent network connections to virtual machines. The network label is intended to be a functional descriptor for the network connection. ESX Server represents both virtual switches and port groups to virtual machines by assigning them a network label.

You can only change the network label for a switch when it is not being used by a virtual machine.

Binding Physical Adapters

You can group physical adapters by "binding" them together. This is the functional equivalent for NIC teaming in ESX Server. Certain options you can configure through the Service Console refer to grouped adapters as a "bond".

You should bind together similar physical adapters whenever possible. ESX Server uses only features or capabilities common to all adapters when defining the functionality of a bonded switch. For example, ESX Server can use a hardware acceleration feature for a bond only if all adapters in the bond include that feature.

Hardware acceleration features supported by ESX Server include:

VLAN tag handling
Checksum calculations
TCP Segmentation Offloading
Binding together identical models of physical adapters ensures that all features of the adapter can be used by ESX Server.

When you choose a network connection for a virtual machine, ESX Server links it to the associated virtual switch. The operation of the virtual machine depends on the configuration of its network connection. Thus, you cannot bind or detach physical adapters while a virtual switch is being used by a virtual machine.

You can bind up to ten physical adapters to each virtual switch.

Finding Bonds and Adapters in the Service Console

When you bind together adapters in a virtual switch, ESX Server assigns a bond number identifying the new logical grouping of physical adapters. You will need to know the bond number in order to configure the bond options described below. Check /etc/vmware/netmap.conf to determine the bond number assigned to a virtual switch.

You may also need to know the device name ESX Server assigns to a physical adapter. Certain options use the device name to designate a specific adapter. ESX Server defines device names with the string vmnic, for which is the same adapter number displayed for an adapter in the Management Interface. For example, the physical adapter identified as Outbound Adapter 1 would have the device name vmnic1.

You can also determine the device name by searching /etc/vmware/devnames.conf for the name definition. Note the PCI bus address of the adapter in the Management Interface and search for the corresponding name definition. For example, to find the device name for the adapter at PCI 2:4.0:

Log into the Service Console.
Search /etc/vmware/devnames.conf:
$ grep 2:04.0 /etc/vmware/devnames.conf

002:04.0 nic vmnic0

The device name is vmnic0.

Creating a Virtual Switch

You can find basic instructions for creating and modifying virtual switches in Changing Network Connections.

Note: The configuration options described below are used for optimizing virtual switches for complex operating conditions. You can create and use a virtual switch without changing these options for most configurations.

Choosing a Load Balancing Mode

You can choose one of three modes for determining how ESX Server distributes traffic among the network adapters assigned to a virtual switch:

MAC address balancing
IP address balancing
You select the load balancing mode by setting the load_balance_mode option for a virtual switch. All options for virtual switches are defined in /etc/vmware/hwconfig, which you can modify through the Service Console.

MAC address load balancing distributes networking traffic based on the MAC hardware addresses of the source network adapters. Select MAC address balancing by setting load_balance_mode to out-mac.

Note: MAC address balancing is the default load balancing mode in ESX Server.

IP address load balancing distributes networking traffic based on IP addresses. ESX Server distributes network traffic not using the IP protocol on a fixed-volume sequential cycle. Select IP address balancing by setting load_balance_mode to out-ip.

Standby mode designates a specific adapter to use as the primary connection. Use Standby mode for redundant connection switches, as described in the next section.

This example describes how to set the load balancing mode for bond1 to IP address load balancing:

Log into the Service Console as root.
Edit /etc/vmware/hwconfig.
Define the load balancing mode for bond1:
nicteam.bond1.load_balance_mode = "out-ip"

If you previously defined the option for this switch, just change the current mode value to out-ip.

Save the file and close it.

Configuring the Bond Failure Mode

You can select one physical adapter to be the primary network connection for a virtual switch. In this configuration, ESX Server routes all traffic through the primary adapter and reserves the other adapters in case of connection failure. This type of redundant connection switch is defined as using a "failover" configuration.

Select a primary adapter by setting the home_link option for a virtual switch:

Log into the Service Console as root.
Edit /etc/vmware/hwconfig.
Define the primary adapter. For example, to choose vmnic2 for bond1:
nicteam.bond1.home_link = "vmnic2"

If you previously defined the option for this switch, just change the current mode value to vmnic2.

Save the file and close it.
Note: Designating a primary link for a virtual switch overrides the load balancing mode. If you set the home_link option, ESX Server ignores the value of load_balance_mode.

ESX Server monitors the primary link for physical connection failures. When the primary adapter loses contact, ESX Server transfers the network traffic to one of the secondary adapters while continuing to monitor the primary adapter. When ESX Server detects that the physical connection of the primary link has been restored, it transfers the connection for the virtual switch back to the primary.

This basic failure detection mode passively monitors an adapter for loss of physical connection to an external switch. You can configure ESX Server to actively search for network failures using beacon monitoring.

Using Beacon Monitoring

The beacon monitoring feature broadcasts beacon packets on the external network linked to the server to detect distributed connection failures. ESX Server issues beacon packets from one adapter addressed to other adapters assigned to a virtual switch. By monitoring beacon reception, the server can determine the state of connections in a multi-point network route. You can configure beacon monitoring for each virtual switch and for the entire server.

Beacon monitoring is designed to be used in configurations where the multiple adapters assigned to a virtual switch are connected to more than one external switch. Physical link monitoring only indicates whether an adapter is communicating with one external switch. Beacon failures can detect connection failures between external switches or routing errors among switches in a distributed network domain.

ESX Server uses beacon monitoring as a variable indicator of network connection failure. The server indicates a connection loss after it fails to receive a set number of broadcast beacons in succession. Only when the number of failed beacons exceeds the failure threshold will the server identify a link as disconnected and switch to another adapter.

By default, the beacon failure threshold is set to zero for each virtual switch. You can enable beacon monitoring by setting the failure threshold to two or greater.

ESX Server also allows you to determine the frequency with which it issues beacons. The rate at which the server broadcasts beacons, in conjunction with the failure threshold, determines the total monitoring interval before the server identifies a link as isolated:

Beacon Interval (in seconds) X Beacon Failure Threshold = Total Beacon Failure Interval

You set the failure threshold for an individual switch with the switch_failover_threshold option. This example describes how to set the failure threshold for bond1 to 2 beacons:

Log into the Service Console as root.
Edit /etc/vmware/hwconfig.
Set the beacon failure threshold for bond1:
nicteam.bond1.switch_failover_threshold = "2"

Save the file and close it.
ESX server broadcasts beacons with the same frequency for all switches. The SwitchFailoverBeaconInterval option sets this value. The server also defines an overall failure threshold for all switches with the SwitchFailoverThreshold option, but switch_failover_threshold overrides this value for each individual switch.

You can set the values of the SwitchFailoverBeaconInterval and SwitchFailoverThreshold options in the Advanced Settings panel of the Management Interface. See Changing Advanced Settings for details.

Beacon monitoring can cause false indications of network connection failure. External switches may trap beacon packets, causing ESX Server to declare a switch failure for a connection that is functioning normally. When the the server switches to a secondary link, traffic from the primary may still be transmitted because the connection has not actually failed. This can result in an external switch receiving duplicate packets from both links.

Note: ESX Server uses beacon monitoring as a secondary method to detect network failures. When the server detects a physical link failure for the primary adapter, it will switch to a secondary adapter without regard to whether beacon monitoring indicates a failed connection.

Using Beacon Monitoring with a Port Group

You can enable beacon monitoring for a port group with the SwitchFailoverBeaconVlanID option. If you select a port group with this option, ESX Server broadcasts monitor beacons tagged with the VLAN ID of that port group. This effectively limits the server to monitoring connection failures within the external VLAN.

You can set the value of the SwitchFailoverBeaconVlanID in the Advanced Settings... panel of the Management Interface. See Changing Advanced Settings for details.

Note: The value of SwitchFailoverBeaconVlanID applies to all virtual switches. You cannot choose a separate VLAN to monitor for each switch.

Configuring External Network Switches

IP Load Balancing — With this load balancing mode enabled, ESX Server may present duplicate MAC addresses to an external network switch. The external switch should be set static 802.3ad (EtherChannel) mode to avoid external routing errors.
SwitchFailoverBeaconEtherType — This option sets the Ether type of monitor beacons. You may wish to change this value so that your external switches correctly handle monitor beacons.
Beacon Monitoring with Multiple Switches — All external switches connected to a virtual switch using beacon monitoring must be within the same network broadcast domain.
Spanning-Tree Protocol — If an adapter loses the physical connection to an external switch that is using the Spanning-Tree Protocol, the switch may induce a delay in reconnecting the link while it applies the protocol to check for duplicate active connections. ESX Server can only detect that the link has been physically restored, but not that the port is blocked by the Spanning-Tree check.
Portfast Mode — You can use the Portfast mode to reduce errors caused by Spanning-Tree checks. If you cannot disable the Spanning-Tree Protocol for an external switch, configure the ports connected to the server to use Portfast mode. This reduces Spanning-Tree delays, resulting in fewer false indications of link failures.


If, while booting your virtual machine, you see an error message stating that the Ethernet device cannot be detected, then check the following:

Network Connections page — Be sure that the correct physical adapters are assigned to a bond
VM Configuration page — Be sure the correct bond is selected for the specified Ethernet device and that the selected vmnic is not already assigned to a bond device or already in use.
Make the appropriate change(s), then reboot your virtual machine to see if the error message persists.