Symptoms

You notice that network performance between VMs in Compute cluster  is low: private networks based on physical networks with MTU 1500 only have 1/3 of network bandwidth (i.e. ~3.5Gb at 10Gb network).

Cause

By default, MTU is set to 1500, because different hardware has different maximum MTU supported.

Solution

Increase MTU manually.

1. Make sure that network switch and network card support MTU more than 1500.
2. On each node of the cluster, increase MTU on host physical interfaces up to the maximum supported by network equipment.
   
   We recommend applying this setting before the cluster deployment due to the fact, that during cluster creation private network will be created and additional steps will be required.
   • Either increase the MTU during the OS installation procedure in network interface settings.
   • Or increase the MTU on the installed node using CLI by issuing:
     # ip link set mtu <value> dev eth0
     where <value> is the maximum supported MTU
     For example, to set MTU to 9000, run:
     # ip link set mtu 9000 dev eth0
     You can then verify that the new MTU has taken effect by running:
     # ip link show dev eth0
3. Update the interface’s configuration file so that this change persists the next time the server is rebooted.
   To edit the MTU for the eth0 interface, do one of the following:
   • either open the /etc/sysconfig/network-scripts/ifcfg-eth0 file for editing and a new line:
     MTU=<value>
     where <value> is the maximum supported MTU
   • or increase the MTU for the interface using the Web Management panel: open Infrastructure -> Nodes -> select the node -> Networks -> choose the NIC, that you are using for the private network -> Click Configure on the right and change MTU setting  -> Click Done to apply changes.

If a private network had been created already (cluster had been deployed), private network MTU change is also required.

To find out network ID use the following command:
# vinfra service compute network list --long -c id -c type | awk '$(NF-1)~/vxlan/ {print $2}'
MTU should be changed for all these networks.

To check current private network MTU run:
# openstack --insecure network show <networkID>

Change MTU for current network:
# openstack --insecure network set <networkID> --mtu <new value>
where <new value> is <value>-50
For example, if you changed MTU to 9000, you need to set MTU 8950 on private network:
# openstack --insecure network set <networkID> --mtu 8950

Apply these changes on all nodes in the cluster.

To speed up the changes you may execute this in cycle, e.g:
for b in $(vinfra service compute network list --long -c id -c type | awk '$(NF-1)~/vxlan/ {print $2}'); do openstack --insecure network set $b --mtu 8950; done