Lustre tip

Since I had a hard time finding this… If you’re setting up Lustre on a system with multiple interfaces, it seems to default to picking eth0. If you’re looking to route traffic over another interface instead (such as an internal network, a VPN, etc.) you’ll need to make a tweak on your MGS. In /etc/modprobe.conf (or equivalent file, depending on your distro), add:

options lnet networks=tcp(eth1)

Obviously, substitute eth1 for whatever your applicable interface is. Obviously, you’ll need to reboot for this to take effect. (There may be a way to change it without a reboot using the lctl tool, but I didn’t find one.)

Failure to make this change will result in OST mounts failing like so:

mount.lustre: mount /dev/lustre/ost1 at /opt/lustre/ost1 failed: Input/output error
Is the MGS running?

This is because, having selected eth0 as its only network ID, it refuses connections from other hosts (even though the actual daemon binds to… a little odd, but, whatever). The appropriate snipped from dmesg:

LustreError: 120-3: Refusing connection from for [email protected]: No matching NI
LustreError: 4695:0:(socklnd_cb.c:1714:ksocknal_recv_hello()) Error -104 reading HELLO from
LustreError: 11b-b: Connection to [email protected] at host on port 988 was reset: is it running a compatible version of Lustre and is [email protected] one of its NIDs?

Hopefully this will help solve someone else’s headaches 🙂