Lustre tip

Since I had a hard time finding this… If you’re setting up Lustre on a system with multiple interfaces, it seems to default to picking eth0. If you’re looking to route traffic over another interface instead (such as an internal network, a VPN, etc.) you’ll need to make a tweak on your MGS. In /etc/modprobe.conf (or equivalent file, depending on your distro), add:

options lnet networks=tcp(eth1)

Obviously, substitute eth1 for whatever your applicable interface is. Obviously, you’ll need to reboot for this to take effect. (There may be a way to change it without a reboot using the lctl tool, but I didn’t find one.)

Failure to make this change will result in OST mounts failing like so:

mount.lustre: mount /dev/lustre/ost1 at /opt/lustre/ost1 failed: Input/output error
Is the MGS running?

This is because, having selected eth0 as its only network ID, it refuses connections from other hosts (even though the actual daemon binds to 0.0.0.0… a little odd, but, whatever). The appropriate snipped from dmesg:

LustreError: 120-3: Refusing connection from 192.168.0.100 for [email protected]: No matching NI
LustreError: 4695:0:(socklnd_cb.c:1714:ksocknal_recv_hello()) Error -104 reading HELLO from 192.168.0.100
LustreError: 11b-b: Connection to [email protected] at host 192.168.0.100 on port 988 was reset: is it running a compatible version of Lustre and is [email protected] one of its NIDs?

Hopefully this will help solve someone else’s headaches 🙂

Tagged , , . Bookmark the permalink.

Comments are closed.