The cheapest 10Gigabit Network Switch available for sale on newegg.com goes for $749 NETGEAR XS708Ev2 ProSAFE 8-Port 10-Gigabit Ethernet Web Managed (Plus) Switch. The cheapest 10Gigabit Network Card however is the Mellanox ConnectX-2 10GbE PCIe 2.0 x8 Low Profile Network Interface Card, MNPA19-XTR which sells for $24. A cable to connect the two goes for $39.99, Ipolex for Cisco SFP-H10GB-CU2M,10GBASE-CU Direct Attach Copper Cable, Twinax Cable, Passive,2-Meter.
So now I have 2 linux machines, each of them are running Fedora 24 and they have 1 of the Mellanox cards mentioned above and are connected to each other via the Direct Attach Copper Cable. How do we set this up?
Well, my local area network is using a Class C network of 192.168.0.x. So I will configure each Mellanox card with a Class A network of 10.x.x.x. How do I do this? Well, first I have to identify the Mellanox Card. You can do this with the following command:
john@localhost ~]$lspci | grep Mellanox
The output of this command should look like the following and will give you the PCI bus address, which in this case is 02:00.0.
02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
Now you can use this PCI bus address to get the MAC or hardware address of the card,
john@localhost ~]$sudo lspci -vvv -s 02:00.0 | grep Device\ Serial Capabilities: [148 v1] Device Serial Number 00-02-c9-03-00-55-08-44
In my case, the mac address is 00-02-c9-03-00-55-08-44. Funny, this number has 16 octets as opposed to the normal 12 octets. After running ifconfig, I see that the true MAC address is the first three and the last three octets, which is 00-02-c9-55-08-44. To see how Mellanox maps to 00-02-c9, visit Mac Vendor Lookup at: http://www.macvendorlookup.com/
So now we can identify which configuration file is the appropriate one to modify for this network card by examining the output of ifconfig:
john@localhost ~]$ifconfig enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::202:c9ff:fe55:844 prefixlen 64 scopeid 0x20<link> ether 00:02:c9:55:08:44 txqueuelen 1000 (Ethernet) RX packets 189 bytes 21776 (21.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 443 bytes 47771 (46.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.4 netmask 255.255.255.0 broadcast 192.168.0.255 inet6 fe80::be5f:f4ff:feac:12e3 prefixlen 64 scopeid 0x20<link> ether bc:5f:f4:ac:12:e3 txqueuelen 1000 (Ethernet) RX packets 723088 bytes 1093485296 (1.0 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 337046 bytes 23950834 (22.8 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 16 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1 (Local Loopback) RX packets 308 bytes 32354 (31.5 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 308 bytes 32354 (31.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Notice how there is no inet (IPv4) address, but there is a defualt IPv6 address, the Ethernet or MAC or Hardware address does match the output that we saw from lspci. So now we know that the device is assigned network device name of enp2s0. Now if you are using Fedora 24, or a close version, you have to become root and go to the /etc/sysconfig/network-scripts/ directory. Then search each file that starts with ifcfg and contains the MAC address. Note: I only used the first 6 octets, because the first 6 octets are used to identify the manufacturer of a network card and that should be unique to my system since I only have one Mellanox card.
[root@localhost network-scripts]# grep -i 00:02:c9 * ifcfg-Wired_connection_1:HWADDR=00:02:C9:55:08:44
So now I know that I have to modify the file named ifcfg-Wired_connection_1. What I want to do is the following:
- Disable IPv6
- Set an IPv4 IP Address using the Class A network
- Set the network mask corresponding to the Class A network
To disable IPv6, simply make sure the following line exists:
To set the IPv4 IP Address, set the following:
To set the IPv4 network mask:
Now, go to the other linux machine which has the other card, and follow these instructions and set the ip address to 10.0.1.101.
So after I did this, on both machines I took down my network interface by using the “ifdown enp3s0” and “ifdown enp2s0” commands. Then I brought the interfaces back up by using the “ifup enp2s0” and “ifup enp2s0” commands.
Then I tested my connections by pinging computer A from computer B and computer B from computer A. Now normally I would have done more exhaustive tests, but I was staring right at the network card when I saw the activity light turn green at exactly when I saw the ping response, regardless of which computer I was on. Nevertheless, I used ssh to connect to computer A from computer B and vice versa and I verified that I indeed was on each computer by running the ifconfig command.
Now, what about a speed test? I am going to write a quick Python script and let’s see what happens…
The cheapest 10Gigabit Network Switch available for sale on newegg.com goes for $749 NETGEAR XS708Ev2 ProSAFE 8-Port 10-...