Configuring Multiple Interfaces and Multiple Default Routes in Linux

Scenario: Multiple network interfaces in your server and each connected to different network and getting the IP via DHCP
Objective: Make all IP pingable and accessible remotely e.g. SSH
Solution: Configure Gateway Routing or setting up multiple default routes for each interfaces

Initially when you connect different links/network (regardless it’s in the same subnet or not) for each of the interfaces, it will end up only one interface is reachable. Why? Because by default, it can only have one default route or gateway on a system.

Example: 5 NIC in 1 Linux Server
Noted: Tested in Redhat based Linux (CentOS,Fedora). Assuming the IP dynamically assigned by the DHCP server:

eth1=ip:192.168.10.100/24 gw:192.168.10.1
eth2=ip:192.168.20.100/24 gw:192.168.20.1
eth3=ip:192.168.30.100/24 gw:192.168.30.1
eth4=ip:192.168.40.100/24 gw:192.168.40.1
eth5=ip:192.168.50.100/24 gw:192.168.50.1

1. Set eth1 as the default route and disable it from other interfaces
a. add the line “DEFROUTE=yes” (without the quote) to /etc/sysconfig/network-scripts/ifcfg-eth1
b. add the line “DEFROUTE=no” (without the quote) to /etc/sysconfig/network-scripts/ifcfg-eth2  .. to ifcfg-eth5

2. Restart the network and the IP routing table should be something like this:

[tux@freelinux ~]$ netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
192.168.20.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
192.168.30.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
192.168.40.0 0.0.0.0 255.255.255.0 U 0 0 0 eth4
192.168.50.0 0.0.0.0 255.255.255.0 U 0 0 0 eth5
0.0.0.0 192.168.10.1 0.0.0.0 UG 0 0 0 eth1

From the example above, the only pingable/reachable would be the eth1 IP only as it carries the default gateway.

3. Setup additional routing table
a. Edit the file  /etc/iproute2/rt_tables and set the eth1 with preference 1, and eth2 with 2, and so on and so forth. Any name to represent it can work, this is basically creating a policy in the routing table that can be called later.  Sample content of that file will be something like:
[tux@freelinux ~]$ cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
1 ilove
2 free
3 linux
4 tutorials

b. Configure the routing table. Firstly, configure the 192.168.20.0/24 to tell that it can be reached in “ilove” and to find its gateway. Secondly, set interface’s default gateway
ip route add 192.168.20.0/24 dev eth2 src 192.168.20.100 table ilove
ip route add default via 192.168.20.1 dev eth2 table ilove

c. Set the policy routes for the system to know when to use that new routing table
ip rule add from 192.168.20.100 /32 table ilove
ip rule add to 192.168.20.100/32 table ilove

Do the same thing for eth3-5. These will be the configuration for the rest:
for eth3:
ip route add 192.168.30.0/24 dev eth3 src 192.168.30.100 table free
ip route add default via 192.168.30.1 dev eth3 table free
ip rule add from 192.168.30.100 /32 table free
ip rule add to 192.168.30.100/32 table free

for eth4:
ip route add 192.168.40.0/24 dev eth4 src 192.168.40.100 table linux
ip route add default via 192.168.40.1 dev eth4 table linux
ip rule add from 192.168.40.100 /32 table linux
ip rule add to 192.168.40.100/32 table linux

for eth5:
ip route add 192.168.50.0/24 dev eth5 src 192.168.50.100 table tutorials
ip route add default via 192.168.50.1 dev eth5 table tutorials
ip rule add from 192.168.50.100 /32 table tutorials
ip rule add to 192.168.50.100/32 table tutorials

Check: Ping and ssh all the IP address see if it’s already reachable
Verification (sample output)

[tux@freelinux ~]$ ip rule show
0: from all lookup local
32732: from all to 192.168.50.100 lookup tutorials
32733: from 192.168.50.100 lookup tutorials
32734: from all to 192.168.40.100 lookup linux
32735: from 192.168.40.100 lookup linux
32736: from all to 192.168.30.100 lookup free
32737: from 192.168.30.100 lookup free
32738: from all to 192.168.20.100 lookup ilove
32739: from 192.168.20.100 lookup ilove
32740: from all lookup main
32741: from all lookup default

Simple explanation: Rule 32732 means that all traffic going to eth5 will use the “tutorials” routing table.  Rule 32733 means that traffic coming from eth5 will use the”tutorials” routing table. And so on, and so forth

[tux@freelinux ~]$ ip route show table all

192.168.20.0/24 dev eth2 table ilove scope link src 192.168.20.100
default via 192.168.20.1 dev eth2 table ilove
192.168.30.0/24 dev eth3 table free scope link src 192.168.30.100
default via 192.168.30.1 dev eth3 table free
192.168.40.0/24 dev eth4 table linux scope link src 192.168.40.100
default via 192.168.40.1 dev eth4 table linux
192.168.50.0/24 dev eth5 table tutorials scope link src 192.168.50.100
default via 192.168.50.1 dev eth5 table tutorials
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.100
default via 192.168.10.1 dev eth1

Enjoy! Cheers!