How to create a multi-LXC infrastructure using custom NAT and DHCP server

I am creating complex infrastructures with LXC. This time I wanted to create a realistic infrastructure that would simulate a real multi computer infrastructure, and I wanted to control all the components. I know that LXC will provide IP addresses through its DHCP server, and it will NAT the internal IPs, but I want to use my own DHCP server to provide multiple IP ranges. So this time…

I learned how to create a multi-LXC infrastructure using custom NAT router and DHCP server with multiple IP ranges

My virtual container infrastructure will consist of:

  • 1 container that will have 2 network interfaces: 1 to a “public” network (the LXC network) and 1 to a “private” network (to a private bridge).
  • 1 container in a “private network” (with IP in the range of 10.0.0.x)
  • 1 container in other “private network” (with IP in the range of 10.1.0.x)

The containers in the private network will be able to NAT to the outern world through the router.

I am using unprivileged containers (to learn how to create them, please read my previous post), but it is easy to execute all of this using the privileged containers (i.e. sudo lxc-* commands).

Creating the “private network”

First I create a bridge that will act as the switch for the private network:

calfonso@mmlin:~$ sudo su -
root@mmlin:~# cat >> /etc/network/interfaces << EOT
auto privbr
iface privbr inet manual
bridge_ports none
EOT
root@mmlin:~# ifup privbr

Then I will give permissions for my user to be able to add devices to that bridge

(* this is a unprivileged container specific step)

calfonso@mmlin:~$ sudo bash -c 'echo "calfonso veth privbr 100" >> /etc/lxc/lxc-usernet'

Setting the router

Now I will create a container named “router”

$ lxc-create -t download -n router -- -d ubuntu -r xenial -a amd64

And I will edit the configuration to set the proper network configuration. Edit the file $HOME/.local/share/lxc/router/config and modify it to leave it like this one:

# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.include = /usr/share/lxc/config/ubuntu.userns.conf
lxc.arch = x86_64

# Container specific configuration
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536
lxc.rootfs = /home/calfonso/.local/share/lxc/router/rootfs
lxc.rootfs.backend = dir
lxc.utsname = router

# Network configuration
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:9b:1b:70

# Private interface for the first range
lxc.network.type = veth
lxc.network.link = privbr
lxc.network.flags = up
lxc.network.ipv4 = 10.0.0.1/24
lxc.network.hwaddr = 20:00:00:9b:1b:70

# Additional interface for the other private range
lxc.network.type = veth
lxc.network.link = privbr
lxc.network.flags = up
lxc.network.ipv4 = 10.1.0.1/24
lxc.network.hwaddr =20:20:00:9b:1b:70

Now you can start your container, and check that you have your network devices:

calfonso@mmlin:~$ lxc-start -n router 
calfonso@mmlin:~$ lxc-attach -n router 
root@router:/# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:16:3e:9b:1b:70 
 inet addr:10.0.3.87 Bcast:10.0.3.255 Mask:255.255.255.0
 ...
eth1 Link encap:Ethernet HWaddr 20:00:00:9b:1b:70 
 inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0
 ...
eth2 Link encap:Ethernet HWaddr 20:20:00:9b:1b:70 
 inet addr:10.1.0.1 Bcast:10.1.0.255 Mask:255.255.255.0
...
lo Link encap:Local Loopback 
...

DHCP Server

And now, install DNSMASQ to be used as the nameserver and the DHCP server:

root@router:/# apt-get update
root@router:/# apt-get -y dist-upgrade
root@router:/# apt-get install -y dnsmasq
root@router:/# cat >> /etc/dnsmasq.d/priv-network.conf << EOT
except-interface=eth0
bind-interfaces
dhcp-range=tag:if1,10.0.0.2,10.0.0.100,1h
dhcp-range=tag:if2,10.1.0.2,10.1.0.100,1h
dhcp-host=20:00:00:*:*:*,set:if1
dhcp-host=20:20:00:*:*:*,set:if2
EOT
root@router:/# service dnsmasq restart

Setting up as router

I will use iptables to NAT the internal networks. So we’ll need to install iptables:

root@router:/# apt-get install iptables

Then I have to add the following lines to the /etc/rc.local file (just before the exit 0 line)

echo "1" > /proc/sys/net/ipv4/ip_forward

iptables -t nat -A POSTROUTING -s 10.0.0.0/24 ! -d 10.0.0.0/24 -j MASQUERADE
iptables -A FORWARD -d 10.0.0.0/24 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -s 10.0.0.0/24 -i eth1 -j ACCEPT

iptables -t nat -A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
iptables -A FORWARD -d 10.1.0.0/24 -o eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -s 10.1.0.0/24 -i eth2 -j ACCEPT

And now it is time to reboot the container

root@router:/# exit
calfonso@mmlin:~$ lxc-stop -n router 
calfonso@mmlin:~$ lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6 
myunprivilegedcont STOPPED 0 - - - 
router STOPPED 0 - - - 
calfonso@mmlin:~$ lxc-start -n router 
calfonso@mmlin:~$ lxc-attach -n router 
root@router:/# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -s 10.0.0.0/24 ! -d 10.0.0.0/24 -j MASQUERADE
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
root@router:/#

And that’s all for our router.

(*) All these steps would also work for a common server (either physical o virtual) that will act as a router and DHCP server.

Creating the other containers

Now we are ready to create containers in our subnet. We’ll create a container in network 10.0.0.x (which will be named node_in_0) and other container in network 10.1.0.x (which will be named node_in_1).

Creating a container in network 10.0.0.x

First we create a container (as we did with the router)

calfonso@mmlin:~$ lxc-create -t download -n node_in_0 -- -d ubuntu -r xenial -a amd64

And now we’ll edit its configuration file .local/share/lxc/node_in_0/config to set it like this

# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.include = /usr/share/lxc/config/ubuntu.userns.conf
lxc.arch = x86_64

# Container specific configuration
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536
lxc.rootfs = /home/calfonso/.local/share/lxc/node_in_0/rootfs
lxc.rootfs.backend = dir
lxc.utsname = node_in_0

# Network configuration
lxc.network.type = veth
lxc.network.link = privbr
lxc.network.flags = up
lxc.network.hwaddr = 20:00:00:e9:79:10

(*) The most noticeable modifications are related to the network device: set the private intarface in lxc.network.link and set the first octects to the hwaddr to the mask set in the DNSMASQ server (I left the other as those that LXC generated by itself).

Now you can start your container and it will get an IP in the range 10.0.0.x.

calfonso@mmlin:~$ lxc-start -n node_in_0 
calfonso@mmlin:~$ lxc-attach -n node_in_0 
root@node_in_0:/# ifconfig
eth0 Link encap:Ethernet HWaddr 20:00:00:e9:79:10 
 inet addr:10.0.0.53 Bcast:10.0.0.255 Mask:255.255.255.0
 ...

lo Link encap:Local Loopback 
...

Creating a container in network 10.1.0.x

Again, we have to create a container

calfonso@mmlin:~$ lxc-create -t download -n node_in_1 -- -d ubuntu -r xenial -a amd64

And now we’ll edit its configuration file .local/share/lxc/node_in_1/config to set it like this

# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.include = /usr/share/lxc/config/ubuntu.userns.conf
lxc.arch = x86_64

# Container specific configuration
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536
lxc.rootfs = /home/calfonso/.local/share/lxc/node_in_0/rootfs
lxc.rootfs.backend = dir
lxc.utsname = node_in_0

# Network configuration
lxc.network.type = veth
lxc.network.link = privbr
lxc.network.flags = up
lxc.network.hwaddr = 20:20:00:92:45:86

And finally, start the container and check that it has the expected IP address

calfonso@mmlin:~$ lxc-start -n node_in_1 
calfonso@mmlin:~$ lxc-attach -n node_in_1 
root@node_in_1:/# ifconfig
eth0 Link encap:Ethernet HWaddr 20:20:00:92:45:86 
 inet addr:10.1.0.12 Bcast:10.1.0.255 Mask:255.255.255.0
...
lo Link encap:Local Loopback 
...

Ah, you can check that these containers are able to access the internet 😉

root@node_in_1:/# ping -c 3 www.google.es
PING www.google.es (216.58.201.131) 56(84) bytes of data.
64 bytes from mad06s25-in-f3.1e100.net (216.58.201.131): icmp_seq=1 ttl=54 time=8.14 ms
64 bytes from mad06s25-in-f3.1e100.net (216.58.201.131): icmp_seq=2 ttl=54 time=7.91 ms
64 bytes from mad06s25-in-f3.1e100.net (216.58.201.131): icmp_seq=3 ttl=54 time=7.86 ms

--- www.google.es ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 7.869/7.973/8.140/0.119 ms

(*) If you can check that our router is effectively the router (I you don’t trust me 😉 ), you can check it

root@node_in_1:/# apt-get install traceroute
root@node_in_1:/# traceroute www.google.es
traceroute to www.google.es (216.58.201.131), 30 hops max, 60 byte packets
 1 10.1.0.1 (10.1.0.1) 0.101 ms 0.047 ms 0.054 ms
 2 10.0.3.1 (10.0.3.1) 0.102 ms 0.053 ms 0.045 ms
...
8 google-router.red.rediris.es (130.206.255.2) 7.694 ms 7.561 ms 7.659 ms
 9 72.14.235.18 (72.14.235.18) 8.100 ms 7.973 ms 11.170 ms
10 216.239.40.217 (216.239.40.217) 8.077 ms 8.114 ms 8.040 ms
11 mad06s25-in-f3.1e100.net (216.58.201.131) 7.976 ms 7.910 ms 8.106 ms

Last words on this…

Wow, there are a lot of learned things here…

  1. Creating a basic DHCP and DNS server with DNSMASQ
  2. Creating a NAT router with iptables
  3. Creating a bridge without a device attached to it

Each topic would could have be a post in this blog, but the important thing in here is that all these tasks are made inside LXC containers 🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s