Adventures in Datacenter Networking

Adventures in Datacenter Networking

datacenter networking freebsd

Field notes from bringing our first datacenter server's networking online.

How we got here

When I saw that 37signals made the decision to leave the cloud, I got curious and started looking at the tradeoffs for us to bring our upcoming products online using the cloud vs. a datacenter. To my surprise, the cost for what we needed to launch was very similar. But what makes running in a datacenter better than the cloud are the customer service and networking guarantees. For instance, I have real humans I can contact if something goes wrong, and they even reached out to me directly to be proactive about my network when they saw that my network link was down (self inflicted). The network guarantees are also phenomenal. They have me within 5ms(!) of their object storage system, which I'm also going to be heavily using, and they provide a bonded 10GB network connection to my server.

What is a bonded network connection, you ask?

Yeah, I also had to learn what that was, but it's really neat. A bonded network is a network connection that combines multiple physical network connections into a single logical connection. This allows for increased bandwidth and reliability because it can fail over to a backup connection if one of the physical connections fails.

FreeBSD and bonded network connections.

As neat as the bonded connections are, there was definitely a learning curve in getting set up on my FreeBSD server running in the datacenter. I learned the hard way that the first question you should ask your datacenter provider is whether they require your bonded circuit to be VLAN tagged. In many datacenters, your switch ports carry traffic inside a VLAN, so your server must tag packets correctly to be accepted upstream. I spent countless attempts trying to figure out what was wrong with my bonded connection settings, only to discover that I needed to tag it. For my own sanity in the future, I've written down the steps I took to get my bonded network connection working on FreeBSD:

# Bring up the two physical interfaces that are part of the bonded connection.
ifconfig ix2 up
ifconfig ix3 up

# Create the lacp bonded interface.
ifconfig lagg0 create
ifconfig lagg0 laggproto lacp laggport ix2 laggport ix3 up

# Verify. You should see both ports listed as active.
ifconfig lagg0

# If your datacenter provider requires it, create the VLAN they assigned you on top of your bonded interface.
ifconfig vlan226 create
ifconfig vlan226 vlan 226 vlandev lagg0 up

# Assign your IP to the VLAN interface (NOT the bonded interface)
# Note: If the VLAN isn't required by your datacenter provider, then you should assign the IP to your bonded interface.
ifconfig vlan226 inet [assigned IP address] netmask 255.255.255.240

# Add the default route to your assigned gateway.
route add default [assigned gateway IP address]

# Test connectivity to both your assigned gateway and the public internet.
ping -c 2 [assigned gateway IP address]
ping -c 2 1.1.1.1

Congrats, your fancy bonded network connection is now online :)

Make the settings permanent

Edit your /etc/rc.conf as follows:

cloned_interfaces="lagg0 vlan226"

ifconfig_ix2="up"
ifconfig_ix3="up"

ifconfig_lagg0="laggproto lacp laggport ix2 laggport ix3 up"

ifconfig_vlan226="inet [assigned IP address] netmask 255.255.255.240 vlan 226 vlandev lagg0"

defaultrouter="[assigned gateway IP address]"

This looks all fairly straightforward, but it actually took me a while to figure out how to do it. I hope this helps someone else out there.

Cheers 🥂

More Articles