High Availability on Vultr with Floating IP and BGP

By combining Border Gateway Protocol (BGP) and Vultr's Reserved IP feature, you can create a Floating IP that provides high availability for your application by automatically redirecting traffic among available instances. This guide explains how to configure a Floating IP using BIRD, a popular routing daemon.

• Learn more about Vultr Reserved IPs.
• For an introduction to BGP, including how to bring your own IP space to Vultr, see Configuring BGP on Vultr.

Example Values

This guide uses example values. Please replace them with your information.

• Instance IPv4 address: 192.0.2.111
• Reserved IPv4 (the Floating IP): 192.0.2.2
• Example BGP AS number: 65500

Look for your instance IP address and BGP AS number on the Vultr Console server information page, and the reserved IP on the Network Reserved IP page.

Vultr's Side of the BGP Session

All Vultr VPS cloud server instances use these BGP values:

• The neighbor IPv4 address is: 169.254.169.254
• Vultr's AS number is: 64515

If you use Vultr Bare Metal, please use these values:

• The neighbor IPv4 address is: 169.254.1.1
• Vultr's AS number is: 20473

Using the Floating IP in Production

You can ensure high availability for your site by using multiple VPS instances with the same BGP configuration. When an instance is unavailable, BGP anycast dynamically makes a one-of-many selection and redirects traffic to an available instance. This routing strategy allows a high number of instances with the same configuration to share a single reserved IP as a floating address. BGP anycast routes traffic from the reserved address to a single instance at any given time. The reserved address, and all available instances, must be in the same location.

BGP anycast distributes traffic with an Equal-Cost Multi-Path (ECMP) routing strategy. ECMP routing means that traffic will be randomly distributed between any instances in the same location announcing an IP address.

Prerequisites

To follow the steps in this guide:

• Deploy multiple Vultr cloud server instances in the same location. This guide uses Ubuntu 20.10 as the example OS. You may need to adapt some instructions for your operating system.
• Add a Reserved IP address to use as the floating IP. Learn more about Reserved IPs. Leave your Reserved IP unattached. This method for floating IPs requires the Reserved IP address remain unattached from all instances. The reserved IP and instances must be in the same location.
• Open a ticket requesting BGP activation for your account. Please request a private ASN assignment unless you have a public ASN. You must restart your instances through the Vultr Console after Vultr activates BGP in your account.

The steps below explain how to set up a single instance. To achieve high availability, repeat these steps on two or more instances, using the corresponding IP address for each instance.

1. Configure a Dummy Network

1. SSH to the instance as root.

2. Create a dummy network interface.

    # ip link add dev dummy1 type dummy
   

3. Bring the interface up.

    # ip link set dummy1 up
   

4. Bind the dummy interface to the Reserved IP address. Replace the example 192.0.2.2 with your reserved IP.

    # ip addr add dev dummy1 192.0.2.2/32
   

5. Confirm the network configuration.

    # ip addr show dev dummy1
   
    3: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
        link/ether 00:00:5e:00:53:11 brd ff:ff:ff:ff:ff:ff
        inet 192.0.2.2/32 scope global dummy1

2. Configure BIRD

1. Verify your firewall permits traffic on TCP port 179.

2. In a web browser, navigate to the Vultr Console and locate the server information page for your instance.

3. Click the BGP tab.

4. Look for the link at the bottom of the page:

   

5. Click the BGP configuration link. Refer to this example configuration for the next few steps.

6. Install BIRD. This guide uses BIRD version 1.6.8.

    # apt install bird
   

7. Edit your BIRD configuration. On Ubuntu, the BIRD configuration file is /etc/bird/bird.conf.

    # nano /etc/bird/bird.conf

   On a new installation, the configuration file looks like this:

    router id 192.0.2.111;
   
    protocol kernel {
        scan time 60;
        import none;
    }
   
    protocol device {
        scan time 60;
    }
   

8. Edit the router id line to match the IP address in your instance BGP example.

9. Add this section to the bottom of the file.

    protocol direct
    {
        interface "dummy1";
    }
   

10. Add the protocol bgp vultr block from your instance BGP example to the bottom of the file. When finished, your file should look like this. Substitute your values for router id, local as, source address, and password.

     router id 192.0.2.111;
    
     protocol kernel {
         scan time 60;
         import none;
     }
    
     protocol device {
         scan time 60;
     }
    
     protocol direct
     {
         interface "dummy1";
     }
    
     protocol bgp vultr
     {
         local as 65500;
         source address 192.0.2.111;
         import none;
         export all;
         graceful restart on;
         multihop 2;
         neighbor 169.254.169.254 as 64515;
         password "YOUR_PASSWORD";
     }

👋 If you use Bare Metal, replace the neighbor line with the Bare Metal values in the Example Values section.

With this configuration, BIRD looks for the dummy1 interface and advertises its IP to Vultr's infrastructure via BGP. As soon as your instance is running, it receives traffic. If it crashes, traffic will stop.

3. Verify BGP Connectivity

1. Start the BIRD service.

      systemctl start bird
   

2. Wait a few seconds. Check that the BGP session is established.

    # birdc show proto all vultr
   
    BIRD 1.6.8 ready.
    name     proto    table    state  since       info
    vultr    BGP      master   up     20:58:23    Established
      Preference:     100
      Input filter:   REJECT
      Output filter:  ACCEPT
      Routes:         0 imported, 1 exported, 0 preferred
      Route change stats:     received   rejected   filtered    ignored   accepted
        Import updates:              0          0          0          0          0
        Import withdraws:            0          0        ---          0          0
        Export updates:              1          0          0        ---          1
        Export withdraws:            0        ---        ---        ---          0
      BGP state:          Established
        Neighbor address: 169.254.169.254
        Neighbor AS:      64515
        Neighbor ID:
        Neighbor caps:    refresh restart-aware AS4 add-path-rx
        Session:          external multihop AS4
        Source address:   192.0.2.111
        Hold timer:       141/180
        Keepalive timer:  50/60

If everything is working, you'll see Established next to the BGP state.

Debugging

If there are configuration issues, you may see an error such as:

# birdc show proto all vultr
BIRD 1.6.8 ready.
vultr is not a protocol

• The most common reason is a firewall blocking TCP port 179.
• If you deployed this instance before Vultr enabled the BGP feature on your account, it must be restarted from the Vultr Console before BGP is available. Rebooting the operating system is not sufficient.
• For additional debugging, see your log file, typically in /var/log/bird.

4. Test the Floating IP

1. Verify BIRD is advertising the route to your floating IP.

    # birdc show route
    BIRD 1.6.8 ready.
    192.0.2.2/32 dev dummy1 [direct1 20:58:21] * (240)
   

2. Open two terminal sessions. We'll call the session connected to the server the SSH session and a second session at your local workstation the Local session.

3. In the Local session, ping the Reserved IP address. You should see replies.

    $ ping 192.0.2.2
    64 bytes from 192.0.2.2: icmp_seq=12 ttl=44 time=42.863 ms
    64 bytes from 192.0.2.2: icmp_seq=13 ttl=44 time=49.471 ms
    64 bytes from 192.0.2.2: icmp_seq=14 ttl=44 time=42.547 ms
    64 bytes from 192.0.2.2: icmp_seq=15 ttl=44 time=43.489 ms
   

4. In SSH session, take the dummy interface down. Repeat this step on all instances.

    # ip link set dummy1 down

   You'll see the pings begin to fail in the Local session.

    Request timeout for icmp_seq 68
    Request timeout for icmp_seq 69
    Request timeout for icmp_seq 70
    Request timeout for icmp_seq 71
   

5. In the SSH session, check the route again.

    # birdc show route

   BIRD notices that the interface has disappeared and withdraws the route. Repeat this command on all instances to verify.

6. In the SSH session, enable the dummy1 interface on one instance only.

    # ip link set dummy1 up
   

7. Notice that the pings resume in the Local session.

    64 bytes from 192.0.2.2: icmp_seq=82 ttl=44 time=42.863 ms
    64 bytes from 192.0.2.2: icmp_seq=83 ttl=44 time=49.471 ms
    64 bytes from 192.0.2.2: icmp_seq=84 ttl=44 time=42.547 ms
    64 bytes from 192.0.2.2: icmp_seq=85 ttl=44 time=43.489 ms
   

8. Disable the dummy1 interface on the selected instance again. You'll see the pings begin to fail in the Local session again.

9. Repeat these steps on each instance in turn. You'll see the pings on the floating IP resume as each instance becomes available. Repeat your tests until you are satisfied the floating IP is performing as desired.

Use BGP Prepends to Assign Traffic Priority

You can use BGP prepends to assign preferential traffic ordering if you want to control traffic distribution to your instances. Assume there are three instances:

• A Primary instance that receives all traffic in normal conditions.
• A First Backup instance that receives traffic when the Primary instance is down.
• A Second Backup instance that receives traffic when the both the Primary and First Backup instances are down.

1. Configure three instances as previously described.

2. Edit the BIRD configuration on the First Backup instance.

    # nano /etc/bird/bird.conf
   

3. Insert the following block inside the protocol bgp vultr section. Replace the example AS number 65500 with your AS number.

    export filter {
        # Artificially increase path length
        # by prepending the local AS number.
        bgp_path.prepend(65500);
        accept;
    };

   For reference, the full example looks like this:

    router id 192.0.2.111;
   
    protocol kernel {
        scan time 60;
        import none;
    }
   
    protocol device {
        scan time 60;
    }
   
    protocol direct
    {
        interface "dummy1";
    }
   
    protocol bgp vultr
    {
        local as 65500;
        source address 192.0.2.111;
        import none;
        export all;
        graceful restart on;
        multihop 2;
        neighbor 169.254.169.254 as 64515;
        password "YOUR_PASSWORD";
   
        export filter {
            # Artificially increase path length
            # by prepending the local AS number.
            bgp_path.prepend(65500);
            accept;
        };
    }
   

4. On the Second Backup instance, add a similar section, with two bgp_path.prepend lines, like this:

    export filter {
        # Artificially increase path length
        # by prepending the local AS number two times.
        bgp_path.prepend(65500);
        bgp_path.prepend(65500);
        accept;
    };

In normal operation, all traffic goes to the Primary instance. In case of failure, BGP directs traffic to the First Backup. If both of these are down, BGP directs floating IP traffic to the Second Backup. You can extend this method to more instances by adding additional bgp_path.prepend lines on lower-priority backup instances.

Using Floating IPs with IPv6

This process also works with IPv6 reserved subnets. In general, the process for IPv6 is similar to IPv4. You can follow the process described above with a few changes:

• Substitute bird6 for bird
• Substitute birdc6 for birdc.
• Use IPv6 addresses.

All instance types at Vultr use 2001:19f0:ffff::1 as the neighbor IPv6 address. Use the AS number 64515 for all instances except for Bare Metal, which uses 20473.

An abbreviated example is shown below that uses two IPv6 addresses from a reserved /64 subnet. The example values are:

• Instance IPv4 address: 192.0.2.111
• Instance IPv6 address is 2001:0db8:1111::1
• Reserved IPv6 /64 subnet: 2001:0db8:2222::/64
• The example AS number for your instances is: 65500

Configure the IPv6 Dummy Adapter

👋 Note: The instance must have a public IPv6 subnet and address assigned for Floating IPv6 BGP to work.

1. Create a dummy network interface.

    # ip link add dev dummy1 type dummy
   

2. Bring the interface up.

    ip link set dummy1 up
   

3. Bind the dummy interface to one or more IPv6 addresses in the Reserved IP /64 subnet. Use our IPv6 Calculator to see the range of usable IP addresses. You can assign as many IPv6 addresses within the Reserved /64 subnet as you need. This example configures two addresses.

    # ip addr add dev dummy1 2001:0db8:2222::1/64
    # ip addr add dev dummy1 2001:0db8:2222::2/64
   

4. Verify the interface:

    # ip addr show dev dummy1
   
    3: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
        link/ether 00:00:5e:00:53:11 brd ff:ff:ff:ff:ff:ff
        inet 192.0.2.111/32 scope global dummy6
        valid_lft forever preferred_lft forever
        inet6 2001:0db8:2222::1/64 scope global
        valid_lft forever preferred_lft forever
        inet6 2001:0db8:2222::2/64 scope global
        valid_lft forever preferred_lft forever

BIRD6 Configuration

1. Create a /etc/bird/bird.conf file:

    log "/var/log/bird.log" all;
   
    router id 192.0.2.111;
   
    protocol device
    {
        scan time 60;
    }
   
    protocol direct
    {
        interface "dummy1";
    }
   
    protocol bgp vultr
    {
        local as 65500;
        source address 2001:0db8:1111::1;
        import none;
        export all;
        graceful restart on;
        next hop self;
        multihop 2;
        neighbor 2001:19f0:ffff::1 as 64515;
        password "YOUR_PASSWORD";
    }

   👋 If you use Bare Metal, replace the neighbor line with the Bare Metal values:

    neighbor 2001:19f0:ffff::1 as 20473;
   

2. Start the Bird6 process.

    # systemctl start bird6
   

3. Check the Bird6 console.

    # birdc6 show route
   
    BIRD 1.6.8 ready.
    2001:0db8:2222::/64 dev dummy6 [direct1 18:17:21] * (240)
   
    # birdc6 show proto all vultr
   
    BIRD 1.6.8 ready.
    name     proto    table    state  since       info
    vultr    BGP      master   up     18:17:24    Established
      Preference:     100
      Input filter:   REJECT
      Output filter:  ACCEPT
      Routes:         0 imported, 1 exported, 0 preferred
      Route change stats:     received   rejected   filtered    ignored   accepted
        Import updates:         279366          0     279366          0          0
        Import withdraws:         2692          0        ---     282058          0
        Export updates:              1          0          0        ---          1
        Export withdraws:            0        ---        ---        ---          0
      BGP state:          Established
        Neighbor address: 2001:19f0:ffff::1
        Neighbor AS:      64515
        Neighbor ID:
        Neighbor caps:    refresh restart-aware AS4 add-path-rx
        Session:          external multihop AS4
        Source address:   2001:0db8:1111::1
        Hold timer:       170/180
        Keepalive timer:  53/60
   

4. Use ping6 from your local workstation to test connectivity on the two IPv6 addresses.

    # ping6 -c2 2001:0db8:2222::1
   
    PING 2001:0db8:2222::1(2001:0db8:2222::1) 56 data bytes
    64 bytes from 2001:0db8:2222::1: icmp_seq=1 ttl=44 time=154 ms
    64 bytes from 2001:0db8:2222::1: icmp_seq=2 ttl=44 time=55.5 ms
    --- 2001:0db8:2222::1 ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1001ms
    rtt min/avg/max/mdev = 55.530/104.768/154.006/49.238 ms
   
    # ping6 -c2 2001:0db8:2222::2
   
    PING 2001:0db8:2222::2(2001:0db8:2222::2) 56 data bytes
    64 bytes from 2001:0db8:2222::2: icmp_seq=1 ttl=44 time=55.4 ms
    64 bytes from 2001:0db8:2222::2: icmp_seq=2 ttl=44 time=51.9 ms
    --- 2001:0db8:2222::2 ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1000ms
    rtt min/avg/max/mdev = 51.939/53.672/55.405/1.733 ms

It may take up to 20 minutes for Vultr's network to pick up the IPv6 routes.

FreeBSD Configuration Notes

• If you use FreeBSD 12 or 13 instead of Linux, you may need to load the tcpmd5 kernel module by adding this line to /boot/loader.conf.

    tcpmd5_load="YES"

  Note: loader.conf is only read at boot. If you'd like to test the configuration, you can load it immediately with:

    kldload /boot/kernel/tcpmd5.ko
  

• Older versions may need to recompile the kernel for TCP MD5 signature support. Those instructions are outside of the scope of this article.

If your BSD kernel does not support TCP MD5 signatures, you will see the following output in the BIRD log.

    $ cat /var/log/bird.log
    2017-12-15 01:35:00 <INFO> Started
    2017-12-15 01:35:00 <ERR> vultr: Socket error: Kernel does not support TCP MD5 signatures

The BIRD configuration file is located at /usr/local/etc/bird.conf on BSD.

Manage Reserved IPs via API

The Vultr API offers several endpoints to manage reserved IPs.

• Create a new reserved IP.
• List all reserved IPs in your account.
• Get information about a reserved IP.
• Convert the IP address on an instance into a reserved IP.
• Delete a reserved IP.