Twitter this

Powered by MariaDB Powered by nginx ...

Setting up a haproxy/keepalived loadbalancer on a XEN virtual machine – AKA a cloud load balancer

To enable haproxy/keepalived failovers on a virtual machine, some kernel configuration tuning is needed. The settings on the VM host and client need to be adapted from a real physical server. The case I will describe below is built upon a simple bridged setup. At the time of install there where plenty of issues running openvswitch on ubuntu. The package wasn't stable. I haven't looked back yet to see if the problem still exists since then but like one says, don't catch a falling knife and don't fix what isn't broken. It works fine like this.

Let take a look at some configs. He're is the relevant part of the config from the Virtual machine itself. We are using keepalived to arrange the floating IP on a bridged VM network with the host. All IP's are in the private space realm.

1
2
3
net.ipv4.ip_nonlocal_bind=1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_syncookies = 1

On the VM host you'll need to set these

1
2
3
4
5
6
7
8
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.eth1.proxy_arp = 1
net.ipv4.conf.eth0.proxy_arp = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Adjust this for your own interface labels. If you set these, bridged traffic between VM HOST and CLIENT will not be subjected to the firewall and you have no arp issues. In case you want to see if keepalived instances are talking to eachother, just use tcpdump:

1
2
3
4
5
tcpdump ether multicast
13:01:17.365565 IP 192.168.128.4 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 101, authtype none, intvl 1s, length 20
13:01:17.366613 IP 192.168.128.4 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 52, prio 101, authtype none, intvl 1s, length 20
13:01:17.367712 IP 192.168.128.4 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 53, prio 101, authtype none, intvl 1s, length 20
13:01:17.368760 IP 192.168.128.4 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 54, prio 101, authtype none, intvl 1s, length 20

That's it. When you set up 2 of these successfully, you created yourself a 'cloud load balancer' .

Using HAPROXY as an SSL gateway

Haproxy is a pretty nifty product. Probably not the least due to the fact that it's author, Willy Tarreau spends hours of his life helping others in setting it up the way they want, sometimes fixing a bug in the process. The latest release marks the end to a 4-year long bugfix that finally went live. It is this sort of things, features that have been wanted for years, now incorporated into the software helping sysadmins and programmers around the world in fulfilling corporate IT needs. Haprpxy to me is more than a loadbalancer, you can piece together solutions as a workaround for more fundamental problems.

That's what I recently did. A case of a failing network (NAT problems) that wasn't easily solvable due to the need of having a box around with 2 patched network cards. NAT using the same interface is just not working right. We just needed some outgoing SSL calls to go to the internet and back, through some portforwarded upstream firewall setup. That setup sucks by being limited to only having a private IP range, we have public IP's that map to the internal one but that is a firewall hell as packets arrive at the port from a range that doesn't really belong to the network cards IP range.

So in comes HAPROXY, let's reverse the thinking and consider putting haproxy 'in front' of those SSL requests at port 443. We will just hack up the /etc/hosts file or the local DNS to make sure they endup on haproxy instead of directly to the target public IP:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
global
   log 127.0.0.1  local0
   log 127.0.0.1  local1 notice
   #log loghost   local0 info
   maxconn 4096
   # chroot /usr/share/haproxy
   user haproxy
   group haproxy
   daemon
   #debug
   #quiet

defaults
   log   global
   mode  http
   option   httplog
   option   dontlognull
   retries  3
   option redispatch
   maxconn  2000
   contimeout  5000
   clitimeout  50000
   srvtimeout  50000

# Host HA-Proxy web stats on Port 3306 (that will confuse those script kiddies)
listen HAProxy-Statistics *:3306
    mode http
    option httplog
    option httpclose
    stats enable
    stats uri /haproxy?stats
    stats refresh 20s
    stats show-node
    stats show-legends
    stats show-desc Workaround haproxy for SSL
    stats auth admin:ifIruledTheWorld

frontend ssl_relay 192.168.128.21:443
    # this only works with 1.5 haproxy
    mode tcp
    option tcplog
    option socket-stats
    # option nolinger
    maxconn  300

    # use tcp content accepts to detects ssl client and server hello.
    # acl clienthello req_ssl_hello_type 1 -> seems to not work

    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }

    use_backend ssl_testdomain_prod if { req_ssl_sni -i www.testdomain.nl }
    use_backend ssl_testdomain_stag if { req_ssl_sni -i test.testdomain.nl }

    default_backend ssl_testdomain_stag

backend ssl_testdomain_stag
   mode tcp
   #option nolinger
   option tcplog
   balance roundrobin
   hash-type consistent
   option srvtcpka

    # maximum SSL session ID length is 32 bytes.
    stick-table type binary len 32 size 30k expire 30m

    # make sure we cover type 1 (fallback)
    acl clienthello req_ssl_hello_type 1
    acl serverhello rep_ssl_hello_type 2

    # use tcp content accepts to detects ssl client and server hello.
    tcp-request inspect-delay 5s
    tcp-request content accept if clienthello

    # no timeout on response inspect delay by default.
    tcp-response content accept if serverhello

    # SSL session ID (SSLID) may be present on a client or server hello.
    # Its length is coded on 1 byte at offset 43 and its value starts
    # at offset 44.
    # Match and learn on request if client hello.
    stick on payload_lv(43,1) if clienthello

    # Learn on response if server hello.
    stick store-response payload_lv(43,1) if serverhello

    #option ssl-hello-chk

    server x_testdomain_stag 123.123.123.123:443


backend ssl_testdomain_prod
   mode tcp
   #option nolinger
   option tcplog
   balance roundrobin
   hash-type consistent
   option srvtcpka

    # maximum SSL session ID length is 32 bytes.
    stick-table type binary len 32 size 30k expire 30m

    # make sure we cover type 1 (fallback)
    acl clienthello req_ssl_hello_type 1
    acl serverhello rep_ssl_hello_type 2

    # use tcp content accepts to detects ssl client and server hello.
    tcp-request inspect-delay 5s
    tcp-request content accept if clienthello

    # no timeout on response inspect delay by default.
    tcp-response content accept if serverhello

    # SSL session ID (SSLID) may be present on a client or server hello.
    # Its length is coded on 1 byte at offset 43 and its value starts
    # at offset 44.
    # Match and learn on request if client hello.
    stick on payload_lv(43,1) if clienthello

    # Learn on response if server hello.
    stick store-response payload_lv(43,1) if serverhello

    #option ssl-hello-chk

    server x_testdomain_prod 123.123.111.111:443

So what does it do? We will send all request to our internal IP (frontend .21) , who will split this up according to SNI information , if that fails it will go to the 'staging' backend. This should only happen to software which doesn't support this feature. Curl for example knows how to use SNI. Wget also knows. All modern browsers do in fact.

It will analyse the SNI information and play SSL proxy for us. Haproxy is a good puppy! And as sugar on the pie, you can check the stats to see how much has been routed. It's the other way around, an internal IP is the frontend, while the backends reside on a public IP.

This works for me on the following version

1
2
root@server:/usr/sbin# haproxy -V
HA-Proxy version 1.5-dev19 2013/06/17

The only thing left to do is trick DNS by overriding the official target IP's by your internal IP address.

recruitment
recruitment
recruitment
recruitment