[lvs-users] Multiple HTTPS (per real-server) on LVS-DR does not work

[Michael Moody]

I have LVS-DR on gentoo 2006.1, kernel 2.6.20.

Apache 2.0

I am running multiple ssl vhosts (ip-based) on each realserver.

The load balancer has two vips, 10.0.0.20(https site 1), and
10.0.0.24(https site 2)

I have two rip's on each realserver,
[192.168.1.20(https1) and 192.168.1.23(https2) (server1)]
[198.168.1.54(https1) and 192.168.1.24(https2) (server2)]

The vhost conf looks like this:

Server 1:
NameVirtualHost 192.168.1.20:443
<VirtualHost 192.168.1.20:443>

NameVirtualHost 192.168.1.23:443
<VirtualHost 192.168.1.23:443>

Server2:

NameVirtualHost 192.168.1.24:443
<VirtualHost 192.168.1.24:443>

NameVirtualHost 192.168.1.54:443
<VirtualHost 192.168.1.54:443>

However, if I go to the vip, via https://10.0.0.20, I get an ssl error.
What it appears like to me is that since apache is listening on
192.168.1.24, it can't respond to requests from the load balancer since
it's not also listening on the vip. Is there a way to make it listen on
the vip as well? What am I doing wrong?

Thanks,
Michael
--
Michael S. Moody
Systems Engineer
Global Systems Consulting
Direct: [phone]
Web: http://www.GlobalSystemsConsulting.com

Engineering Support: ***@gsc.cc
Billing Support: ***@gsc.cc
Customer Support Portal: http://my.gsc.cc


NOTICE - This message contains privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message, you are hereby notified that you must not disseminate, copy or take any action in reliance on it. If you have received this message in error, please immediately notify Global Systems Consulting, its subsidiaries or associates. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the view of Global Systems Consulting, its subsidiaries and associates.

[Joseph Mack NA3T]

Post by Michael Moody
I have LVS-DR on gentoo 2006.1, kernel 2.6.20.
Apache 2.0
I am running multiple ssl vhosts (ip-based) on each realserver.
The load balancer has two vips, 10.0.0.20(https site 1), and
10.0.0.24(https site 2)
I have two rip's on each realserver,

you should have one RIP (192.168.x.x) and two VIPs
(10.x.x.x)

Joe

[Graeme Fowler]

Post by Michael Moody
I have LVS-DR on gentoo 2006.1, kernel 2.6.20.

OK...

Post by Michael Moody
I am running multiple ssl vhosts (ip-based) on each realserver.

OK... remember however that although you bind a given SSL virtual host
to a single IP address, the certificate is in the *name* of the site,
not the IP.

Post by Michael Moody
However, if I go to the vip, via https://10.0.0.20, I get an ssl error.

This is to be expected. It is a well-known catch-22 using SSL for web
hosting - the TLS/SSL session is established over the IP connection
*before* the HTTP application layer comes into play.

During the TLS negotiation, the server sends back the public part of the
certificate; this contains a "Subject" attribute something like this:

Subject: C=GB, ST=Leicestershire, L=Loughborough, O=Graeme Fowler,
OU=linuxvirtualserver.org, CN=www.linuxvirtualserver.org

The CN attribute in that line is what the browser then compares against
the virtual host being requested - if they don't match, it throws an
error.

Only when the TLS session is established can the client say:

GET / HTTP/1.1
Host: 10.0.0.20

(which is what the browser says when using the URI you gave as an
example).

Post by Michael Moody
What it appears like to me is that since apache is listening on
192.168.1.24, it can't respond to requests from the load balancer since
it's not also listening on the vip. Is there a way to make it listen on
the vip as well? What am I doing wrong?

Don't connect to the VIP by IP address using SSL. If you must, your
browser will not be able to validate the certificate and will throw an
error.

As an interesting exercise, try connecting to the realserver's IP
address from a machine local to it; you'll get the same problem.

Aside from the validation problem, is anything else the matter?

Graeme

[Michael Moody]

It may be my failing, but perhaps I didn't explain well enough. I'll
attempt to do so here, with a cute ascii diagram, as best my ability:
User
|
Firewall ----> VIP (load balancer,
10.0.0.24=https://www.https1.com, 10.0.0.34=https://www.https2.com)

^ ^
^ ^

192.168.1.23(srv1) 192.1681.25(srv2) 192.168.1.24(srv1)
192.168.1.26(srv2)

I have 2 servers, each behind the load balancer. Each server has 2 ssl
sites, 2 different ssl certificates. www.https1.com, www.https2.com,
both reside on each server.

The vhost is configured so that each ssl is bound to it's own ip, as
name based hosting doesn't work for ssl sites in apache (I'm aware of this).

Each realserver has the dummy eth module installed, and multiple ip's,
in a fashion of dummy0=10.0.0.24, dummy0:1=10.0.0.34.

If I bind the ssl in apache vhosts to an ip, like this <virtualhost
192.168.1.23:443>, then for whatever reason, apache refuses to work with
the load balancer, perhaps because it doesn't know to answer requests
using the vip, however, if I seperate them, and have one server with
<virtualhost *:443>
vhost1 settings
</virtualhost>

and the other server with
<virtualhost *:443>
vhost2 settings
</virtualhost>

Things work fine, but there's no load balancing, and no redundancy.

Any ideas at all?

Michael

Post by Graeme Fowler

Post by Michael Moody
I have LVS-DR on gentoo 2006.1, kernel 2.6.20.

OK...

Post by Michael Moody
I am running multiple ssl vhosts (ip-based) on each realserver.

OK... remember however that although you bind a given SSL virtual host
to a single IP address, the certificate is in the *name* of the site,
not the IP.

Post by Michael Moody
However, if I go to the vip, via https://10.0.0.20, I get an ssl error.

This is to be expected. It is a well-known catch-22 using SSL for web
hosting - the TLS/SSL session is established over the IP connection
*before* the HTTP application layer comes into play.
During the TLS negotiation, the server sends back the public part of the
Subject: C=GB, ST=Leicestershire, L=Loughborough, O=Graeme Fowler,
OU=linuxvirtualserver.org, CN=www.linuxvirtualserver.org
The CN attribute in that line is what the browser then compares against
the virtual host being requested - if they don't match, it throws an
error.
GET / HTTP/1.1
Host: 10.0.0.20
(which is what the browser says when using the URI you gave as an
example).

Post by Michael Moody
What it appears like to me is that since apache is listening on
192.168.1.24, it can't respond to requests from the load balancer since
it's not also listening on the vip. Is there a way to make it listen on
the vip as well? What am I doing wrong?

Don't connect to the VIP by IP address using SSL. If you must, your
browser will not be able to validate the certificate and will throw an
error.
As an interesting exercise, try connecting to the realserver's IP
address from a machine local to it; you'll get the same problem.
Aside from the validation problem, is anything else the matter?
Graeme

--
Michael S. Moody
Systems Engineer
Global Systems Consulting
Direct: [phone]
Web: http://www.GlobalSystemsConsulting.com

Engineering Support: ***@gsc.cc
Billing Support: ***@gsc.cc
Customer Support Portal: http://my.gsc.cc


NOTICE - This message contains privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message, you are hereby notified that you must not disseminate, copy or take any action in reliance on it. If you have received this message in error, please immediately notify Global Systems Consulting, its subsidiaries or associates. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the view of Global Systems Consulting, its subsidiaries and associates.

[Joseph Mack NA3T]

Post by Michael Moody
If I bind the ssl in apache vhosts to an ip, like this <virtualhost
192.168.1.23:443>, then for whatever reason, apache refuses to work with
the load balancer,

apache has to bind to the VIP (in your case 10.x.x.x)

Joe

[Michael M.]

I thought that was the case, however, apache doesn't seem to support binding
a vhost to multiple ip's, unless perhaps I was to create multiple vhosts? I
might try that. If I bind it to 10.0.0.x, then apache won't answer on the
192.168.1.x address that the load balancer is looking for. Recommendations?

Thanks,
Michael

-----Original Message-----
From: Joseph Mack NA3T [mailto:***@wm7d.net]
Sent: Wednesday, October 17, 2007 12:34 PM
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] Multiple HTTPS (per real-server) on LVS-DR does not
work

Post by Michael Moody
If I bind the ssl in apache vhosts to an ip, like this <virtualhost
192.168.1.23:443>, then for whatever reason, apache refuses to work with
the load balancer,

apache has to bind to the VIP (in your case 10.x.x.x)

Joe

[Michael Moody]

For those of you who may have a similar problem, here is the solution:

(192.168.1.20 and 192.168.1.23 are https1 and https2 repectively on
server1, 192.168.1.54 and 192.168.1.24 are https1 and https2 repectively
on server2) 10.0.0.20 is the vip for https1, 10.0.0.24 is the vip for
https2.

First thing, I set up my apache to have dual vhosts, like this, both on
the vip and rip:

------------Vhost for https1 site one on server 1--------------
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
SSLPassPhraseDialog builtin
SSLSessionCache shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout 60
SSLMutex default
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCipherSuite
ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

Listen 192.168.1.20:443
Listen 10.0.0.20:443

NameVirtualHost 192.168.1.20:443

<VirtualHost 192.168.1.20:443>
ServerAdmin ***@somewhere.com
DocumentRoot /var/www/https1
ServerName https1.site.com
SSLCertificateKeyFile /etc/apache2/ssl/https1.site.com.key
SSLCertificateFile /etc/apache2/ssl/https1.site.com.crt
SSLEngine on

<Directory "/var/www/https1">
Options FollowSymlinks Multiviews
Order allow,deny
Allow from all
</Directory>
ErrorDocument 404 http://www.site.com/error.html
</VirtualHost>

NameVirtualHost 10.0.0.20:443

<VirtualHost 10.0.0.20:443>
ServerAdmin ***@somewhere.com
DocumentRoot /var/www/https1
ServerName https1.site.com
SSLCertificateKeyFile /etc/apache2/ssl/https1.site.com.key
SSLCertificateFile /etc/apache2/ssl/https1.site.com.crt
SSLEngine on

<Directory "/var/www/https1">
Options FollowSymlinks Multiviews
Order allow,deny
Allow from all
</Directory>
ErrorDocument 404 http://www.site.com/error.html
</VirtualHost>

------------Vhost for https2 site one on server 1--------------
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
SSLPassPhraseDialog builtin
SSLSessionCache shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout 60
SSLMutex default
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCipherSuite
ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

Listen 192.168.1.23:443
Listen 10.0.0.24:443

NameVirtualHost 192.168.1.23:443

<VirtualHost 192.168.1.23:443>
ServerAdmin ***@somewhere.com
DocumentRoot /var/www/https2
ServerName https2.site.com
SSLCertificateKeyFile /etc/apache2/ssl/https2.site.com.key
SSLCertificateFile /etc/apache2/ssl/https2.site.com.crt
SSLEngine on

<Directory "/var/www/https2">
Options FollowSymlinks Multiviews
Order allow,deny
Allow from all
</Directory>
ErrorDocument 404 http://www.site.com/error.html
</VirtualHost>

NameVirtualHost 10.0.0.24:443

<VirtualHost 10.0.0.24:443>
ServerAdmin ***@somewhere.com
DocumentRoot /var/www/https2
ServerName https1.site.com
SSLCertificateKeyFile /etc/apache2/ssl/https2.site.com.key
SSLCertificateFile /etc/apache2/ssl/https2.site.com.crt
SSLEngine on

<Directory "/var/www/https2">
Options FollowSymlinks Multiviews
Order allow,deny
Allow from all
</Directory>
ErrorDocument 404 http://www.site.com/error.html
</VirtualHost>

Or course, these vhosts are similar on server2, with the ip's changed.

This configuration enables apache to both listen and answer on both the
rip and vip, and then the loadbalancer ldirectord.cf

virtual=10.0.0.20:443 #https1
real=192.168.1.54:443 gate 1 #server2
real=192.168.1.20:443 gate 1 #server1
service=https
scheduler=wlc
persistent=300
#netmask=255.255.255.255
protocol=tcp

virtual=10.0.0.24:443 #https2
real=192.168.1.24:443 gate 1 #server2
real=192.168.1.23:443 gate 1 #server1
service=https
scheduler=wlc
persistent=60
#netmask=255.255.255.255
protocol=tcp


Hope this helps anyone else who runs into this.

Post by Michael M.
I thought that was the case, however, apache doesn't seem to support binding
a vhost to multiple ip's, unless perhaps I was to create multiple vhosts? I
might try that. If I bind it to 10.0.0.x, then apache won't answer on the
192.168.1.x address that the load balancer is looking for. Recommendations?
Thanks,
Michael
-----Original Message-----
Sent: Wednesday, October 17, 2007 12:34 PM
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] Multiple HTTPS (per real-server) on LVS-DR does not
work

Post by Michael Moody
If I bind the ssl in apache vhosts to an ip, like this <virtualhost
192.168.1.23:443>, then for whatever reason, apache refuses to work with
the load balancer,

apache has to bind to the VIP (in your case 10.x.x.x)
Joe

--
Michael S. Moody
Sr. Systems Engineer
Global Systems Consulting
Direct: [phone]
Web: http://www.GlobalSystemsConsulting.com

Engineering Support: ***@gsc.cc
Billing Support: ***@gsc.cc
Customer Support Portal: http://my.gsc.cc


NOTICE - This message contains privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message, you are hereby notified that you must not disseminate, copy or take any action in reliance on it. If you have received this message in error, please immediately notify Global Systems Consulting, its subsidiaries or associates. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the view of Global Systems Consulting, its subsidiaries and associates.

[Joseph Mack NA3T]

Let me see if I understand your posting.

You've given the config of one of the realservers, which
listens on 2 VIPs as it would if it were a standalone
server. The server has two certificates.

As well the realserver listens on two RIPs whose only
purpose is to answer healthchecking of the https server from
the director.

(This last point took a while to figure. I chastised some
poor fellow recently for having two RIPs in an https setup -
not realising what he was doing. When I do the check on the
service on the VIP on the realserver from the director, I
ssh to the RIP and then do a health check on the service
listening on the VIP - there's no service listening on the
RIP. I'll e-mail Horms and Alexandre and see if I can get

Post by Michael Moody
virtual=10.0.0.20:443 #https1

you duplicate the realserver (with suitably adjusted RIPs,
but keeping the same certificates on the duplicated
realserver) and then setup the the director to load
balance two independant https services.

So there's there's nothing particular about having n https
services on an LVS? You just n-fold replicate the service on
the realservers?

Thanks Joe

[Michael M.]

I'm going to reply inline to your questions below....My comments prefix with

Post by Michael Moody

for easier reading.

-----Original Message-----
From: Joseph Mack NA3T [mailto:***@wm7d.net]
Sent: Friday, October 26, 2007 6:25 PM
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] Multiple HTTPS (per real-server) on LVS-DR does not
work
Let me see if I understand your posting.

You've given the config of one of the realservers, which
listens on 2 VIPs as it would if it were a standalone
server. The server has two certificates.

As well the realserver listens on two RIPs whose only
purpose is to answer healthchecking of the https server from
the director.

Post by Michael Moody

This is basically correct, yes. I think the RIP also allows apache to

answer the packets which technically come in on the RIP, so apache would be
listening on that port, otherwise, it would not.


(This last point took a while to figure. I chastised some
poor fellow recently for having two RIPs in an https setup -
not realising what he was doing. When I do the check on the
service on the VIP on the realserver from the director, I
ssh to the RIP and then do a health check on the service
listening on the VIP - there's no service listening on the
RIP. I'll e-mail Horms and Alexandre and see if I can get

Post by Michael Moody
virtual=10.0.0.20:443 #https1

you duplicate the realserver (with suitably adjusted RIPs,
but keeping the same certificates on the duplicated
realserver) and then setup the the director to load
balance two independant https services.

Post by Michael Moody

I'm not sure I understand what you're saying above, and I'm afraid I

don't know who Horms or Alexandre are. I've managed to get this working with
2 realservers, and my config is complete, so if you need any help, just
simply ask (this goes for anyone who comes across this)

So there's there's nothing particular about having n https
services on an LVS? You just n-fold replicate the service on
the realservers?

Post by Michael Moody

This seems to be the key, yes. As long as you're listening on the

service on both the rip and vip, it should work, no matter what the service
is. SSH should do the same thing.


Thanks Joe