On the one hand, we are not even entirely sure if we’ll stick to MySQL or switch to any other DB server type, hence we must not use SQL dialect specific code within the application. On the other hand, we want to provide a SOAP endpoint internally, which allows us to access data from all sorts of applications and clearly separates the business logic of all sorts of clients from our actual data store.

In addition, caching of object structures which are mapped against data structures (or tables) would be very neat. Consequently, I suggested and started using Java with Hibernate and JAX-WS on a Glassfish Java Application Server. We might integrate an additional caching layer later. Terracotta is one of the candidates, which integrates seamlessly and can increase speed by factor 2 to 10. 

After one week of researching, developing proofs of concept and presenting a first prototype, I am sooo excited about all that.  I always loved Java, and again I remember why.

Once you’ve set it up and got it running, you of course need a slave server as well. So all you got to do is either to install another PowerDNS/MySQL instance on another machine and connect it to the same database (you wouldn’t really want to do that), or connect it to a second MySQL server, which ideally is a replication slave of the first one. Then you have a fully redundant setup and don’t need to worry about failures of either of those. All DNS record changes go directly to the master DB server, which will be picked up by the slave in almost real time. Plus, you don’t need to worry about serial numbers. Change a record in the DB and this change will be published instantly. 

If you fancy web interfaces, you can either go for an open source one, or you can write one on your own. As the DB structure of PowerDNS is kept very simple, it’s easy to add/modify records via script also. Do whatever you want and don’t worry about the stupid restrictions and configuration hurdles you used to run into when going for a BIND server.

Here’s a list of features I want to implement:

• full failover
• high security
• scalability
• load balancing
• prioritisation based on the distance to the micro data centres
• clustered filesystem across micro data centres

The idea is to put a couple of virtual machines on both of the “micro data centre boxes:”

• pfSense firewall
• Pound load-balancer
• MySQL-Cluster nodes (NDB Cluster)
• GlusterFS cluster storage nodes
• PowerDNS with geo plugin (to deliver IP resolutions depending on the source of a query)
• Apache instance(s) accessing the GlusterFS storage and the MySQL cluster

The only VM which has a public IP is the pfSense firewall. It deals with the requests and does the inter-box traffic shaping to ensure stability for the cluster filesystem (which replicates itself between the “micro data centres” in real-time) and the MySQL cluster, which is also spread over both physical boxes. The PowerDNS server resolves domain queries depending on the source of the requests. One server resides in the US, one is located in the UK (for now); so European customers will always be redirected to the UK server, whereas American customers will always see the US based server first.

The reason, why everything lives in its own VM is that choosing this approach, it is possible to easily move a VM onto a more powerful machine if necessary. The whole “micro data centre” can grow as required. But from the beginning — that is the key interest of this experiment — everything is fully redundant and it does not matter, if a single VM or one of the two boxes entirely fails. That’s much closer to 100% uptime than any single-data-centre approach (which you will most commonly find).

Actually, I have never had a look behind the curtains of a Content Delivery Network (aka CDN). But I am pretty sure that the idea is similar, although they certainly do not use VMs or only two servers. So the implementation will differ. Anyway, I’d like to see how my approach works. It’s an experiment using free software only (which includes ESXi since it was released with a new license end of July).

I started this, because I like VMware ESXi and want to stress-test it a bit. Also I like the pfSense firewall (which is FreeBSD based). And I do have good experience with GlusterFS (also not yet in an inter-continental setup). MySQL NDB is new to me. So is location-based domain resolution based on PowerDNS.

I’m pretty sure that VMware Infrastructure with the HA-AddOn would be much easier (and way more expensive!), but this is about learning new things and combining them with existing knowledge… Yes, call me a freak.

So, you now know the idea. In the next articles of this series, I will go into detail about all the things I do and how I am getting on with the project. Stay tuned.

[Added August 17th] Here’s a small diagram how I thing it could look like:

It’s a prejudice that virtualisation is only interesting for so-called Virtual Private Server providers or for big companies who need to run loads of tests for their software releases on different platforms and configurations. Also, you do not need a bunch of servers or a blade-center to take an advantage on virtualisation. In this article I’d like to give an example of what can be achieved with virtualisation apart from those typical and well-known scenarios.

Firstly, have a look at the load average of your own server(s). If it stays below 1 (per CPU/core) most of the time, you are actually not making use of all the power your machine has to offer. Hence, you are actually wasting money! You might say: “I am not yet making use of it, but my business is supposed to grow and some day I will need all of the idle resources.”

That’s absolutely fine, and 90% of all small companies will agree with your approach. However, I would like to outline what virtualisation can do for everybody who owns (or rents) a dedicated server. I’d like to show how

• you can improve access to your system for maintenance tasks
• you can use idle resources and free them when they are needed for other purposes
• virtualisation can increase stability and security of your server

First things first, virtualisation is not as complicated to set up as commonly expected. My favourite implementations are VMware Server and VMware ESXi (free since 28/07/2008), because they enable you to run any operating system within virtual machines without any changes to their kernels whatsoever. Moreover, WMware products have a long history and have proven that they are rock-solid. However, if you are sure that you will not run anything but Linux, you might also want to have a look at XEN, OpenVZ, Linux vserver or other implementations. See the previous article for an overview of available products.

The easiest installation is offered by VMware ESXi Installable. All you have to do is to insert its bootable CD (if you have a SuperMicro KVM-over-IP, a Raritan eRIC G4 card or similar, you can do that remotely as well). It will install itself onto the server, ask you some questions, and that’s it. ESXi is a hypervisor which does not require any host operating system to run on. You are ready to install any piece of operating system you want as a virtual machine through the VMware Infrastructure Client (Windows application, for free, part of ESXi).

If you don’t have KVM-over-IP or local physical access to the server, you could also ask your server provider to do that for you. It takes less than half an hour and is very easy to do.

Anyway, the subject of this article is not how to set up virtualisation. I just wanted to give an example. The topic here is, why you may want to consider using virtualisation.

How to improve access to your system for maintenance tasks

Does you server have KVM-over-IP or do you have physical access to the server? Then this might not apply to you. All others regularly generate costs when they want to do simple things as an ReiserFS or ext3 filesystem check, or when they want to compile a new kernel and it does not work out as expected: They have to ask their server providers to grant access to their servers via KVM-over-IP (if possible). With some hosts that can be kind of a nightmare!

So how does virtualisation help here? In case you are using VMware, each virtual machine can be accessed via a remote console. You can change BIOS settings, monitor the boot process and access your machine even if it does not have a SSH daemon running, as if you were sitting in front of it. The filesystem is corrupted? Just insert an ISO image of your favourite rescue CD into the virtual CDROM drive and boot from it. Do your filesystem maintenance or fix the problems with the custom kernel, eject the CD image, and boot again. There you go: Within minutes your problems can be solved — anytime you want and without any additional costs.

As for VMware you could also have a tftp server running in another virtual machine and boot VMs via PXE! That’s quite advanced but very helpful, should you ever need it.

How to use idle resources

Would be a shame to waste resources (and money!) on a machine, wouldn’t it? So why not running the main VM with most of the resources assigned, while still keeping spare resources available to install completely different things on the same machine?

Of course you can usually do that on a non-virtualised server as well. But how do you control resources of less important services and tasks? And would you really want to mix experimental stuff with your production servers? What about security in that case? Maybe you may want to test other Linux distributions? That’s all easily possible with a virtualised server.

Just tell the hypervisor to prioritize your most important VM(s) or hard-limit resources of your additional VMs. Then you will not see any impact of your experimental stuff on the production services at all! You do not need to worry about security, stability, clean un-installs of failed experiments. You will not experience a single second of downtime of your production VM while you are doing the most sophisticated experiments on another VM!

You also might to do more than just experiments on the same server. For example, you are starting with your business and want to keep costs at a minimum in the beginning. However, you expect your business to grow quickly. What you could do is to start with a single virtualised dedicated server and split it into logical units, each of them running on a separate virtual machine. As soon as you realise that the server may reach its performance limits, you simply migrate one or more of the virtual machines onto another physical virtualised server. The interesting thing here is, that you do not need to worry about the hardware it is running on. All VMs can have the same set of virtual hardware components. To the guest operating system, they all look the same, no matter what network cards or RAID controllers physically exist in a server. That makes migration quite easy, even if you do not use VMware or its migration tools.

Virtualisation can increase stability and security of your server

Okay, this one is a bit more complicated to explain. So when does a server usually become unstable?

• when it runs out of memory and swap space, so that the kernel randomly has to kill tasks/services
• when the load gets too high (often in conjunction with heavy swapping due to lack of memory)
• when it is being attacked from outside (DDoS)

The problem with these reasons is that the results are unpredictable and may lead to data loss or data inconsistency. Sometimes a hardware reset is necessary, as no remote access is possible any more (SSH daemon might have crashed already or takes ages to establish the SSH session).

As long as it’s not a DDoS attack which causes the problems, virtualisation together with it’s resource limitations for VMs and a remote console (as in VMware) can help to log onto your VM and fix the issues. In the worst case you might have to shut it down, but that’s not a hard-reset via remote power bar control (which is the worst thing you can do to a server). You just have to restart the VM, which is so much faster than restarting a physical server!

And even in case of a DDoS attack, you might be better of with virtualisation, if you have two NICs connected one of which is on a local/maintenance network. Then you still could have full access from there and could use the remote console to block ports or source IPs. That works even better, if you run a firewall within one of the VMs and have a virtual local network infrastructure set up within the virtualisation.

Now that sounds a bit complicated, doesn’t it? You might want to read about my example setup, which is not very complicated but very effective.

My example setup

To start playing with the free VMware ESXi, I ordered a nice machine from SoftLayer which comes with KVM-over-IP (and allows remote CD image mounts). So I literally do have full control over the server. But you could also choose any other host and do not need a KVM at all, if the host agrees to install VMware ESXi for you (which is easy and does not take more than half an hour).

After successful installation, I used the Virtual Infrastructure Client to set up a couple of virtual switches:

• vSwitch 0 is connected to the public interface card
• vSwitch 1 is a host-only switch for local networking between the VMs (not accessible from outside)
• vSwitch 2 is connected to the interface card on the management network (SoftLayer provides access to this special infrastructure via VPN — very nice!)

My first VM was meant to become the firewall. Now, you may ask: “Why the hell do you put the firewall onto a VM? You could use iptables, pf, ipfw within the VM.” Sure, I could. But why should I want to maintain firewalls for each single VM when I can do that centrally? Moreover, why should I reinvent the wheel? There are plenty of good firewall solutions out there, which come with so many extra features out of the box.

I went for pfSense. It comes with literally everything you might want to implement:

• stateful firewall
• NAT port-forwarding and 1:1 NAT (interesting if your dedicated machine has multiple IPs)
• web interface
• different VPN services (PPTP, IPSec, OpenVPN)
• traffic shaping (queues, prioritisation etc.)
• bandwith monitoring
• netflow hooks
• SNMP
• DHCP/DNS
• and a bunch of other packages which can easily be added to the configuration

The pfSense VM connects exclusively to vSwitch0 (public network) and to vSwitch1 (host-only local network). That means, all traffic to the other VMs goes through this firewall. There’s no way to circumvent that — neither for public sources nor for the VMs, which are only connected to vSwitch1 locally and listening on private IP ranges (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12). The VMs get their local IPs assigned by  pfSense’s DHCP (nice for quick experimental VM setups) and have to use pfSense also as gateway and DNS.

Currently I am experimenting with TFTP to boot VMs via the network/PXE, which is possible with VMware ESXi and VMware Server. That will allow extremely quick and slim VM installations.

Thanks to the many features of VMware ESXi, you can literally set up a virtual data centre in a box. Ok, a small one. And you gain full control about everything which happens in your small data centre.

So, to cut a long story short: Virtualisation helps to

• increase control over and maintainability of your services
• centralise certain tasks (like the firewall or bandwith control)
• make use of spare resources and reduce costs
• ease migration in case the hardware is no longer sufficient (you don’t need to worry about the hardware platform the VMs are running on)
• speed up development/experiments as a VM is deployed much quicker than a dedicated box

I hope, this article helped at least a bit to make you curious about virtualisation. Personally, I think that virtualisation does not only target big companies or VPS hosts. It is interesting for everybody who owns/rents dedicated boxes. And it is certainly worth the effort to give it a go!

This article is about virtualisation. Caution: Long article!

As I am professionally dealing with all sorts of Linux flavours (Debian/Ubuntu, RedHat, CentOS, Fedora) and BSD/Unix derivates (Solaris, FreeBSD, OpenBSD), I have always been keen on a real virtualisation solution. I have come across many different (para-)virtualisation solutions in desktop and server environments over the last couple of years. Just to give an outline, I will now list all of them (without any particular order). I’d like to give an overview of the main differences later. So here’s the list for virtualisation approaches:

• VMware ESXi / VMware Server
• XEN
• OpenVZ (Virtuozzo open source fork)
• Sun xVM (for some reason, many people do not see this one)
• QEMU (honestly, who of you knows that one? )
• FreeBSD jails
• Linux vserver
• VMware Fusion for Mac
• Parallels Desktop for Mac
• Linux Virtual Server (LVS)

Okay, here we have a bunch of different things in one big pot… First, we have a couple of desktop based products (Parallels Desktop, VMware Fusion). I am not going to go into details about them. Personally I think that for professional testing/dev environments, VMware Fusion is slightly better than Parallels Desktop, because its networking components are more sophisticated and also allow to boot via PXE/tftp from within a virtual machine! On the other hand Parallels Desktop is said to be slightly ahead regarding graphics performance. I did not see much difference. Apart from that, both do pretty much the same and allow to run a variety of operating systems on top of the MacOS platform. I am using both heavily and have never had any real problems (tested with Windows XP, FreeBSD, Solaris and a bunch of Linuxes as guests). Okay, enough about Desktop virtualisation, especially as neither VMware Fusion nor Parallels Desktop are for free. Well, they are almost and definitely worth the money. Should you ever need to run different OS on your Mac, I can recommend both.

Sun’s xVM sits somewhere in between desktop and server virtualisation. The Mac implementation is not yet very mature, so it is out of question for me. But on Linux it is known to run quite well. Certainly better than the Wine patchwork thingy.

And here we go. Finally, eh? How to virtualise your server for developement, testing, or production use…

The rising star and increasingly popular is certainly XEN. I used to work with it way back in the past (early 2.x, if I remember correctly). That time, it was hardly more than an experimental virtualisation. It was great to use for running a couple of Linux instances on one machine. But you could not expect much more from it. Quite annoying was the fact that you had to patch the host and guest kernels. Also, it was anything but well documented. As it is becoming more and more popular, I would expect that it has become more mature over time. For sure I can tell that it performed quite well. When it comes to more than “just” Linux guests, XEN does not have much to offer. Support for Windows seems to become better, but FreeBSD or Solaris are still out of question.

If I needed to go for a para-virtualisation which allows to run Linux guests only, I would go for OpenVZ. It is documented very well and offers loads of opportunities to assign ressources to its guests. As an Open Source fork of Virtuozzo, it moreover has proven reliability very well. It comes with pre-compiled Linux kernels and a couple of tools. All of them are easy to install and work right out of the box. As from my experience, the performance is awesome.

Neither XEN nor OpenVZ do have their own graphical interface. That’s where a tool named HyperVM kicks in. It’s not for free but claims to be the only graphical front-end for both XEN and OpenVZ. I’ve never tried it, but people who fancy graphical tools rather than command line tweaking, might want to give it a go.

Other Linux-only virtualisation approaches are Linux Virtual Server and Linux vServer. I’ve tested both in an early stage, but they really did not convince me. None of them was capable of assigning resources to guests (resources in terms of at least RAM and CPU soft and hard limits). OpenVZ does that much better, as it enables you to limit almost everything. However, Linux vServer at least runs very stable (we used that on Ubuntu Server production machines in the last company I worked for).

A bit of a special role takes QEMU. Unlike the other mentioned virtualisation approaches, it does not need any kernel manipulation or sophisticated configuration. It’s merely a userland program which provides an environment for a single virtual computer. This is great if you quickly want to test something on another platform. “Quickly” in terms of easy to set up, not in terms of performance. Compared to kernel-level virtualisations it performs rather poorly. Where that isn’t important, it however is a great tool.
I used it to prepare FreeBSD images for a server which I did not have KVM access to. After preparing the images on my local client, I uploaded them to the rescue system on the server (booted via PXE) and then wrote the image directly to disk. Those days, there was no other way for me to do it; and it worked unexpectedly great!

Another special approach can be found on FreeBSD: so-called jails. It is pretty similar to Linux-only virtualisation on Linux systems, although it’s obviously for FreeBSD only. You get fully separated guests (jails), which have to get an IP address assigned (kind of bridged mode) and operate completely independently from the host system. Together with layered file-systems, read-only and null mounts, you can set up very slim but secure guests. Also, it is possible to run guests on higher secure-levels (a FreeBSD thing), and make them extraordinary safe. They perform incredibly well.

And here we go: VMware Server and ESXi Installable. It’s not a secret that I like them more than any other virtualisation approach, because they allow to run any imaginable operating system as guests. Moreover they offer extremely good tools to create and manage guests and to overlook the health and performance. With the VMware Converter, it is possible to transfer guests between different dedicated servers. All of this for free!

When ESXi 3.5 was released for free on Monday, July 28th, I decided to install it immediately on a private server, because I wanted to find out in which way it differs from VMware Server (which has been for free since its first release).
First of all, it does not depend on the host operating system, because it is just a hypervisor which then allows to install any operating systems as guests. So basically, we do not have a host operating system at all here, which certainly improves stability, performance and maybe also security. Secondly, it allows to set up a virtual network infrastructure as well. Virtual switches, VLANs,… Basically it means, that ESXi allows you to run a small data centre virtually. This of course offers great opportunities for a variety of testing scenarios.

Apart from that, it does not really differ from VMware Server. Both ESXi and Server can be maintained either through Perl scripts (remotely) or through the VMware Infrastructure Client (also free, but requires Windows to run). It’s needless to say that VMware products are rock-solid and reliable due to the fact that they are widely spread and have been available for many years. Only the EULA of VMware ESXi and Server is still a bit mistakebly. I started a discussion thread in the VMware community forum to get that clarified.

In another blog entry, I’d like to explain why virtualisation is much more interesting for many people than they might think. And I’ll also give an example with my own private server setup.

Under very high load and many concurrent read requests (I set up the company’s mail server with ZFS and root from ZFS), the two disks in the Raid array repeatedly lost sync, forcing an automatic re-silvering (auto healing) process to be started, which blocked the system as everything (except /boot) was running from that ZFS arrray. As far as I figured out, the system halted entirely as there was another inconsistency occuring while the re-silvering was still in progress.

I would have investigated further, if it wasn’t a crucial production machine. And that kind of traffic is very difficult to simulate under laboratory situations (maybe I can do that when I have more time). So I had to revert back to UFS as the downtime had to be minimized. It’s a shame, really, because I love the features ZFS offers. On my private server it runs very smoothly, but traffic, load and I/O are not comparable to the mail server in question.

But as I took a closer view at it recently, it turned out to be a pretty simple task. Only people with loads of domain names and/or restricted nameserver control might run into problems.

In a nutshell, you only need to do this in order to get SPF running on your own server and to tell other mail servers how to deal with your domain name:

• add a policy daemon to your MTA (e.g. postfix-policyd-spf) — pretty easy, really!
• add SPF/TXT records to your zonefiles as described here

That’s it, honestly. Assuming, you have successfully finished the two tasks, your mail server will block all mails that originate from domains, which have SPF records set and been delivered through other than the allowed hosts. And your domains will be protected from being misused on all other mail servers which use SPF as well.

Example:

example.com.       IN MX 10  mail
mail.example.com.  IN TXT "v=spf1 mx -all
mail.example.com.  IN A 1.2.3.4

Now, mails from johndoe@example.com must be delivered through the mailserver mail.example.com. All mailservers which implement SPF will refuse to accept mails from other than that server. Especially the big players like Googlemail do make use of SPF. Although some don’t block mails, they at least add a telling header which makes spam-filtering easier:

Received-SPF: fail (google.com: domain of mail@***.co.uk does not designate
85.***.***.*** as permitted sender) client-ip=85.***.***.***
Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of
mail@***.co.uk does not designate 85.***.***.*** as permitted sender)
smtp.mail=mail@***.co.uk

So what are you waiting for? The more people make SPF mandatory on their servers, the better is its protection against SPAM.

After having set up the minimal FreeBSD and doing some tuning (such as creating the ZFS volumes), I ran some tests. You won’t believe me, but writing a 10GB file (/dev/random to the ZFS volume) resulted in a transfer rate at about 160MB/sec and reading (cp testfile /dev/null) was done at a speed of more than 270MB/sec!!

To be continued…

Consequently we asked for quotes on bigger NetApp devices. Unfortunately, they cost more than a good car. The investigation on other solutions began…

After doing some research and testing as well performance as failure behaviour, the decision was made: I will setup a storage cluster based on i386 hardware and GlusterFS. Its speed was quite impressive. And also the fact that mirroring (for data security) and striping (for performance) can be combined, is very convincing.

We will start with two huge servers which run partly in mirrored and partly in striped mode. The only thing which needs to be tested before hand is, if FreeBSD’s UFS in combination with its snapshot feature makes sense here (this could be a bit tricky). If it works, this solution will be as good as the proprietary offer. But it costs less than 1/5!

I will keep you posted.