A System Administrator’s Diary

Waking up from hibernation

After years of Systems Administration I have completely forgotten how much I used to like developing in Java. My current employer gave me that opportunity recently. So I took ownership and responsibility to de-hibernate myself (my Java knowledge became a bit rusty) and to start working on a project which separates the website’s business logic layer from the database layer. That becomes necessary as we want to re-design our infrastructure in a way that enables us to be as platform independent and as scalable as possible.  

On the one hand, we are not even entirely sure if we’ll stick to MySQL or switch to any other DB server type, hence we must not use SQL dialect specific code within the application. On the other hand, we want to provide a SOAP endpoint internally, which allows us to access data from all sorts of applications and clearly separates the business logic of all sorts of clients from our actual data store.

In addition, caching of object structures which are mapped against data structures (or tables) would be very neat. Consequently, I suggested and started using Java with Hibernate and JAX-WS on a Glassfish Java Application Server. We might integrate an additional caching layer later. Terracotta is one of the candidates, which integrates seamlessly and can increase speed by factor 2 to 10. 

After one week of researching, developing proofs of concept and presenting a first prototype, I am sooo excited about all that.  I always loved Java, and again I remember why.

Posted in Java | No Comments »

PowerDNS w/ MySQL — should have tried earlier

I was quite fed up with BIND (zonefile editing, serial number increasing, master/slave setups), so I was searching for alternatives to run Master/Slave DNS services. More or less randomly, I again came across PowerDNS. Again, because I have heard a couple of times that big providers quite often use it. So I decided to give it a go. On FreeBSD it’s quite easy. You just install it together with its MySQL modules, activate the modules in the configuration files, import the database schemata, and fill it with content. A script to import BIND zone files directly to the DB is also included. Couldn’t be easier. 

Once you’ve set it up and got it running, you of course need a slave server as well. So all you got to do is either to install another PowerDNS/MySQL instance on another machine and connect it to the same database (you wouldn’t really want to do that), or connect it to a second MySQL server, which ideally is a replication slave of the first one. Then you have a fully redundant setup and don’t need to worry about failures of either of those. All DNS record changes go directly to the master DB server, which will be picked up by the slave in almost real time. Plus, you don’t need to worry about serial numbers. Change a record in the DB and this change will be published instantly. 

If you fancy web interfaces, you can either go for an open source one, or you can write one on your own. As the DB structure of PowerDNS is kept very simple, it’s easy to add/modify records via script also. Do whatever you want and don’t worry about the stupid restrictions and configuration hurdles you used to run into when going for a BIND server.

Posted in Software | No Comments »

Distributed Data Centre in-a-box

Weird title, right? Well, it’s also a weird project, which I am doing just for fun and as a kind of proof-of-concept. The idea is to build a “micro data centre” on a single VMware ESXi based machine and to fully replicate all parts of it onto another single ESXi based machine, which is located in another real data centre. That sounds easy, right? But hold on a second.

Read the rest of this entry »

Posted in Experimental Projects, Virtualisation | 3 Comments »

Why should you bother to use virtualisation?

Be honest: You are running at least one dedicated server, and you certainly have asked yourself whether you should use virtualisation. You might have found “no” to be the answer, as you have one server for each purpose and do not plan to migrate to other hardware machines or to “sub let” your system. That’s ok. But on the other hand, everybody likes to reduce hardware costs, or make more of the hardware they have.

It’s a prejudice that virtualisation is only interesting for so-called Virtual Private Server providers or for big companies who need to run loads of tests for their software releases on different platforms and configurations. Also, you do not need a bunch of servers or a blade-center to take an advantage on virtualisation. In this article I’d like to give an example of what can be achieved with virtualisation apart from those typical and well-known scenarios.

Read the rest of this entry »

Posted in Virtualisation | No Comments »

Free Virtualisation at its best!

Did you realise that VMware released ESXi for nothing recently? If not, get it and try it! All you need is a server with full KVM access (either over IP or locally) including the opportunity to install CD images (remotely). In my case, I went for a neat SuperMicro machine from SoftLayer, which offers full KVM-over-IP with remote CD mounts (ISO images on network storages or on the client computer). But that’s not important here, although they actually offer Double-RAM/Double-Drive deals, so that I am paying only US$ 311 for a Quad-Core Xeon, 6GB of RAM and 2×146 GB SAS at an Adaptec RAID controller. It’s an awesome offer, especially because they have an incredible user interface and are extremely flexible when it comes to updates/changes. Okay, okay. Enough advertising

This article is about virtualisation. Caution: Long article!

Read the rest of this entry »

Posted in Virtualisation | No Comments »

ZFS on FreeBSD 7 — experimental for a reason!

Yes, it has a reason that ZFS is not yet marked stable on FreeBSD! I had to learn the hard lesson today.

Under very high load and many concurrent read requests (I set up the company’s mail server with ZFS and root from ZFS), the two disks in the Raid array repeatedly lost sync, forcing an automatic re-silvering (auto healing) process to be started, which blocked the system as everything (except /boot) was running from that ZFS arrray. As far as I figured out, the system halted entirely as there was another inconsistency occuring while the re-silvering was still in progress.

I would have investigated further, if it wasn’t a crucial production machine. And that kind of traffic is very difficult to simulate under laboratory situations (maybe I can do that when I have more time). So I had to revert back to UFS as the downtime had to be minimized. It’s a shame, really, because I love the features ZFS offers. On my private server it runs very smoothly, but traffic, load and I/O are not comparable to the mail server in question.

Posted in BSD, Storage, Work | No Comments »

SPF — Sender Policy Framework

Did you ever receive spam mails which seem to originate from your own mail address? Or did anybody else complain about you sending those emails? Then you should take a look at SPF. In a nutshell, it plugs the holes in the SMTP protocoll, which does not allow to verify if a sender (or anybody who pretends to be that sender) really may use a particular mail server to transmit their mails. Unfortunately, SPF is not yet very wide-spread. Almost everybody (including me) has come across this abbreviation and/or heard that it might protect misuse of mail addresses. But most people (including me) cannot be bothered to implement it.

Read the rest of this entry »

Posted in Software, general | No Comments »

Two neat storage machines

Yesterday I set up one of our new storage machines for testing: Dell 2950, Quad Xeon, 8GB, 6×750 GB HDD. I installed FreeBSD 7 with ZFS (following up this article). Firstly it seemed to be a bit tricky, because the PERC/6i controller configuration is — sorry — crap from the usuability point of view. It seemed not to support non-RAID configurations, but taking a closer look it turned out to be a wrong assumption. Six RAID-0 Arrays with only one drive each is in fact the same as no RAID at all. (The reason why RAID does not make sense is, that ZFS will do this job, and its auto-healing is much better than any hardware controller’s auto-healing)

After having set up the minimal FreeBSD and doing some tuning (such as creating the ZFS volumes), I ran some tests. You won’t believe me, but writing a 10GB file (/dev/random to the ZFS volume) resulted in a transfer rate at about 160MB/sec and reading (cp testfile /dev/null) was done at a speed of more than 270MB/sec!!

To be continued…

Posted in BSD, Storage, Work | 2 Comments »

Solaris’ ZFS on FreeBSD 7.0

Wasn’t it annoying that FreeBSD never had a volume manager which allowed to resize volumes including the contained filesystem on the fly? Well, it was possible to use gvinum in conjunction with UFS’ growfs tool to make filesystems bigger at least. But that’s not as flexible as Solaris’ ZFS. Moreover, UFS does not support snapshots as well as ZFS does. But finally we have a solution: Since FreeBSD 7.0, ZFS is part of the OS! Although it is still considered experimental and lacks some of the original’s features, it works quite well. Also, it is possible to use ZFS as a boot partition! See this great article.

Posted in BSD | 1 Comment »

(C|G)lustered Storage

As the company I work with has to store many media files, backups, rapidly changing documents and so on, they used to run a NetworkAppliance FAS2020 storage machine, which is quite neat. Unfortunately, the current setup does not allow to scale the volumes any more. So we needed to find an alternative.

Read the rest of this entry »

Posted in BSD, Storage, Work | No Comments »