I have started using nginx web server for testing Enomalism (to supply the https proxy), and I am really impressed. I also tried to run some php/static serving stuff over it, and the speed increase blew me away. Then I checked out the capacity to handle massively concurrent connections, and was REALLY blown away. I was so excited I actually sent benchmark results to Khaz.

Anyhow, I think that I would like to investigate using nginx with fastCGI and PHP for future high traffic sites, and I am thinking that the combo might actually be the magic bullet for memory heavy sites like Typo3 where the apache process, even with extraneous modules disabled, easily adds another 10M or so per pageview. This becomes even more important on VPS servers where RAM is the most limiting resource you have.

As an added bonus, nobody is targeting nginx with attacks right now.

Benchmarks here:
http://blog.kovyrin.net/2006/08/28/ruby-performance-results/

I have been running into some unusual problems in Apache lately, and thought I would document them.

If you get this error when trying to start Apache, and only a reboot will fix it:

[emerg] (28)No space left on device: Couldn't create accept lock

Run this bash command:

ipcs -s | grep apache | awk ' { print $2 } ' | xargs ipcrm sem

I would also recommend adding this function to the end of the stop function in /etc/init.d/httpd if the problem persists. The cause of the problem seems to be crashed apache forks that refuse to respond to a SIGTERM, and must be killed. The problem is that the semaphores are used up for the Apache user, and that results in the apache startup failing. One possible workaround is reducing the max pages served by a process, but testing hasn't proven this out yet.

While working on my Enomalism SimpleFirewall module, I had some difficulties getting a reliable firewall up and running. It turns out that XEN firewalls are a little tricky to configure. Who knew.

So while you COULD use Enomalism to easily set up your virtual machines, some people INSIST on doing things the hard way, so here are some quick details...

When XEN creates a domain in it's default configuration (xen-bridge and vif-bridge), it creates a virtual interface with a particular numbering scheme. If you create a virtual machine named george, which starts as domid 2, it will have a virtual interface labelled as vif2.0 for it's network traffic. If you have a virtual machine that contains iptables (using either pygrub or HVM), you can easily configure iptables on the DomU to protect this machine. Sadly, this is not usually the case, and there are some SNAFUs if you are running a linux non pygrub machine (which is normally my preference).

(sorry for the scarcity lately, but Enomalism is going 0.7 this very week, and the enterprise version just got limited beta released, so they have kept me pretty busy again).

I was doing some research on best practices scalability recently, and found that there is really very little useful information available that covered all of the bases. Because of this, I am building out a web platform for DomU deployment that will provide the following:
- MySQL replication for Drupal scalability
- Pound clustering for load balancing (userspace for DomU tools)!
- Squid for caching and efficiency.
- Gluster for a shared file upload and content storage directory.
- Add a dash of Stunnel for cert based security on Gluster.

The entire thing will be build into an easy to configure DomU image, based on Debian 3.1.

Watch and wait!

I realize that I have been a little scarce on the Enomalism fora for the last few days, but it is not due to laziness... I have been frantically adding new features:

• XVM (Xen Virtual Machine) image file format, with an easily extensible GPL library for creating and editing XEN machines. With compression and portability
• Another.... secret file format, which should be announced soon enough. Support for various machine types, including hardware virtualization
• Completely rewritten provisioning scheme
• REAL SSH for the virtual terminals
• Transaction backend (hopefully on this release, but maybe next week) which allows for throttling of the asynchronous backend tasks like provisioning and hard drive resizing
• Further dummyproofing of the installer to avoid some of the issues in the forum

Enomalism 0.6.1 got pushed out by one day today. I had some issues on my main test server which resulted from upgrading to 3.0.4 directly from 3.0.2 when testing against the various Xen platforms. Needless to say, I am going to re-install the OS, but that will take some time as well.

Troubles aside, I should be able to get 0.6.1 out later today, or tomorrow morning at the latest.

It only took a few months ;) , but we finally churned out the 0.5.5 release, followed immediately by the 0.6 release just this monday. Next up we are going to have (wait for it), the 0.6.1 release on Monday. I expect that this release will have a clever management import script so that people who started without Enomalism can migrate over without too much trouble, and automatically import their old Virtual Machines.

Many other "alpha" quality features should make it into this release. Keep yer' eyes open!

enomalism.com

How the hell do you do that anyways?!

Pictures soon!

I have completed a beta 0.5 version of Enomalism, with new installer functionality, and a much easier to use interface. I plan to release an improved version with full support for IE, and an automatic retrieval system for new DomU images, sometime in the middle of December.

Updated installer, with better detection of python requirements is attached. Also, improved sysV scripts (for Ubuntu) installed.

Possible bug in xen.lowlevel.xs python libs here: Bug 824 (not externally verified)

Enomalism 0.5 With Installer (Download)
Enomalism 0.5.1 With Installer (Download)
Enomalism 0.5.2 With Installer (Download)

To install:

cd /opt
tar xzvf enomalism-0.5.tgz
cd enomalism
./preinstall.py
(answer all the questions, follow the instructions)
/etc/init.d/enomalism start
/etc/init.d/httpd restart (if you installed the apache ssl proxy in the installer)

This story about the risks of software development, and the methodology for mitigating them (original manifesto in PDF Format), really encapsulates the problems that a developer can hve when working for a client.

What is really great is that they propose workable solutions to prevent these problems, and methods for handling them when they occur. While a lot of the methods should be familiar for a seasoned vet, there were still some ideas in there which I hadn't really thought of before. Even better, the document is written in a way that informs the client without putting them down, so you can print it out and hand it to them!