Turns out that when you are interfacing something to the kernel, it is a REALLY bad idea to log to stderr, ESPECIALLY if it is exceedingly rare, and only in your debug logging code. Grrr.

I had some code that was writing to stderr when sockets failed on their Amazon S3 connections, but only in the debug logging mode, and it only happens every couple of hours. That led to my machine deadlocking and dying due to the kernel blocking while waiting for a response from my pseudo loopback device, which was blocking while waiting for stderr to get written to a standard file/stderr , which couldn't be written due to waiting for a write to the loopback device, which couldn't................... BARF!

Mental note: use klog. Really.