Annoying Twisted Python Problem

So... Twisted has an issue. A very annoying one.

Subprocess doesn't work. The reason? No idea ;) Twisted installs default signal handlers for EVERYTHING, which results in some odd behavior. The specific problem we are running into with Enomalism is here (via bug 2535):

Twisted currently needs to install a SIGCHLD handler for reactor.spawnProcess to work reliably. This is problematic when Twisted is used with other libraries that depend on SIGCHLD (e.g. the standard library's subprocess module depends on SIGCHLD being left as SIG_DFL, otherwise it can get EINTR at unexpected times). Hence the reactor can be run with the installSignalHandlers=False flag, but then reactor.spawnProcess isn't reliable.

Seriously though, launching subprocess calls from inside Twisted python is a good way to end up with a dead reactor. To get around this, make sure that you call your subprocess Popens from inside a Fork, whenever possible. This means that if you pull libraries in that need to use subprocess, the twisted web methods MUST fork before calling. You can check the return codes using return values, or redirect to a stdout file descriptor if necessary. Actually, if you want (unlike me) to not be an idiot, this little snippet completely works around the problem AFAICT except for the fact that reactor.spawnProcess will no longer work (nope. It still crashes, it just takes longer): THERE IS A COMPLETE WORKING LIBRARY POSTED WAY BELOW

import signal 
def handle_signal(signum,stackframe):
    if signum in [signal.SIGHUP,signal.SIGINT   ]:
        reactor.stop()
#The internal twisted signal handling sux0rs, so we run our own.
signal.signal(signal.SIGHUP,handle_signal)
signal.signal(signal.SIGINT,handle_signal)
reactor.run(installSignalHandlers=False)

Yes Virginia, I am writing a module that gives complete subprocess functionality from inside the fork (well, most of it), but I plan to just migrate to tgWebServices eventually anyhow.

UPDATE: I created this little library that handles subprocess calls. You DO NOT WANT THIS VERSION. YOU WANT THE NEW ONE. THERE IS A BUG IN PTY THAT MAKES THIS ONE UNSTABLE. There is a new version available here: New Version of subprocess_helper

#!/usr/bin/python2.4
"""Process helper application"""
#Enomalism; Web based XEN Management console
#Copyright (C) 2007 Enomaly
#
#This program is free software; you can redistribute it and/or
#modify it under the terms of the GNU Lesser General Public
#License as published by the Free Software Foundation; either
#version 2.1 of the License, or (at your option) any later version.
#
#This program is distributed in the hope that it will be useful,
#but WITHOUT ANY WARRANTY; without even the implied warranty of
#MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
#Lesser General Public License for more details.
#
#You should have received a copy of the GNU Lesser General Public
#License along with this program; if not, write to the Free Software
#Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
#
#Contact Enomaly at:
#180 Senneville,
#Montreal, Quebec H9X 3X2 Canada

import subprocess,os,sys,signal,pty,time,errno,thread

        
def smart_wait_for_subprocess(sp,timeout=30):
    """
    Will wait for Process given to expire, and then kill it if
    the time elapses (or never if timeout==0). 
    @param sp: Subprocess to watch
    @param timeout: timeout length in seconds
    """
    try:
        sleeps=timeout*4
        #Just wait around for termination...
        while sp.poll()==None:
            sleeps-=1
            time.sleep(.25)
            if sleeps<=0 and timeout:
                os.kill(sp.pid,signal.SIGTERM)
                break
        #Did it REALLY exit? 
        while sp.poll()==None:
            sleeps-=1
            time.sleep(.25)
            #No? KILL IT!
            if sleeps<=0 and timeout:
                os.kill(sp.pid,signal.SIGKILL)
                break
    except:
        #Make REALLY effing sure it exited.
        os.kill(sp.pid,signal.SIGKILL)
    
    
        
    
def call_subprocess(args,timeout=30,stdin=None):
    """
    Will take a set of arguments and run them as a subprocess
    call inside of a thread, which will always exit after
    timeout seconds (or never if timeout==0). Pushes stdin
    into the pipe on the communicate call if it is given.    
    Designed to be called by run_in_subprocess, as it doesn't 
    have any forking in it.
    Returns tuple of (returncode,stdout,stderr)
    @param args: Subprocess arguments
    @param timeout: max time to wait for completion
    @param stdin: Input to stuff into stdin
    """
    P=subprocess.Popen(args,stdout=subprocess.PIPE,stderr=subprocess.PIPE,stdin=subprocess.PIPE)
    smart_wait_for_subprocess(P,10)
    try:
        returncode=P.returncode
        stdout=P.stdout.read()
        stderr=P.stderr.read()
        if not stderr:
            stderr=""
        del P
    except Exception,e:
        stdout="ERR:",str(e)
    if not stdout:
        stdout='\n'
    return (returncode,stdout,stderr)
        

def run_in_subprocess(args,timeout=30,stdin=None):
    """
    Will take a set of arguments and run them as a subprocess
    call, which will always exit after
    timeout seconds. Pushes stdin
    into the pipe on the communicate call if it is given.    
    Uses fork, and requires no other signals to work. Good for
    running inside of twisted.
    Returns tuple of (returncode,stdout,stderr)
    @param args: Subprocess arguments
    @param timeout: max time to wait for completion
    @param stdin: Input to stuff into stdin
    """
    returncode,stdout,stderr=(1,"","")
    pid,fd=pty.fork()
    if pid: 
        #We are in the original process. Wait for output with waitpid...
        returned=os.waitpid(pid,0)
        returncode=os.WEXITSTATUS(returned[1])
        try:
            sof=os.fdopen(fd,'r')
            stdout=sof.read()
        except Exception,e:
            if returncode==0:
                stdout=None
                stderr="No returned data"                
    else: 
        #We are in the forked process
        try:
            try:
                returncode,stdout,stderr=call_subprocess(args,timeout)
                sys.stdout.write(str(stdout)+str(stderr))
                os._exit(returncode)
            except:
                os._exit(1)    
        finally:
            os._exit(returncode)
        os._exit(1) #Are we REALLY sure?
    return returncode,stdout,stderr
    
    
if __name__=='__main__':
    #print run_in_subprocess(['/bin/cat','/proc/cpuinfo',])
    #print run_in_subprocess(['/bin/sleep','3',],15)
    print run_in_subprocess(['/usr/bin/sudo','/usr/sbin/xm','create','/xen/06faaf7e3a238bfbea3c729353a13e16_pvtest2/xen.cfg'],10)
    time.sleep(5)
    print run_in_subprocess(['/usr/bin/sudo','/usr/sbin/xm','shutdown','/xen/06faaf7e3a238bfbea3c729353a13e16_pvtest2/xen.cfg'],10)
    print run_in_subprocess(['/usr/bin/sudo','/sbin/iptables','-L','FORWARD'],10)
    print run_in_subprocess(['/usr/bin/sudo','/sbin/iptables','-L','SF_moo92'],10)


__all__=['run_in_subprocess',]
Home Home