Posts Tagged ‘File descriptor’

Debugging python (multi)processing

Thursday, January 7th, 2010
CPython
Image via Wikipedia

My goal is to get the pdb shell from the worker processes i spawned with Process() from python-processing. The “classic” approach to spawning the pdb shell miserably fails:

(Pdb) > /home/redduck666/dev/abj/bin/feeds.py(639)__init__()
-> self.timeout = timeout
Process Process-3:2:
Traceback (most recent call last):
  File "/var/lib/python-support/python2.5/processing/process.py", line 227, in _bootstrap
    self.run()
  File "/var/lib/python-support/python2.5/processing/process.py", line 85, in run
    self._target(*self._args, **self._kwargs)
  File "./feeds.py", line 639, in __init__
    self.timeout = timeout
  File "./feeds.py", line 639, in __init__
    self.timeout = timeout
  File "/usr/lib/python2.5/bdb.py", line 48, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/lib/python2.5/bdb.py", line 66, in dispatch_line
    self.user_line(frame)
  File "/usr/lib/python2.5/pdb.py", line 144, in user_line
    self.interaction(frame, None)
  File "/usr/lib/python2.5/pdb.py", line 187, in interaction
    self.cmdloop()
  File "/usr/lib/python2.5/cmd.py", line 130, in cmdloop
    line = raw_input(self.prompt)
ValueError: I/O operation on closed file

The problem here is that processing closes the file descriptors for the processes it spawns, so a straight forward approach like that will not work. Due to the same reason using sys.__std(out|in|err)__ will not work.

The solution for me was to tell explicitly python to use my current stdin/stdout. The ‘r+’ flag is needed as pdb needs to read from stdin.

pdb.Pdb(stdin=open('/dev/stdin', 'r+'), stdout=open('/dev/stdout', 'r+')).set_trace()

I use this on Linux, AFAIK it should work across Unix world (and is probably horribly broken on Windows).

Blog Widget by LinkWithin