Python subprocess and stderr

[zaitcev]

Suppose you want to create a pipeline with the subprocess and you want to capture the stderr. A colleague of mine upstream wrote this:

p1 = subprocess.Popen(cmd1, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True) p2 = subprocess.Popen(cmd2, stdin=p1.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True) p1.stdout.close() p1_stderr = p1.communicate() p2_stderr = p2.communicate() return p1.returncode or p2.returncode, p1_stderr, p2_stderr

Unfortunately, the above saves the p1.stdout in memory, which may come back to bite the user once the amount piped becomes large enough.

I think the right answer is this:

with tempfile.TemporaryFile() as errfile: p1 = subprocess.Popen(cmd1, stdout=subprocess.PIPE, stderr=errfile, close_fds=True) p2 = subprocess.Popen(cmd2, stdin=p1.stdout, stdout=subprocess.PIPE, stderr=errfile, close_fds=True) p1.stdout.close() p2.communicate() p1.wait() errfile.seek(0) px_stderr = errfile.read() return p1.returncode or p2.returncode, px_stderr

Stackoverflow is overflowing with noise on this topic. Just ignore it.