librelist archives

« back to archive

Question about how stdout/stderr are stored

Question about how stdout/stderr are stored

Kelsey Prantis
2014-11-21 @ 17:49
Hello there,

We have been using paramiko in our testing, and have a little convenience
function for sending ssh commands to our various test servers easily and
returning the stdout/stderr/exit status. We were considering potential
performance impacts for commands that return a *very* large amount of
output to stdout or stderr, and were wondering if someone might help with
some understanding about how that data is stored. Please bear with me if
anything I say below is nonsense since my understanding of the paramiko
code is essentially nil.

So, if we do a foo=channel.makefile('rb').read(), foo is now going to carry
around the entire stdout output, which could potentially be very large and
not even be used wherever the function is being called.

If we returned the channel.makefile('rb') itself, and read() only when its
used, we were having issues with it occasionally returning an empty string
(presumably because the channel stream had closed when the channel was
garbage collected?)

We thought about returning the channel itself, so it wouldn't be garbage
collected and the caller could decide to read from the channel only if
needed, but that left us wondering, how *is* that result actually stored?
If we pass around the channel, is that long stdout all stored in memory
along with it anyways so it isn't any practically different than the first
option? Or does it actually wait to read it from the remote server until
you call channel.makefile('rb').read() and this would save putting a large
stdout that wasn't used later into memory?

Hope this question makes sense,