librelist archives

« back to archive

Logging to file from workers

Logging to file from workers

Matthias Seise
2014-04-04 @ 07:46
Dear joblist users,

I'm trying to work out how to (correctly) log from multiple processes
(using joblib.Parallel) to a single file. How do you handle it?

(I'm working in Windows 7, python 2.7.2 64 bit) - The "standard" way using
a network logger as in the logging Cookbook doesn't work, the network
logger crashes after a while (~1hour).

Any pointers would be appreciated!


Re: [joblib] Logging to file from workers

Olivier Grisel
2014-04-04 @ 08:34
I would make each worker process write results in its own file
(possibly with timestamp information) and from time to time have a
single finalizer process that
consolidate the results in the final file.