librelist archives

« back to archive

joblib and max user procs

joblib and max user procs

2015-03-25 @ 21:14
I'm on centos 6.3 and when I launch processes via joblib.Parallel with
the multiprocess backend, my processes all hang (inside Parallel, i
don't remember which line of code), after Parallel has launched
n_procs * max user procs for my machine (i.e. ulimit -u). I've
verified this by changing the os setting. However, I don't understand
why this is the case, since the process limit is usually related to
the number of processes running simultaneously. It seems like joblib
or multiprocessing is reusing whatever subshell it creates to launch
processes even after a process has exited. But beyond that I don't
know what's going on or how to workaround it. Just wondering if anyone
has any insights about this.


Re: [joblib] joblib and max user procs

Olivier Grisel
2015-03-26 @ 12:54
I replied on github. Let's move the discussion there: