librelist archives

« back to archive

IPython extension for transparent use of joblib.Parallel

IPython extension for transparent use of joblib.Parallel

From:
Dan Stahlke
Date:
2013-09-29 @ 02:16
I created an IPython extension to allow convenient use of joblib.Parallel:
https://gist.github.com/dstahlke/6748551

Usage is like so:  (note the "parfor" keyword)

 >>> %load_ext parfor
parfor extension activated.
 >>> [2*x parfor x in range(10)]
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
 >>> np.sum([ 
np.max(np.fft.fft(np.random.standard_normal(10000000)).real) parfor i in 
range(10) ])
110820.94298349788

To set custom options:

 >>> _parfor = joblib.Parallel(n_jobs=-1, verbose=5)
 >>> [2*x parfor x in range(10)]
[Parallel(n_jobs=-1)]: Done   1 out of  10 | elapsed:    0.0s 
remaining:    0.0s
[Parallel(n_jobs=-1)]: Done   3 out of  10 | elapsed:    0.0s 
remaining:    0.0s
[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.0s 
remaining:    0.0s
[Parallel(n_jobs=-1)]: Done   9 out of  10 | elapsed:    0.0s 
remaining:    0.0s
[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.0s finished
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

Please let me know of any suggestions you may have.

- Dan