librelist archives

« back to archive

Re: [joblib] joblib can't pickle function object?

Re: [joblib] joblib can't pickle function object?

From:
Gael Varoquaux
Date:
2014-03-07 @ 22:15
Yes. This cannot work because your function is only defined in the closure
of the function. Therfore it cannot be imported which means that it is 
much harder to do parallel computing. 

We could try using dill pickle. That might help. Do you want to try to do 
a pull request on that? 

Gaël

<div>-------- Original message --------</div><div>From: Skipper Seabold 
<jsseabold@gmail.com> </div><div>Date:07/03/2014  22:40  (GMT+01:00) 
</div><div>To: joblib@librelist.com </div><div>Subject: [joblib] joblib 
can't pickle function object? </div><div>
</div>Apparently I have to send this e-mail twice? I'm a bit baffled by 
this and am hoping to get some help to understand. I would assume in the 
second example that they're in the same scope so there's no closure. I'm 
on joblib 0.7.1 installed using pip on python 2.7.

These work

from joblib import Parallel, delayed

def sqr(i):
    return i ** 2
 
Parallel(n_jobs=4)(delayed(sqr)(i) for i in range(50))

def func(x):
   return Parallel(n_jobs=4)(delayed(sqr)(i) for i in x)

func(range(50))

But this doesn't work?

def func(x):
    def sqr(i):
        return i ** 2
    return Parallel(n_jobs=4)(delayed(sqr)(i) for i in x)

func(range(50))

Re: [joblib] joblib can't pickle function object?

From:
Skipper Seabold
Date:
2014-03-10 @ 21:12
On Fri, Mar 7, 2014 at 5:15 PM, Gael Varoquaux
<gael.varoquaux@normalesup.org> wrote:

Hi Gaël,

> Yes. This cannot work because your function is only defined in the closure
> of the function. Therfore it cannot be imported which means that it is much
> harder to do parallel computing.

I guess that makes sense. I assumed that the fork would have the same
local environment namespace that Parallel was called from.

>
> We could try using dill pickle. That might help. Do you want to try to do a
> pull request on that?

TBH, nope, unfortunately. Not at the moment. I'm not familiar with
this code at all.

Thanks,

Skipper

>
> Gaël
>
>
> -------- Original message --------
> From: Skipper Seabold
> Date:07/03/2014 22:40 (GMT+01:00)
> To: joblib@librelist.com
> Subject: [joblib] joblib can't pickle function object?
>
> Apparently I have to send this e-mail twice? I'm a bit baffled by this and
> am hoping to get some help to understand. I would assume in the second
> example that they're in the same scope so there's no closure. I'm on joblib
> 0.7.1 installed using pip on python 2.7.
>
> These work
>
> from joblib import Parallel, delayed
>
> def sqr(i):
>     return i ** 2
>
> Parallel(n_jobs=4)(delayed(sqr)(i) for i in range(50))
>
> def func(x):
>    return Parallel(n_jobs=4)(delayed(sqr)(i) for i in x)
>
> func(range(50))
>
> But this doesn't work?
>
> def func(x):
>     def sqr(i):
>         return i ** 2
>     return Parallel(n_jobs=4)(delayed(sqr)(i) for i in x)
>
> func(range(50))