librelist archives

« back to archive

Using Parallel with Memory.cache'd functions

Using Parallel with Memory.cache'd functions

From:
Nicolas Pinto
Date:
2011-06-29 @ 23:28
Hello,

What is the recommended way of using Parallel with functions that have
been decorated with Memory(...).cache ?

Thanks for your time.

Regards,

-- 
Nicolas Pinto
http://web.mit.edu/pinto

Re: Using Parallel with Memory.cache'd functions

From:
Nicolas Pinto
Date:
2011-06-29 @ 23:34
Here is an example of what I'd like to get working:
https://gist.github.com/1055274

Right now, this throws a "TypeError: can't pickle function objects".

 <https://gist.github.com/1055274>HTH

Thanks.

N

On Wed, Jun 29, 2011 at 7:28 PM, Nicolas Pinto <nicolas.pinto@gmail.com>wrote:

> Hello,
>
> What is the recommended way of using Parallel with functions that have
> been decorated with Memory(...).cache ?
>
> Thanks for your time.
>
> Regards,
>
> --
> Nicolas Pinto
> http://web.mit.edu/pinto
>



-- 
Nicolas Pinto
http://web.mit.edu/pinto

Re: [joblib] Re: Using Parallel with Memory.cache'd functions

From:
Gael Varoquaux
Date:
2011-06-30 @ 12:11
On Wed, Jun 29, 2011 at 07:34:49PM -0400, Nicolas Pinto wrote:
>    Here is an example of what I'd like to get working:
>    [1]https://gist.github.com/1055274
>    Right now, this throws a "TypeError: can't pickle function objects".

Don't use the '@', do something like:

sqrt3 = mem.cache(sqrt2)

print Parallel(n_jobs=4, verbose=True)(delayed(sqrt3)(i**2) for i in
    range(10))

G

Re: [joblib] Re: Using Parallel with Memory.cache'd functions

From:
Nicolas Pinto
Date:
2011-06-30 @ 15:56
Great, thank you Gael.

Did I miss this in the docs?

N

On Thu, Jun 30, 2011 at 8:11 AM, Gael Varoquaux
<gael.varoquaux@normalesup.org> wrote:
> On Wed, Jun 29, 2011 at 07:34:49PM -0400, Nicolas Pinto wrote:
>>    Here is an example of what I'd like to get working:
>>    [1]https://gist.github.com/1055274
>>    Right now, this throws a "TypeError: can't pickle function objects".
>
> Don't use the '@', do something like:
>
> sqrt3 = mem.cache(sqrt2)
>
> print Parallel(n_jobs=4, verbose=True)(delayed(sqrt3)(i**2) for i in
>    range(10))
>
> G
>



-- 
Nicolas Pinto
http://web.mit.edu/pinto