librelist archives

« back to archive

Combining Memory and Parallel

Combining Memory and Parallel

From:
Andreas Hilboll
Date:
2015-04-29 @ 08:40
Hi,

I'd like to use the Memory caching functionality together with the
Parallel function -- so all of joblib combined together.

Here's what I have:

   import dill
   from joblib import Parallel, delayed, Memory


   memory = Memory(cachedir="./cache/joblib_cache", verbose=0)


   @memory.cache
   def do_this_in_parallel(value):
       return value * value


   Parallel(n_jobs=1, verbose=50)(delayed(do_this_in_parallel)(c)
                                          for c in range(10))


However, I get the following error:

   TypeError: can't pickle function objects

Without decorating the do_this_in_parallel function, everything works as
expected.

Is there any way to combine the Parallel and Memory features of joblib
together?

Cheers,
  Andreas.

Re: [joblib] Combining Memory and Parallel

From:
Gael Varoquaux
Date:
2015-04-29 @ 10:35
On Wed, Apr 29, 2015 at 10:40:33AM +0200, Andreas Hilboll wrote:
>    @memory.cache
>    def do_this_in_parallel(value):
>        return value * value


>    Parallel(n_jobs=1, verbose=50)(delayed(do_this_in_parallel)(c)
>                                           for c in range(10))

The following might work:

    def do_this_in_parallel(value):
        return value * value


    Parallel(n_jobs=1, verbose=50)(delayed(memory.cache(do_this_in_parallel))(c)
                                           for c in range(10))


Gaƫl

Re: [joblib] Combining Memory and Parallel

From:
Andreas Hilboll
Date:
2015-04-29 @ 11:20
On 29.04.2015 12:35, Gael Varoquaux wrote:
>     Parallel(n_jobs=1, verbose=50)(delayed(memory.cache(do_this_in_parallel))(c)
>                                            for c in range(10))

It does indeed work like you describe it, thanks, Gael!

-- Andreas.

Re: [joblib] Combining Memory and Parallel

From:
Hanan Shteingart
Date:
2015-04-29 @ 09:02
Hi Andreas,

I also had trouble with this and asked the forum about it few weeks back
(no answer...)
I had the "do_this_in_parallel" function as a method in a class. Once I
made it have a global scope it worked.

Hanan

*HS*

On Wed, Apr 29, 2015 at 11:40 AM, Andreas Hilboll <lists@hilboll.de> wrote:

> Hi,
>
> I'd like to use the Memory caching functionality together with the
> Parallel function -- so all of joblib combined together.
>
> Here's what I have:
>
>    import dill
>    from joblib import Parallel, delayed, Memory
>
>
>    memory = Memory(cachedir="./cache/joblib_cache", verbose=0)
>
>
>    @memory.cache
>    def do_this_in_parallel(value):
>        return value * value
>
>
>    Parallel(n_jobs=1, verbose=50)(delayed(do_this_in_parallel)(c)
>                                           for c in range(10))
>
>
> However, I get the following error:
>
>    TypeError: can't pickle function objects
>
> Without decorating the do_this_in_parallel function, everything works as
> expected.
>
> Is there any way to combine the Parallel and Memory features of joblib
> together?
>
> Cheers,
>   Andreas.
>