librelist archives

« back to archive

using joblib's Memory.cache() with decorated functions (bug report included)

using joblib's Memory.cache() with decorated functions (bug report included)

From:
Nicolas Pinto
Date:
2011-03-17 @ 21:48
Hello,

First of all: thanks a ton for writing joblib, it has helped us tremendously!

I'm trying to get joblib's Memory.cache() working on some (already)
decorated functions and methods. For some reason, when I use *args and
**kwargs, the cache doesn't work. See below for details, sorry I
haven't had time to dig deeper (but I guess this has to do with the
known limitation briefly mentioned at
http://packages.python.org/joblib/memory.html#gotchas i.e. "memory
cannot be used on some complex objects, eg a callable object with a
__call__ method")

Thank you.

Best,

-- 
Nicolas Pinto, PhD
http://web.mit.edu/pinto


# to reproduce the bug
from joblib import Memory

mem = Memory(cachedir='./bug', verbose=False)
mem.clear()

import numpy as np

def add4(func):
    def _add4(*args, **kwargs):
        val = args[0]
        return func(val+4)
    return _add4

def add4_no_args(func):
    def _add4_no_args(val):
        return func(val+4)
    return _add4_no_args

square = np.square
add4_square = add4(np.square)
add4_no_args_square = add4_no_args(np.square)

print "Ground truth:"
for val in 1,2:
    print square(val), add4_square(val), add4_no_args_square(val)

print "with joblib's Memory.cache():"
square = mem.cache(np.square)
add4_square = mem.cache(add4(np.square))
add4_no_args_square = mem.cache(add4_no_args(np.square))
for val in 1,2:
    print square(val), add4_square(val), add4_no_args_square(val)

# output:
Ground truth:
1 25 25
4 36 36
with joblib's Memory.cache():
1 25 25
4 25 36

Re: [joblib] using joblib's Memory.cache() with decorated functions (bug report included)

From:
Gael Varoquaux
Date:
2011-03-18 @ 13:16
Hey Nicolas,

I have gotten your mail, but since then things have been hectic and I
haven't had even time to read it carefully. I think I'll look at it in
the train, going off to the Alps for the week end. I might not be able to
report back until Monday.

Sorry for the delay

Gael

On Thu, Mar 17, 2011 at 09:48:19PM +0000, Nicolas Pinto wrote:
> Hello,

> First of all: thanks a ton for writing joblib, it has helped us tremendously!

> I'm trying to get joblib's Memory.cache() working on some (already)
> decorated functions and methods. For some reason, when I use *args and
> **kwargs, the cache doesn't work. See below for details, sorry I
> haven't had time to dig deeper (but I guess this has to do with the
> known limitation briefly mentioned at
> http://packages.python.org/joblib/memory.html#gotchas i.e. "memory
> cannot be used on some complex objects, eg a callable object with a
> __call__ method")

> Thank you.

> Best,

Re: [joblib] using joblib's Memory.cache() with decorated functions (bug report included)

From:
Gael Varoquaux
Date:
2011-03-19 @ 11:39
On Fri, Mar 18, 2011 at 02:16:45PM +0100, Gael Varoquaux wrote:
> I have gotten your mail, but since then things have been hectic and I
> haven't had even time to read it carefully. I think I'll look at it in
> the train, going off to the Alps for the week end. I might not be able to
> report back until Monday.

I have had a look in the train. It is indeed a bug. The simplest test
case is::

    from nose.tools import assert_equal

    from joblib.func_inspect import filter_args

    def f(*args, **kwargs):
	return None
	
    assert_equal(filter_args(f, [], 1), {'*': [1], '**': {}})

I am working on it, but I am not supposed to work this week end, ... So I
am picking up my computer when nobody is looking.

G

Re: [joblib] using joblib's Memory.cache() with decorated functions (bug report included)

From:
Nicolas Pinto
Date:
2011-03-19 @ 13:13
Thanks a lot Gael for the blazing fast answer. Let me know if there is
anything I can help you with.

Best,

N

On Sat, Mar 19, 2011 at 7:39 AM, Gael Varoquaux
<gael.varoquaux@normalesup.org> wrote:
> On Fri, Mar 18, 2011 at 02:16:45PM +0100, Gael Varoquaux wrote:
>> I have gotten your mail, but since then things have been hectic and I
>> haven't had even time to read it carefully. I think I'll look at it in
>> the train, going off to the Alps for the week end. I might not be able to
>> report back until Monday.
>
> I have had a look in the train. It is indeed a bug. The simplest test
> case is::
>
>    from nose.tools import assert_equal
>
>    from joblib.func_inspect import filter_args
>
>    def f(*args, **kwargs):
>        return None
>
>    assert_equal(filter_args(f, [], 1), {'*': [1], '**': {}})
>
> I am working on it, but I am not supposed to work this week end, ... So I
> am picking up my computer when nobody is looking.
>
> G
>



-- 
Nicolas Pinto, PhD
http://web.mit.edu/pinto

Re: [joblib] using joblib's Memory.cache() with decorated functions (bug report included)

From:
Gael Varoquaux
Date:
2011-03-19 @ 18:47
Ok, should be fixed in branch 0.5.X on github. Can you please confirm.

Gael

----- Original message -----
> Hey Nicolas,
> 
> I have gotten your mail, but since then things have been hectic and I
> haven't had even time to read it carefully. I think I'll look at it in
> the train, going off to the Alps for the week end. I might not be able to
> report back until Monday.
> 
> Sorry for the delay
> 
> Gael
> 
> On Thu, Mar 17, 2011 at 09:48:19PM +0000, Nicolas Pinto wrote:
> > Hello,
> 
> > First of all: thanks a ton for writing joblib, it has helped us
> > tremendously!
> 
> > I'm trying to get joblib's Memory.cache() working on some (already)
> > decorated functions and methods. For some reason, when I use *args and
> > **kwargs, the cache doesn't work. See below for details, sorry I
> > haven't had time to dig deeper (but I guess this has to do with the
> > known limitation briefly mentioned at
> > http://packages.python.org/joblib/memory.html#gotchas i.e. "memory
> > cannot be used on some complex objects, eg a callable object with a
> > __call__ method")
> 
> > Thank you.

Re: [joblib] using joblib's Memory.cache() with decorated functions (bug report included)

From:
Nicolas Pinto
Date:
2011-03-19 @ 19:33
Fabulous! It works now.

Thanks a ton for your help!

N

On Sat, Mar 19, 2011 at 2:47 PM, Gael Varoquaux
<gael.varoquaux@normalesup.org> wrote:
> Ok, should be fixed in branch 0.5.X on github. Can you please confirm.
>
> Gael
>
> ----- Original message -----
>> Hey Nicolas,
>>
>> I have gotten your mail, but since then things have been hectic and I
>> haven't had even time to read it carefully. I think I'll look at it in
>> the train, going off to the Alps for the week end. I might not be able to
>> report back until Monday.
>>
>> Sorry for the delay
>>
>> Gael
>>
>> On Thu, Mar 17, 2011 at 09:48:19PM +0000, Nicolas Pinto wrote:
>> > Hello,
>>
>> > First of all: thanks a ton for writing joblib, it has helped us
>> > tremendously!
>>
>> > I'm trying to get joblib's Memory.cache() working on some (already)
>> > decorated functions and methods. For some reason, when I use *args and
>> > **kwargs, the cache doesn't work. See below for details, sorry I
>> > haven't had time to dig deeper (but I guess this has to do with the
>> > known limitation briefly mentioned at
>> > http://packages.python.org/joblib/memory.html#gotchas i.e. "memory
>> > cannot be used on some complex objects, eg a callable object with a
>> > __call__ method")
>>
>> > Thank you.
>
>



-- 
Nicolas Pinto, PhD
http://web.mit.edu/pinto