librelist archives

« back to archive

Joblib AttributeError

Joblib AttributeError

From:
Nick Snels
Date:
2010-12-11 @ 13:12
Hi,

I'm trying to install Joblib alongsite Python 2.5. I started with the
lastest version and that didn't work, now I'm at version 0.4.2 and it
still isn't working. I get the following error:

Traceback (most recent call last):
  File "D:\Python\mle\src\test.py", line 9, in <module>
    Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
  File "C:\Python25\lib\site-packages\joblib\parallel.py", line 220, in __call__
    for index, (function, args, kwargs) in enumerate(iterable):
  File "D:\Python\mle\src\test.py", line 9, in <genexpr>
    Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
  File "C:\Python25\lib\site-packages\joblib\parallel.py", line 94, in delayed
    @functools.wraps(function)
  File "C:\Python25\lib\functools.py", line 35, in update_wrapper
    getattr(wrapper, attr).update(getattr(wrapped, attr))
AttributeError: 'builtin_function_or_method' object has no attribute '__dict__'

Does anybody know how to resolve this. Thanks. My code is one of the examples:

from joblib import Parallel, delayed
from math import sqrt

if __name__=='__main__':
    Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))

Kind regards,

Nick

Re: [joblib] Joblib AttributeError

From:
Gael Varoquaux
Date:
2010-12-11 @ 13:44
Hi Nick,

I am pretty sure that joblib does work on python2.5, as I am stuck with
2.5 on one of my computers. However, right now, I am stuck at an airport
with only 2.6, so I can't really check. You mention version 0.4.2,
however, the latest version is 0.4.6. Would you mind checking that you
still get the problem with 0.4.6? If you still have the problem with the
latest version, I'll try my best to reproduce the problem, and fix it.

Best,

Gaël

On Sat, Dec 11, 2010 at 02:12:51PM +0100, Nick Snels wrote:
> Hi,

> I'm trying to install Joblib alongsite Python 2.5. I started with the
> lastest version and that didn't work, now I'm at version 0.4.2 and it
> still isn't working. I get the following error:

> Traceback (most recent call last):
>   File "D:\Python\mle\src\test.py", line 9, in <module>
>     Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>   File "C:\Python25\lib\site-packages\joblib\parallel.py", line 220, in __call__
>     for index, (function, args, kwargs) in enumerate(iterable):
>   File "D:\Python\mle\src\test.py", line 9, in <genexpr>
>     Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>   File "C:\Python25\lib\site-packages\joblib\parallel.py", line 94, in delayed
>     @functools.wraps(function)
>   File "C:\Python25\lib\functools.py", line 35, in update_wrapper
>     getattr(wrapper, attr).update(getattr(wrapped, attr))
> AttributeError: 'builtin_function_or_method' object has no attribute '__dict__'

> Does anybody know how to resolve this. Thanks. My code is one of the examples:

> from joblib import Parallel, delayed
> from math import sqrt

> if __name__=='__main__':
>     Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))

> Kind regards,

> Nick

-- 
    Gael Varoquaux
    Research Fellow, INSERM
    Associate researcher, INRIA
    Laboratoire de Neuro-Imagerie Assistee par Ordinateur
    NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
    Phone:  ++ 33-1-69-08-78-35
    Mobile: ++ 33-6-28-25-64-62
    http://gael-varoquaux.info

Re: [joblib] Joblib AttributeError

From:
Pietro Berkes
Date:
2010-12-11 @ 13:56
Dear Nick,

this seems to be a bug in the module functools in Python 2.5, as
described here http://bugs.python.org/issue1576241 . Builtin
functions, like math.sqrt, don't have a __dict__ attribute, and
functools.wraps used to rely on it.

I'm also stuck with Python 2.5 at the moment, but I think this should
be a quick workaround:
define a standard function
my_sqrt = lambda x: math.sqrt(x)
and use it instead of math.sqrt in your example.

Of course, in this case the docstring and the signature of the my_sqrt
won't match those of math.sqrt .

Hope it helps,
Pietro


On Sat, Dec 11, 2010 at 8:44 AM, Gael Varoquaux
<gael.varoquaux@normalesup.org> wrote:
> Hi Nick,
>
> I am pretty sure that joblib does work on python2.5, as I am stuck with
> 2.5 on one of my computers. However, right now, I am stuck at an airport
> with only 2.6, so I can't really check. You mention version 0.4.2,
> however, the latest version is 0.4.6. Would you mind checking that you
> still get the problem with 0.4.6? If you still have the problem with the
> latest version, I'll try my best to reproduce the problem, and fix it.
>
> Best,
>
> Gaël
>
> On Sat, Dec 11, 2010 at 02:12:51PM +0100, Nick Snels wrote:
>> Hi,
>
>> I'm trying to install Joblib alongsite Python 2.5. I started with the
>> lastest version and that didn't work, now I'm at version 0.4.2 and it
>> still isn't working. I get the following error:
>
>> Traceback (most recent call last):
>>   File "D:\Python\mle\src\test.py", line 9, in <module>
>>     Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>>   File "C:\Python25\lib\site-packages\joblib\parallel.py", line 220, in
__call__
>>     for index, (function, args, kwargs) in enumerate(iterable):
>>   File "D:\Python\mle\src\test.py", line 9, in <genexpr>
>>     Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>>   File "C:\Python25\lib\site-packages\joblib\parallel.py", line 94, in delayed
>>     @functools.wraps(function)
>>   File "C:\Python25\lib\functools.py", line 35, in update_wrapper
>>     getattr(wrapper, attr).update(getattr(wrapped, attr))
>> AttributeError: 'builtin_function_or_method' object has no attribute '__dict__'
>
>> Does anybody know how to resolve this. Thanks. My code is one of the examples:
>
>> from joblib import Parallel, delayed
>> from math import sqrt
>
>> if __name__=='__main__':
>>     Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>
>> Kind regards,
>
>> Nick
>
> --
>    Gael Varoquaux
>    Research Fellow, INSERM
>    Associate researcher, INRIA
>    Laboratoire de Neuro-Imagerie Assistee par Ordinateur
>    NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
>    Phone:  ++ 33-1-69-08-78-35
>    Mobile: ++ 33-6-28-25-64-62
>    http://gael-varoquaux.info
>

Re: [joblib] Joblib AttributeError

From:
Gael Varoquaux
Date:
2010-12-11 @ 14:10
Hi Pietro,

Thanks a lot for the reply, I wasn't aware of this issue (I should have
discovered it if I were careful enough with my testing :( ). I'll see
what I can do to work around it in the next joblib release.

Best,

Gaël

On Sat, Dec 11, 2010 at 08:56:56AM -0500, Pietro Berkes wrote:
> this seems to be a bug in the module functools in Python 2.5, as
> described here http://bugs.python.org/issue1576241 . Builtin
> functions, like math.sqrt, don't have a __dict__ attribute, and
> functools.wraps used to rely on it.

> I'm also stuck with Python 2.5 at the moment, but I think this should
> be a quick workaround:
> define a standard function
> my_sqrt = lambda x: math.sqrt(x)
> and use it instead of math.sqrt in your example.

> Of course, in this case the docstring and the signature of the my_sqrt
> won't match those of math.sqrt .

Re: [joblib] Joblib AttributeError

From:
Nick Snels
Date:
2010-12-11 @ 15:42
Hi Gaël and Petro,

I did two things, as you guys suggested. I reinstalled the latest
version of Joblib 0.4.6. And I added the my_sqrt function. My code now
is:

from joblib import Parallel, delayed
import math

my_sqrt = lambda x: math.sqrt(x)

if __name__=='__main__':
    Parallel(n_jobs=1)(delayed(my_sqrt)(i**2) for i in range(10))

When I run it this time, I get the following error:

Traceback (most recent call last):
  File "D:\Python\mle\src\test.py", line 12, in <module>
    Parallel(n_jobs=1)(delayed(my_sqrt)(i**2) for i in range(10))
  File "C:\Python25\Lib\site-packages\joblib\parallel.py", line 239, in __call__
    for index, (function, args, kwargs) in enumerate(iterable):
  File "D:\Python\mle\src\test.py", line 12, in <genexpr>
    Parallel(n_jobs=1)(delayed(my_sqrt)(i**2) for i in range(10))
  File "C:\Python25\Lib\site-packages\joblib\parallel.py", line 85, in delayed
    pickle.dumps(function)
  File "C:\Python25\lib\copy_reg.py", line 69, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

Any ideas what may be causing this? Thanks.

Kind regards,

Nick

Re: [joblib] Joblib AttributeError

From:
Gael Varoquaux
Date:
2010-12-11 @ 15:48
On Sat, Dec 11, 2010 at 04:42:01PM +0100, Nick Snels wrote:
> from joblib import Parallel, delayed
> import math

> my_sqrt = lambda x: math.sqrt(x)

> if __name__=='__main__':
>     Parallel(n_jobs=1)(delayed(my_sqrt)(i**2) for i in range(10))

> When I run it this time, I get the following error:

> Traceback (most recent call last):
>   File "D:\Python\mle\src\test.py", line 12, in <module>
>     Parallel(n_jobs=1)(delayed(my_sqrt)(i**2) for i in range(10))
>   File "C:\Python25\Lib\site-packages\joblib\parallel.py", line 239, in __call__
>     for index, (function, args, kwargs) in enumerate(iterable):
>   File "D:\Python\mle\src\test.py", line 12, in <genexpr>
>     Parallel(n_jobs=1)(delayed(my_sqrt)(i**2) for i in range(10))
>   File "C:\Python25\Lib\site-packages\joblib\parallel.py", line 85, in delayed
>     pickle.dumps(function)
>   File "C:\Python25\lib\copy_reg.py", line 69, in _reduce_ex
>     raise TypeError, "can't pickle %s objects" % base.__name__
> TypeError: can't pickle function objects

> Any ideas what may be causing this? Thanks.

Yes: Python cannot picle lambda functions, and pickling is required for
parallel computing. In the long run we can work around this limitation by
implementing our own persitence framework[*], but in the short run we
have to live with it.

To solve this issue, you need to use an explicit 'def' to define your
function. In addition, you will probably have to put this function in a
separate module that you import from, depending on your system and how
you run this Python file. I need to add a note on this in the docs...

HTH,

Gaël

[*] This is not as bad as one might think: Mike McKerns has already done
the hard work: http://dev.danse.us/trac/pathos/browser/dill

Re: [joblib] Joblib AttributeError

From:
Pietro Berkes
Date:
2010-12-11 @ 15:54
My apologies, Nick: I'm using Python 2.6 (there's a typo in the
previous email) and I based my suggestion on the fact that the lambda
has got a __dict__ attribute... the error is due to pickle this time,
Gael's message is the way to go
P.


On Sat, Dec 11, 2010 at 10:48 AM, Gael Varoquaux
<gael.varoquaux@normalesup.org> wrote:
> On Sat, Dec 11, 2010 at 04:42:01PM +0100, Nick Snels wrote:
>> from joblib import Parallel, delayed
>> import math
>
>> my_sqrt = lambda x: math.sqrt(x)
>
>> if __name__=='__main__':
>>     Parallel(n_jobs=1)(delayed(my_sqrt)(i**2) for i in range(10))
>
>> When I run it this time, I get the following error:
>
>> Traceback (most recent call last):
>>   File "D:\Python\mle\src\test.py", line 12, in <module>
>>     Parallel(n_jobs=1)(delayed(my_sqrt)(i**2) for i in range(10))
>>   File "C:\Python25\Lib\site-packages\joblib\parallel.py", line 239, in
__call__
>>     for index, (function, args, kwargs) in enumerate(iterable):
>>   File "D:\Python\mle\src\test.py", line 12, in <genexpr>
>>     Parallel(n_jobs=1)(delayed(my_sqrt)(i**2) for i in range(10))
>>   File "C:\Python25\Lib\site-packages\joblib\parallel.py", line 85, in delayed
>>     pickle.dumps(function)
>>   File "C:\Python25\lib\copy_reg.py", line 69, in _reduce_ex
>>     raise TypeError, "can't pickle %s objects" % base.__name__
>> TypeError: can't pickle function objects
>
>> Any ideas what may be causing this? Thanks.
>
> Yes: Python cannot picle lambda functions, and pickling is required for
> parallel computing. In the long run we can work around this limitation by
> implementing our own persitence framework[*], but in the short run we
> have to live with it.
>
> To solve this issue, you need to use an explicit 'def' to define your
> function. In addition, you will probably have to put this function in a
> separate module that you import from, depending on your system and how
> you run this Python file. I need to add a note on this in the docs...
>
> HTH,
>
> Gaël
>
> [*] This is not as bad as one might think: Mike McKerns has already done
> the hard work: http://dev.danse.us/trac/pathos/browser/dill
>