librelist archives

« back to archive

problem with IPython notebooks vs .py scripts - execution occurs twice

problem with IPython notebooks vs .py scripts - execution occurs twice

From:
Shabnam Kadir
Date:
2014-04-01 @ 11:33
Hi,
    I'd like to report a problem I am having with joblib. If I execute a
function (defined in a separate module) with the same dictionary argument
from:

1) A .py script
2) A IPython notebook

Then this is *sometimes* treated as two separate executions and is cached
by joblib twice. What is worrying is that this happens sporadically. I am
happy to send some code reproducing the problem. Two separate directories
are created in the joblib cache directory - they are bit for bit identical!
So the input_args.json files are identical as are the output pickle files.
I also call the modules in the same way in both the .py script and the
IPython notebook - I use exactly the same commands.

It would be great if this could be looked into, as I think joblib is
otherwise wonderful! I tried using ordered dictionaries instead of
dictionaries as I understand there have been issues concerning this in the
past, but alas this does not solve the problem.

Thanks,
Shabnam

Re: problem with IPython notebooks vs .py scripts - execution occurs twice

From:
Shabnam Kadir
Date:
2014-04-15 @ 11:06
I have not yet found the cause to the above problem, but I noticed today
that job lib is sensitive to the line numbers in the definition of a
function. This seems to be a new 'feature' rather than a bug. Perhaps this
is beneficial to some people, but is there any way of disabling it (rather
than having to use an older version)?


On Tue, Apr 1, 2014 at 12:33 PM, Shabnam Kadir <shabnam.kadir@gmail.com>wrote:

> Hi,
>     I'd like to report a problem I am having with joblib. If I execute a
> function (defined in a separate module) with the same dictionary argument
> from:
>
> 1) A .py script
> 2) A IPython notebook
>
> Then this is *sometimes* treated as two separate executions and is cached
> by joblib twice. What is worrying is that this happens sporadically. I am
> happy to send some code reproducing the problem. Two separate directories
> are created in the joblib cache directory - they are bit for bit identical!
> So the input_args.json files are identical as are the output pickle files.
> I also call the modules in the same way in both the .py script and the
> IPython notebook - I use exactly the same commands.
>
> It would be great if this could be looked into, as I think joblib is
> otherwise wonderful! I tried using ordered dictionaries instead of
> dictionaries as I understand there have been issues concerning this in the
> past, but alas this does not solve the problem.
>
> Thanks,
> Shabnam
>
>
>


-- 

----------------------------------------------------------------------------------------------------------------
Dr. Shabnam Kadir
Institute of Neurology, Department of Neuroscience, Physiology, and
Pharmacology
University College London
21 University Street
London WC1E 6DE
Tel: +44 (0)20 3108 2407

----------------------------------------------------------------------------------------------------------------