librelist archives

« back to archive

Memoization in long-running processes

Memoization in long-running processes

From:
Colin Fleming
Date:
2014-09-30 @ 10:05
Hi all,

A while ago, I filed #1378
<https://github.com/technomancy/leiningen/issues/1378>, the upshot of which
is that Leiningen memoizes values which are not referentially transparent.
This is actually a much more pervasive problem than I originally realised -
memoize is also used in a lot of the profile gathering code, for example,
which mostly loads the profiles from disk somewhere. This means that when
Leiningen is run embedded in a long-running process such as Cursive or
Counterclockwise, the user will have to restart their IDE every time they
change one of these files on disk. In general, the use of memoization
almost makes using lein in an embedded fashion impractical.

Would there be interest in a patch which allowed memoization to be disabled
entirely for lein using a dynamic var (*allow-memoization* or similar)
which could then be turned off in these sorts of environments?

Cheers,
Colin

Re: [leiningen] Memoization in long-running processes

From:
Laurent Petit
Date:
2014-10-02 @ 11:10
This seems indeed interesting, or else the only solution for IDEs is to
recreate / maintain a kind of pool, etc.

2014-09-30 12:05 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:

> Hi all,
>
> A while ago, I filed #1378
> <https://github.com/technomancy/leiningen/issues/1378>, the upshot of
> which is that Leiningen memoizes values which are not referentially
> transparent. This is actually a much more pervasive problem than I
> originally realised - memoize is also used in a lot of the profile
> gathering code, for example, which mostly loads the profiles from disk
> somewhere. This means that when Leiningen is run embedded in a long-running
> process such as Cursive or Counterclockwise, the user will have to restart
> their IDE every time they change one of these files on disk. In general,
> the use of memoization almost makes using lein in an embedded fashion
> impractical.
>
> Would there be interest in a patch which allowed memoization to be
> disabled entirely for lein using a dynamic var (*allow-memoization* or
> similar) which could then be turned off in these sorts of environments?
>
> Cheers,
> Colin
>



-- 
Laurent Petit

Re: [leiningen] Memoization in long-running processes

From:
Colin Fleming
Date:
2014-10-02 @ 14:10
Well, as the code is right now you can't do anything unfortunately. I think
there are two options: allow disabling memoization entirely with a
*memoize?* flag or similar, or have custom memoization in Leiningen that
memoizes to a central atom that could be cleared after calling into lein.
Disabling it entirely is certainly easiest, what I don't know is what
effect this will have on performance.

Certainly some of the memoization currently in lein could probably be
removed - as far as I can tell (although I haven't analysed it thoroughly)
the profile calculation is only called once per "normal" lein operation,
and it doesn't look that performance critical. I mentioned this to
technomancy on IRC and he said he thought it was for historical reasons and
might not be relevant now.

On 3 October 2014 00:10, Laurent PETIT <laurent.petit@gmail.com> wrote:

> This seems indeed interesting, or else the only solution for IDEs is to
> recreate / maintain a kind of pool, etc.
>
> 2014-09-30 12:05 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>
>> Hi all,
>>
>> A while ago, I filed #1378
>> <https://github.com/technomancy/leiningen/issues/1378>, the upshot of
>> which is that Leiningen memoizes values which are not referentially
>> transparent. This is actually a much more pervasive problem than I
>> originally realised - memoize is also used in a lot of the profile
>> gathering code, for example, which mostly loads the profiles from disk
>> somewhere. This means that when Leiningen is run embedded in a long-running
>> process such as Cursive or Counterclockwise, the user will have to restart
>> their IDE every time they change one of these files on disk. In general,
>> the use of memoization almost makes using lein in an embedded fashion
>> impractical.
>>
>> Would there be interest in a patch which allowed memoization to be
>> disabled entirely for lein using a dynamic var (*allow-memoization* or
>> similar) which could then be turned off in these sorts of environments?
>>
>> Cheers,
>> Colin
>>
>
>
>
> --
> Laurent Petit
>

Re: [leiningen] Memoization in long-running processes

From:
Laurent Petit
Date:
2014-10-02 @ 14:56
2014-10-02 16:10 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:

> Well, as the code is right now you can't do anything unfortunately. I
> think there are two options: allow disabling memoization entirely with a
> *memoize?* flag or similar, or have custom memoization in Leiningen that
> memoizes to a central atom that could be cleared after calling into lein.
> Disabling it entirely is certainly easiest, what I don't know is what
> effect this will have on performance.
>
> Certainly some of the memoization currently in lein could probably be
> removed - as far as I can tell (although I haven't analysed it thoroughly)
> the profile calculation is only called once per "normal" lein operation,
> and it doesn't look that performance critical. I mentioned this to
> technomancy on IRC and he said he thought it was for historical reasons and
> might not be relevant now.
>

Hmm, so getting rid of it could also be a satisfying option? :-)


>
> On 3 October 2014 00:10, Laurent PETIT <laurent.petit@gmail.com> wrote:
>
>> This seems indeed interesting, or else the only solution for IDEs is to
>> recreate / maintain a kind of pool, etc.
>>
>> 2014-09-30 12:05 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>>
>>> Hi all,
>>>
>>> A while ago, I filed #1378
>>> <https://github.com/technomancy/leiningen/issues/1378>, the upshot of
>>> which is that Leiningen memoizes values which are not referentially
>>> transparent. This is actually a much more pervasive problem than I
>>> originally realised - memoize is also us ed in a lot of the profile
>>> gathering code, for example, which mostly loads the profiles from disk
>>> somewhere. This means that when Leiningen is run embedded in a long-running
>>> process such as Cursive or Counterclockwise, the user will have to restart
>>> their IDE every time they change one of these files on disk. In general,
>>> the use of memoization almost makes using lein in an embedded fashion
>>> impractical.
>>>
>>> Would there be interest in a patch which allowed memoization to be
>>> disabled entirely for lein using a dynamic var (*allow-memoization* or
>>> similar) which could then be turned off in these sorts of environments?
>>>
>>> Cheers,
>>> Colin
>>>
>>
>>
>>
>> --
>> Laurent Petit
>>
>
>


-- 
Laurent Petit

Re: [leiningen] Memoization in long-running processes

From:
Phil Hagelberg
Date:
2014-10-02 @ 20:59
Laurent PETIT <laurent.petit@gmail.com> writes:

> 2014-10-02 16:10 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>
>> Certainly some of the memoization currently in lein could probably be
>> removed - as far as I can tell (although I haven't analysed it thoroughly)
>> the profile calculation is only called once per "normal" lein operation,
>> and it doesn't look that performance critical. I mentioned this to
>> technomancy on IRC and he said he thought it was for historical reasons and
>> might not be relevant now.
>
> Hmm, so getting rid of it could also be a satisfying option? :-)

Definitely; as long as it isn't called more than a handful of times we
should just get rid of it. I suspect the original reason is gone.

-Phil