librelist archives

« back to archive

Getting leiningen classpath

Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-14 @ 22:37
Hello,

For Counterclockwise, I'd like to change the way I'm integrated with
Leiningen, aka stop using leiningen-core as an embedded library, and start
calling leiningen as an external process (still forging the java command
line, though).

The first use case I want to do the switch for is invoking leiningen in
some way that I get back (presumably on stdout) all required information
about what's on the classpath given a certain profile.

To give you an idea of what I expect, here's how Counterclockwise
serializes the build path on disk for caching purposes:

({:path

"/Users/laurentpetit/.m2/repository/org/clojure/clojure/1.6.0/clojure-1.6.0.jar",
  :native-path
  "/Users/laurentpetit/tmp/clp/target/native/macosx/x86_64",
  :source-attachment-path


"/Users/laurentpetit/.m2/repository/org/clojure/clojure/1.6.0/clojure-1.6.0-sources.jar"}
 {:path


"/Users/laurentpetit/.m2/repository/clojure-complete/clojure-complete/0.2.3/clojure-complete-0.2.3.jar",
  :native-path
  "/Users/laurentpetit/tmp/clp/target/native/macosx/x86_64"}
 ...)

What's missing here is a way to distinguish between binary libraries and
source/resource paths, but that's a minor annoyance since it's easy to find
out which entries are source path (no native path, no source attachment
path, and located within the project's directory but its target/
subdirectory).

So my question is: how would you invoke leiningen so that it can give me
the same level of information ?
Is it provided out of the box and I missed it ? Should I write a plugin ,
etc.


-- 
Laurent Petit


-- 
Laurent Petit

Re: [leiningen] Getting leiningen classpath

From:
Phil Hagelberg
Date:
2015-08-15 @ 01:02
Laurent PETIT <laurent.petit@gmail.com> writes:

> What's missing here is a way to distinguish between binary libraries
> and source/resource paths, but that's a minor annoyance since it's
> easy to find out which entries are source path (no native path, no
> source attachment path, and located within the project's directory but
> its target/ subdirectory).
>
> So my question is: how would you invoke leiningen so that it can give
> me the same level of information ?
> Is it provided out of the box and I missed it ? Should I write a
> plugin , etc.

You can try `lein assoc :eval-in :pprint run -m clojure.main`; this will
pprint the form to be run and print the classpath and JVM args instead
of actually running it.

  https://github.com/technomancy/lein-assoc

Bootstrapping it such that the plugin is on the classpath could be
tricky though.

However, this raises a larger question of why you want to move away from
leiningen-core. Counterclockwise is basically the textbook use-case for
the reason this library exists, so if there are problems around
instability or conflicts maybe that's something we can do better about
in Leiningen itself?

-Phil

Re: [leiningen] Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-15 @ 01:28
Hello,

I'm not totally sure I understand the command-line you gave.

As to why I want to move away from leiningen-core, there are several
reasons.
But please note that it's still an experimentation phase, nothing's written
in stone yet.

Anyway:
- leiningen-core with all its dependencies makes the CCW Plugin heavy. I
have never really liked it.
- I have never liked that leiningen-core is embedded since CCW is then at
the mercy of a bad behaving plugin. Thus disabling middlewares / plugins,
but then some user desired features may not work. I'd like to give user
full power, not risking CCW stability (if lein crashes, it crashes its own
JVM)
- lots of tweaks inside CCW to work around some limitations. I'd like to
limit - or externalize as much as possible outside CCW codebase - those.
- leiningen is not alone now. CCW is tightly coupled to it right now. I'd
like to have a cleaner interface between CCW and whatever tool the user has
defined its project dependencies in. Forging java command lines seems like
the right interface for me. And maybe I'm still too careful of sticking to
the JVM. A time where CCW would have to launch / call into native processes
may well be around the corner (thinking about self hosted clojurescript, of
course).

So all these considerations make me want to redefine the interface between
CCW and Leiningen. Leiningen-core made the job, we have learned from it,
and now is the time to look beyond it.




2015-08-15 3:02 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:

> Laurent PETIT <laurent.petit@gmail.com> writes:
>
> > What's missing here is a way to distinguish between binary libraries
> > and source/resource paths, but that's a minor annoyance since it's
> > easy to find out which entries are source path (no native path, no
> > source attachment path, and located within the project's directory but
> > its target/ subdirectory).
> >
> > So my question is: how would you invoke leiningen so that it can give
> > me the same level of information ?
> > Is it provided out of the box and I missed it ? Should I write a
> > plugin , etc.
>
> You can try `lein assoc :eval-in :pprint run -m clojure.main`; this will
> pprint the form to be run and print the classpath and JVM args instead
> of actually running it.
>
>   https://github.com/technomancy/lein-assoc
>
> Bootstrapping it such that the plugin is on the classpath could be
> tricky though.
>
> However, this raises a larger question of why you want to move away from
> leiningen-core. Counterclockwise is basically the textbook use-case for
> the reason this library exists, so if there are problems around
> instability or conflicts maybe that's something we can do better about
> in Leiningen itself?
>
> -Phil
>



-- 
Laurent Petit

Re: [leiningen] Getting leiningen classpath

From:
Phil Hagelberg
Date:
2015-08-15 @ 01:04
Phil Hagelberg <phil@hagelb.org> writes:

> You can try `lein assoc :eval-in :pprint run -m clojure.main`; this will
> pprint the form to be run and print the classpath and JVM args instead
> of actually running it.

Wow, I'm rusty. Use the update-in task instead; it's built-in.

-Phil

Re: [leiningen] Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-15 @ 01:50
OK understood this works as expected: lein update-in : assoc :eval-in
:pprint -- run -m clojure.main

BTW, does that mean that it's possible to emulate TRAMPOLINE by storing the
result of :pprint (and thus preventing the scenario where there's 2 JVMs
loaded, one for lein itself, and one for the project) ?

2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:

> Phil Hagelberg <phil@hagelb.org> writes:
>
> > You can try `lein assoc :eval-in :pprint run -m clojure.main`; this will
> > pprint the form to be run and print the classpath and JVM args instead
> > of actually running it.
>
> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>
> -Phil
>



-- 
Laurent Petit

Re: [leiningen] Getting leiningen classpath

From:
Colin Fleming
Date:
2015-08-17 @ 10:37
Curiously enough, I'm also about to totally change the way I integrate
Leiningen in Cursive.

To answer Phil's question about embedding leiningen-core, there are many
many disadvantages to doing so:

   1. You're tied to a specific version of lein. Currently I bundle 2.5.0 -
   I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users encountered bugs so
   I downgraded to 2.5.0. Some users currently experience bugs with 2.5.0, but
   they're easier to work around. Anyone using older versions on their project
   is out of luck.
   2. Lein incorrectly memoises many things internally, which is a gigantic
   pain. Currently users have to restart Cursive every time they change
   profiles.clj, and additionally they also have to be aware of the bug to
   know to do that. I've also lost countless hours debugging problems like
   #1378 <https://github.com/technomancy/leiningen/issues/1378>. A more
   general report is #1768
   <https://github.com/technomancy/leiningen/issues/1768>. This is slated
   to be fixed in lein 3.0, but a) who knows when that will be and b) I'm
   still at the mercy of plugins doing the same thing anyway.
   3. Java cannot change the CWD of the current process, which means that
   local-repo always appears somewhere in the IntelliJ bin directory. This
   also fails for users' home grown versioning systems when they do things
   like (slurp versions.txt). More examples here
   <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
   (there are probably more, but that's the easiest search). Essentially, one
   of lein's big selling points is that it's declarative, but the fact that
   project.clj is actually executed means that it's not truly declarative.
   4. Many things in lein (gpg, proxies etc) are configured using
   environment variables, which are a pain to pass when embedding. Cursive
   hooks leiningen fairly heavily using robert.hooke to supply values for
   these instead of getting them from env vars.
   5. Often you need access to the lein tasks, not just core. Cursive has
   bundled all of lein for a long time so that it can access the repl task. If
   you don't do this you end up duplicating a lot of the code from the tasks
   anyway.
   6. There are also lots of minor issues - I wanted to hook into the
   Aether transfer listener so that I could provide updates in the Cursive UI
   when libs are downloaded. Currently I do this in Cursive by hooking
   leiningen.core.classpath/warn and checking if it looks like a "Retrieving"
   message and parsing it if so.
   7. As Laurent says, lein, even just core, is really big. The Cursive
   download is about 3x the size it would be if it didn't bundle lein.

That said, while I understand Laurent's worry about stability, that's never
been a problem. I've never received a single bug report about stability
that was as a result of bundling lein.

There is also one significant benefit to bundling lein - it's fast.
Starting a JVM process for lein every time Cursive needs to sync the deps
is going to be way too slow. This will be particularly true for large
multi-module projects where to do everything totally accurately (in
particular, the CWD) requires starting a new JVM per module - this is
unworkable. I considered caching the last-modified of the project.clj and
only re-parsing a particular module if that had changed, but I'd need to do
that for all possible contributing files which is impossible to calculate.
Apart from profiles.clj and other "official" files which might affect the
result, I'd also need to test the version.txt type files from #3, and
there's no way I can detect all of those.

Since I don't think that stability is actually a major problem, I'm
planning to continue running lein in-process but I'm going to isolate its
classpath using ShimDandy. I have a prototype of this but it's still pretty
primitive, and I have a lot of other things going on so it's advancing
slowly. I run lein in a shim, and I create the shim's classpath using
Aether so I don't actually bundle lein, I pull it in when creating the
shim. This means that users can choose the lein version they want, and the
files are stored in .m2 rather than being downloaded with Cursive. I'll
then throw the shim away when done so memoisation won't be a problem. I can
amortize the performance problem by eagerly creating the shim and loading
the namespaces into it before I need it, similar to boot's pod caching.
This is much less costly than eagerly starting a JVM and keeping that
around until needed.

I'm also going to try to fiddle much less with the lein internals since
that has caused problems when upgrading lein. Using this new mechanism I
can call the functions representing the lein tasks as if they were being
called from main, which greatly reduces my dependency on lein internals.
I'll still have to hook for gpg/proxies etc, but I'll be hooking aether
rather than lein for those, which is much more stable since it's basically
abandonware at this point - I should have done this ages ago. I can also
hook into the transfer listener by hooking aether.

My interface to the shim will be to pass it a list of file paths to the
project.clj files and eventually a list of profiles, and it will return a
map containing all the information I need about the projects.

The upshot of all this is that this should fix all my current problems
except #3, which there's no workaround for. I'll fix the local-repo problem
by hooking lein or fiddling with the project.clj, but the only way to fix
the slurp problem is to start a JVM per lein project, or convince users not
to do that. Or hook slurp, I guess.

Laurent, currently I run the REPL within Cursive using the trampoline
mechanism - IMO you don't have to emulate it, you can just use it directly.
Get lein to generate a trampoline file then parse that - it's very easy to
parse since all the elements always appear in the same order. Currently I
hook the lein internals to do this, but as described above I'm about to
start calling lein trampoline directly from the shim.

Cheers,
Colin


On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com> wrote:

> OK understood this works as expected: lein update-in : assoc :eval-in
> :pprint -- run -m clojure.main
>
> BTW, does that mean that it's possible to emulate TRAMPOLINE by storing
> the result of :pprint (and thus preventing the scenario where there's 2
> JVMs loaded, one for lein itself, and one for the project) ?
>
> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>
>> Phil Hagelberg <phil@hagelb.org> writes:
>>
>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`; this will
>> > pprint the form to be run and print the classpath and JVM args instead
>> > of actually running it.
>>
>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>
>> -Phil
>>
>
>
>
> --
> Laurent Petit
>

Re: [leiningen] Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-17 @ 11:05
Well said Colin,
You've written down a much more accurate version of all the problems I've
had with embedded leiningen-core.
It's been a while that CCW uses classlojure for isolating one classloader
per project, but I had chosen to still try to use leiningen-core directly,
and it proved as painful as you described. Small things that have added up
with time.

I'll take a look at the direct trampoline mechanism, thanks for the tip.

Also, the #3 item is currently solved by CCW in an inefficient way: I
serialize access to lein classloaders and temporarily change the CWD system
property. This is really bad.

What you suggest makes me reconsider starting JVMs per project. Maybe it's
more in the way I have tied CCW code to leiningen core instead of going
through leiningen.main ...

Good food for thoughts, thanks!





2015-08-17 12:37 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:

> Curiously enough, I'm also about to totally change the way I integrate
> Leiningen in Cursive.
>
> To answer Phil's question about embedding leiningen-core, there are many
> many disadvantages to doing so:
>
>    1. You're tied to a specific version of lein. Currently I bundle 2.5.0
>    - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users encountered bugs
>    so I downgraded to 2.5.0. Some users currently experience bugs with 2.5.0,
>    but they're easier to work around. Anyone using older versions on their
>    project is out of luck.
>    2. Lein incorrectly memoises many things internally, which is a
>    gigantic pain. Currently users have to restart Cursive every time they
>    change profiles.clj, and additionally they also have to be aware of the bug
>    to know to do that. I've also lost countless hours debugging problems like
>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A more
>    general report is #1768. This is slated to be fixed in lein 3.0, but
>    a) who knows when that will be and b) I'm still at the mercy of plugins
>    doing the same thing anyway.
>    3. Java cannot change the CWD of the current process, which means that
>    local-repo always appears somewhere in the IntelliJ bin directory. This
>    also fails for users' home grown versioning systems when they do things
>    like (slurp versions.txt). More examples here
>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>    (there are probably more, but that's the easiest search). Essentially, one
>    of lein's big selling points is that it's declarative, but the fact that
>    project.clj is actually executed means that it's not truly declarative.
>    4. Many things in lein (gpg, proxies etc) are configured using
>    environment variables, which are a pain to pass when embedding. Cursive
>    hooks leiningen fairly heavily using robert.hooke t o supply values for
>    these instead of getting them from env vars.
>    5. Often you need access to the lein tasks, not just core. Cursive has
>    bundled all of lein for a long time so that it can access the repl task. If
>    you don't do this you end up duplicating a lot of the code from the tasks
>    anyway.
>    6. There are also lots of minor issues - I wanted to hook into the
>    Aether transfer listener so that I could provide updates in the Cursive UI
>    when libs are downloaded. Currently I do this in Cursive by hooking
>    leiningen.core.classpath/warn and checking if it looks like a "Retrieving"
>    message and parsing it if so.
>    7. As Laurent says, lein, even just core, is really big. The Cursive
>    download is about 3x the size it would be if it didn't bundle lein.
>
> That said, while I understand Laurent's worry about stability, that's
> never been a problem. I've never received a single bug report about
> stability that was as a result of bundling lein.
>
> There is also one significant benefit to bundling lein - it's fast.
> Starting a JVM process for lein every time Cursive needs to sync the deps
> is going to be way too slow. This will be particularly true for large
> multi-module projects where to do everything totally accurately (in
> particular, the CWD) requires starting a new JVM per module - this is
> unworkable. I considered caching the last-modified of the project.clj and
> only re-parsing a particular module if that had changed, but I'd need to do
> that for all possible contributing files which is impossible to calculate.
> Apart from profiles.clj and other "official" files which might affect the
> result, I'd also need to test the version.txt type files from #3, and
> there's no way I can detect all of those.
>
> Since I don't think that stability is actually a major problem, I'm
> planning to continue running lein in-process but I'm going to isolate its
> classpath using ShimDandy. I have a prototype of this but it's still pretty
> primitive, and I have a lot of other things going on so it's advancing
> slowly. I run lein in a shim, and I create the shim's classpath using
> Aether so I don't actually bundle lein, I pull it in when creating the
> shim. This means that users can choose the lein version they want, and the
> files are stored in .m2 rather than being downloaded with Cursive. I'll
> then throw the shim away when done so memoisation won't be a problem. I can
> amortize the performance problem by eagerly creating the shim and loading
> the namespaces into it before I need it, similar to boot's pod caching.
> This is much less costly than eagerly starting a JVM and keeping that
> around until needed.
>
> I'm also going to try to fiddle much less with the lein internals since
> that has caused problems when upgrading lein. Using this new mechanism I
> can call the functions representing the lein tasks as if they were being
> called from main, whic h greatly reduces my dependency on lein internals.
> I'll still have to hook for gpg/proxies etc, but I'll be hooking aether
> rather than lein for those, which is much more stable since it's basically
> abandonware at this point - I should have done this ages ago. I can also
> hook into the transfer listener by hooking aether.
>
> My interface to the shim will be to pass it a list of file paths to the
> project.clj files and eventually a list of profiles, and it will return a
> map containing all the information I need about the projects.
>
> The upshot of all this is that this should fix all my current problems
> except #3, which there's no workaround for. I'll fix the local-repo problem
> by hooking lein or fiddling with the project.clj, but the only way to fix
> the slurp problem is to start a JVM per lein project, or convince users not
> to do that. Or hook slurp, I guess.
>
> Laurent, currently I run the REPL within Cursive using the trampoline
> mechanism - IMO you don't have to emulate it, you can just use it directly.
> Get lein to generate a trampoline file then parse that - it's very easy to
> parse since all the elements always appear in the same order. Currently I
> hook the lein internals to do this, but as described above I'm about to
> start calling lein trampoline directly from the shim.
>
> Cheers,
> Colin
>
>
> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com> wrote:
>
>> OK understood this works as expected: lein update-in : assoc :eval-in
>> :pprint -- run -m clojure.main
>>
>> BTW, does that mean that it's possible to emulate TRAMPOLINE by storing
>> the result o f :pprint (and thus preventing the scenario where there's 2
>> JVMs loaded, one for lein itself, and one for the project) ?
>>
>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>
>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>
>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`; this
>>> will
>>> > pprint the form to be run and print the classpath and JVM args instead
>>> > of actually running it.
>>>
>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>
>>> -Phil
>>>
>>
>>
>>
>> --
>> Laurent Petit
>>
>
>


-- 
Laurent Petit

Re: [leiningen] Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-17 @ 11:21
More thoughts, Colin:

2015-08-17 12:37 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:

> Curiously enough, I'm also about to totally change the way I integrate
> Leiningen in Cursive.
>
> To answer Phil's question about embedding leiningen-core, there are many
> many disadvantages to doing so:
>
>    1. You're tied to a specific version of lein. Currently I bundle 2.5.0
>    - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users encountered bugs
>    so I downgraded to 2.5.0. Some users currently experience bugs with 2.5.0,
>    but they're easier to work around. Anyone using older versions on their
>    project is out of luck.
>    2. Lein incorrectly memoises many things internally, which is a
>    gigantic pain. Currently users have to restart Cursive every time they
>    change profiles.clj, and additionally they also have to be aware of the bug
>    to know to do that. I've also lost countless hours debugging problems like
>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A more
>    general report is #1768. This is slated to be fixed in lein 3.0, but
>    a) who knows when that will be and b) I'm still at the mercy of plugins
>    doing the same thing anyway.
>    3. Java cannot change the CWD of the current process, which means that
>    local-repo always appears somewhere in the IntelliJ bin directory. This
>    also fails for users' home grown versioning systems when they do things
>    like (slurp versions.txt). More examples here
>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>    (there are probably more, but that's the easiest search). Essentially, one
>    of lein's big selling points is that it's declarative, but the fact that
>    project.clj is actually executed means that it's not truly declarative.
>    4. Many things in lein (gpg, proxies etc) are configured using
>    environment variables, which are a pain to pass when embedding. Cursive
>    hooks leiningen fairly heavily using robert.hooke t o supply values for
>    these instead of getting them from env vars.
>    5. Often you need access to the lein tasks, not just core. Cursive has
>    bundled all of lein for a long time so that it can access the repl task. If
>    you don't do this you end up duplicating a lot of the code from the tasks
>    anyway.
>    6. There are also lots of minor issues - I wanted to hook into the
>    Aether transfer listener so that I could provide updates in the Cursive UI
>    when libs are downloaded. Currently I do this in Cursive by hooking
>    leiningen.core.classpath/warn and checking if it looks like a "Retrieving"
>    message and parsing it if so.
>    7. As Laurent says, lein, even just core, is really big. The Cursive
>    download is about 3x the size it would be if it didn't bundle lein.
>
> That said, while I understand Laurent's worry about stability, that's
> never been a problem. I've never received a single bug report about
> stability that was as a result of bundling lein.
>

I admit that it has been more a fear than an issue.


> There is also one significant benefit to bundling lein - it's fast.
> Starting a JVM process for lein every time Cursive needs to sync the deps
> is going to be way too slow. This will be particularly true for large
> multi-module projects where to do everything totally accurately (in
> particular, the CWD) requires starting a new JVM per module - this is
> unworkable. I considered caching the last-modified of the project.clj and
> only re-parsing a particular module if that had changed, but I'd need to do
> that for all possible contributing files which is impossible to calculate.
> Apart from profiles.clj and other "official" files which might affect the
> result, I'd also need to test the version.txt type files from #3, and
> there's no way I can detect all of those.
>
> Since I don't think that stability is actually a major problem, I'm
> planning to continue running lein in-process but I'm going to isolate its
> classpath using ShimDandy. I have a prototype of this but it's still pretty
> primitive, and I have a lot of other things going on so it's advancing
> slowly. I run lein in a shim, and I create the shim's classpath using
> Aether so I don't actually bundle lein, I pull it in when creating the
> shim. This means that users can choose the lein version they want, and the
> files are stored in .m2 rather than being downloaded with Cursive.
>

I had considered this for CCW, but chose to embed leiningen because back in
time (and still currently in practice, despite the growing interest in
boot) leiningen was the only option.
Having the latest leiningen downloaded at the same time as CCW is easier
for users, I think. They have to wait once for things to download, and then
they're good to go.
By downloading lein lazily, you're at risk for users going offline, wiping
out their ~/.m2/repository (I do this from time to time). That's one more
moving part, not to be considered too lightly, IMHO, but YMMV.


> I'll then throw the shim away when done so memoisation won't be a problem.
> I can amortize the performance problem by eagerly creating the shim and
> loading the namespaces into it before I need it, similar to boot's pod
> caching. This is much less costly than eagerly starting a JVM and keeping
> that around until needed.
>

Have you checked the time required for the JVM itself to start versus the
time to load leiningen.main and all its dependencies? I think I'll do it
again, just to check that the ratio proves your point. We might be
surprised discovering that the JVM startup time is not the problem, leading
to the conclusion that reloading everything of lein in a new classloader
will not be that different (performance wise) than starting a new JVM.


>
> I'm also going to try to fiddle much less with the lein internals since
> that has caused problems when upgrading lein. Using this new mechanism I
> can call the functions representing the lein tasks as if they were being
> called from main, whic h greatly reduces my dependency on lein internals.
> I'll still have to hook for gpg/proxies etc, but I'll be hooking aether
> rather than lein for those, which is much more stable since it's basically
> abandonware at this point - I should have done this ages ago. I can also
> hook into the transfer listener by hooking aether.
>
> My interface to the shim will be to pass it a list of file paths to the
> project.clj files and eventually a list of profiles, and it will return a
> map containing all the information I need about the projects.
>
> The upshot of all this is that this should fix all my current problems
> except #3, which there's no workaround for. I'll fix the local-repo problem
> by hooking lein or fiddling with the project.clj, but the only way to fix
> the slurp problem is to start a JVM per lein project, or convince users not
> to do that. Or hook slurp, I guess.
>
> Laurent, currently I run the REPL within Cursive using the trampoline
> mechanism - IMO you don't have to emulate it, you can just use it directly.
> Get lein to generate a trampoline file then parse that - it's very easy to
> parse since all the elements always appear in the same order. Currently I
> hook the lein internals to do this, but as described above I'm about to
> start calling lein trampoline directly from the shim.
>
> Cheers,
> Colin
>
>
> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com> wrote:
>
>> OK understood this works as expected: lein update-in : assoc :eval-in
>> :pprint -- run -m clojure.main
>>
>> BTW, does that mean that it's possible to emulate TRAMPOLINE by storing
>> the result o f :pprint (and thus preventing the scenario where there's 2
>> JVMs loaded, one for lein itself, and one for the project) ?
>>
>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>
>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>
>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`; this
>>> will
>>> > pprint the form to be run and print the classpath and JVM args instead
>>> > of actually running it.
>>>
>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>
>>> -Phil
>>>
>>
>>
>>
>> --
>> Laurent Petit
>>
>
>


-- 
Laurent Petit

Re: [leiningen] Getting leiningen classpath

From:
Colin Fleming
Date:
2015-08-17 @ 11:40
Thanks for the thoughts, Laurent - very interesting. Do you have a link to
the CCW source that changes the CWD property? I read about doing that a
while back, but since IntelliJ doesn't isolate classloaders like Eclipse
does I couldn't do it. Now that I'm going to use ShimDandy I can probably
do that. If you isolate classloaders, are the system properties then
isolated too?

In terms of lazily downloading Leiningen, it'll happen as soon as they open
a lein project, so that's generally almost immediately. If they're offline
then they'll have problems, right, but at that point they'll almost
certainly need to be connected to download deps anyway so I'm not sure
it'll be a problem. I think the most important thing is that the user
doesn't have to explicitly download it separately from Cursive,
particularly on Windows where that seems to be pretty painful.

WRT JVM startup time, I'm not worried about that - that is dwarfed by the
time to load the Clojure/lein namespaces. But by doing that in-process I
can start a shim and load the namespaces into it ahead of time so it's
ready when I need it and will work immediately. To do that with a
standalone JVM would mean starting up an external JVM eagerly and having it
hanging around - I'm more worried about memory use than time for that.
Obviously that will still occupy memory in the IntelliJ heap but I'm paying
that price now anyway - it will be more due to isolation, but still less
than the 300mb that a JVM takes up.


On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com> wrote:

> More thoughts, Colin:
>
> 2015-08-17 12:37 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>
>> Curiously enough, I'm also about to totally change the way I integrate
>> Leiningen in Cursive.
>>
>> To answer Phil's question about embedding leiningen-core, there are many
>> many disadvantages to doing so:
>>
>>    1. You're tied to a specific version of lein. Currently I bundle
>>    2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users encountered
>>    bugs so I downgraded to 2.5.0. Some users currently experience bugs with
>>    2.5.0, but they're easier to work around. Anyone using older versions on
>>    their project is out of luck.
>>    2. Lein incorrectly memoises many things internally, which is a
>>    gigantic pain. Currently users have to restart Cursive every time they
>>    change profiles.clj, and additionally they also have to be aware of the bug
>>    to know to do that. I've also lost countless hours debugging problems like
>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A more
>>    general report is #1768. This is slated to be fixed in lein 3.0, but
>>    a) who knows when that will be and b) I'm still at the mercy of plugins
>>    doing the same thing anyway.
>>    3. Java cannot change the CWD of the current process, which means
>>    that local-repo always appears somewhere in the IntelliJ bin directory.
>>    This also fails for users' home grown versioning systems when they do
>>    things like (slurp versions.txt). More examples here
>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>    (there are probably more, but that's the easiest search). Essentially, one
>>    of lein's big s elling points is that it's declarative, but the fact that
>>    project.clj is actually executed means that it's not truly declarative.
>>    4. Many things in lein (gpg, proxies etc) are configured using
>>    environment variables, which are a pain to pass when embedding. Cursive
>>    hooks leiningen fairly heavily using robert.hooke t o supply values for
>>    these instead of getting them from env vars.
>>    5. Often you need access to the lein tasks, not just core. Cursive
>>    has bundled all of lein for a long time so that it can access the repl
>>    task. If you don't do this you end up duplicating a lot of the code from
>>    the tasks anyway.
>>    6. There are also lots of minor issues - I wanted to hook into the
>>    Aether transfer listener so that I could provide updates in the Cursive UI
>>    when libs are downloaded. Currently I do this in Cursive by hooking
>>    leiningen.core.classpath/warn and checking if it looks like a "Retrieving"
>>    message and parsing it if so.
>>    7. As Laurent says, lein, even just core, is really big. The Cursive
>>    download is about 3x the size it would be if it didn't bundle lein.
>>
>> That said, while I understand Laurent's worry about stability, that's
>> never been a problem. I've never received a single bug report about
>> stability that was as a result of bundling lein.
>>
>
> I admit that it has been more a fear than an issue.
>
>
>> There is also one significant benefit to bundling lein - it's fast.
>> Starting a JVM process for lein every time Cursive needs to sync the deps
>> is going to be way too slow. This will be particularly true for large
>> multi-module projects where to do everything totally accurately (in
>> particular, the CWD) requires starting a new JVM per module - this is
>> unworkable. I considered caching the last-modified of the project.clj and
>> only re-parsing a particular module if that had changed, but I'd need to do
>> that for all possible contributing files which is impossible to calculate.
>> Apart from profiles.clj and other "official" files which might affect the
>> result, I'd also need to test the ver sion.txt type files from #3, and
>> there's no way I can detect all of those.
>>
>> Since I don't think that stability is actually a major problem, I'm
>> planning to continue running lein in-process but I'm going to isolate its
>> classpath using ShimDandy. I have a prototype of this but it's still pretty
>> primitive, and I have a lot of other things going on so it's advancing
>> slowly. I run lein in a shim, and I create the shim's classpath using
>> Aether so I don't actually bundle lein, I pull it in when creating the
>> shim. This means that users can choose the lein version they want, and the
>> files are stored in .m2 rather than being downloaded with Cursive.
>>
>
> I had considered this for CCW, but chose to embed leiningen because back
> in time (and still currently in practice, despite the growing interest in
> boot) leiningen was the only option.
> Having the latest leiningen downloaded at the same time as CCW is easier
> for users, I think. They have to wait once for things to download, and then
> they're good to go.
> By downloading lein lazily, you're at risk for users going offline, wiping
> out their ~/.m2/repository (I do this from time to time). That's one more
> moving part, not to be considered too lightly, IMHO, but YMMV.
>
>
>> I'll then throw the shim away when done so memoisation won't be a
>> problem. I can amortize the performance problem by eagerly creating the
>> shim and loading the namespaces into it before I need it, similar to boot's
>> pod caching. This is much less costly than eagerly starting a JVM and
>> keeping that around until needed.
>>
>
> Have you checked the time required for the JVM itself to start versus the
> time to load leiningen.main and all its dependencies? I think I'll do it
> again, just to check that the ratio proves your point. We might be
> surprised discovering that the JVM startup time is not the problem, leading
> to the conclusion that reloading everything of lein in a new classloader
> will n ot be that different (performance wise) than starting a new JVM.
>
>
>>
>> I'm also going to try to fiddle much less with the lein internals since
>> that has caused problems when upgrading lein. Using this new mechanism I
>> can call the functions representing the lein tasks as if they were being
>> called from main, whic h greatly reduces my dependency on lein internals.
>> I'll still have to hook for gpg/proxies etc, but I'll be hooking aether
>> rather than lein for those, which is much more stable since it's basically
>> abandonware at this point - I should have done this ages ago. I can also
>> hook into the transfer listener by hooking aether.
>>
>> My interface to the shim will be to pass it a list of file paths to the
>> project.clj files and eventually a list of profiles, and it will return a
>> map containing all the information I need about the projects.
>>
>> The upshot of all this is that this should fix all my current problems
>> except #3, which there's no workaround for. I'll fix the local-repo problem
>> by hooking lein or fiddling with the project.clj, but the only way to fix
>> the slurp problem is to start a JVM per lein project, or convince users not
>> to do that. Or hook slurp, I guess.
>>
>> Laurent, currently I run the REPL within Cursive using the trampoline
>> mechanism - IMO you don't have to emulate it, you can just use it directly.
>> Get lein to generate a trampoline file then parse that - it's very easy to
>> parse since all the elements always appear in the same order. Currently I
>> hook the lein internals to do this, but as described above I'm about to
>> start calling lein trampoline directly from the shim.
>>
>> Cheers,
>> Colin
>>
>>
>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com>
>> wrote:
>>
>>> OK understood this works as expected: lein update-in : assoc :eval-in
>>> :pprint -- run -m clojure.main
>>>
>>> BTW, does that mean that it's possible t o emulate TRAMPOLINE by storing
>>> the result o f :pprint (and thus preventing the scenario where there's 2
>>> JVMs loaded, one for lein itself, and one for the project) ?
>>>
>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>
>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>
>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`; this
>>>> will
>>>> > pprint the form to be run and print the classpath and JVM args instead
>>>> > of actually running it.
>>>>
>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>
>>>> -Phil
>>>>
>>>
>>>
>>>
>>> --
>>> Laurent Petit
>>>
>>
>>
>
>
> --
> Laurent Petit
>

Re: [leiningen] Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-17 @ 11:55
2015-08-17 13:40 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:

> Thanks for the thoughts, Laurent - very interesting. Do you have a link to
> the CCW source that changes the CWD property? I read about doing that a
> while back, but since IntelliJ doesn't isolate classloaders like Eclipse
> does I couldn't do it. Now that I'm going to use ShimDandy I can probably
> do that. If you isolate classloaders, are the system properties then
> isolated too?
>

I won't be able to do so ... because I was wrong. There's no trace of
"user.dir" in CCW codebase. So I may have thought about it in my shower,
but never did it in practice ...


>
> In terms of lazily downloading Leiningen, it'll happen as soon as they
> open a lein project, so that's generally almost immediately. If they're
> offline then they'll have problems, right, but at that point they'll almost
> certainly need to be connected to download deps anyway so I'm not sure
> it'll be a problem. I think the most important thing is that the user
> doesn't have to explicitly download it separately from Cursive,
> particularly on Windows where that seems to be pretty painful.
>

Fair enough


> WRT JVM startup time, I'm not worried about that - that is dwarfed by the
> time to load the Clojure/lein namespaces. But by doing that in-process I
> can start a shim and load the namespaces into it ahead of time so it's
> ready when I need it and will work immediately. To do that with a
> standalone JVM would mean starting up an external JVM eagerly and having it
> hanging around - I'm more worried about memory use than time for that.
> Obviously that will still occupy memory in the IntelliJ heap but I'm paying
> that price now anyway - it will be more due to isolation, but still less
> than the 300mb that a JVM takes up.
>

Well, yes, always keeping ahead of time one classloader is certainly easier
and more light-weight for the user's computer memory.
But still, interoperating through a new JVM rather than through
classloaders appeals more to me:
- fixes all the problems you listed, without compromise
- introduces the problem of more RAM if a pool of JVM processes is
maintained. This could be leveraged by letting the user choose the pool
size (setting it to 0 would be friendly for computers with low memory.
Setting it to 5 would be a boost for users with modern computers and multi
module projects. Setting it to 1 by default would probably be a good
average).
- more flexible to adapt to boot, planck or similar for clojurescript, or
plain processes.

I think I will stick to my idea of moving from embedded to external. Thanks
for sharing the thoughts!


>
>
> On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com> wrote:
>
>> More thoughts, Colin:
>>
>> 2015-08-17 12:37 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>>
>>> Curiously enough, I'm also about to totally change the way I integrate
>>> Leiningen in Cursive.
>>>
>>> To answer Phil's question about embedding leiningen-core, there are many
>>> many disadvantages to doing so:
>>>
>>>    1. You're tied to a specific version of lein. Currently I bundle
>>>    2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users encountered
>>>    bugs so I downgraded to 2.5.0. Some users currently experience bugs with
>>>    2.5.0, but they're easier to work around. Anyone using older versions on
>>>    their project is out of luck.
>>>    2. Lein incorrectly memoises many things internally, which is a
>>>    gigantic pain. Currently users have to restart Cursive every time they
>>>    change profiles.clj, and additionally they also have to be aware of the bug
>>>    to know to do that. I've also lost countless hours debugging problems like
>>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A more
>>>    general report is #1768. This is slated to be fixed in lein 3.0, but
>>>    a) who knows when that will be and b) I'm still at the mercy of plugins
>>>    doing the same thing anyway.
>>>    3. Java cannot change the CWD of the current process, which means
>>>    that local-repo always appears somewhere in the IntelliJ bin directory.
>>>    This also fails for users' home grown versioning systems when they do
>>>    things like (slurp versions.txt). More examples here
>>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>>    (there are probably more, but that's the easiest search). Essentially, one
>>>    of lein' s big s elling points is that it's declarative, but the fact that
>>>    project.clj is actually executed means that it's not truly declarative.
>>>    4. Many things in lein (gpg, proxies etc) are configured using
>>>    environment variables, which are a pain to pass when embedding. Cursive
>>>    hooks leiningen fairly heavily using robert.hooke t o supply values for
>>>    these instead of getting them from env vars.
>>>    5. Often you need access to the lein tasks, not just core. Cursive
>>>    has bundled all of lein for a long time so that it can access the repl
>>>    task. If you don't do this you end up duplicating a lot of the code from
>>>    the tasks anyway.
>>>    6. There are also lots of minor issues - I wanted to hook into the
>>>    Aether transfer listener so that I could provide updates in the Cursive UI
>>>    when libs are downloaded. Currently I do this in Cursive by hooking
>>>    leiningen.core.classpath/warn and checking if it looks like a "Retrieving"
>>>    message and parsing it if so.
>>>    7. As Laurent says, lein, even just core, is really big. The Cursive
>>>    download is about 3x the size it would be if it didn't bundle lein.
>>>
>>> That said, while I understand Laurent's worry about stability, that's
>>> never been a problem. I've never received a single bug report about
>>> stability that was as a result of bundling lein.
>>>
>>
>> I admit that it has been more a fear than an issue.
>>
>>
>>> There is also one significant benefit to bundling lein - it's fast.
>>> Starting a JVM process for lein every time Cursive needs to sync the deps
>>> is going to be way too slow. This will be particularly true for large
>>> multi-module projects where to do everything totally accurately (in
>>> particular, the CWD) requires starting a new JVM per module - this is
>>> unworkable. I considered caching the last-modified of the project.clj and
>>> only re-parsing a particular module if that had changed, but I'd need to do
>>> that for all possible contributing files which is impossible to calculate.
>>> Apart from profiles.clj and other "official" files which might a ffect the
>>> result, I'd also need to test the ver sion.txt type files from #3, and
>>> there's no way I can detect all of those.
>>>
>>> Since I don't think that stability is actually a major problem, I'm
>>> planning to continue running lein in-process but I'm going to isolate its
>>> classpath using ShimDandy. I have a prototype of this but it's still pretty
>>> primitive, and I have a lot of other things going on so it's advancing
>>> slowly. I run lein in a shim, and I create the shim's classpath using
>>> Aether so I don't actually bundle lein, I pull it in when creating the
>>> shim. This means that users can choose the lein version they want, and the
>>> files are stored in .m2 rather than being downloaded with Cursive.
>>>
>>
>> I had considered this for CCW, but chose to embed leiningen because back
>> in time (and still currently in practice, despite the growing interest in
>> boot) leiningen was the only option.
>> Having the latest leiningen downloaded at the same time as CCW is easier
>> for users, I think. They have to wait once for things to download, and then
>> they're good to go.
>> By downloading lein lazily, you're at risk for users going offline,
>> wiping out their ~/.m2/repository (I do this from time to time). That 's
>> one more moving part, not to be considered too lightly, IMHO, but YMMV.
>>
>>
>>> I'll then throw the shim away when done so memoisation won't be a
>>> problem. I can amortize the performance problem by eagerly creating the
>>> shim and loading the namespaces into it before I need it, similar to boot's
>>> pod caching. This is much less costly than eagerly starting a JVM and
>>> keeping that around until needed.
>>>
>>
>> Have you checked the time required for the JVM itself to start versus the
>> time to load leiningen.main and all its dependencies? I think I'll do it
>> again, just to check that the ratio proves your point. We might be
>> surprised discovering that the JVM startup time is not the problem, leading
>> to the conclusion that reloading everything of lein in a new classloader
>> will n ot be that different (performance wise) than starting a new JVM.
>>
>>
>>>
>>> I'm also going to try to fiddle much less with the lein internals since
>>> that has caused problems when upgrading lein. Using this new mechanism I
>>> can call the functions representing the lein tasks as if they were being
>>> called from main, whic h greatly reduces my dependency on lein internals.
>>> I'll still have to hook for gpg/proxies etc, but I'll be hooking aether
>>> rather than lein for those, which is much more stable since it's basically
>>> abandonware at this point - I should have done this ages ago. I can also
>>> hook into the transfer listener by hooking aether.
>>>
>>> My interface to the shim will be to pass it a list of file paths to the
>>> project.clj files and eventually a list of profiles, and it will return a
>>> map containing all the information I need about the projects.
>>>
>>> The upshot of all this is that this should fix all my current problems
>>> except #3, which there's no workaround for. I'll fix the local-repo problem
>>> by hooking lein or fiddling with the project.clj, but the only way to fix
>>> the slurp problem is to start a JVM per lein project, or convince users not
>>> to do that. Or hook slurp, I guess.
>>>
>>> Laurent, currently I run the REPL withi n Cursive using the trampoline
>>> mechanism - IMO you don't have to emulate it, you can just use it directly.
>>> Get lein to generate a trampoline file then parse that - it's very easy to
>>> parse since all the elements always appear in the same order. Currently I
>>> hook the lein internals to do this, but as described above I'm about to
>>> start calling lein trampoline directly from the shim.
>>>
>>> Cheers,
>>> Colin
>>>
>>>
>>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com>
>>> wrote:
>>>
>>>> OK understood this works as expected: lein update-in : assoc :eval-in
>>>> :pprint -- run -m clojure.main
>>>>
>>>> BTW, does t hat mean that it's possible t o emulate TRAMPOLINE by
>>>> storing the result o f :pprint (and thus preventing the scenario where
>>>> there's 2 JVMs loaded, one for lein itself, and one for the project) ?
>>>>
>>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>>
>>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>>
>>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`; this
>>>>> will
>>>>> > pprint the form to be run and print the classpath and JVM args
>>>>> instead
>>>>> > of actually running it.
>>>>>
>>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>>
>>>>> -Phil
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Laurent Petit
>>>>
>>>
>>>
>>
>>
>> --
>> Laurent Petit
>>
>
>


-- 
Laurent Petit

Re: [leiningen] Getting leiningen classpath

From:
Colin Fleming
Date:
2015-08-17 @ 13:38
Interesting about the user.dir thing, I also investigated that, I'll try it
and report back. If not I'll probably just hook slurp, which is obviously
not very general but is probably a good 90% case.

I've been thinking about an external tooling JVM for a long time. I'll have
to do it sometime, since boot support will require it - boot needs Java 1.7
but I have to support 1.6 in Cursive. So that will have to be an external
JVM. I'm planning to run shims within that JVM to support functionality
(such as macroexpansion or core.typed checking) that users currently have
to start a REPL for. They may not want to do so, or may not want to modify
their running REPL just to see a macroexpansion, so I will start a shim
inside the JVM and use that.

However note that this is still not compromise-free. Apart from the memory
use, in the case of a multi-module process you'll still need to have the
CWD correct for each lein project. A Cursive user tweeted the other day
about his project containing 66 modules (lein projects), 3068 files and
500k LOC. Starting 66 JVMs just to sync the dependencies is never going to
work. While the single-project case is probably by far the most common, I
still need to support the large project case properly and I can't see a way
to do that with no compromises at all. I could work around the worst of it
with some complex configuration saying how many JVMs to start up, and
specifying which modules should use which JVMs, but it starts to sound like
a nightmare at that point, both for me and the user.

On 17 August 2015 at 13:55, Laurent PETIT <laurent.petit@gmail.com> wrote:

>
>
> 2015-08-17 13:40 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>
>> Thanks for the thoughts, Laurent - very interesting. Do you have a link
>> to the CCW source that changes the CWD property? I read about doing that a
>> while back, but since IntelliJ doesn't isolate classloaders like Eclipse
>> does I couldn't do it. Now that I'm going to use ShimDandy I can probably
>> do that. If you isolate classloaders, are the system properties then
>> isolated too?
>>
>
> I won't be able to do so ... because I was wrong. There's no trace of
> "user.dir" in CCW codebase. So I may have thought about it in my shower,
> but never did it in pract ice ...
>
>
>>
>> In terms of lazily downloading Leiningen, it'll happen as soon as they
>> open a lein project, so that's generally almost immediately. If they're
>> offline then they'll have problems, right, but at that point they'll almost
>> certainly need to be connected to download deps anyway so I'm not sure
>> it'll be a problem. I think the most important thing is that the user
>> doesn't have to explicitly download it separately from Cursive,
>> particularly on Windows where that seems to be pretty painful.
>>
>
> Fair enough
>
>
>> WRT JVM startup time, I'm not worried about that - that is dwarfed by the
>> time to load the Clojure/lein namespa ces. But by doing that in-process I
>> can start a shim and load the namespaces into it ahead of time so it's
>> ready when I need it and will work immediately. To do that with a
>> standalone JVM would mean starting up an external JVM eagerly and having it
>> hanging around - I'm more worried about memory use than time for that.
>> Obviously that will still occupy memory in the IntelliJ heap but I'm paying
>> that price now anyway - it will be more due to isolation, but still less
>> than the 300mb that a JVM takes up.
>>
>
> Well, yes, always keeping ahead of time one classloader is certainly
> easier and more light-weight for the user's computer memory.
> But still, interoperating through a new JVM rather than through
> classloaders appeals more to me:
> - fixes all the problems you listed, without compromise
> - introduces the problem of more RAM if a pool of JVM processes is
> maintained. This could be leveraged by letting t he user choose the pool
> size (setting it to 0 would be friendly for computers with low memory.
> Setting it to 5 would be a boost for users with modern computers and multi
> module projects. Setting it to 1 by default would probably be a good
> average).
> - more flexible to adapt to boot, planck or similar for clojurescript, or
> plain processes.
>
> I think I will stick to my idea of moving from embedded to external.
> Thanks for sharing the thoughts!
>
>
>>
>>
>> On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com>
>> wrote:
>>
>>> Mor e thoughts, Colin:
>>>
>>> 2015-08-17 12:37 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>>>
>>>> Curiously enough, I'm also about to totally change the way I integrate
>>>> Leiningen in Cursive.
>>>>
>>>> To answer Phil's question about embedding leiningen-core, there are
>>>> many many disadvantages to doing so:
>>>>
>>>>    1. You're tied to a specific version of lein. Currently I bundle
>>>>    2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users 
encountered
>>>>    bugs so I downgraded to 2.5.0. Some users currently experience bugs with
>>>>    2.5.0, but they're easier to work around. Anyone using older versions on
>>>>    their project is ou t of luck.
>>>>    2. Lein incorrectly memoises many things internally, which is a
>>>>    gigantic pain. Currently users have to restart Cursive every time they
>>>>    change profiles.clj, and additionally they also have to be aware 
of the bug
>>>>    to know to do that. I've also lost countless hours debugging problems like
>>>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A
>>>>    more general report is #1768. This is slated to be fixed in lein
>>>>    3.0, but a) who knows when that will be and b) I'm still at the mercy of
>>>>    plugins doing the same thing anyway.
>>>>    3. Java cannot change the CWD of the current process, which means
>>>>    that local-repo always appears somewhere in the IntelliJ bin directory.
>>>>    This also fails for users' home grown versioning systems when they do
>>>>    things like (slurp versions.txt). More examples here
>>>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>>>    (there are probably more, but that's the easiest search). Essentially, one
>>>>    of le in' s big s elling points is that it's declarative, but the 
fact that
>>>>    project.clj is actually executed means that it's not truly declarative.
>>>>    4. Many things in lein (gpg, proxies etc) are configured using
>>>>    environment variables, which are a pain to pass when embedding. Cursive
>>>>    hooks leiningen fairly heavily using robert.hooke t o supply values for
>>>>    these instead of getting them from env vars.
>>>>    5. Often you need access to the lein tasks, not just core. Cursive
>>>>    has bundled all of lein for a long time so that it can access the repl
>>>>    task. If you don't do this you end up duplicating a lot of the code from
>>>>    the tasks anyway.
>>>>    6. There are also lots of minor issues - I wanted to hook into the
>>>>    Aether transfer listener so that I could provide updates in the Cursive UI
>>>>    when libs are downloaded. Currently I do this in Cursive by hooking
>>>>    leiningen.core.classpath/warn and checking if it looks like a "Retrieving"
>>>>    message and parsing it if so.
>>>>    7. As Laurent says, lein, even just core, is really big. The
>>>>    Cursive download is about 3x the size it would be if it didn't 
bundle lein.
>>>>
>>>> That said, while I understand Laurent's worry about stability, that's
>>>> never been a problem. I've never received a single bug report about
>>>> stability that was as a result of bundling lein.
>>>>
>>>
>>> I admit that it has been more a fear than an issue.
>>>
>>>
>>>> There is also one significant benefit to bundling lein - it's fast.
>>>> Starting a JVM process for lein every time Cursive needs to sync the deps
>>>> is going to be way too slow. This will be particularly true for large
>>>> multi-module projects where to do everything totally accurately (in
>>>> particular, the CWD) requires starting a new JVM per module - this is
>>>> unworkable. I considered caching the last-modified of the project.clj and
>>>> only re-parsing a particular module if that had changed, but I'd need to do
>>>> that for all possible contributing files which is impossible to calculate.
>>>> Apart from profiles.clj and other "offic ial" files which might a ffect the
>>>> result, I'd also need to test the ver sion.txt type files from #3, and
>>>> there's no way I can detect all of those.
>>>>
>>>> Since I don't think that stability is actually a major problem, I'm
>>>> planning to continue running lein in-process but I'm going to isolate its
>>>> classpath using ShimDandy. I have a prototype of this but it's still pretty
>>>> primitive, and I have a lot of other things going on so it's advancing
>>>> slowly. I run lein in a shim, and I create the shim's classpath using
>>>> Aether so I don't actually bundle lein, I pull it in when creating the
>>>> shim. This means that users can choose the lein version they want, and the
>>>> files are stored in .m2 rather than being downloaded with Cursive.
>>>>
>>>
>>> I had considered this for CCW, but chose to embed leiningen because back
>>> in time (and still currently in practice, despite the growing interest in
>>> boot) leiningen was the only option.
>>> Having the latest leiningen downloaded at the same time as CCW is easier
>>> for users, I think. They have to wait once for things to download, and then
>>> they're good to go.
>>> By downloading lein lazily, you're at risk for users going offline,
>>> wiping out their ~/.m2/repository (I do this fr om time to time). That 's
>>> one more moving part, not to be considered too lightly, IMHO, but YMMV.
>>>
>>>
>>>> I'll then throw the shim away when done so memoisation won't be a
>>>> problem. I can amortize the performance problem by eagerly creating the
>>>> shim and loading the namespaces into it before I need it, similar to boot's
>>>> pod caching. This is much less costly than eagerly starting a JVM and
>>>> keeping that around until needed.
>>>>
>>>
>>> Have you checked the time required for the JVM itself to start versus
>>> the time to load leiningen.main and all its dependencies? I think I'll do
>>> it again, just to check that the ratio proves your point. We might be
>>> surprised discovering that the JVM startup time is not the problem, leading
>>> to the conclusion that reloading everything of lein in a new classloader
>>> will n ot be that different (performance wise) than starting a new JVM.
>>>
>>>
>>>>
>>>> I'm also going to try to fiddle much less with the lein internals since
>>>> that has caused problems when upgrading lein. Using this new mechanism I
>>>> can call the functions representing the lein tasks as if they were being
>>>> called from main, whic h greatly reduces my dependency on lein internals.
>>>> I'll still have to hook for gpg/proxies etc, but I'll be hooking aether
>>>> rather than lein for those, which is much more stable since it's basically
>>>> abandonware at this point - I should have done this ages ago. I can also
>>>> hook into the transfer listener by hooking aether.
>>>>
>>>> My interface to the shim will be to pass it a list of file paths to the
>>>> project.clj files and eventually a list of profiles, and it will return a
>>>> map containing all the information I need about the projects.
>>>>
>>>> The upshot of all this is that this should fix all my current problems
>>>> except #3, which there's no workaround for. I'll fix the local-repo problem
>>>> by hooking lein or fiddling with the project.clj, but the only way to fix
>>>> the slurp problem is to start a JVM per lein project, or convince users not
>>>> to do that. Or hook slurp, I guess.
>>>>
>>>> Laurent, currently I run the REPL withi n Cursive using the trampoline
>>>> mechanism - IMO you don't have to emulate it, you can just use it directly.
>>>> Get lein to generate a trampoline file then parse that - it's very easy to
>>>> parse since all the elements always appear in the same order. Currently I
>>>> hook the lein internals to do this, but as described above I'm about to
>>>> start calling lein trampoline directly from the shim.
>>>>
>>>> Cheers,
>>>> Colin
>>>>
>>>>
>>>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com>
>>>> wrote:
>>>>
>>>>> OK understood this works as expected: lein update-in : assoc :eval-in
>>>>> :pprint -- run -m clojure.main
>>>>>
>>>>> BTW, does t hat mean that it's possible t o emulate TRAMPOLINE by
>>>>> storing the result o f :pprint (and thus preventing the scenario where
>>>>> there's 2 JVMs loaded, one for lein itself, and one for the project) ?
>>>>>
>>>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>>>
>>>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>>>
>>>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`; this
>>>>>> will
>>>>>> > pprint the form to be run and print the classpath and JVM args
>>>>>> instead
>>>>>> > of actually running it.
>>>>>>
>>>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>>>
>>>>>> -Phil
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Laurent Petit
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Laurent Petit
>>>
>>
>>
>
>
> --
> Laurent Petit
>

Re: [leiningen] Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-17 @ 14:27
2015-08-17 15:38 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:

> Interesting about the user.dir thing, I also investigated that, I'll try
> it and report back. If not I'll probably just hook slurp, which is
> obviously not very general but is probably a good 90% case.
>

It's probably the only option if you want to support concurrency.


> I've been thinking about an external tooling JVM for a long time. I'll
> have to do it sometime, since boot support will require it - boot needs
> Java 1.7 but I have to support 1.6 in Cursive. So that will have to be an
> external JVM. I'm planning to run shims within that JVM to support
> functionality (such as macroexpansion or core.typed checking) that users
> currently have to start a REPL for. They may not want to do so, or may not
> want to modify their running REPL just to see a macroexpansion, so I will
> start a shim inside the JVM and use that.
>

Indeed, and so if something goes wrong, you'll be able to scratch this
external JVM and create a fresh one, without the user noticing and be
forced to restart Cursive.


> However note that this is still not compromise-free. Apart from the memory
> use, in the case of a multi-module process you'll still need to have the
> CWD correct for each lein p roject. A Cursive user tweeted the other day
> about his project containing 66 modules (lein projects), 3068 files and
> 500k LOC. Starting 66 JVMs just to sync the dependencies is never going to
> work.
>

Wow, *that* big. Indeed. So instead of starting 66 JVMs, you intend to
create (and immediately ditch after usage) 66 shims. Will take a look at
shim, since I haven't followed what it brings to the table. I've stayed
with classlojure which AFAIK is mostly abandonware atm.


> While the single-project case is probably by far the most common, I still
> need to support the large project case properly and I can't see a way to do
> that with no compromises at all. I could work around the worst of it with
> some complex configuration saying how many JVMs to start up, and specifying
> which modules should use which JVMs, but it starts to sound like a
> nightmare at that point, both for me and the user.
>



>
> On 17 August 2015 at 13:55, Laurent PETIT <laurent.petit@gmail.com> wrote:
>
>>
>>
>> 2015-08-17 13:40 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>>
>>> Thanks for the thoughts, Laurent - very interesting. Do you have a link
>>> to the CCW source that changes the CWD property? I read about doing that a
>>> while back, but since IntelliJ doesn't isolate classloaders like Eclipse
>>> does I couldn't do it. Now that I'm going to use ShimDandy I can probably
>>> do that. If you isolate classloaders, are the system properties then
>>> isolated too?
>>>
>>
>> I won't be able to do so ... because I was wrong. There's no trace of
>> "user.dir" in CCW codebase. So I may have thought about it in my shower,
>> but never did it in pract ice ...
>>
>>
>>>
>>> In terms of lazily downloading Leiningen, it'll happen as soon as they
>>> open a lein project, so that's generally almost immediately. If they're
>>> offline then they'll have problems, right, but at that point they'll almost
>>> certainly need to be connected to download deps anyway so I'm not sure
>>> it'll be a problem. I think the most important thing is that the user
>>> doesn't have to explicitly download it separately from Cursive,
>>> particularly on Windows where that seems to be pretty painful.
>>>
>>
>> Fair enough
>>
>>
>>> WRT JVM startup time, I'm not worried about that - that is dwarfed by
>>> the time to load th e Clojure/lein namespa ces. But by doing that
>>> in-process I can start a shim and load the namespaces into it ahead of time
>>> so it's ready when I need it and will work immediately. To do that with a
>>> standalone JVM would mean starting up an external JVM eagerly and having it
>>> hanging around - I'm more worried about memory use than time for that.
>>> Obviously that will still occupy memory in the IntelliJ heap but I'm paying
>>> that price now anyway - it will be more due to isolation, but still less
>>> than the 300mb that a JVM takes up.
>>>
>>
>> Well, yes, always keeping ahead of time one classloader is certainly
>> easier and more light-weight for the user's computer memory.
>> But still, interoperating through a new JVM rather than through
>> classloaders appeals more to me:
>> - fixes all the problems you listed, without compromise
>> - introduces the problem of more RAM if a pool of JVM processes is
>> maintained. This could be leveraged by letting t he user choose the pool
>> size (setting it to 0 would be friendly for computers with low memory.
>> Setting it to 5 would be a boost for users with modern computers and multi
>> module projects. Setting it to 1 by default would probably be a good
>> average).
>> - more flexible to adapt to boot, planck or similar for clojurescript, or
>> plain processes.
>>
>> I think I will stick to my idea of moving from embedded to external.
>> Thanks for sharing the thoughts!
>>
>>
>>>
>>>
>>> On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com>
>>> wrote:
>>>
>>>> Mor e thoughts, Colin:
>>>>
>>>> 2015-08-17 12:37 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>>>>
>>>>> Curiously enough, I'm also about to totally change the way I integrate
>>>>> Leiningen in Cursive.
>>>>>
>>>>> To answer Phil's question about embedding leiningen-core, there are
>>>>> many many disadvantages to doing so:
>>>>>
>>>>>    1. You're tied to a specific version of lein. Currently I bundle
>>>>>    2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users 
encountered
>>>>>    bugs so I downgraded to 2.5.0. Some users currently experience bugs with
>>>>>    2.5.0, but they're easier to work around. Anyone using older 
versions on th
>>>>>    eir project is ou t of luck.
>>>>>    2. Lein incorrectly memoises many things internally, which is a
>>>>>    gigantic pain. Currently users have to restart Cursive every time they
>>>>>    change profiles.clj, and additionally they also have to be aware 
of the bug
>>>>>    to know to do that. I've also lost countless hours debugging 
problems like
>>>>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A
>>>>>    more general report is #1768. This is slated to be fixed in lein
>>>>>    3.0, but a) who knows when that will be and b) I'm still at the mercy of
>>>>>    plugins doing the same thing anyway.
>>>>>    3. Java cannot change the CWD of the current process, which means
>>>>>    that local-repo always appears somewhere in the IntelliJ bin directory.
>>>>>    This also fails for users' home grown versioning systems when they do
>>>>>    things like (slurp versions.txt). More examples here
>>>>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>>>>    (there are probably more, but that's the easiest search). 
Essentially, on e
>>>>>    of le in' s big s elling points is that it's declarative, but the
fact that
>>>>>    project.clj is actually executed means that it's not truly declarative.
>>>>>    4. Many things in lein (gpg, proxies etc) are configured using
>>>>>    environment variables, which are a pain to pass when embedding. Cursive
>>>>>    hooks leiningen fairly heavily using robert.hooke t o supply values for
>>>>>    these instead of getting them from env vars.
>>>>>    5. Often you need access to the lein tasks, not just core. Cursive
>>>>>    has bundled all of lein for a long time so that it can access the repl
>>>>>    task. If you don't do this you end up duplicating a lot of the code from
>>>>>    the tasks anyway.
>>>>>    6. There are also lots of minor issues - I wanted to hook into the
>>>>>    Aether transfer listener so that I could provide updates in the 
Cursive UI
>>>>>    when libs are downloaded. Currently I do this in Cursive by hooking
>>>>>    leiningen.core.classpath/warn and checking if it looks like a 
"Retrieving"
>>>>>    message and parsing it if so.
>>>>>    7. As Laurent says, lein, even just core, is really big. The
>>>>>    Cursive download is about 3x the size it would be if it didn't 
bundle lein.
>>>>>
>>>>> That said, while I understand Laurent's worry about stability, that's
>>>>> never been a problem. I've never received a single bug report about s
>>>>> tability that was as a result of bundling lein.
>>>>>
>>>>
>>>> I admit that it has been more a fear than an issue.
>>>>
>>>>
>>>>> There is also one significant benefit to bundling lein - it's fast.
>>>>> Starting a JVM process for lein every time Cursive needs to sync the deps
>>>>> is going to be way too slow. This will be particularly true for large
>>>>> multi-module projects where to do everything totally accurately (in
>>>>> particular, the CWD) requires starting a new JVM per module - this is
>>>>> unworkable. I considered caching the last-modified of the project.clj and
>>>>> only re-parsing a particular module if that had changed, but I'd need to do
>>>>> that for all possible contributing files which is impossible to calculate.
>>>>> Apart from profiles.clj a nd other "offic ial" files which might a ffect
>>>>> the result, I'd also need to test the ver sion.txt type files from #3, and
>>>>> there's no way I can detect all of those.
>>>>>
>>>>> Since I don't think that stability is actually a major problem, I'm
>>>>> planning to continue running lein in-process but I'm going to isolate its
>>>>> classpath using ShimDandy. I have a prototype of this but it's still pretty
>>>>> primitive, and I have a lot of other things going on so it's advancing
>>>>> slowly. I run lein in a shim, and I create the shim's classpath using
>>>>> Aether so I don't actually bundle lein, I pull it in when creating the
>>>>> shim. This means that users can choose the lein version they want, and the
>>>>> files are stored in .m2 rather than being downloaded with Cursive.
>>>>>
>>>>
>>>> I had considered this for CCW, but chose to embed leiningen because
>>>> back in time (and still currently in practice, despite the growing interest
>>>> in boot) leiningen was the only option.
>>>> Having the latest leiningen downloaded at the same time as CCW is
>>>> easier for users, I think. They have to wait once for things to download,
>>>> and then they're good to go.
>>>> By downloading lein lazily, you're at risk for users going offline,
>>>> wiping out their ~/.m2/repo sitory (I do this fr om time to time). That 's
>>>> one more moving part, not to be considered too lightly, IMHO, but YMMV.
>>>>
>>>>
>>>>> I'll then throw the shim away when done so memoisation won't be a
>>>>> problem. I can amortize the performance problem by eagerly creating the
>>>>> shim and loading the namespaces into it before I need it, similar to boot's
>>>>> pod caching. This is much less costly than eagerly starting a JVM and
>>>>> keeping that around until needed.
>>>>>
>>>>
>>>> Have you checked the time required for the JVM itself to start versus
>>>> the time to load leiningen.main and all its dependencies? I think I'll do
>>>> it again, just to check that the ratio proves your point. We might be
>>>> surprised discovering that the JVM startup time is not the problem, leading
>>>> to the conclusion that reloading everything of lein in a new classloader
>>>> will n ot be that different (performance wise) than starting a new JVM.
>>>>
>>>>
>>>>>
>>>>> I'm also going to try to fiddle much less with the lein internals
>>>>> since that has caused problems when upgrading lein. Using this new
>>>>> mechanism I can call the functions representing the lein tasks as if they
>>>>> were being called from main, whic h greatly reduces my dependency on lein
>>>>> internals. I'll still have to hook for gpg/proxies etc, but I'll be hooking
>>>>> aether rather than lein for those, which is much more stable since it's
>>>>> basically abandonware at this point - I should have done this ages ago. I
>>>>> can also hook into the transfer listener by hooking aether.
>>>>>
>>>>> My interface to the shim will be to pass it a list of file paths to
>>>>> the project.clj files and eventually a list of profiles, and it will return
>>>>> a map containing all the information I need about the projects.
>>>>>
>>>>> The upshot of all this is that this should fix all my current problems
>>>>> except #3, which there's no workaround for. I'll fix the local-repo problem
>>>>> by hooking lein or fiddling with the project.clj, but the only way to fix
>>>>> the slurp problem is to start a JVM per lein project, or convince users not
>>>>> to do that. Or hook slurp, I guess.
>>>>>
>>>>> Laurent, currently I run the REPL withi n Cursive using the trampoline
>>>>> mechanism - IMO you don't have to emulate it, you can just use it directly.
>>>>> Get lein to generate a trampoline file then parse that - it's very easy to
>>>>> parse since all the elements always appear in the same order. Currently I
>>>>> hook the lein internals to do this, but as described above I'm about to
>>>>> start calling lein trampoline directly from the shim.
>>>>>
>>>>> Cheers,
>>>>> Colin
>>>>>
>>>>>
>>>>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> OK understood this works as expected: lein update-in : assoc :eval-in
>>>>>> :pprint -- run -m clojure.main
>>>>>> BTW, does t hat mean that it's possible t o emulate TRAMPOLINE by
>>>>>> storing the result o f :pprint (and thus preventing the scenario where
>>>>>> there's 2 JVMs loaded, one for lein itself, and one for the project) ?
>>>>>>
>>>>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>>>>
>>>>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>>>>
>>>>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`;
>>>>>>> this will
>>>>>>> > pprint the form to be run and print the classpath and JVM args
>>>>>>> instead
>>>>>>> > of actually running it.
>>>>>>>
>>>>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>>>>
>>>>>>> -Phil
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Laurent Petit
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Laurent Petit
>>>>
>>>
>>>
>>
>>
>> --
>> Laurent Petit
>>
>
>


-- 
Laurent Petit

Re: [leiningen] Getting leiningen classpath

From:
Colin Fleming
Date:
2015-08-17 @ 14:58
On 17 August 2015 at 16:27, Laurent PETIT <laurent.petit@gmail.com> wrote:

> It's probably the only option if you want to support concurrency.
>

Hopefully I won't actually have to support concurrency - see below.

>
> Wow, *that* big. Indeed.
>

Yeah - that's a very big one but I'm aware of a lot of Cursive users with
projects with a couple of dozen modules.


> So instead of starting 66 JVMs, you intend to create (and immediately
> ditch after usage) 66 shims.
>

Initially no, I'll just create a single shim and use that for all modules,
so it will be very fast. There's not really a big advantage to multiple
shims AFAIK, since memoisation should be ok for a single cross-project
operation. I think the worst offenders are the dependencies and the
profiles, both of which can be memoised for a single call across the whole
project. If I can do the user.dir hack then I can do it for each project
one by one within the same shim as I sync each one.



> While the single-project case is probably by far the most common, I still
>> need to support the large project case properly and I can't see a way to do
>> that with no compromises at all. I could work around the worst of it with
>> some complex configuration saying how many JVMs to start up, and specifying
>> which modules should use which JVMs, but i t starts to sound like a
>> nightmare at that point, both for me and the user.
>>
>
>
>
>>
>> On 17 August 2015 at 13:55, Laurent PETIT <laurent.petit@gmail.com>
>> wrote:
>>
>>>
>>>
>>> 2015-08-17 13:40 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>>>
>>>> Thanks for the thoughts, Laurent - very interesting. Do you have a link
>>>> to the CCW source that changes the CWD property? I read about doing that a
>>>> while back, but since IntelliJ doesn't isolate classloaders like Eclipse
>>>> does I couldn't do it. Now that I'm going to use ShimDandy I can probably
>>>> do that. If you isolate classloaders, are the system properties then
>>>> isolated too?
>>>>
>>>
>>> I won't be able to do so ... because I was wrong. There's no trace of
>>> "user.dir" in CCW codebase. So I may have thought about it in my shower,
>>> but never did it in pract ice ...
>>>
>>>
>>>>
>>>> In terms of lazily downloading Leiningen, it'll happen as soon as they
>>>> open a lein project, so that's generally almost immediately. If they're
>>>> offline then they'll have problems, right, but at that point they'll almost
>>>> certainly need to be connected to download deps anyway so I'm not sure
>>>> it'll be a problem. I think the most important thing is that the user
>>>> doesn't have to explicitly download it separately from Cursive,
>>>> particularly on Windows where that seems to be pretty painful.
>>>>
>>>
>>> Fair enough
>>>
>>>
>>>> WRT JVM startup time, I'm not worried about that - that is dwarfed by
>>>> the ti me to load th e Clojure/lein namespa ces. But by doing that
>>>> in-process I can start a shim and load the namespaces into it ahead of time
>>>> so it's ready when I need it and will work immediately. To do that with a
>>>> standalone JVM would mean starting up an external JVM eagerly and having it
>>>> hanging around - I'm more worried about memory use than time for that.
>>>> Obviously that will still occupy memory in the IntelliJ heap but I'm paying
>>>> that price now anyway - it will be more due to isolation, but still less
>>>> than the 300mb that a JVM takes up.
>>>>
>>>
>>> Well, yes, always keeping ahead of time one classloader is certainly
>>> easier and more light-weight for the user's computer memory.
>>> But still, interoperating through a new JVM rather than through
>>> classloaders appeals more to me:
>>> - fixes all the problems you listed, without compromise
>>> - introduces the problem of more RAM if a pool of JVM processes is
>>> maintained. This could be leveraged by letting t he user choose the pool
>>> size (setting it to 0 would be friendly for computers with low memory.
>>> Setting it to 5 would be a boost for users with modern computers and multi
>>> module projects. Setting it to 1 by default would probably be a good
>>> average).
>>> - more flexible to adapt to boot, planck or similar for clojurescript,
>>> or plain processes.
>>>
>>> I think I will stick to my idea of moving from embedded to external.
>>> Thanks for sharing the thoughts!
>>>
>>>
>>>>
>>>>
>>>> On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com>
>>>> wrote:
>>>>
>>>>> Mor e thoughts, Colin:
>>>>>
>>>>> 2015-08-17 12:37 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>
>>>>> :
>>>>>
>>>>>> Curiously enough, I'm also about to totally change the way I
>>>>>> integrate Leiningen in Cursive.
>>>>>>
>>>>>> To answer Phil's question about embedding leiningen-core, there are
>>>>>> many many disadvantages to doing so:
>>>>>>
>>>>>>    1. You're tied to a specific version of lein. Currently I bundle
>>>>>>    2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users 
encountered
>>>>>>    bugs so I downgraded to 2.5.0. Some users currently experience bugs with
>>>>>>    2.5.0, but they're easier to work around. Anyone using older 
versions on th
>>>>>>    eir project is ou t of luck.
>>>>>>    2. Lein incorrectly memoises many things internally, which is a
>>>>>>    gigantic pain. Currently users have to restart Cursive every time they
>>>>>>    change profiles.clj, and additionally they also have to be aware
of the bug
>>>>>>    to know to do that. I've also lost countless hours debugging 
problems like
>>>>>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A
>>>>>>    more general report is #1768. This is slated to be fixed in lein
>>>>>>    3.0, but a) who knows when that will be and b) I'm still at the mercy of
>>>>>>    plugins doing the same thing anyway.
>>>>>>    3. Java cannot change the CWD of the current process, which means
>>>>>>    that local-repo always appears somewhere in the IntelliJ bin directory.
>>>>>>    This also fails for users' home grown versioning systems when they do
>>>>>>    things like (slurp versions.txt). More examples here
>>>>>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>>>>>    (there are probably more, but that's the easiest search). 
Essentia lly, on
>>>>>>    e of le in' s big s elling points is that it's declarative, but the fact
>>>>>>    that project.clj is actually executed means that it's not truly 
declarative.
>>>>>>    4. Many things in lein (gpg, proxies etc) are configured using
>>>>>>    environment variables, which are a pain to pass when embedding. Cursive
>>>>>>    hooks leiningen fairly heavily using robert.hooke t o supply values for
>>>>>>    these instead of getting them from env vars.
>>>>>>    5. Often you need access to the lein tasks, not just core.
>>>>>>    Cursive has bundled all of lein for a long time so that it can 
access the
>>>>>>    repl task. If you don't do this you end up duplicating a lot of the code
>>>>>>    from the tasks anyway.
>>>>>>    6. There are also lots of minor issues - I wanted to hook into
>>>>>>    the Aether transfer listener so that I could provide updates in 
the Cursive
>>>>>>    UI when libs are downloaded. Currently I do this in Cursive by hooking
>>>>>>    leiningen.core.classpath/warn and checking if it looks like a 
"Retrieving"
>>>>>>    message and parsing it if so.
>>>>>>    7. As Laurent says, lein, even just core, is really big. The
>>>>>>    Cursive download is about 3x the size it would be if it didn't 
bundle lein.
>>>>>>
>>>>>> That said, while I understand Laurent's worry about stability, that's
>>>>>> never been a problem. I've never received a single bug report about s
>>>>>> tability that was as a result of bundling lein.
>>>>>>
>>>>>
>>>>> I admit that it has been more a fear than an issue.
>>>>>
>>>>>
>>>>>> There is also one significant benefit to bundling lein - it's fast.
>>>>>> Starting a JVM process for lein every time Cursive needs to sync the deps
>>>>>> is going to be way too slow. This will be particularly true for large
>>>>>> multi-module projects where to do everything totally accurately (in
>>>>>> particular, the CWD) requires starting a new JVM per module - this is
>>>>>> unworkable. I considered caching the last-modified of the project.clj and
>>>>>> only re-parsing a particular module if that had changed, but I'd need to do
>>>>>> that for all possible contributing files which is impossible to calculate.
>>>>>> Apart from p rofiles.clj a nd other "offic ial" files which might a ffect
>>>>>> the result, I'd also need to test the ver sion.txt type files from #3, and
>>>>>> there's no way I can detect all of those.
>>>>>>
>>>>>> Since I don't think that stability is actually a major problem, I'm
>>>>>> planning to continue running lein in-process but I'm going to isolate its
>>>>>> classpath using ShimDandy. I have a prototype of this but it's still pretty
>>>>>> primitive, and I have a lot of other things going on so it's advancing
>>>>>> slowly. I run lein in a shim, and I create the shim's classpath using
>>>>>> Aether so I don't actually bundle lein, I pull it in when creating the
>>>>>> shim. This means that users can choose the lein version they want, and the
>>>>>> files are stored in .m2 rather than being downloaded with Cursive.
>>>>>>
>>>>>
>>>>> I had considered this for CCW, but chose to embed leiningen because
>>>>> back in time (and still currently in practice, despite the growing interest
>>>>> in boot) leiningen was the only option.
>>>>> Having the latest leiningen downloaded at the same time as CCW is
>>>>> easier for users, I think. They have to wait once for things to download,
>>>>> and then they're good to go.
>>>>> By downloading lein lazily, you're at risk for users going offline,
>>>>> wiping out their ~/.m2/repo sitory (I do this fr om time to time). That 's
>>>>> one more moving part, not to be considered too lightly, IMHO, but YMMV.
>>>>>
>>>>>
>>>>>> I'll then throw the shim away when done so memoisation won't be a
>>>>>> problem. I can amortize the performance problem by eagerly creating the
>>>>>> shim and loading the namespaces into it before I need it, similar to boot's
>>>>>> pod caching. This is much less costly than eagerly starting a JVM and
>>>>>> keeping that around until needed.
>>>>>>
>>>>>
>>>>> Have you checked the time required for the JVM itself to start versus
>>>>> the time to load leiningen.main and all its dependencies? I think I'll do
>>>>> it again, just to check that the ratio proves your point. We might be
>>>>> surprised discovering that the JVM startup time is not the problem, leading
>>>>> to the conclusion that reloading everything of lein in a new classloader
>>>>> will n ot be that different (performance wise) than starting a new JVM.
>>>>>
>>>>>
>>>>>>
>>>>>> I'm also going to try to fiddle much less with the lein internals
>>>>>> since that has caused problems when upgrading lein. Using this new
>>>>>> mechanism I can call the functions representing the lein tasks as if they
>>>>>> were being called from main, whic h greatly reduces my dependency on lein
>>>>>> internals. I'll still have to hook for gpg/proxies etc, but I'll be hooking
>>>>>> aether rather than lein for those, which is much more stable since it's
>>>>>> basically abandonware at this point - I should have done this ages ago. I
>>>>>> can also hook into the transfer listener by hooking aether.
>>>>>>
>>>>>> My interface to the shim will be to pass it a list of file paths to
>>>>>> the project.clj files and eventually a list of profiles, and it will return
>>>>>> a map containing all the information I need about the projects.
>>>>>>
>>>>>> The upshot of all this is that this should fix all my current
>>>>>> problems except #3, which there's no workaround for. I'll fix the
>>>>>> local-repo problem by hooking lein or fiddling with the project.clj, but
>>>>>> the only way to fix the slurp problem is to start a JVM per lein project,
>>>>>> or convince users not to do that. Or hook slurp, I guess.
>>>>>>
>>>>>> Laurent, currently I run the REPL withi n Cursive using the
>>>>>> trampoline mechanism - IMO you don't have to emulate it, you can just use
>>>>>> it directly. Get lein to generate a trampoline file then parse that - it's
>>>>>> very easy to parse since all the elements always appear in the same order.
>>>>>> Currently I hook the lein internals to do this, but as described above I'm
>>>>>> about to start calling lein trampoline directly from the shim.
>>>>>>
>>>>>> Cheers,
>>>>>> Colin
>>>>>>
>>>>>>
>>>>>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> OK understood this works as expected: lein update-in : assoc
>>>>>>> :eval-in :pprint -- run -m clojure.main
>>>>>>> BTW, does t hat mean that it's possible t o emulate TRAMPOLINE by
>>>>>>> storing the result o f :pprint (and thus preventing the scenario where
>>>>>>> there's 2 JVMs loaded, one for lein itself, and one for the project) ?
>>>>>>>
>>>>>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>>>>>
>>>>>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>>>>>
>>>>>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`;
>>>>>>>> this will
>>>>>>>> > pprint the form to be run and print the classpath and JVM args
>>>>>>>> instead
>>>>>>>> > of actually running it.
>>>>>>>>
>>>>>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>>>>>
>>>>>>>> -Phil
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Laurent Petit
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Laurent Petit
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Laurent Petit
>>>
>>
>>
>
>
> --
> Laurent Petit
>

Re: [leiningen] Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-17 @ 21:54
When talking about 66 "modules" - corresponding to 66 Idea projects ? -, do
users generally do so by having 66 sibling folders which are automatically
resolved by Idea as dependent project (using some heuristic like matching
artifactId and project name), or is it by using leiningen checkouts?

2015-08-17 16:58 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:

> On 17 August 2015 at 16:27, Laurent PETIT <laurent.petit@gmail.com> wrote:
>
>> It's probably the only option if you want to support concurrency.
>>
>
> Hopefully I won't actually have to support concurrency - see below.
>
>>
>> Wow, *that* big. Indeed.
>>
>
> Yeah - that's a very big one but I'm aware of a lot of Cursive users with
> projects with a couple of dozen modules.
>
>
>> So instead of starting 66 JVMs, you intend to create (and immediately
>> ditch after usage) 66 shims.
>>
>
> Initially no, I'll just create a single shim and use that for all modules,
> so it will be very fast. There's not really a big advantage to multiple
> shims AFAIK, since memoisation should be ok for a single cross-project
> operation. I think the worst offenders are the dependencies and the
> profiles, both of which can be memoised for a single call across the whole
> project. If I can do the user.dir hack then I can do it for each project
> one by one within the same shim as I sync each one.
>
>
>
>> While the single-project case is probably by far the most common, I still
>>> need to support the large project case properly and I can't see a way to do
>>> that with no compromises at all. I could work around the worst of it with
>>> some complex configuration saying how many JVMs to start up, and specifying
>>> which modules should use which JVMs, but i t starts to sound like a
>>> nightmare at that point, both for me and the user.
>>>
>>
>>
>>
>>>
>>> On 17 August 2015 at 13:55, Laurent PETIT <laurent.petit@gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> 2015-08-17 13:40 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>>>>
>>>>> Thanks for the thoughts, Laurent - very interesting. Do you have a
>>>>> link to the CCW source that changes the CWD property? I read about doing
>>>>> that a while back, but since IntelliJ doesn't isolate classloaders like
>>>>> Eclipse does I couldn't do it. Now that I'm going to use ShimDandy I can
>>>>> probably do that. If you isolate classloaders, are the system properties
>>>>> then isolated too?
>>>>>
>>>>
>>>> I won't be able to do so ... because I was wrong. There's no trace of
>>>> "user.dir" in CCW codebase. So I may have thought about it in my shower,
>>>> but never did it in pract ice ...
>>>>
>>>>
>>>>>
>>>>> In terms of lazily downloading Leiningen, it'll happen as soon as they
>>>>> open a lein project, so that's generally almost immediately. If they're
>>>>> offline then they'll have problems, right, but at that point they'll almost
>>>>> certainly need to be connected to download deps anyway so I'm not sure
>>>>> it'll be a problem. I think the most important thing is that the user
>>>>> doesn't have to explicitly download it separately from Cursive,
>>>>> particularly on Windows where that seems to be pretty painful.
>>>>>
>>>>
>>>> Fair enough
>>>>
>>>>
>>>>> WRT JVM startup time, I'm not worried about that - that is dwarfed by
>>>>> the ti me to load th e Clojure/lein namespa ces. But by doing that
>>>>> in-process I can start a shim and load the namespaces into it ahead of time
>>>>> so it's ready when I need it and will work immediately. To do that with a
>>>>> standalone JVM would mean starting up an external JVM eagerly and having it
>>>>> hanging around - I'm more worried about memory use than time for that.
>>>>> Obviously that will still occupy memory in the IntelliJ heap but I'm paying
>>>>> that price now anyway - it will be more due to isolation, but still less
>>>>> than the 300mb that a JVM takes up.
>>>>>
>>>>
>>>> Well, yes, always keeping ahead of time one classloader is certainly
>>>> easier and more light-weight for the user's computer memory.
>>>> But still, interoperating through a new JVM rather than through
>>>> classloaders appeals more to me:
>>>> - fixes all the problems you listed, without compromise
>>>> - introduces the problem of more RAM if a pool of JVM processes is
>>>> maintained. This cou ld be leveraged by letting t he user choose the pool
>>>> size (setting it to 0 would be friendly for computers with low memory.
>>>> Setting it to 5 would be a boost for users with modern computers and multi
>>>> module projects. Setting it to 1 by default would probably be a good
>>>> average).
>>>> - more flexible to adapt to boot, planck or similar for clojurescript,
>>>> or plain processes.
>>>>
>>>> I think I will stick to my idea of moving from embedded to external.
>>>> Thanks for sharing the thoughts!
>>>>
>>>>
>>>>>
>>>>>
>>>>> On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Mor e thoughts, Colin:
>>>>>>
>>>>>> 2015-08-17 12:37 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com
>>>>>> >:
>>>>>>
>>>>>>> Curiously enough, I'm also about to totally change the way I
>>>>>>> integrate Leiningen in Cursive.
>>>>>>>
>>>>>>> To answer Phil's question about embedding leiningen-core, there are
>>>>>>> many many disadvantages to doing so:
>>>>>>>
>>>>>>>    1. You're tied to a specific version of lein. Currently I bundle
>>>>>>>    2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users
encountered
>>>>>>>    bugs so I downgraded to 2.5.0. Some users currently experience 
bugs with
>>>>>>>    2.5.0, but they're easi er to work around. Anyone using older 
versions on
>>>>>>>    th eir project is ou t of luck.
>>>>>>>    2. Lein incorrectly memoises many things internally, which is a
>>>>>>>    gigantic pain. Currently users have to restart Cursive every time they
>>>>>>>    change profiles.clj, and additionally they also have to be 
aware of the bug
>>>>>>>    to know to do that. I've also lost countless hours debugging 
problems like
>>>>>>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A
>>>>>>>    more general report is #1768. This is slated to be fixed in lein
>>>>>>>    3.0, but a) who knows when that will be and b) I'm still at the
mercy of
>>>>>>>    plugins doing the same thing anyway.
>>>>>>>    3. Java cannot change the CWD of the current process, which
>>>>>>>    means that local-repo always appears somewhere in the IntelliJ bin
>>>>>>>    directory. This also fails for users' home grown versioning 
systems when
>>>>>>>    they do things like (slurp versions.txt). More examples here
>>>>>>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>>>>>>    (there are probably more, but that's the easiest search). E 
ssentia lly, on
>>>>>>>    e of le in' s big s elling points is that it's declarative, but
the fact
>>>>>>>    that project.clj is actually executed means that it's not truly
declarative.
>>>>>>>    4. Many things in lein (gpg, proxies etc) are configured using
>>>>>>>    environment variables, which are a pain to pass when embedding. Cursive
>>>>>>>    hooks leiningen fairly heavily using robert.hooke t o supply values for
>>>>>>>    these instead of getting them from env vars.
>>>>>>>    5. Often you need access to the lein tasks, not just core.
>>>>>>>    Cursive has bundled all of lein for a long time so that it can 
access the
>>>>>>>    repl task. If you don't do this you end up duplicating a lot of
the code
>>>>>>>    from the tasks anyway.
>>>>>>>    6. There are also lots of minor issues - I wanted to hook into
>>>>>>>    the Aether transfer listener so that I could provide updates in
the Cursive
>>>>>>>    UI when libs are downloaded. Currently I do this in Cursive by hooking
>>>>>>>    leiningen.core.classpath/warn and checking if it looks like a 
"Retrieving"
>>>>>>>    message and parsing it if so.
>>>>>>>    7. As Laurent says, lein, even just core, is really big. The
>>>>>>>    Cursive download is about 3x the size it would be if it didn't 
bundle lein.
>>>>>>>
>>>>>>> That said, while I understand Laurent's worry about stability,
>>>>>>> that's never been a problem. I've never received a single bug 
report ab out
>>>>>>> s tability that was as a result of bundling lein.
>>>>>>>
>>>>>>
>>>>>> I admit that it has been more a fear than an issue.
>>>>>>
>>>>>>
>>>>>>> There is also one significant benefit to bundling lein - it's fast.
>>>>>>> Starting a JVM process for lein every time Cursive needs to sync the deps
>>>>>>> is going to be way too slow. This will be particularly true for large
>>>>>>> multi-module projects where to do everything totally accurately (in
>>>>>>> particular, the CWD) requires starting a new JVM per module - this is
>>>>>>> unworkable. I considered caching the last-modified of the project.clj and
>>>>>>> only re-parsing a particular module if that had changed, but I'd 
need to do
>>>>>>> that for all possible contributing files which is impossible to calculate.
>>>>>>> Apart from p rofiles.clj a nd other "offic ial" files which might a ffect
>>>>>>> the result, I'd also need to test the ver sion.txt type files from #3, and
>>>>>>> there's no way I can detect all of those.
>>>>>>>
>>>>>>> Since I don't think that stability is actually a major problem, I'm
>>>>>>> planning to continue running lein in-process but I'm going to isolate its
>>>>>>> classpath using ShimDandy. I have a prototype of this but it's 
still pretty
>>>>>>> primitive, and I have a lot of other things going on so it's advancing
>>>>>>> slowly. I run lein in a shim, and I create the shim's classpath using
>>>>>>> Aether so I don't actually bundle lein, I pull it in when creating the
>>>>>>> shim. This means that users can choose the lein version they want, and the
>>>>>>> files are stored in .m2 rather than being downloaded with Cursive.
>>>>>>>
>>>>>>
>>>>>> I had considered this for CCW, but chose to embed leiningen because
>>>>>> back in time (and still currently in practice, despite the growing interest
>>>>>> in boot) leiningen was the only option.
>>>>>> Having the latest leiningen downloaded at the same time as CCW is
>>>>>> easier for users, I think. They have to wait once for things to download,
>>>>>> and then they're good to go.
>>>>>> By downloading lein lazily, you're at risk for users go ing offline,
>>>>>> wiping out their ~/.m2/repo sitory (I do this fr om time to time). That 's
>>>>>> one more moving part, not to be considered too lightly, IMHO, but YMMV.
>>>>>>
>>>>>>
>>>>>>> I'll then throw the shim away when done so memoisation won't be a
>>>>>>> problem. I can amortize the performance problem by eagerly creating the
>>>>>>> shim and loading the namespaces into it before I need it, similar 
to boot's
>>>>>>> pod caching. This is much less costly than eagerly starting a JVM and
>>>>>>> keeping that around until needed.
>>>>>>>
>>>>>>
>>>>>> Have you checked the time required for the JVM itself to start versus
>>>>>> the time to load leiningen.main and all its dependencies? I think I'll do
>>>>>> it again, just to check that the ratio proves your point. We might be
>>>>>> surprised discovering that the JVM startup time is not the p roblem,
>>>>>> leading to the conclusion that reloading everything of lein in a new
>>>>>> classloader will n ot be that different (performance wise) than starting a
>>>>>> new JVM.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I'm also going to try to fiddle much less with the lein internals
>>>>>>> since that has caused problems when upgrading lein. Using this new
>>>>>>> mechanism I can call the functions representing the lein tasks as if they
>>>>>>> were being called from main, whic h greatly reduces my dependency on lein
>>>>>>> internals. I'll still have to hook for gpg/proxies etc, but I'll 
be hooking
>>>>>>> aether rather than lein for those, which is much more stable since it's
>>>>>>> basically abandonware at this point - I should have done this ages ago. I
>>>>>>> can also hook into the transfer listener by hooking aether.
>>>>>>>
>>>>>>> My interface to the shim will be to pass it a list of file paths to
>>>>>>> the project.clj files and eventually a list of profiles, and it 
will return
>>>>>>> a map containing all the information I need about the projects.
>>>>>>>
>>>>>>> The upshot of all this is that this should fix all my current
>>>>>>> problems except #3, which there's no workaround for. I'll fix the
>>>>>>> local-repo problem by hooking lein or fiddling with the project.clj, but
>>>>>>> the only way to fix the slurp problem is to start a JVM per lein project,
>>>>>>> or convince users not to do that. Or hook slurp, I guess.
>>>>>>>
>>>>>>> Laurent, currently I run the REPL withi n Cursive using the
>>>>>>> trampoline mechanism - IMO you don't have to emulate it, you can just use
>>>>>>> it directly. Get lein to generate a trampoline file then parse that - it's
>>>>>>> very easy to parse since all the elements always appear in the same order.
>>>>>>> Currently I hook the lein internals to do this, but as described above I'm
>>>>>>> about to start calling lein trampoline directly from the shim.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Colin
>>>>>>>
>>>>>>>
>>>>>>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> OK understood this works as expected: lein update-in : assoc
>>>>>>>> :eval-in :pprint -- run -m clojure.main
>>>>>>>> BTW, does t hat mean that it's possible t o emulate TRAMPOLINE by
>>>>>>>> storing the result o f :pprint (and thus preventing the scenario where
>>>>>>>> there's 2 JVMs loaded, one for lein itself, and one for the project) ?
>>>>>>>>
>>>>>>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>>>>>>
>>>>>>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>>>>>>
>>>>>>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`;
>>>>>>>>> this will
>>>>>>>>> > pprint the form to be run and print the classpath and JVM args
>>>>>>>>> instead
>>>>>>>>> > of actually running it.
>>>>>>>>>
>>>>>>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>>>>>>
>>>>>>>>> -Phil
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Laurent Petit
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Laurent Petit
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Laurent Petit
>>>>
>>>
>>>
>>
>>
>> --
>> Laurent Petit
>>
>
>


-- 
Laurent Petit

Re: [leiningen] Getting leiningen classpath

From:
Colin Fleming
Date:
2015-08-18 @ 10:33
So this is complicated. Firstly, there's some terminology differences (from
IntelliJ's Eclipse Migration Guide
<https://www.jetbrains.com/idea/help/terminology.html>): In IntelliJ, a
project corresponds to an Eclipse workspace, and a module corresponds to an
Eclipse project. I'm not very familiar with Eclipse, but my impression is
that that is not quite right - in IntelliJ, I would only have related
things in a project, whereas IIRC in Eclipse it's common to have multiple
unrelated things in a workspace, and you can open and close projects in it
depending on what you're working on. So in IntelliJ, Cursive is a project
containing 3 or 4 modules - is the CCW source all in a single workspace? If
you had another similar project, would that always be in a separate
workspace?

Cursive creates modules for each Leiningen project. These are either
detected on project import, or can be added later. IntelliJ has the concept
of dependencies between modules, so a module that depends on another can
access classes (and namespaces, for Clojure) from those modules. These do
not have to be sibling directories, so a module can contain sub-modules.
Leiningen, for example, has the leiningen project in the root and
leiningen-core and lein-pprint in directories inside the root. When Cursive
syncs the dependencies from the lein projects, I work out which of their
dependencies are actually inter-module dependencies within the same project
and don't add those as libraries - they're added as module dependencies
instead. This means that the up-to-date code from those dependencies is
available to the depending modules without needing to use checkouts. If the
user uses checkouts for compatibility with other tooling Cursive supports
that, it will work out the required module dependencies automatically.

There are additional complications syncing the dependencies for large
projects such as these, since their up-to-date artifacts may not be
installed to .m2. Currently I hack this by modifying the project.clj files
before passing them to lein, removing the dependencies which correspond to
the inter-module deps. However this gives subtle problems such as slightly
different versions for dependencies, presumably because the transitive
closure is modified by doing this so Aether occasionally returns different
dependency versions from the ones that would be returned on the command
line. I'm planning to fix this by instead publishing up-to-date poms for
the project artifacts with empty jars to a temporary repository during the
resolve process, which will mean that I don't have to fiddle with the
project files. Then after the resolve I'll work out which dependencies are
actually inter-module ones and remove them from the dep list before library
creation happens.



On 17 August 2015 at 23:54, Laurent PETIT <laurent.petit@gmail.com> wrote:

> When talking about 66 "modules" - corresponding to 66 Idea projects ? -,
> do users generally do so by having 66 sibling folders which are
> automatically resolved by Idea as dependent project (using some heuristic
> like matching artifactId and project name), or is it by using leiningen
> checkouts?
>
> 2015-08-17 16:58 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>
>> On 17 August 2015 at 16:27, Laurent PETIT <laurent.petit@gmail.com>
>> wrote:
>>
>>> It's probably the only option if you want to support concurrency.
>>>
>>
>> Hopefully I won't actually have to support concurrency - see below.
>>
>>>
>>> Wow, *that* big. Indeed.
>>>
>>
>> Yeah - that's a very big one but I'm aware of a lot of Cursive users with
>> projects with a couple of dozen modules.
>>
>>
>>> So instead of starting 66 JVMs, you intend to create (and immediately
>>> ditch after usage) 66 shims.
>>>
>>
>> Initially no, I'll just create a single shim and use that for all
>> modules, so it will be very fast. There's not really a big advantage to
>> multiple shims AFAIK, since memoisation should be ok for a single
>> cross-project operation. I think the worst offenders are the dependencies
>> and the profiles, both of which can be memoised for a single call across
>> the whole project. If I can do the user.dir hack then I can do it for each
>> project one by one within the same shim as I sync each one.
>>
>>
>>
>>> While the single-project case is probably by far the most common, I
>>>> still need to support the large project case properly and I can't see a way
>>>> to do t hat with no compromises at all. I could work around the worst of it
>>>> with some complex configuration saying how many JVMs to start up, and
>>>> specifying which modules should use which JVMs, but i t starts to sound
>>>> like a nightmare at that point, both for me and the user.
>>>>
>>>
>>>
>>>
>>>>
>>>> On 17 August 2015 at 13:55, Laurent PETIT <laurent.petit@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> 2015-08-17 13:40 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>
>>>>> :
>>>>>
>>>>>> Thanks for the thoughts, Laurent - very interesting. Do you have a
>>>>>> link to the CCW source that changes the CWD property? I read about doing
>>>>>> that a while back, but since IntelliJ doesn't isolate classloaders like
>>>>>> Eclipse does I couldn't do it. Now that I'm going to use ShimDandy I can
>>>>>> probably do that. If you isolate classloaders, are the system properties
>>>>>> then isolated too?
>>>>>>
>>>>>
>>>>> I won't be able to do so ... because I was wrong. There's no trace of
>>>>> "user.dir" in CCW codebase. So I may have thought about it in my shower,
>>>>> but never did it in p ract ice ...
>>>>>
>>>>>
>>>>>>
>>>>>> In terms of lazily downloading Leiningen, it'll happen as soon as
>>>>>> they open a lein project, so that's generally almost immediately. If
>>>>>> they're offline then they'll have problems, right, but at that point
>>>>>> they'll almost certainly need to be connected to download deps anyway so
>>>>>> I'm not sure it'll be a problem. I think the most important thing is that
>>>>>> the user doesn't have to explicitly download it separately from Cursive,
>>>>>> particularly on Windows where that seems to be pretty painful.
>>>>>>
>>>>>
>>>>> Fair enough
>>>>>
>>>>>
>>>>>> WRT JVM startup time, I'm not worried about that - that is dwarfed by
>>>>>> t he ti me to load th e Clojure/lein namespa ces. But by doing that
>>>>>> in-process I can start a shim and load the namespaces into it ahead of time
>>>>>> so it's ready when I need it and will work immediately. To do that with a
>>>>>> standalone JVM would mean starting up an external JVM eagerly and having it
>>>>>> hanging around - I'm more worried about memory use than time for that.
>>>>>> Obviously that will still occupy memory in the IntelliJ heap but I'm paying
>>>>>> that price now anyway - it will be more due to isolation, but still less
>>>>>> than the 300mb that a JVM takes up.
>>>>>>
>>>>>
>>>>> Well, yes, always keeping ahead of time one classloader is certainly
>>>>> easier and more light-weight for the user's computer memory.
>>>>> But still, interoperating through a new JVM rather than through
>>>>> classloaders appeals more to me:
>>>>> - fixes all the problems you listed, without compromise
>>>>> - introduces the problem of more RAM if a pool of JVM processes is
>>>>> maintai ned. This cou ld be leveraged by letting t he user choose the pool
>>>>> size (setting it to 0 would be friendly for computers with low memory.
>>>>> Setting it to 5 would be a boost for users with modern computers and multi
>>>>> module projects. Setting it to 1 by default would probably be a good
>>>>> average).
>>>>> - more flexible to adapt to boot, planck or similar for clojurescript,
>>>>> or plain processes.
>>>>>
>>>>> I think I will stick to my idea of moving from embedded to external.
>>>>> Thanks for sharing the thoughts!
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Mor e thoughts, Colin:
>>>>>>
>>>>>> 2015-08-17 12:37 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com
>>>>>> >:
>>>>>>
>>>>>>> Curiously enough, I'm also about to totally change the way I
>>>>>>> integrate Leiningen in Cursive.
>>>>>>>
>>>>>>> To answer Phil's question about embedding leiningen-core, there are
>>>>>>> many many disadvantages to doing so:
>>>>>>>
>>>>>>>    1. You're tied to a specific version of lein. Currently I bundle
>>>>>>>    2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users
encountered
>>>>>>>    bugs so I downgraded to 2.5.0. Some users currently experience 
bugs with 2
>>>>>>>    .5.0, but they're easi er to work around. Anyone using older 
versions on th
>>>>>>>    eir project is ou t of luck.
>>>>>>>    2. Lein incorrectly memoises many things internally, which is a
>>>>>>>    gigantic pain. Currently users have to restart Cursive every time they
>>>>>>>    change profiles.clj, and additionally they also have to be 
aware of the bug
>>>>>>>    to know to do that. I've also lost countless hours debugging 
problems like
>>>>>>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A
>>>>>>>    more general report is #1768. This is slated to be fixed in lein
>>>>>>>    3.0, but a) who knows when that will be and b) I'm still at the
mercy of
>>>>>>>    plugins doing the same thing anyway.
>>>>>>>    3. Java cannot change the CWD of the current process, which
>>>>>>>    means that local-repo always appears somewhere in the IntelliJ bin
>>>>>>>    directory. This also fails for users' home grown versioning 
systems when
>>>>>>>    they do things like (slurp versions.txt). More examples here
>>>>>>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>>>>>>    (there are probably more, but that's the easiest sea rch). E 
ssentia lly,
>>>>>>>    on e of le in' s big s elling points is that it's declarative, 
but the fact
>>>>>>>    that project.clj is actually executed means that it's not truly
declarative.
>>>>>>>    4. Many things in lein (gpg, proxies etc) are configured using
>>>>>>>    environment variables, which are a pain to pass when embedding. Cursive
>>>>>>>    hooks leiningen fairly heavily using robert.hooke t o supply values for
>>>>>>>    these instead of getting them from env vars.
>>>>>>>    5. Often you need access to the lein tasks, not just core.
>>>>>>>    Cursive has bundled all of lein for a long time so that it can 
access the
>>>>>>>    repl task. If you don't do this you end up duplicating a lot of
the code
>>>>>>>    from the tasks anyway.
>>>>>>>    6. There are also lots of minor issues - I wanted to hook into
>>>>>>>    the Aether transfer listener so that I could provide updates in
the Cursive
>>>>>>>    UI when libs are downloaded. Currently I do this in Cursive by hooking
>>>>>>>    leiningen.core.classpath/warn and checking if it looks like a 
"Retrieving"
>>>>>>>    message and parsing it if so.
>>>>>>>    7. As Laurent says, lein, even just core, is really big. The
>>>>>>>    Cursive download is about 3x the size it would be if it didn't 
bundle lein.
>>>>>>>
>>>>>>> That said, while I understand Laurent's worry about stability,
>>>>>>> that's never been a problem. I've never received a single bug re port ab
>>>>>>> out s tability that was as a result of bundling lein.
>>>>>>>
>>>>>>
>>>>>> I admit that it has been more a fear than an issue.
>>>>>>
>>>>>>
>>>>>>> There is also one significant benefit to bundling lein - it's fast.
>>>>>>> Starting a JVM process for lein every time Cursive needs to sync the deps
>>>>>>> is going to be way too slow. This will be particularly true for large
>>>>>>> multi-module projects where to do everything totally accurately (in
>>>>>>> particular, the CWD) requires starting a new JVM per module - this is
>>>>>>> unworkable. I considered caching the last-modified of the project.clj and
>>>>>>> only re-parsing a particular module if that had changed, but I'd 
need to do
>>>>>>> that for all possible contributing files which is impossible to calculate.
>>>>>>> Apart from p rofiles.clj a nd other "offic ial" files which might a ffect
>>>>>>> the result, I'd also need to test the ver sion.txt type files from #3, and
>>>>>>> there's no way I can detect all of those.
>>>>>>>
>>>>>>> Since I don't think that stability is actually a major problem, I'm
>>>>>>> planning to continue running lein in-process but I'm going to isolate its
>>>>>>> classpath using ShimDandy. I have a prototype of this but it's 
still pretty
>>>>>>> primitive, and I have a lot of other things going on so it's advancing
>>>>>>> slowly. I run lein in a shim, and I create the shim's classpath using
>>>>>>> Aether so I don't actually bundle lein, I pull it in when creating the
>>>>>>> shim. This means that users can choose the lein version they want, and the
>>>>>>> files are stored in .m2 rather than being downloaded with Cursive.
>>>>>>>
>>>>>>
>>>>>> I had considered this for CCW, but chose to embed leiningen because
>>>>>> back in time (and still currently in practice, despite the growing interest
>>>>>> in boot) leiningen was the only option.
>>>>>> Having the latest leiningen downloaded at the same time as CCW is
>>>>>> easier for users, I think. They have to wait once for things to download,
>>>>>> and then they're good to go.
>>>>>> By downloading lein lazily, you're at risk for users go ing offline,
>>>>>> wiping out their ~/.m2/repo sitory (I do this fr om time to time). That 's
>>>>>> one more moving part, not to be considered too lightly, IMHO, but YMMV.
>>>>>>
>>>>>>
>>>>>>> I'll then throw the shim away when done so memoisation won't be a
>>>>>>> problem. I can amortize the performance problem by eagerly creating the
>>>>>>> shim and loading the namespaces into it before I need it, similar 
to boot's
>>>>>>> pod caching. This is much less costly than eagerly starting a JVM and
>>>>>>> keeping that around until needed.
>>>>>>>
>>>>>>
>>>>>> Have you checked the time required for the JVM itself to start versus
>>>>>> the time to load leiningen.main and all its dependencies? I think I'll do
>>>>>> it again, just to check that the ratio proves your point. We might be
>>>>>> surprised discovering that the JVM startup time is not t he p roblem,
>>>>>> leading to the conclusion that reloading everything of lein in a new
>>>>>> classloader will n ot be that different (performance wise) than starting a
>>>>>> new JVM.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I'm also going to try to fiddle much less with the lein internals
>>>>>>> since that has caused problems when upgrading lein. Using this new
>>>>>>> mechanism I can call the functions representing the lein tasks as if they
>>>>>>> were being called from main, whic h greatly reduces my dependency on lein
>>>>>>> internals. I'll still have to hook for gpg/proxies etc, but I'll 
be hooking
>>>>>>> aether rather than lein for those, which is much more stable since it's
>>>>>>> basically abandonware at this point - I should have done this ages ago. I
>>>>>>> can also hook into the transfer listener by hooking aether.
>>>>>>>
>>>>>>> My interface to the shim will be to pass it a list of file paths to
>>>>>>> the project.clj files and eventually a list of profiles, and it 
will return
>>>>>>> a map containing all the information I need about the projects.
>>>>>>>
>>>>>>> The upshot of all this is that this should fix all my current
>>>>>>> problems except #3, which there's no workaround for. I'll fix the
>>>>>>> local-repo problem by hooking lein or fiddling with the project.clj, but
>>>>>>> the only way to fix the slurp problem is to start a JVM per lein project,
>>>>>>> or convince users not to do that. Or hook slurp, I guess.
>>>>>>>
>>>>>>> Laurent, currently I run the REPL withi n Cursive using the
>>>>>>> trampoline mechanism - IMO you don't have to emulate it, you can just use
>>>>>>> it directly. Get lein to generate a trampoline file then parse that - it's
>>>>>>> very easy to parse since all the elements always appear in the same order.
>>>>>>> Currently I hook the lein internals to do this, but as described above I'm
>>>>>>> about to start calling lein trampoline directly from the shim.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Colin
>>>>>>>
>>>>>>>
>>>>>>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> OK understood this works as expected: lein update-in : assoc
>>>>>>>> :eval-in :pprint -- run -m clojure.main
>>>>>>>> BTW, does t hat mean that it's possible t o emulate TRAMPOLINE by
>>>>>>>> storing the result o f :pprint (and thus preventing the scenario where
>>>>>>>> there's 2 JVMs loaded, one for lein itself, and one for the project) ?
>>>>>>>>
>>>>>>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>>>>>>
>>>>>>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>>>>>>
>>>>>>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`;
>>>>>>>>> this will
>>>>>>>>> > pprint the form to be run and print the classpath and JVM args
>>>>>>>>> instead
>>>>>>>>> > of actually running it.
>>>>>>>>>
>>>>>>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>>>>>>
>>>>>>>>> -Phil
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Laurent Petit
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Laurent Petit
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Laurent Petit
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Laurent Petit
>>>
>>
>>
>
>
> --
> Laurent Petit
>

Re: Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-18 @ 20:25
Saying that a module corresponds to an Eclipse project seems right.
To draw another analogy: a leiningen project corresponds to an Eclipse
project.
But in a workspace, you're free to have multiple related or unrelated
projects. Eclipse imposes nothing there.

For a long time, it had been impossible to have an eclipse project's folder
as a child of another eclipse project's folder. Since Eclipse 4, it seems
it is now possible, I'll have to explore this some time or another.

The CCW source is all in a single workspace, but I have also the clojure
source as a maven project, pomegranate source as a leininigen project, etc.

It is possible for a project to "export" part of its classpath, and then
another project can depend on this exported classpath. So while I haven't
done it yet, it's totally within reach to have multiproject / multimodule
in Eclipse, it's not a second-class citizen feature.

When you say you "work out which of their dependencies are actually
inter-module dependencies", how do you that? That's a question I've asked
myself, because indeed it's never guaranteed that you've got the right
source code checked out corresponding to the declared dependency version?

But when you delegate REPL creation to leiningen, how do you deal with
intermodule dependencies, then? I understand that you can remove
dependencies from the list, but I'm curious to know how you add modules
classpath back...

Le mardi 18 août 2015, Colin Fleming <colin.mailinglist@gmail.com> a écrit :

> So this is complicated. Firstly, there's some terminology differences
> (from IntelliJ's Eclipse Migration Guide
> <https://www.jetbrains.com/idea/help/terminology.html>): In IntelliJ, a
> project corresponds to an Eclipse workspace, and a module corresponds to an
> Eclipse project. I'm not very familiar with Eclipse, but my impression is
> that that is not quite right - in IntelliJ, I would only have related
> things in a project, whereas IIRC in Eclipse it's common to have multiple
> unrelated things in a workspace, and you can open and close projects in it
> depending on what you're working on. So in IntelliJ, Cursive is a project
> containing 3 or 4 modules - is the CCW source all in a single workspace? If
> you had another similar project, would that always be in a separate
> workspace?
>
> Cursive creates modules for each Leiningen project. These are either
> detected on project import, or can be added later. IntelliJ has the concept
> of depe ndencies between modules, so a module that depends on another can
> access classes (and namespaces, for Clojure) from those modules. These do
> not have to be sibling directories, so a module can contain sub-modules.
> Leiningen, for example, has the leiningen project in the root and
> leiningen-core and lein-pprint in directories inside the root. When Cursive
> syncs the dependencies from the lein projects, I work out which of their
> dependencies are actually inter-module dependencies within the same project
> and don't add those as libraries - they're added as module dependencies
> instead. This means that the up-to-date code from those dependencies is
> available to the depending modules without needing to use checkouts. If the
> user uses checkouts for compatibility with other tooling Cursive supports
> that, it will work out the required module dependencies automatically.
>
> There are additional complications syncing the dependencies for large
> projects such as these, since their up-to-date artifacts may not be
> installed to .m2. Currently I hack this by modifying the project.clj files
> before passing them to lein, removing the dependencies which correspond to
> the inter-module deps. However this gives subtle problems such as slightly
> different versions for dependencies, presumably because the transitive
> closure is modified by doing this so Aether occasionally returns different
> dependency versions from the ones that would be returned on the command
> line. I'm planning to fix this by instead publishing up-to-date poms for
> the project artifacts with empty jars to a temporary repository during the
> resolve process, which will mean that I don't have to fiddle with the
> project files. Then after the resolve I'll work out which dependencies are
> actually inter-module ones and remove them from the dep list before library
> creation happens.
>
>
>


-- 
Laurent Petit

Re: [leiningen] Re: Getting leiningen classpath

From:
Colin Fleming
Date:
2015-08-19 @ 09:51
I work out the inter-module dependencies using the artifact ID. I know all
the lein modules in the project, so if another project.clj in the project
has a dependency on an artifact declared by one of those other modules,
then that's an inter-module dependency. So the leiningen project has three
modules: leiningen 2.5.3-SNAPSHOT, leiningen-core 2.5.3-SNAPSHOT and
lein-pprint
1.1.2. Since leiningen has a [leiningen-core "2.5.3-SNAPSHOT"] dependency,
I know that that is something I have to add as a module dependency, and not
as a library. So in general, I never create libraries for modules that
exist in the project.

When I delegate creation to the REPL, I just call lein and don't do any
fiddling, but I also have an option to add source paths from dependent
modules to the classpath so those will be picked up ahead of the libraries
if required. Obviously this will only work for Clojure, though. The
semantics around that were very tricky, I'd need to go back and read the
issue to remind myself.

On 18 August 2015 at 22:25, Laurent PETIT <laurent.petit@gmail.com> wrote:

> Saying that a module corresponds to an Eclipse project seems right.
> To draw another analogy: a leiningen project corresponds to an Eclipse
> project.
> But in a workspace, you're free to have multiple related or unrelated
> projects. Eclipse imposes nothing there.
>
> For a long time, it had been impossible to have an eclipse project's
> folder as a child of another eclipse project's folder. Since Eclipse 4, it
> seems it is now possible, I'll have to explore this some time or another.
>
> The CCW source is all in a single workspace, but I have also the clojure
> source as a maven project, pomegranate source as a leininigen project, etc.
>
> It is possible for a project to "export" part of its classpath, and then
> another project can depend on this exported classpath. So while I haven't
> done it yet, it's totally within reach to have multiproject / multimodule
> in Eclipse, it's not a second-class citizen feature.
>
> When you say you "work out which of their dependencies are actually inte
> r-module dependencies", how do you that? That's a question I've asked
> myself, because indeed it's never guaranteed that you've got the right
> source code checked out corresponding to the declared dependency version?
>
> But when you delegate REPL creation to leiningen, how do you deal with
> intermodule dependencies, then? I understand that you can remove
> dependencies from the list, but I'm curious to know how you add modules
> classpath back...
>
> Le mardi 18 août 2015, Colin Fleming <colin.mailinglist@gmail.com> a
> écrit :
>
>> So this is complicated . Firstly, there's some terminology differences
>> (from IntelliJ's Eclipse Migration Guide
>> <https://www.jetbrains.com/idea/help/terminology.html>): In IntelliJ, a
>> project corresponds to an Eclipse workspace, and a module corresponds to an
>> Eclipse project. I'm not very familiar with Eclipse, but my impression is
>> that that is not quite right - in IntelliJ, I would only have related
>> things in a project, whereas IIRC in Eclipse it's common to have multiple
>> unrelated things in a workspace, and you can open and close projects in it
>> depending on what you're working on. So in IntelliJ, Cursive is a project
>> containing 3 or 4 modules - is the CCW source all in a single workspace? If
>> you had another similar project, would that always be in a separate
>> workspace?
>>
>> Cursive creates modules for each Leiningen project. These are either
>> detected on project import, or can be added later. IntelliJ has the concept
>> of depe ndencies between modules, so a module that depends on another can
>> access classes (and namespaces, for Clojure) from those modules. These do
>> not have to be sibling directories, so a module can contain sub-modules.
>> Leiningen, for example, has the leiningen project in the root and
>> leiningen-core and lein-pprint in directories inside the root. When Cursive
>> syncs the dependencies from the lein projects, I work out which of their
>> dependencies are actually inter-module dependencies within the same project
>> and don't add those as libraries - they're added as module dependencies
>> instead. This means that the up-to-date code from those dependencies is
>> available to the depending modules without needing to use checkouts. If the
>> user uses checkouts for compatibility with other tooling Cursive supports
>> that, it will work out the required module dependencies automatically.
>>
>> There are additional complications syncing the dependencies for large
>> projects such as these, since their up-to-date artifacts may not be
>> installed to .m2. Currently I hack this by modifying the project.clj files
>> before passing them to lein, removing the dependencies which correspond to
>> the inter-module deps. However this gives subtle problems such as slightly
>> different versions for dependencies, presumably because the transitive
>> closure is modified by doing this so Aether occasionally returns different
>> dependency versions from the ones that would be returned on the command
>> line. I'm planning to fix this by instead publishing up-to-date poms for
>> the project artifacts with empty jars to a temporary repository during the
>> resolve process, which will mean that I don't have to fiddle with the
>> project files. Then after the resolve I'll work out which dependencies are
>> actually inter-module ones and remove them from the dep list before library
>> creation happens.
>>
>>
>>
> < br>
> --
> Laurent Petit
>
>

Re: [leiningen] Re: Getting leiningen classpath

From:
Colin Fleming
Date:
2015-09-14 @ 22:29
So I've been trying to modify the CWD dynamically via clojure.java.io as
suggested by Laurent. It's tricky, to say the least. Extending
make-input-stream and make-output-stream as Laurent suggested seems to work
without any major problems, at least during some fairly minimal testing,
which is great.

However I'd also like io/file to work, since people want to do things like
this: https://github.com/cursiveclojure/cursive/issues/573 and it's also
used for local-repo. Initially I tried to extend io/Coercions to String, so
that as-file for a string would return an absolute file when given a
relative file, but that's not always what you want. In particular,
io/as-relative-path will then throw exceptions since it calls as-file on
its argument and throws if it's not relative. This is used when
constructing files from a base like (io/file base path).

I then tried just hooke'ing io/file, and for the single-arg form I call the
original then absolutise it if it returns a relative file. That works, but
then breaks leiningen.core.project/absolutize which checks if the file
returned by io/file is absolute or not. When it is returns the original
path, not the actual file returned from io/file which seems slightly dodgy,
at least in part because that means that one arm of the if returns a string
path and the other returns a file.

That's where I am now. I can hook absolutize to fix its behaviour but this
is starting to feel like a house of cards. Any suggestions for improvements
very welcome.

On 19 August 2015 at 21:51, Colin Fleming <colin.mailinglist@gmail.com>
wrote:

> I work out the inter-module dependencies using the artifact ID. I know all
> the lein modules in the project, so if another project.clj in the project
> has a dependency on an artifact declared by one of those other modules,
> then that's an inter-module dependency. So the leiningen project has three
> modules: leiningen 2.5.3-SNAPSHOT, leiningen-core 2.5.3-SNAPSHOT and lein-pprint
> 1.1.2. Since leiningen has a [leiningen-core "2.5.3-SNAPSHOT"]
> dependency, I know that that is something I have to add as a module
> dependency, and not as a library. So in general, I never create libraries
> for modules that exist in the project.
>
> When I delegate creation to the REPL, I just call lein and don't do any
> fiddling, but I also have an option to add source paths from dependent
> modules to the classpath so those will be picked up ahead of the libraries
> if required. Obviously this will only work for Clojure, though. The
> semantics around that were very tricky, I'd need to go back and read the
> issue to remind myself.
>
> On 18 August 2015 at 22:25, Laurent PETIT <laurent.petit@gmail.com> wrote:
>
>> Saying that a module corresponds to an Eclipse project seems right.
>> To draw another analogy: a leiningen project corresponds to an Eclipse
>> project.
>> But in a workspace, you're free to have multiple related or unrelated
>> projects. Eclipse imposes nothing there.
>>
>> For a long time, it had been impossible to have an eclipse project's
>> folder as a child of another eclipse project's folder. Since Eclipse 4, it
>> seems it is now possible, I'll have to explore this some time or another.
>>
>> The CCW source is all in a single workspace, but I have also the clojure
>> source as a maven project, pomegranate source as a leininigen project, etc.
>>
>> It is possible for a project to "export" part of its classpath, and then
>> another project can depend on this exported classpath. So while I haven't
>> done it yet, it's totally within reach to have multiproject / multimodule
>> in Eclipse, it's not a second-class citizen feature.
>>
>> When you say you "work out which of their dependencies are actually inte
>> r-module dependencies", how do you that? That's a question I've asked
>> myself, because indeed it's never guaranteed that you've got the right
>> source code checked out corresponding to the declared dependency version?
>>
>> But when you delegate REPL creation to leiningen, how do you deal with
>> intermodule dependencies, then? I understand that you can remove
>> dependencies from the list, but I'm curious to know how you add modules
>> classpath back...
>>
>> Le mardi 18 août 2015, Colin Fleming <colin.mailinglist@gmail.com> a
>> écrit :
>>
>>> So this is complicated . Firstly, there's some terminology differences
>>> (from IntelliJ's Eclipse Migration Guide
>>> <https://www.jetbrains.com/idea/help/terminology.html>): In IntelliJ, a
>>> project corresponds to an Eclipse workspace, and a module corresponds to an
>>> Eclipse project. I'm not very familiar with Eclipse, but my impression is
>>> that that is not quite right - in IntelliJ, I would only have related
>>> things in a project, whereas IIRC in Eclipse it's common to have multiple
>>> unrelated things in a workspace, and you can open and close projects in it
>>> depending on what you're working on. So in IntelliJ, Cursive is a project
>>> containing 3 or 4 modules - is the CCW source all in a single workspace? If
>>> you had another similar project, would that always be in a separate
>>> workspace?
>>>
>>> Cursive creates modules for each Leiningen project. These are either
>>> detected on project import, or can be added later. IntelliJ has the concept
>>> of depe ndencies between modules, so a module that depends on another can
>>> access classes (and namespaces, for Clojure) from those modules. These do
>>> not have to be sibling directories, so a module can contain sub-modules.
>>> Leiningen, for example, has the leiningen project in the root and
>>> leiningen-core and lein-pprint in directories inside the root. When Cursive
>>> syncs the dependencies from the lein projects, I work out which of their
>>> dependencies are actually inter-module dependencies within the same project
>>> and don't add those as libraries - they're added as module dependencies
>>> instead. This means that the up-to-date code from those dependencies is
>>> available to the depending modules without needing to use checkouts. If the
>>> user uses checkouts for compatibility with other tooling Cursive supports
>>> that, it will work out the required module dependencies automatically.
>>>
>>> There are additional complications syncing the dependencies for large
>>> projects such as these, since their up-to-date artifacts may not be
>>> installed to .m2. Currently I hack this by modifying the project.clj files
>>> before passing them to lein, removing the dependencies which correspond to
>>> the inter-module deps. However this gives subtle problems such as slightly
>>> different versions for dependencies, presumably because the transitive
>>> closure is modified by doing this so Aether occasionally returns different
>>> dependency versions from the ones that would be returned on the command
>>> line. I'm planning to fix this by instead publishing up-to-date poms for
>>> the project artifacts with empty jars to a temporary repository during the
>>> resolve process, which will mean that I don't have to fiddle with the
>>> project files. Then after the resolve I'll work out which dependencies are
>>> actually inter-module ones and remove them from the dep list before library
>>> creation happens.
>>>
>>>
>>>
>> < br>
>> --
>> Laurent Petit
>>
>>
>

Re: [leiningen] Re: Getting leiningen classpath

From:
Colin Fleming
Date:
2015-09-14 @ 22:44
Actually, looking at the code again, absolutize calls str on the result of
the if, so it'll return the path in either case.

On 15 September 2015 at 10:29, Colin Fleming <colin.mailinglist@gmail.com>
wrote:

> So I've been trying to modify the CWD dynamically via clojure.java.io as
> suggested by Laurent. It's tricky, to say the least. Extending
> make-input-stream and make-output-stream as Laurent suggested seems to work
> without any major problems, at least during some fairly minimal testing,
> which is great.
>
> However I'd also like io/file to work, since people want to do things like
> this: https://github.com/cursiveclojure/cursive/issues/573 and it's also
> used for local-repo. Initially I tried to extend io/Coercions to String, so
> that as-file for a string would return an absolute file when given a
> relative file, but that's not always what you want. In particular,
> io/as-relative-path will then throw exceptions since it calls as-file on
> its argument and throws if it's not relative. This is used when
> constructing files from a base like (io/file base path).
>
> I then tried just hooke'ing io/file, and for the single-arg form I call
> the original then absolutise it if it returns a relative file. That works,
> but then breaks leiningen.core.project/absolutize which checks if the file
> returned by io/file is absolute or not. When it is returns the original
> path, not the actual file returned from io/file which seems slightly dodgy,
> at least in part because that means that one arm of the if returns a string
> path and the other returns a file.
>
> That's where I am now. I can hook absolutize to fix its behaviour but this
> is starting to feel like a house of cards. Any suggestions for improvements
> very welcome.
>
> On 19 August 2015 at 21:51, Colin Fleming <colin.mailinglist@gmail.com>
> wrote:
>
>> I work out the inter-module dependencies using the artifact ID. I know
>> all the lein modules in the project, so if another project.clj in the
>> project has a dependency on an artifact declared by one of those other
>> modules, then that's an inter-module dependency. So the leiningen project
>> has three modules: leiningen 2.5.3-SNAPSHOT, leiningen-core
>> 2.5.3-SNAPSHOT and lein-pprint 1.1.2. Since leiningen has a [leiningen-core
>> "2.5.3-SNAPSHOT"] dependency, I know that that is something I have to
>> add as a module dependency, and not as a library. So in general, I never
>> create libraries for modules that exist in the project.
>>
>> When I delegate creation to the REPL, I just call lein and don't do any
>> fiddling, but I also have an option to add source paths from dependent
>> modules to the classpath so those will be picked up ahead of the libraries
>> if required. Obviously this will only work for Clojure, though. The
>> semantics around that were very tricky, I'd need to go back and read the
>> issue to remind myself.
>>
>> On 18 August 2015 at 22:25, Laurent PETIT <laurent.petit@gmail.com>
>> wrote:
>>
>>> Saying that a module corresponds to an Eclipse project seems right.
>>> To draw another analogy: a leiningen project corresponds to an Eclipse
>>> project.
>>> But in a workspace, you're free to have multiple related or unrelated
>>> projects. Eclipse imposes nothing there.
>>>
>>> For a long time, it had been impossible to have an eclipse project's
>>> folder as a child of another eclipse project's folder. Since Eclipse 4, it
>>> seems it is now possible, I'll have to explore this some time or another.
>>>
>>> The CCW source is all in a single workspace, but I have also the clojure
>>> source as a maven project, pomegranate source as a leininigen project, etc.
>>>
>>> It is possible for a project to "export" part of its classpath, and then
>>> another project can depend on this exported classpath. So while I haven't
>>> done it yet, it's totally within reach to have multiproject / multimodule
>>> in Eclipse, it's not a second-class citizen feature.
>>>
>>> When you say you "work out which of their dependencies are actually inte
>>> r-module dependencies", how do you that? That's a question I've asked
>>> myself, because indeed it's never guaranteed that you've got the right
>>> source code checked out corresponding to the declared dependency version?
>>>
>>> But when you delegate REPL creation to leiningen, how do you deal with
>>> intermodule dependencies, then? I understand that you can remove
>>> dependencies from the list, but I'm curious to know how you add modules
>>> classpath back...
>>>
>>> Le mardi 18 août 2015, Colin Fleming <colin.mailinglist@gmail.com> a
>>> écrit :
>>>
>>>> So this is complicated . Firstly, there's some terminology differences
>>>> (from IntelliJ's Eclipse Migration Guide
>>>> <https://www.jetbrains.com/idea/help/terminology.html>): In IntelliJ,
>>>> a project corresponds to an Eclipse workspace, and a module corresponds to
>>>> an Eclipse project. I'm not very familiar with Eclipse, but my impression
>>>> is that that is not quite right - in IntelliJ, I would only have related
>>>> things in a project, whereas IIRC in Eclipse it's common to have multiple
>>>> unrelated things in a workspace, and you can open and close projects in it
>>>> depending on what you're working on. So in IntelliJ, Cursive is a project
>>>> containing 3 or 4 modules - is the CCW source all in a single workspace? If
>>>> you had another similar project, would that always be in a separate
>>>> workspace?
>>>>
>>>> Cursive creates modules for each Leiningen project. These are either
>>>> detected on project import, or can be added later. IntelliJ has the concept
>>>> of depe ndencies between modules, so a module that depends on another can
>>>> access classes (and namespaces, for Clojure) from those modules. These do
>>>> not have to be sibling directories, so a module can contain sub-modules.
>>>> Leiningen, for example, has the leiningen project in the root and
>>>> leiningen-core and lein-pprint in directories inside the root. When Cursive
>>>> syncs the dependencies from the lein projects, I work out which of their
>>>> dependencies are actually inter-module dependencies within the same project
>>>> and don't add those as libraries - they're added as module dependencies
>>>> instead. This means that the up-to-date code from those dependencies is
>>>> available to the depending modules without needing to use checkouts. If the
>>>> user uses checkouts for compatibility with other tooling Cursive supports
>>>> that, it will work out the required module dependencies automatically.
>>>>
>>>> There are additional complications syncing the dependencies for large
>>>> projects such as these, since their up-to-date artifacts may not be
>>>> installed to .m2. Currently I hack this by modifying the project.clj files
>>>> before passing them to lein, removing the dependencies which correspond to
>>>> the inter-module deps. However this gives subtle problems such as slightly
>>>> different versions for dependencies, presumably because the transitive
>>>> closure is modified by doing this so Aether occasionally returns different
>>>> dependency versions from the ones that would be returned on the command
>>>> line. I'm planning to fix this by instead publishing up-to-date poms for
>>>> the project artifacts with empty jars to a temporary repository during the
>>>> resolve process, which will mean that I don't have to fiddle with the
>>>> project files. Then after the resolve I'll work out which dependencies are
>>>> actually inter-module ones and remove them from the dep list before library
>>>> creation happens.
>>>>
>>>>
>>>>
>>> < br>
>>> --
>>> Laurent Petit
>>>
>>>
>>
>

Re: [leiningen] Re: Getting leiningen classpath

From:
Colin Fleming
Date:
2015-09-15 @ 04:58
Here's what I ended up with. I manually absolutise local-repo, and hook to
modify the protocols and hook io/file only during project/read-raw to
minimise interference with other things. The pedantic stuff is just to show
the user the pedantic messages instead of aborting or throwing an exception.

(defn read-raw-hook [original file]
  (let [cwd (project-dir file)
        project (let [prev-factory (get (:impls io/IOFactory) File)
                      {:keys [make-input-stream make-output-stream]}
prev-factory
                      absolute #(if (.isAbsolute ^File %) % (io/file cwd
%))]
                  (try
                    (extend File
                      io/IOFactory
                      (merge prev-factory
                             {:make-input-stream  #(make-input-stream
(absolute %1) %2)
                              :make-output-stream #(make-output-stream
(absolute %1) %2)}))
                    (hooke/add-hook #'io/file :file-hook (fn
                                                           ([original arg]
                                                            (let [result
(original arg)]
                                                              (if
(instance? File result)
                                                                (absolute
result)
                                                                result)))
                                                           ([original
parent child]
                                                            (original
parent child))
                                                           ([original
parent child & more]
                                                            (apply original
parent child more))))
                    (original file)
                    (finally
                      (hooke/remove-hook #'io/file :file-hook)
                      (extend File io/IOFactory prev-factory))))]
    (if (and (:local-repo project)
             (not (.isAbsolute (io/file (:local-repo project)))))
      (assoc project :local-repo (-> (io/file cwd (:local-repo project))
                                     (.getCanonicalPath)))
      project)))

(defn read-project [file]
  (let [writer (StringWriter.)]
    (try
      (hooke/add-hook #'classpath/pedantic-do :pedantic-hook (pedantic-hook
writer))
      (hooke/add-hook #'project/read-raw :read-raw-hook read-raw-hook)
      (let [project (project/read file)
            pedantic-message (str writer)]
        (if-not (str/blank? pedantic-message)
          (let [{:keys [group name version]} project]
            (with-warning project
                          (str "Pedantic warning for " (artifact-name group
name version))
                          (str "<pre>" (str/trim pedantic-message)
"</pre>")))
          project))
      (finally
        (hooke/remove-hook #'project/read-raw :read-raw-hook)
        (hooke/remove-hook #'classpath/pedantic-do :pedantic-hook)))))


On 15 September 2015 at 10:44, Colin Fleming <colin.mailinglist@gmail.com>
wrote:

> Actually, looking at the code again, absolutize calls str on the result of
> the if, so it'll return the path in either case.
>
> On 15 September 2015 at 10:29, Colin Fleming <colin.mailinglist@gmail.com>
> wrote:
>
>> So I've been trying to modify the CWD dynamically via clojure.java.io as
>> suggested by Laurent. It's tricky, to say the least. Extending
>> make-input-stream and make-output-stream as Laurent suggested seems to work
>> without any major problems, at least during some fairly minimal testing,
>> which is great.
>>
>> However I'd also like io/file to work, since people want to do things
>> like this: https://github.com/cursiveclojure/cursive/issues/573 and it's
>> also used for local-repo. Initially I tried to extend io/Coercions to
>> String, so that as-file for a string would return an absolute file when
>> given a relative file, but that's not always what you want. In particular,
>> io/as-relative-path will then throw exceptions since it calls as-file on
>> its argument and throws if it's not relative. This is used when
>> constructing files from a base like (io/file base path).
>>
>> I then tried just hooke'ing io/file, and for the single-arg form I call
>> the original then absolutise it if it returns a relative file. That works,
>> but then breaks leiningen.core.project/absolutize which checks if the file
>> returned by io/file is absolute or not. When it is returns the original
>> path, not the actual file returned from io/file which seems slightly dodgy,
>> at least in part because that means that one arm of the if returns a string
>> path and the other returns a file.
>>
>> That's where I am now. I can hook absolutize to fix its behaviour but
>> this is starting to feel like a house of cards. Any suggestions for
>> improvements very welcome.
>>
>> On 19 August 2015 at 21:51, Colin Fleming <colin.mailinglist@gmail.com>
>> wrote:
>>
>>> I work out the inter-module dependencies using the artifact ID. I know
>>> all the lein modules in the project, so if another project.clj in the
>>> project has a dependency on an artifact declared by one of those other
>>> modules, then that's an inter-module dependency. So the leiningen project
>>> has three modules: leiningen 2.5.3-SNAPSHOT, leiningen-core
>>> 2.5.3-SNAPSHOT and lein-pprint 1.1.2. Since leiningen has a [leiningen-core
>>> "2.5.3-SNAPSHOT"] dependency, I know that that is something I have to
>>> add as a module dependency, and not as a library. So in general, I never
>>> create libraries for modules that exist in the project.
>>>
>>> When I delegate creation to the REPL, I just call lein and don't do any
>>> fiddling, but I also have an option to add source paths from dependent
>>> modules to the classpath so those will be picked up ahead of the libraries
>>> if required. Obviously this will only work for Clojure, though. The
>>> semantics around that were very tricky, I'd need to go back and read the
>>> issue to remind myself.
>>>
>>> On 18 August 2015 at 22:25, Laurent PETIT <laurent.petit@gmail.com>
>>> wrote:
>>>
>>>> Saying that a module corresponds to an Eclipse project seems right.
>>>> To draw another analogy: a leiningen project corresponds to an Eclipse
>>>> project.
>>>> But in a workspace, you're free to have multiple related or unrelated
>>>> projects. Eclipse imposes nothing there.
>>>>
>>>> For a long time, it had been impossible to have an eclipse project's
>>>> folder as a child of another eclipse project's folder. Since Eclipse 4, it
>>>> seems it is now possible, I'll have to explore this some time or another.
>>>>
>>>> The CCW source is all in a single workspace, but I have also the
>>>> clojure source as a maven project, pomegranate source as a leininigen
>>>> project, etc.
>>>>
>>>> It is possible for a project to "export" part of its classpath, and
>>>> then another project can depend on this exported classpath. So while I
>>>> haven't done it yet, it's totally within reach to have multiproject /
>>>> multimodule in Eclipse, it's not a second-class citizen feature.
>>>>
>>>> When you say you "work out which of their dependencies are actually
>>>> inte r-module dependencies", how do you that? That's a question I've asked
>>>> myself, because indeed it's never guaranteed that you've got the right
>>>> source code checked out corresponding to the declared dependency version?
>>>>
>>>> But when you delegate REPL creation to leiningen, how do you deal with
>>>> intermodule dependencies, then? I understand that you can remove
>>>> dependencies from the list, but I'm curious to know how you add modules
>>>> classpath back...
>>>>
>>>> Le mardi 18 août 2015, Colin Fleming <colin.mailinglist@gmail.com> a
>>>> écrit :
>>>>
>>>>> So this is complicated . Firstly, there's some terminology differences
>>>>> (from IntelliJ's Eclipse Migration Guide
>>>>> <https://www.jetbrains.com/idea/help/terminology.html>): In IntelliJ,
>>>>> a project corresponds to an Eclipse workspace, and a module corresponds to
>>>>> an Eclipse project. I'm not very familiar with Eclipse, but my impression
>>>>> is that that is not quite right - in IntelliJ, I would only have related
>>>>> things in a project, whereas IIRC in Eclipse it's common to have multiple
>>>>> unrelated things in a workspace, and you can open and close projects in it
>>>>> depending on what you're working on. So in IntelliJ, Cursive is a project
>>>>> containing 3 or 4 modules - is the CCW source all in a single workspace? If
>>>>> you had another similar project, would that always be in a separate
>>>>> workspace?
>>>>>
>>>>> Cursive creates modules for each Leiningen project. These are either
>>>>> detected on project import, or can be added later. IntelliJ has the concept
>>>>> of depe ndencies between modules, so a module that depends on another can
>>>>> access classes (and namespaces, for Clojure) from those modules. These do
>>>>> not have to be sibling directories, so a module can contain sub-modules.
>>>>> Leiningen, for example, has the leiningen project in the root and
>>>>> leiningen-core and lein-pprint in directories inside the root. When Cursive
>>>>> syncs the dependencies from the lein projects, I work out which of their
>>>>> dependencies are actually inter-module dependencies within the same project
>>>>> and don't add those as libraries - they're added as module dependencies
>>>>> instead. This means that the up-to-date code from those dependencies is
>>>>> available to the depending modules without needing to use checkouts. If the
>>>>> user uses checkouts for compatibility with other tooling Cursive supports
>>>>> that, it will work out the required module dependencies automatically.
>>>>>
>>>>> There are additional complications syncing the dependencies for large
>>>>> projects such as these, since their up-to-date artifacts may not be
>>>>> installed to .m2. Currently I hack this by modifying the project.clj files
>>>>> before passing them to lein, removing the dependencies which correspond to
>>>>> the inter-module deps. However this gives subtle problems such as slightly
>>>>> different versions for dependencies, presumably because the transitive
>>>>> closure is modified by doing this so Aether occasionally returns different
>>>>> dependency versions from the ones that would be returned on the command
>>>>> line. I'm planning to fix this by instead publishing up-to-date poms for
>>>>> the project artifacts with empty jars to a temporary repository during the
>>>>> resolve process, which will mean that I don't have to fiddle with the
>>>>> project files. Then after the resolve I'll work out which dependencies are
>>>>> actually inter-module ones and remove them from the dep list before library
>>>>> creation happens.
>>>>>
>>>>>
>>>>>
>>>> < br>
>>>> --
>>>> Laurent Petit
>>>>
>>>>
>>>
>>
>

Re: [leiningen] Getting leiningen classpath

From:
Colin Fleming
Date:
2015-08-17 @ 15:54
So I investigated the CWD thing a bit. Setting user.dir doesn't work, since
new File(".") will not respect it. You can do it using this

<https://github.com/jnr/jnr-posix/blob/master/src/main/java/jnr/posix/POSIX.java#L116>
but I reached out to Allan Rohner who used it for lein-daemon and he said:

changing the CWD works, but it was … weird. Parts of the JVM didn’t expect
> it.



> I would put changing CWD at the level of “dubious hack”. definitely avoid
> if you can for serious work
>

I think I'll just hook local-repo and perhaps slurp and spit, and just not
support the rest.

On 17 August 2015 at 16:58, Colin Fleming <colin.mailinglist@gmail.com>
wrote:

> On 17 August 2015 at 16:27, Laurent PETIT <laurent.petit@gmail.com> wrote:
>
>> It's probably the only option if you want to support concurrency.
>>
>
> Hopefully I won't actually have to support concurrency - see below.
>
>>
>> Wow, *that* big. Indeed.
>>
>
> Yeah - that's a very big one but I'm aware of a lot of Cursive users with
> projects with a couple of dozen modules.
>
>
>> So instead of starting 66 JVMs, you intend to create (and immediately
>> ditch after usage) 66 shims.
>>
>
> Initially no, I'll just create a single shim and use that for all modules,
> so it will be very fast. There's not really a big advantage to multiple
> shims AFAIK, since memoisation should be ok for a single cross-project
> operation. I think the worst offenders are the dependencies and the
> profiles, both of which can be memoised for a single call across the whole
> project. If I can do the user.dir hack then I can do it for each project
> one by one within the same shim as I sync each one.
>
>
>
>> While the single-project case is probably by far the most common, I still
>>> need to support the large project case properly and I can't see a way to do
>>> that with no compromises at all. I could work around the worst of it with
>>> some complex configuration saying how many JVMs to start up, and specifying
>>> which modules should use which JVMs, but i t starts to sound like a
>>> nightmare at that point, both for me and the user.
>>>
>>
>>
>>
>>>
>>> On 17 August 2015 at 13:55, Laurent PETIT <laurent.petit@gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> 2015-08-17 13:40 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>>>>
>>>>> Thanks for the thoughts, Laurent - very interesting. Do you have a
>>>>> link to the CCW source that changes the CWD property? I read about doing
>>>>> that a while back, but since IntelliJ doesn't isolate classloaders like
>>>>> Eclipse does I couldn't do it. Now that I'm going to use ShimDandy I can
>>>>> probably do that. If you isolate classloaders, are the system properties
>>>>> then isolated too?
>>>>>
>>>>
>>>> I won't be able to do so ... because I was wrong. There's no trace of
>>>> "user.dir" in CCW codebase. So I may have thought about it in my shower,
>>>> but never did it in pract ice ...
>>>>
>>>>
>>>>>
>>>>> In terms of lazily downloading Leiningen, it'll happen as soon as they
>>>>> open a lein project, so that's generally almost immediately. If they're
>>>>> offline then they'll have problems, right, but at that point they'll almost
>>>>> certainly need to be connected to download deps anyway so I'm not sure
>>>>> it'll be a problem. I think the most important thing is that the user
>>>>> doesn't have to explicitly download it separately from Cursive,
>>>>> particularly on Windows where that seems to be pretty painful.
>>>>>
>>>>
>>>> Fair enough
>>>>
>>>>
>>>>> WRT JVM startup time, I'm not worried about that - that is dwarfed by
>>>>> the ti me to load th e Clojure/lein namespa ces. But by doing that
>>>>> in-process I can start a shim and load the namespaces into it ahead of time
>>>>> so it's ready when I need it and will work immediately. To do that with a
>>>>> standalone JVM would mean starting up an external JVM eagerly and having it
>>>>> hanging around - I'm more worried about memory use than time for that.
>>>>> Obviously that will still occupy memory in the IntelliJ heap but I'm paying
>>>>> that price now anyway - it will be more due to isolation, but still less
>>>>> than the 300mb that a JVM takes up.
>>>>>
>>>>
>>>> Well, yes, always keeping ahead of time one classloader is certainly
>>>> easier and more light-weight for the user's computer memory.
>>>> But still, interoperating through a new JVM rather than through
>>>> classloaders appeals more to me:
>>>> - fixes all the problems you listed, without compromise
>>>> - introduces the problem of more RAM if a pool of JVM processes is
>>>> maintained. This could be leveraged by letting t he user choose the pool
>>>> size (setting it to 0 would be friendly for computers with low memory.
>>>> Setting it to 5 would be a boost for users with modern computers and multi
>>>> module projects. Setting it to 1 by default would probably be a good
>>>> average).
>>>> - more flexible to adapt to boot, planck or similar for clojurescript,
>>>> or plain processes.
>>>>
>>>> I think I will stick to my idea of moving from embedded to external.
>>>> Thanks for sharing the thoughts!
>>>>
>>>>
>>>>>
>>>>>
>>>>> On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Mor e thoughts, Colin:
>>>>>>
>>>>>> 2015-08-17 12:37 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com
>>>>>> >:
>>>>>>
>>>>>>> Curiously enough, I'm also about to totally change the way I
>>>>>>> integrate Leiningen in Cursive.
>>>>>>>
>>>>>>> To answer Phil's question about embedding leiningen-core, there are
>>>>>>> many many disadvantages to doing so:
>>>>>>>
>>>>>>>    1. You're tied to a specific version of lein. Currently I bundle
>>>>>>>    2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users
encountered
>>>>>>>    bugs so I downgraded to 2.5.0. Some users currently experience 
bugs with
>>>>>>>    2.5.0, but they're easier to work around. Anyone using older 
versions on th
>>>>>>>    eir project is ou t of luck.
>>>>>>>    2. Lein incorrectly memoises many things internally, which is a
>>>>>>>    gigantic pain. Currently users have to restart Cursive every time they
>>>>>>>    change profiles.clj, and additionally they also have to be 
aware of the bug
>>>>>>>    to know to do that. I've also lost countless hours debugging 
problems like
>>>>>>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A
>>>>>>>    more general report is #1768. This is slated to be fixed in lein
>>>>>>>    3.0, but a) who knows when that will be and b) I'm still at the
mercy of
>>>>>>>    plugins doing the same thing anyway.
>>>>>>>    3. Java cannot change the CWD of the current process, which
>>>>>>>    means that local-repo always appears somewhere in the IntelliJ bin
>>>>>>>    directory. This also fails for users' home grown versioning 
systems when
>>>>>>>    they do things like (slurp versions.txt). More examples here
>>>>>>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>>>>>>    (there are probably more, but that's the easiest search). 
Essentia lly, on
>>>>>>>    e of le in' s big s elling points is that it's declarative, but
the fact
>>>>>>>    that project.clj is actually executed means that it's not truly
declarative.
>>>>>>>    4. Many things in lein (gpg, proxies etc) are configured using
>>>>>>>    environment variables, which are a pain to pass when embedding. Cursive
>>>>>>>    hooks leiningen fairly heavily using robert.hooke t o supply values for
>>>>>>>    these instead of getting them from env vars.
>>>>>>>    5. Often you need access to the lein tasks, not just core.
>>>>>>>    Cursive has bundled all of lein for a long time so that it can 
access the
>>>>>>>    repl task. If you don't do this you end up duplicating a lot of
the code
>>>>>>>    from the tasks anyway.
>>>>>>>    6. There are also lots of minor issues - I wanted to hook into
>>>>>>>    the Aether transfer listener so that I could provide updates in
the Cursive
>>>>>>>    UI when libs are downloaded. Currently I do this in Cursive by hooking
>>>>>>>    leiningen.core.classpath/warn and checking if it looks like a 
"Retrieving"
>>>>>>>    message and parsing it if so.
>>>>>>>    7. As Laurent says, lein, even just core, is really big. The
>>>>>>>    Cursive download is about 3x the size it would be if it didn't 
bundle lein.
>>>>>>>
>>>>>>> That said, while I understand Laurent's worry about stability,
>>>>>>> that's never been a problem. I've never received a single bug report about
>>>>>>> s tability that was as a result of bundling lein.
>>>>>>>
>>>>>>
>>>>>> I admit that it has been more a fear than an issue.
>>>>>>
>>>>>>
>>>>>>> There is also one significant benefit to bundling lein - it's fast.
>>>>>>> Starting a JVM process for lein every time Cursive needs to sync the deps
>>>>>>> is going to be way too slow. This will be particularly true for large
>>>>>>> multi-module projects where to do everything totally accurately (in
>>>>>>> particular, the CWD) requires starting a new JVM per module - this is
>>>>>>> unworkable. I considered caching the last-modified of the project.clj and
>>>>>>> only re-parsing a particular module if that had changed, but I'd 
need to do
>>>>>>> that for all possible contributing files which is impossible to calculate.
>>>>>>> Apart from p rofiles.clj a nd other "offic ial" files which might a ffect
>>>>>>> the result, I'd also need to test the ver sion.txt type files from #3, and
>>>>>>> there's no way I can detect all of those.
>>>>>>>
>>>>>>> Since I don't think that stability is actually a major problem, I'm
>>>>>>> planning to continue running lein in-process but I'm going to isolate its
>>>>>>> classpath using ShimDandy. I have a prototype of this but it's 
still pretty
>>>>>>> primitive, and I have a lot of other things going on so it's advancing
>>>>>>> slowly. I run lein in a shim, and I create the shim's classpath using
>>>>>>> Aether so I don't actually bundle lein, I pull it in when creating the
>>>>>>> shim. This means that users can choose the lein version they want, and the
>>>>>>> files are stored in .m2 rather than being downloaded with Cursive.
>>>>>>>
>>>>>>
>>>>>> I had considered this for CCW, but chose to embed leiningen because
>>>>>> back in time (and still currently in practice, despite the growing interest
>>>>>> in boot) leiningen was the only option.
>>>>>> Having the latest leiningen downloaded at the same time as CCW is
>>>>>> easier for users, I think. They have to wait once for things to download,
>>>>>> and then they're good to go.
>>>>>> By downloading lein lazily, you're at risk for users going offline,
>>>>>> wiping out their ~/.m2/repo sitory (I do this fr om time to time). That 's
>>>>>> one more moving part, not to be considered too lightly, IMHO, but YMMV.
>>>>>>
>>>>>>
>>>>>>> I'll then throw the shim away when done so memoisation won't be a
>>>>>>> problem. I can amortize the performance problem by eagerly creating the
>>>>>>> shim and loading the namespaces into it before I need it, similar 
to boot's
>>>>>>> pod caching. This is much less costly than eagerly starting a JVM and
>>>>>>> keeping that around until needed.
>>>>>>>
>>>>>>
>>>>>> Have you checked the time required for the JVM itself to start versus
>>>>>> the time to load leiningen.main and all its dependencies? I think I'll do
>>>>>> it again, just to check that the ratio proves your point. We might be
>>>>>> surprised discovering that the JVM startup time is not the problem, leading
>>>>>> to the conclusion that reloading everything of lein in a new classloader
>>>>>> will n ot be that different (performance wise) than starting a new JVM.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I'm also going to try to fiddle much less with the lein internals
>>>>>>> since that has caused problems when upgrading lein. Using this new
>>>>>>> mechanism I can call the functions representing the lein tasks as if they
>>>>>>> were being called from main, whic h greatly reduces my dependency on lein
>>>>>>> internals. I'll still have to hook for gpg/proxies etc, but I'll 
be hooking
>>>>>>> aether rather than lein for those, which is much more stable since it's
>>>>>>> basically abandonware at this point - I should have done this ages ago. I
>>>>>>> can also hook into the transfer listener by hooking aether.
>>>>>>>
>>>>>>> My interface to the shim will be to pass it a list of file paths to
>>>>>>> the project.clj files and eventually a list of profiles, and it 
will return
>>>>>>> a map containing all the information I need about the projects.
>>>>>>>
>>>>>>> The upshot of all this is that this should fix all my current
>>>>>>> problems except #3, which there's no workaround for. I'll fix the
>>>>>>> local-repo problem by hooking lein or fiddling with the project.clj, but
>>>>>>> the only way to fix the slurp problem is to start a JVM per lein project,
>>>>>>> or convince users not to do that. Or hook slurp, I guess.
>>>>>>>
>>>>>>> Laurent, currently I run the REPL withi n Cursive using the
>>>>>>> trampoline mechanism - IMO you don't have to emulate it, you can just use
>>>>>>> it directly. Get lein to generate a trampoline file then parse that - it's
>>>>>>> very easy to parse since all the elements always appear in the same order.
>>>>>>> Currently I hook the lein internals to do this, but as described above I'm
>>>>>>> about to start calling lein trampoline directly from the shim.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Colin
>>>>>>>
>>>>>>>
>>>>>>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> OK understood this works as expected: lein update-in : assoc
>>>>>>>> :eval-in :pprint -- run -m clojure.main
>>>>>>>> BTW, does t hat mean that it's possible t o emulate TRAMPOLINE by
>>>>>>>> storing the result o f :pprint (and thus preventing the scenario where
>>>>>>>> there's 2 JVMs loaded, one for lein itself, and one for the project) ?
>>>>>>>>
>>>>>>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>>>>>>
>>>>>>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>>>>>>
>>>>>>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`;
>>>>>>>>> this will
>>>>>>>>> > pprint the form to be run and print the classpath and JVM args
>>>>>>>>> instead
>>>>>>>>> > of actually running it.
>>>>>>>>>
>>>>>>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>>>>>>
>>>>>>>>> -Phil
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Laurent Petit
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Laurent Petit
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Laurent Petit
>>>>
>>>
>>>
>>
>>
>> --
>> Laurent Petit
>>
>
>

Re: [leiningen] Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-17 @ 21:40
2015-08-17 17:54 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:

> So I investigated the CWD thing a bit. Setting user.dir doesn't work,
> since new File(".") will not respect it. You can do it using this
> 
<https://github.com/jnr/jnr-posix/blob/master/src/main/java/jnr/posix/POSIX.java#L116>
> but I reached out to Allan Rohner who used it for lein-daemon and he said:
>
> changing the CWD works, but it was … weird. Parts of the JVM didn’t expect
>> it.
>
>
>
>> I would put changing CWD at the level of “dubious hack”. definitely avoid
>> if you can for serious work
>>
>
> I think I'll just hook local-repo and perhaps slurp and spit, and just not
> support the rest.
>

In clojure, everything boils down to clojure.java.io.
In an ideal world, it would have suffice to overwrite the extension of
IOFactory for File. Alas, I can see in Clojure 1.7 that there are places in
clojure.java.io where (FileInputStream. f)  (FileOutputStream. f) are
called in places where (make-input-stream f) / (make-output-stream f)
should have been called.

Anyway, it seems that "fixing" IOFactory for java.io.File (and also maybe
for completeness do-copy for File) will do the job for slurp, spit, and
also for local-repo from pomegranate.

Something like (not tested):

(require '[clojure.java.io :as io])
(import 'java.io.File)
(let [{:keys [make-input-stream make-output-stream] :as prev-impls} (get
(:impls io/IOFactory) File)
       cwd (System/getProperty "user.dir")
        absolute #(if (.isAbsolute %) % (io/file cwd %))]
  (extend File
    IOFactory (merge prev-impls {:make-input-stream (fn [^File f opts]
(make-input-stream (absolute f) opts)) :make-output-stream (fn [^File f
opts] (make-output-stream (absolute f) opts))})))



>
> On 17 August 2015 at 16:58, Colin Fleming <colin.mailinglist@gmail.com>
> wrote:
>
>> On 17 August 2015 at 16:27, Laurent PETIT <laurent.petit@gmail.com>
>> wrote:
>>
>>> It's probably the only option if you want to support concurrency.
>>>
>>
>> Hopefully I won't actually have to support concurrency - see below.
>>
>>>
>>> Wow, *that* big. Indeed.
>>>
>>
>> Yeah - that's a very big one but I'm aware of a lot of Cursive users with
>> projects with a couple of dozen modules.
>>
>>
>>> So instead of starting 66 JVMs, you intend to create (and immediately
>>> ditch after usage) 66 shims.
>>>
>>
>> Initially no, I'll just create a single shim and use that for all
>> modules, so it will be very fast. There's not really a big advantage to
>> multiple shims AFAIK, since memoisation should be ok for a single
>> cross-project operation. I think the worst offenders are the dependencies
>> and the profiles, both of which can be memoised for a single call across
>> the whole project. If I can do the user.dir hack then I can do it for each
>> project one by one within the same shim as I sync each one.
>>
>>
>>
>>> While the single-project case is probably by far the most common, I
>>>> still need to support the large project case properly and I can't see a way
>>>> to do that with no compromises at all. I could work around the worst of it
>>>> with some complex configuration saying how many JVMs to start up, and
>>>> specifying which modules should use which JVMs, but i t starts to sound
>>>> like a nightmare at that point, both for me and the user.
>>>>
>>>
>>>
>>>
>>>>
>>>> On 17 August 2015 at 13:55, Laurent PETIT <laurent.petit@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> 2015-08-17 13:40 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>
>>>>> :
>>>>>
>>>>>> Thanks for the thoughts, Laurent - very interesting. Do you have a
>>>>>> link to the CCW source that changes the CWD property? I read about doing
>>>>>> that a while back, but since IntelliJ doesn't isolate classloaders like
>>>>>> Eclipse does I couldn't do it. Now that I'm going to use ShimDandy I can
>>>>>> probably do that. If you isolate classloaders, are the system properties
>>>>>> then isolated too?
>>>>>>
>>>>>
>>>>> I won't be able to do so ... because I was wrong. There's no trace of
>>>>> "user.dir" in CCW codebase. So I may have thought about it in my shower,
>>>>> but never did it in pract ice ...
>>>>>
>>>>>
>>>>>>
>>>>>> In terms of lazily downloading Leiningen, it'll happen as soon as
>>>>>> they open a lein project, so that's generally almost immediately. If
>>>>>> they're offline then they'll have problems, right, but at that point
>>>>>> they'll almost certainly need to be connected to download deps anyway so
>>>>>> I'm not sure it'll be a problem. I think the most important thing is that
>>>>>> the user doesn't have to explicitly download it separately from Cursive,
>>>>>> particularly on Windows where that seems to be pretty painful.
>>>>>>
>>>>>
>>>>> Fair enough
>>>>>
>>>>>
>>>>>> WRT JVM startup time, I'm not worried about that - that is dwarfed by
>>>>>> the ti me to load th e Clojure/lein namespa ces. But by doing that
>>>>>> in-process I can start a shim and load the namespaces into it ahead of time
>>>>>> so it's ready when I need it and will work immediately. To do that with a
>>>>>> standalone JVM would mean starting up an external JVM eagerly and having it
>>>>>> hanging around - I'm more worried about memory use than time for that.
>>>>>> Obviously that will still occupy memory in the IntelliJ heap but I'm paying
>>>>>> that price now anyway - it will be more due to isolation, but still less
>>>>>> than the 300mb that a JVM takes up.
>>>>>>
>>>>>
>>>>> Well, yes, always keeping ahead of time one classloader is certainly
>>>>> easier and more light-weight for the user's computer memory.
>>>>> But still, interoperating through a new JVM rather than through
>>>>> classloaders appeals more to me:
>>>>> - fixes all the problems you listed, without compromise
>>>>> - introduces the problem of more RAM if a pool of JVM processes is
>>>>> maintained. This could be leveraged by letting t he user choose the pool
>>>>> size (setting it to 0 would be friendly for computers with low memory.
>>>>> Setting it to 5 would be a boost for users with modern computers and multi
>>>>> module projects. Setting it to 1 by default would probably be a good
>>>>> average).
>>>>> - more flexible to adapt to boot, planck or similar for clojurescript,
>>>>> or plain processes.
>>>>>
>>>>> I think I will stick to my idea of moving from embedded to external.
>>>>> Thanks for sharing the thoughts!
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Mor e thoughts, Colin:
>>>>>>>
>>>>>>> 2015-08-17 12:37 GMT+02:00 Colin Fleming <
>>>>>>> colin.mailinglist@gmail.com>:
>>>>>>>
>>>>>>>> Curiously enough, I'm also about to totally change the way I
>>>>>>>> integrate Leiningen in Cursive.
>>>>>>>>
>>>>>>>> To answer Phil's question about embedding leiningen-core, there are
>>>>>>>> many many disadvantages to doing so:
>>>>>>>>
>>>>>>>>    1. You're tied to a specific version of lein. Currently I
>>>>>>>>    bundle 2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users
>>>>>>>>    encountered bugs so I downgraded to 2.5.0. Some users 
currently experience
>>>>>>>>    bugs with 2.5.0, but they're easier to work around. Anyone using older
>>>>>>>>    versions on th eir project is ou t of luck.
>>>>>>>>    2. Lein incorrectly memoises many things internally, which is a
>>>>>>>>    gigantic pain. Currently users have to restart Cursive every time they
>>>>>>>>    change profiles.clj, and additionally they also have to be 
aware of the bug
>>>>>>>>    to know to do that. I've also lost countless hours debugging 
problems like
>>>>>>>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A
>>>>>>>>    more general report is #1768. This is slated to be fixed in
>>>>>>>>    lein 3.0, but a) who knows when that will be and b) I'm still 
at the mercy
>>>>>>>>    of plugins doing the same thing anyway.
>>>>>>>>    3. Java cannot change the CWD of the current process, which
>>>>>>>>    means that local-repo always appears somewhere in the IntelliJ bin
>>>>>>>>    directory. This also fails for users' home grown versioning 
systems when
>>>>>>>>    they do things like (slurp versions.txt). More examples here
>>>>>>>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>>>>>>>    (there are probably more, but that's the easiest search). E 
ssentia lly, on
>>>>>>>>    e of le in' s big s elling points is that it's declarative, 
but the fact
>>>>>>>>    that project.clj is actually executed means that it's not 
truly declarative.
>>>>>>>>    4. Many things in lein (gpg, proxies etc) are configured using
>>>>>>>>    environment variables, which are a pain to pass when 
embedding. Cursive
>>>>>>>>    hooks leiningen fairly heavily using robert.hooke t o supply 
values for
>>>>>>>>    these instead of getting them from env vars.
>>>>>>>>    5. Often you need access to the lein tasks, not just core.
>>>>>>>>    Cursive has bundled all of lein for a long time so that it can
access the
>>>>>>>>    repl task. If you don't do this you end up duplicating a lot 
of the code
>>>>>>>>    from the tasks anyway.
>>>>>>>>    6. There are also lots of minor issues - I wanted to hook into
>>>>>>>>    the Aether transfer listener so that I could provide updates 
in the Cursive
>>>>>>>>    UI when libs are downloaded. Currently I do this in Cursive by hooking
>>>>>>>>    leiningen.core.classpath/warn and checking if it looks like a 
"Retrieving"
>>>>>>>>    message and parsing it if so.
>>>>>>>>    7. As Laurent says, lein, even just core, is really big. The
>>>>>>>>    Cursive download is about 3x the size it would be if it didn't
bundle lein.
>>>>>>>>
>>>>>>>> That said, while I understand Laurent's worry about stability,
>>>>>>>> that's never been a problem. I've never received a single bug 
report ab out
>>>>>>>> s tability that was as a result of bundling lein.
>>>>>>>>
>>>>>>>
>>>>>>> I admit that it has been more a fear than an issue.
>>>>>>>
>>>>>>>
>>>>>>>> There is also one significant benefit to bundling lein - it's fast.
>>>>>>>> Starting a JVM process for lein every time Cursive needs to sync the deps
>>>>>>>> is going to be way too slow. This will be particularly true for large
>>>>>>>> multi-module projects where to do everything totally accurately (in
>>>>>>>> particular, the CWD) requires starting a new JVM per module - this is
>>>>>>>> unworkable. I considered caching the last-modified of the project.clj and
>>>>>>>> only re-parsing a particular module if that had changed, but I'd 
need to do
>>>>>>>> that for all possible contributing files which is impossible to 
calculate.
>>>>>>>> Apart fr om p rofiles.clj a nd other "offic ial" files which 
might a ffect
>>>>>>>> the result, I'd also need to test the ver sion.txt type files 
from #3, and
>>>>>>>> there's no way I can detect all of those.
>>>>>>>>
>>>>>>>> Since I don't think that stability is actually a major problem, I'm
>>>>>>>> planning to continue running lein in-process but I'm going to isolate its
>>>>>>>> classpath using ShimDandy. I have a prototype of this but it's 
still pretty
>>>>>>>> primitive, and I have a lot of other things going on so it's advancing
>>>>>>>> slowly. I run lein in a shim, and I create the shim's classpath using
>>>>>>>> Aether so I don't actually bundle lein, I pull it in when creating the
>>>>>>>> shim. This means that users can choose the lein version they 
want, and the
>>>>>>>> files are stored in .m2 rather than being downloaded with Cursive.
>>>>>>>>
>>>>>>>
>>>>>>> I had considered this for CCW, but chose to embed leiningen because
>>>>>>> back in time (and still currently in practice, despite the growing
interest
>>>>>>> in boot) leiningen was the only option.
>>>>>>> Having the latest leiningen downloaded at the same time as CCW is
>>>>>>> easier for users, I think. They have to wait once for things to download,
>>>>>>> and then they're good to go.
>>>>>>> By downloading lein lazily, you're at risk for users going offli ne,
>>>>>>> wiping out their ~/.m2/repo sitory (I do this fr om time to time). That 's
>>>>>>> one more moving part, not to be considered too lightly, IMHO, but YMMV.
>>>>>>>
>>>>>>>
>>>>>>>> I'll then throw the shim away when done so memoisation won't be a
>>>>>>>> problem. I can amortize the performance problem by eagerly creating the
>>>>>>>> shim and loading the namespaces into it before I need it, similar
to boot's
>>>>>>>> pod caching. This is much less costly than eagerly starting a JVM and
>>>>>>>> keeping that around until needed.
>>>>>>>>
>>>>>>>
>>>>>>> Have you checked the time required for the JVM itself to start
>>>>>>> versus the time to load leiningen.main and all its dependencies? I think
>>>>>>> I'll do it again, just to check that the ratio proves your point. We might
>>>>>>> be surprised discovering that the JVM startup time is not the problem,
>>>>>>> leading to the conclusion that reloading everything of lein in a new
>>>>>>> classloader will n ot be that different (performance wise) than starting a
>>>>>>> new JVM.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I'm also going to try to fiddle much less with the lein internals
>>>>>>>> since that has caused problems when upgrading lein. Using this new
>>>>>>>> mechanism I can call the functions representing the lein tasks as if they
>>>>>>>> were being called from main, whic h greatly reduces my dependency on lein
>>>>>>>> internals. I'll still have to hook for gpg/proxies etc, but I'll 
be hooking
>>>>>>>> aether rather than lein for those, which is much more stable since it's
>>>>>>>> basically abandonware at this point - I should have done this ages ago. I
>>>>>>>> can also hook into the transfer listener by hooking aether.
>>>>>>>>
>>>>>>>> My interface to the shim will be to pass it a list of file paths to
>>>>>>>> the project.clj files and eventually a list of profiles, and it 
will return
>>>>>>>> a map containing all the information I need about the projects.
>>>>>>>>
>>>>>>>> The upshot of all this is that this should fix all my current
>>>>>>>> problems except #3, which there's no workaround for. I'll fix the
>>>>>>>> local-repo problem by hooking lein or fiddling with the project.clj, but
>>>>>>>> the only way to fix the slurp problem is to start a JVM per lein project,
>>>>>>>> or convince users not to do that. Or hook slurp, I guess.
>>>>>>>>
>>>>>>>> Laurent, currently I run the REPL withi n Cursive using the
>>>>>>>> trampoline mechanism - IMO you don't have to emulate it, you can just use
>>>>>>>> it directly. Get lein to generate a trampoline file then parse 
that - it's
>>>>>>>> very easy to parse since all the elements always appear in the 
same order.
>>>>>>>> Currently I hook the lein internals to do this, but as described 
above I'm
>>>>>>>> about to start calling lein trampoline directly from the shim.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Colin
>>>>>>>>
>>>>>>>>
>>>>>>>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> OK understood this works as expected: lein update-in : assoc
>>>>>>>>> :eval-in :pprint -- run -m clojure.main
>>>>>>>>> BTW, does t hat mean that it's possible t o emulate TRAMPOLINE by
>>>>>>>>> storing the result o f :pprint (and thus preventing the scenario where
>>>>>>>>> there's 2 JVMs loaded, one for lein itself, and one for the project) ?
>>>>>>>>>
>>>>>>>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>>>>>>>
>>>>>>>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>>>>>>>
>>>>>>>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`;
>>>>>>>>>> this will
>>>>>>>>>> > pprint the form to be run and print the classpath and JVM args
>>>>>>>>>> instead
>>>>>>>>>> > of actually running it.
>>>>>>>>>>
>>>>>>>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>>>>>>>
>>>>>>>>>> -Phil
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Laurent Petit
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Laurent Petit
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Laurent Petit
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Laurent Petit
>>>
>>
>>
>


-- 
Laurent Petit

Re: [leiningen] Getting leiningen classpath

From:
Colin Fleming
Date:
2015-08-18 @ 10:12
Hmm, very interesting, thanks Laurent! I'll play around with that. That's
definitely beyond my knowledge of protocols :-)


On 17 August 2015 at 23:40, Laurent PETIT <laurent.petit@gmail.com> wrote:

>
>
> 2015-08-17 17:54 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>
>> So I investigated the CWD thing a bit. Setting user.dir doesn't work,
>> since new File(".") will not respect it. You can do it using this
>> 
<https://github.com/jnr/jnr-posix/blob/master/src/main/java/jnr/posix/POSIX.java#L116>
>> but I reached out to Allan Rohner who used it for lein-daemon and he said:
>>
>> changing the CWD works, but it was … weird. Parts of the JVM didn’t
>>> expect it.
>>
>>
>>
>>> I would put changing CWD at the level of “dubious hack”. definitely
>>> avoid if you can for serious work
>>>
>>
>> I think I'll just hook local-repo and perhaps slurp and spit, and just
>> not support the rest.
>>
>
> In clojure, everything boils down to clojure.java.io.
> In an ideal world, it would have suffice to overwrite the extension of
> IOFactory for File. Alas, I can see in Clojure 1.7 that there are places in
> clojure.java.io where (FileInputStream. f)  (FileOutputStream. f) are
> called in places where (make-input-stream f) / (make-output-stream f)
> should have been called.
>
> Anyway, it seems that "fixing" IOFactory for java.io.File (and also maybe
> for completeness do-copy for File) will do the job for slurp, spit, and
> also for local-repo from pomegr anate.
>
> Something like (not tested):
>
> (require '[clojure.java.io :as io])
> (import 'java.io.File)
> (let [{:keys [make-input-stream make-output-stream] :as prev-impls} (get
> (:impls io/IOFactory) File)
>        cwd (System/getProperty "user.dir")
>         absolute #(if (.isAbsolute %) % (io/file cwd %))]
>   (extend File
>     IOFactory (merge prev-impls {:make-input-stream (fn [^File f opts]
> (make-input-stream (absolute f) opts)) :make-output-stream (fn [^File f
> opts] (make-output-stream (absolute f) opts))})))
>
>
>
>>
>> On 17 August 2015 at 16:58, Colin Fleming <colin.mailinglist@gmail.com>
>> wrote:
>>
>>> On 17 August 2015 at 16:27, Laurent PETIT <laurent.petit@gmail.com>
>>> wrote:
>>>
>>>> It's probably the only option if you want to support concurrency.
>>>>
>>>
>>> Hopefully I won't actually have to support concurrency - see below.
>>>
>>>>
>>>> Wow, *that* big. Indeed.
>>>>
>>>
>>> Yeah - that's a very big one but I'm aware of a lot of Cursive users
>>> with projects with a couple of dozen modules.
>>>
>>>
>>>> So instead of starting 66 JVMs, you intend to create (and immediately
>>>> ditch after usage) 66 shims.
>>>>
>>>
>>> Initially no, I'll just create a single shim and use that for all
>>> modules, so it will be very fast. There's not really a big advantage to
>>> multiple shims AFAIK, since memoisation should be ok for a single
>>> cross-project operation. I think the worst offenders are the dependencies
>>> and the profiles, both of which can be memoised for a single call across
>>> the whole project. If I can do the user.dir hack then I can do it for each
>>> project one by one within the same shim as I sync each one.
>>>
>>>
>>>
>>>> While the single-project case is probably by far the most common, I
>>>>> still need to support the large project case properly and I can't see a way
>>>>> to do that with no compromises at all. I could work around the worst of it
>>>>> with some complex configuration saying how many JVMs to start up, and
>>>>> specifying which modules should use which JVMs, but i t starts to sound
>>>>> like a nightmare at that point, both for me and the user.
>>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> On 17 August 2015 at 13:55, Laurent PETIT <laurent.petit@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-08-17 13:40 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com
>>>>>> >:
>>>>>>
>>>>>>> Thanks for the thoughts, Laurent - very interesting. Do you have a
>>>>>>> link to the CCW source that changes the CWD property? I read about doing
>>>>>>> that a while back, but since IntelliJ doesn't isolate classloaders like
>>>>>>> Eclipse does I couldn't do it. Now that I'm going to use ShimDandy I can
>>>>>>> probably do that. If you isolate classloaders, are the system properties
>>>>>>> then isolated too?
>>>>>>>
>>>>>>
>>>>>> I won't be able to do so ... because I was wrong. There's no trace of
>>>>>> "user.dir" in CCW codebase. So I may have thought about it in my shower,
>>>>>> but never did it in p ract ice ...
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> In terms of lazily downloading Leiningen, it'll happen as soon as
>>>>>>> they open a lein project, so that's generally almost immediately. If
>>>>>>> they're offline then they'll have problems, right, but at that point
>>>>>>> they'll almost certainly need to be connected to download deps anyway so
>>>>>>> I'm not sure it'll be a problem. I think the most important thing is that
>>>>>>> the user doesn't have to explicitly download it separately from Cursive,
>>>>>>> particularly on Windows where that seems to be pretty painful.
>>>>>>>
>>>>>>
>>>>>> Fair enough
>>>>>>
>>>>>>
>>>>>>> WRT JVM startup time, I'm not worried about that - that is dwarfed
>>>>>>> by the ti me to load th e Clojure/lein namespa ces. But by doing that
>>>>>>> in-process I can start a shim and load the namespaces into it 
ahead of time
>>>>>>> so it's ready when I need it and will work immediately. To do that with a
>>>>>>> standalone JVM would mean starting up an external JVM eagerly and 
having it
>>>>>>> hanging around - I'm more worried about memory use than time for that.
>>>>>>> Obviously that will still occupy memory in the IntelliJ heap but 
I'm paying
>>>>>>> that price now anyway - it will be more due to isolation, but still less
>>>>>>> than the 300mb that a JVM takes up.
>>>>>>>
>>>>>>
>>>>>> Well, yes, always keeping ahead of time one classloader is certainly
>>>>>> easier and more light-weight for the user's computer memory.
>>>>>> But still, interoperating through a new JVM rather than through
>>>>>> classloaders appeals more to me:
>>>>>> - fixes all the problems you listed, without compromise
>>>>>> - introduces the problem of more RAM if a pool of JVM processes is
>>>>>> maintained. This could be leveraged by letting t he user choose the pool
>>>>>> size (setting it to 0 would be friendly for computers with low memory.
>>>>>> Setting it to 5 would be a boost for users with modern computers and multi
>>>>>> module projects. Setting it to 1 by default would probably be a good
>>>>>> average).
>>>>>> - more flexible to adapt to boot, planck or similar for
>>>>>> clojurescript, or plain processes.
>>>>>>
>>>>>> I think I will stick to my idea of moving from embedded to external.
>>>>>> Thanks for sharing the thoughts!
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Mor e thoughts, Colin:
>>>>>>>
>>>>>>> 2015-08-17 12:37 GMT+02:00 Colin Fleming <
>>>>>>> colin.mailinglist@gmail.com>:
>>>>>>>
>>>>>>>> Curiously enough, I'm also about to totally change the way I
>>>>>>>> integrate Leiningen in Cursive.
>>>>>>>>
>>>>>>>> To answer Phil's question about embedding leiningen-core, there are
>>>>>>>> many many disadvantages to doing so:
>>>>>>>>
>>>>>>>>    1. You're tied to a specific version of lein. Currently I
>>>>>>>>    bundle 2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and several users
>>>>>>>>    encountered bugs so I downgraded to 2.5.0. Some users 
currently experience
>>>>>>>>    bugs with 2 .5.0, but they're easier to work around. Anyone 
using older
>>>>>>>>    versions on th eir project is ou t of luck.
>>>>>>>>    2. Lein incorrectly memoises many things internally, which is a
>>>>>>>>    gigantic pain. Currently users have to restart Cursive every time they
>>>>>>>>    change profiles.clj, and additionally they also have to be 
aware of the bug
>>>>>>>>    to know to do that. I've also lost countless hours debugging 
problems like
>>>>>>>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>. A
>>>>>>>>    more general report is #1768. This is slated to be fixed in
>>>>>>>>    lein 3.0, but a) who knows when that will be and b) I'm still 
at the mercy
>>>>>>>>    of plugins doing the same thing anyway.
>>>>>>>>    3. Java cannot change the CWD of the current process, which
>>>>>>>>    means that local-repo always appears somewhere in the IntelliJ bin
>>>>>>>>    directory. This also fails for users' home grown versioning 
systems when
>>>>>>>>    they do things like (slurp versions.txt). More examples here
>>>>>>>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>>>>>>>    (there are probably more, but that's the easiest search). E 
ssentia lly, on
>>>>>>>>    e of le in' s big s elling points is that it's declarative, 
but the fact
>>>>>>>>    that project.clj is actually executed means that it's not 
truly declarative.
>>>>>>>>    4. Many things in lein (gpg, proxies etc) are configured using
>>>>>>>>    environment variables, which are a pain to pass when 
embedding. Cursive
>>>>>>>>    hooks leiningen fairly heavily using robert.hooke t o supply 
values for
>>>>>>>>    these instead of getting them from env vars.
>>>>>>>>    5. Often you need access to the lein tasks, not just core.
>>>>>>>>    Cursive has bundled all of lein for a long time so that it can
access the
>>>>>>>>    repl task. If you don't do this you end up duplicating a lot 
of the code
>>>>>>>>    from the tasks anyway.
>>>>>>>>    6. There are also lots of minor issues - I wanted to hook into
>>>>>>>>    the Aether transfer listener so that I could provide updates 
in the Cursive
>>>>>>>>    UI when libs are downloaded. Currently I do this in Cursive by hooking
>>>>>>>>    leiningen.core.classpath/warn and checking if it looks like a 
"Retrieving"
>>>>>>>>    message and parsing it if so.
>>>>>>>>    7. As Laurent says, lein, even just core, is really big. The
>>>>>>>>    Cursive download is about 3x the size it would be if it didn't
bundle lein.
>>>>>>>>
>>>>>>>> That said, while I understand Laurent's worry about stability,
>>>>>>>> that's never been a problem. I've never received a single bug 
report ab out
>>>>>>>> s tability that was as a result of bundling lein.
>>>>>>>>
>>>>>>>
>>>>>>> I admit that it has been more a fear than an issue.
>>>>>>>
>>>>>>>
>>>>>>>> There is also one significant benefit to bundling lein - it's fast.
>>>>>>>> Starting a JVM process for lein every time Cursive needs to sync the deps
>>>>>>>> is going to be way too slow. This will be particularly true for large
>>>>>>>> multi-module projects where to do everything totally accurately (in
>>>>>>>> particular, the CWD) requires starting a new JVM per module - this is
>>>>>>>> unworkable. I considered caching the last-modified of the project.clj and
>>>>>>>> only re-parsing a particular module if that had changed, but I'd 
need to do
>>>>>>>> that for all possible contributing files which is impo ssible to 
calculate.
>>>>>>>> Apart fr om p rofiles.clj a nd other "offic ial" files which 
might a ffect
>>>>>>>> the result, I'd also need to test the ver sion.txt type files 
from #3, and
>>>>>>>> there's no way I can detect all of those.
>>>>>>>>
>>>>>>>> Since I don't think that stability is actually a major problem, I'm
>>>>>>>> planning to continue running lein in-process but I'm going to isolate its
>>>>>>>> classpath using ShimDandy. I have a prototype of this but it's 
still pretty
>>>>>>>> primitive, and I have a lot of other things going on so it's advancing
>>>>>>>> slowly. I run lein in a shim, and I create the shim's classpath using
>>>>>>>> Aether so I don't actually bundle lein, I pull it in when creating the
>>>>>>>> shim. This means that users can choose the lein version they 
want, and the
>>>>>>>> files are stored in .m2 rather than being downloaded with Cursive.
>>>>>>>>
>>>>>>>
>>>>>>> I had considered this for CCW, but chose to embed leiningen because
>>>>>>> back in time (and still currently in practice, despite the growing
interest
>>>>>>> in boot) leiningen was the only option.
>>>>>>> Having the latest leiningen downloaded at the same time as CCW is
>>>>>>> easier for users, I think. They have to wait once for things to download,
>>>>>>> and then they're good to go.
>>>>>>> By downloading lein lazily, you're at risk for users going offli ne,
>>>>>>> wiping out their ~/.m2/repo sitory (I do this fr om time to time). That 's
>>>>>>> one more moving part, not to be considered too lightly, IMHO, but YMMV.
>>>>>>>
>>>>>>>
>>>>>>>> I'll then throw the shim away when done so memoisation won't be a
>>>>>>>> problem. I can amortize the performance problem by eagerly creating the
>>>>>>>> shim and loading the namespaces into it before I need it, similar
to boot's
>>>>>>>> pod caching. This is much less costly than eagerly starting a JVM and
>>>>>>>> keeping that around until needed.
>>>>>>>>
>>>>>>>
>>>>>>> Have you checked the time required for the JVM itself to start
>>>>>>> versus the time to load leiningen.main and all its dependencies? I think
>>>>>>> I'll do it again, just to check that the ratio proves your point. We might
>>>>>>> be surprised discovering that the JVM startup time is not t he problem,
>>>>>>> leading to the conclusion that reloading everything of lein in a new
>>>>>>> classloader will n ot be that different (performance wise) than starting a
>>>>>>> new JVM.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I'm also going to try to fiddle much less with the lein internals
>>>>>>>> since that has caused problems when upgrading lein. Using this new
>>>>>>>> mechanism I can call the functions representing the lein tasks as if they
>>>>>>>> were being called from main, whic h greatly reduces my dependency on lein
>>>>>>>> internals. I'll still have to hook for gpg/proxies etc, but I'll 
be hooking
>>>>>>>> aether rather than lein for those, which is much more stable since it's
>>>>>>>> basically abandonware at this point - I should have done this ages ago. I
>>>>>>>> can also hook into the transfer listener by hooking aether.
>>>>>>>>
>>>>>>>> My interface to the shim will be to pass it a list of file paths to
>>>>>>>> the project.clj files and eventually a list of profiles, and it 
will return
>>>>>>>> a map containing all the information I need about the projects.
>>>>>>>>
>>>>>>>> The upshot of all this is that this should fix all my current
>>>>>>>> problems except #3, which there's no workaround for. I'll fix the
>>>>>>>> local-repo problem by hooking lein or fiddling with the project.clj, but
>>>>>>>> the only way to fix the slurp problem is to start a JVM per lein project,
>>>>>>>> or convince users not to do that. Or hook slurp, I guess.
>>>>>>>>
>>>>>>>> Laurent, currently I run the REPL withi n Cursive using the
>>>>>>>> trampoline mechanism - IMO you don't have to emulate it, you can just use
>>>>>>>> it directly. Get lein to generate a trampoline file then parse 
that - it's
>>>>>>>> very easy to parse since all the elements always appear in the 
same order.
>>>>>>>> Currently I hook the lein internals to do this, but as described 
above I'm
>>>>>>>> about to start calling lein trampoline directly from the shim.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Colin
>>>>>>>>
>>>>>>>>
>>>>>>>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> OK understood this works as expected: lein update-in : assoc
>>>>>>>>> :eval-in :pprint -- run -m clojure.main
>>>>>>>>> BTW, does t hat mean that it's possible t o emulate TRAMPOLINE by
>>>>>>>>> storing the result o f :pprint (and thus preventing the scenario where
>>>>>>>>> there's 2 JVMs loaded, one for lein itself, and one for the project) ?
>>>>>>>>>
>>>>>>>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>>>>>>>
>>>>>>>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>>>>>>>
>>>>>>>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`;
>>>>>>>>>> this will
>>>>>>>>>> > pprint the form to be run and print the classpath and JVM args
>>>>>>>>>> instead
>>>>>>>>>> > of actually running it.
>>>>>>>>>>
>>>>>>>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>>>>>>>
>>>>>>>>>> -Phil
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Laurent Petit
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Laurent Petit
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Laurent Petit
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Laurent Petit
>>>>
>>>
>>>
>>
>
>
> --
> Laurent Petit
>

Re: [leiningen] Getting leiningen classpath

From:
Laurent Petit
Date:
2015-08-18 @ 19:39
2015-08-18 12:12 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:

> Hmm, very interesting, thanks Laurent! I'll play around with that. That's
> definitely beyond my knowledge of protocols :-)
>

It was beyond mine also yesterday :-)


>
>
> On 17 August 2015 at 23:40, Laurent PETIT <laurent.petit@gmail.com> wrote:
>
>>
>>
>> 2015-08-17 17:54 GMT+02:00 Colin Fleming <colin.mailinglist@gmail.com>:
>>
>>> So I investigated the CWD thing a bit. Setting user.dir doesn't work,
>>> since new File(".") will not respect it. You can do i t using this
>>> 
<https://github.com/jnr/jnr-posix/blob/master/src/main/java/jnr/posix/POSIX.java#L116>
>>> but I reached out to Allan Rohner who used it for lein-daemon and he said:
>>>
>>> changing the CWD works, but it was … weird. Parts of the JVM didn’t
>>>> expect it.
>>>
>>>
>>>
>>>> I would put changing CWD at the level of “dubious hack”. definitely
>>>> avoid if you can for serious work
>>>>
>>>
>>> I think I'll just hook local-repo and perhaps slurp and spit, and just
>>> not support the rest.
>>>
>>
>> In clojure, everything boils down to clojure.java.io.
>> In an ideal world, it would have suffice to overwrite the extension of
>> IOFactory for File. Alas, I can see in Clojure 1.7 that there are places in
>> clojure.java.io where (FileInputStream. f)  (FileOutputStream. f) are
>> called in places where (make-input-stream f) / (make-output-stream f)
>> should have been called.
>>
>> Anyway, it seems that "fixing" IOFactory for java.io.File (and also maybe
>> for completeness do-copy for File) will do the job for slurp, spit, and
>> also for local-repo from pomegr anate.
>>
>> Something like (not tested):
>>
>> (require '[clojure.java.io :as io])
>> (import 'java.io.File)
>> (let [{:keys [make-input-stream make-output-stream] :as prev-impls} (get
>> (:impls io/IOFactory) File)
>>        cwd (System/getProperty "user.dir")
>>         absolute #(if (.isAbsolute %) % (io/file cwd %))]
>>   (extend File
>>     IOFactory (merge prev-impls {:make-input-stream (fn [^File f opts]
>> (make-input-stream (absolute f) opts)) :make-output-stream (fn [^File f
>> opts] (make-output-stream (absolute f) opts))})))
>>
>>
>>
>>>
>>> On 17 August 2015 at 16:58, Colin Fleming <colin.mailinglist@gmail.com>
>>> wrote:
>>>
>>>> On 17 August 2015 at 16:27, Laurent PETIT <laurent.petit@gmail.com>
>>>> wrote:
>>>>
>>>>> It's probably the only option if you want to support concurrency.
>>>>>
>>>>
>>>> Hopefully I won't actually have to support concurrency - see below.
>>>>
>>>>
>>>> Wow, *that* big. Indeed.
>>>>
>>>>
>>>> Yeah - that's a very big one but I'm aware of a lot of Cursive users
>>>> with projects with a couple of dozen modules.
>>>>
>>>>
>>>>> So instead of starting 66 JVMs, you intend to create (and immediately
>>>>> ditch after usage) 66 shims.
>>>>>
>>>>
>>>> Initially no, I'll just create a single shim and use that for all
>>>> modules, so it will be very fast. There's not really a big advantage to
>>>> multiple shims AFAIK, since memoisation should be ok for a single
>>>> cross-project operation. I think the worst offenders are the dependencies
>>>> and the profiles, both of which can be memoised for a single call across
>>>> the whole project. If I can do the user.dir hack then I can do it for each
>>>> project one by one within the same shim as I sync each one.
>>>>
>>>>
>>>>
>>>>> While the single-project case is probably by far the most common, I
>>>>>> still need to support the large project case properly and I can't see a way
>>>>>> to do that with no compromises at all. I could work around the worst of it
>>>>>> with some complex configuration saying how many JVMs to start up, and
>>>>>> specifying which modules should use which JVMs, but i t starts to sound
>>>>>> like a nightmare at that point, both for me and the user.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On 17 August 2015 at 13:55, Laurent PETIT <laurent.petit@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2015-08-17 13:40 GMT+02:00 Colin Fleming <
>>>>>>> colin.mailinglist@gmail.com>:
>>>>>>>
>>>>>>>> Thanks for the thoughts, Laurent - very interesting. Do you have a
>>>>>>>> link to the CCW source that changes the CWD property? I read about doing
>>>>>>>> that a while back, but since IntelliJ doesn't isolate classloaders like
>>>>>>>> Eclipse does I couldn't do it. Now that I'm going to use ShimDandy I can
>>>>>>>> probably do that. If you isolate classloaders, are the system properties
>>>>>>>> then isolated too?
>>>>>>>>
>>>>>>>
>>>>>>> I won't be able to do so ... because I was wrong. There's no trace
>>>>>>> of "user.dir" in CCW codebase. So I may have thought about it in 
my shower,
>>>>>>> but neve r did it in p ract ice ...
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> In terms of lazily downloading Leiningen, it'll happen as soon as
>>>>>>>> they open a lein project, so that's generally almost immediately. If
>>>>>>>> they're offline then they'll have problems, right, but at that point
>>>>>>>> they'll almost certainly need to be connected to download deps anyway so
>>>>>>>> I'm not sure it'll be a problem. I think the most important thing is that
>>>>>>>> the user doesn't have to explicitly download it separately from Cursive,
>>>>>>>> particularly on Windows where that seems to be pretty painful.
>>>>>>>>
>>>>>>>
>>>>>>> Fair enough
>>>>>>>
>>>>>>>
>>>>>>>> WRT JVM startup time, I'm not worried abo ut that - that is dwarfed
>>>>>>>> by the ti me to load th e Clojure/lein namespa ces. But by doing that
>>>>>>>> in-process I can start a shim and load the namespaces into it 
ahead of time
>>>>>>>> so it's ready when I need it and will work immediately. To do that with a
>>>>>>>> standalone JVM would mean starting up an external JVM eagerly and
having it
>>>>>>>> hanging around - I'm more worried about memory use than time for that.
>>>>>>>> Obviously that will still occupy memory in the IntelliJ heap but 
I'm paying
>>>>>>>> that price now anyway - it will be more due to isolation, but still less
>>>>>>>> than the 300mb that a JVM takes up.
>>>>>>>>
>>>>>>>
>>>>>>> Well, yes, always keeping ahead of time one classloader is certainly
>>>>>>> easier and more light-weight for the user's computer memory.
>>>>>>> But still, interoperating through a new JVM rather than through
>>>>>>> classloaders appeals more to me:
>>>>>>> - fixes all the problems you listed, without compromise
>>>>>>> - introduces the problem of more RAM if a pool of JVM processes is
>>>>>>> maintained. This could be leveraged by letting t he user choose the pool
>>>>>>> size (setting it to 0 would be friendly for computers with low memory.
>>>>>>> Setting it to 5 would be a boost for users with modern computers and multi
>>>>>>> module projects. Setting it to 1 by default would probably be a good
>>>>>>> average).
>>>>>>> - more flexible to adapt to boot, planck or similar for
>>>>>>> clojurescript, or plain processes.
>>>>>>>
>>>>>>> I think I will stick to my idea of moving from embedded to external.
>>>>>>> Thanks for sharing the thoughts!
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 17 August 2015 at 13:21, Laurent PETIT <laurent.petit@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Mor e thoughts, Colin:
>>>>>>>>
>>>>>>>> 2015-08-17 12:37 GMT+02:00 Colin Fleming <
>>>>>>>> colin.mailinglist@gmail.com>:
>>>>>>>>
>>>>>>>>> Curiously enough, I'm also about to totally change the way I
>>>>>>>>> integrate Leiningen in Cursive.
>>>>>>>>>
>>>>>>>>> To answer Phil's question about embedding leiningen-core, there
>>>>>>>>> are many many disadvantages to doing so:
>>>>>>>>>
>>>>>>>>>    1. You're tied to a specific version of lein. Currently I
>>>>>>>>>    bundle 2.5.0 - I upgraded to 2.5.1 from 2.4.3 (IIRC) and 
several users
>>>>>>>>>    encountered bugs so I downgraded to 2.5.0. Some users curren 
tly experience
>>>>>>>>>    bugs with 2 .5.0, but they're easier to work around. Anyone 
using older
>>>>>>>>>    versions on th eir project is ou t of luck.
>>>>>>>>>    2. Lein incorrectly memoises many things internally, which is
>>>>>>>>>    a gigantic pain. Currently users have to restart Cursive 
every time they
>>>>>>>>>    change profiles.clj, and additionally they also have to be 
aware of the bug
>>>>>>>>>    to know to do that. I've also lost countless hours debugging 
problems like
>>>>>>>>>    #1378 <https://github.com/technomancy/leiningen/issues/1378>.
>>>>>>>>>    A more general report is #1768. This is slated to be fixed in
>>>>>>>>>    lein 3.0, but a) who knows when that will be and b) I'm still
at the mercy
>>>>>>>>>    of plugins doing the same thing anyway.
>>>>>>>>>    3. Java cannot change the CWD of the current process, which
>>>>>>>>>    means that local-repo always appears somewhere in the IntelliJ bin
>>>>>>>>>    directory. This also fails for users' home grown versioning 
systems when
>>>>>>>>>    they do things like (slurp versions.txt). More examples here
>>>>>>>>>    <https://github.com/cursiveclojure/cursive/search?q=cwd&type=Issues>
>>>>>>>>>    (there are probably more, but that's the easiest search). E 
ssentia lly, on
>>>>>>>>>    e of le in' s big s elling points is that it's declarative, 
but the fact
>>>>>>>>>    that project.clj is actually executed means that it's not 
truly declarative.
>>>>>>>>>    4. Many things in lein (gpg, proxies etc) are configured using
>>>>>>>>>    environment variables, which are a pain to pass when 
embedding. Cursive
>>>>>>>>>    hooks leiningen fairly heavily using robert.hooke t o supply 
values for
>>>>>>>>>    these instead of getting them from env vars.
>>>>>>>>>    5. Often you need access to the lein tasks, not just core.
>>>>>>>>>    Cursive has bundled all of lein for a long time so that it 
can access the
>>>>>>>>>    repl task. If you don't do this you end up duplicating a lot 
of the code
>>>>>>>>>    from the tasks anyway.
>>>>>>>>>    6. There are also lots of minor issues - I wanted to hook into
>>>>>>>>>    the Aether transfer listener so that I could provide updates 
in the Cursive
>>>>>>>>>    UI when libs are downloaded. Currently I do this in Cursive 
by hooking
>>>>>>>>>    leiningen.core.classpath/warn and checking if it looks like a
"Retrieving"
>>>>>>>>>    message and parsing it if so.
>>>>>>>>>    7. As Laurent says, lein, even just core, is really big. The
>>>>>>>>>    Cursive download is about 3x the size it would be if it 
didn't bundle lein.
>>>>>>>>>
>>>>>>>>> That said, while I understand Laurent's worry about stability,
>>>>>>>>> that's never been a problem. I've never received a single bug 
report ab out
>>>>>>>>> s tability that was as a result of bundling lein.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I admit that it has been more a fear than an issue.
>>>>>>>>
>>>>>>>>
>>>>>>>>> There is also one significant benefit to bundling lein - it's
>>>>>>>>> fast. Starting a JVM process for lein every time Cursive needs 
to sync the
>>>>>>>>> deps is going to be way too slow. This will be particularly true
for large
>>>>>>>>> multi-module projects where to do everything totally accurately (in
>>>>>>>>> particular, the CWD) requires starting a new JVM per module - this is
>>>>>>>>> unworkable. I considered caching the last-modified of the 
project.clj and
>>>>>>>>> only re-parsing a particular module if that had changed, but I'd
need to do
>>>>>>>>> that for all possible contributing files which is impo ssible to
calculate.
>>>>>>>>> Apart fr om p rofiles.clj a nd other "offic ial" files which 
might a ffect
>>>>>>>>> the result, I'd also need to test the ver sion.txt type files 
from #3, and
>>>>>>>>> there's no way I can detect all of those.
>>>>>>>>>
>>>>>>>>> Since I don't think that stability is actually a major problem,
>>>>>>>>> I'm planning to continue running lein in-process but I'm going 
to isolate
>>>>>>>>> its classpath using ShimDandy. I have a prototype of this but it's still
>>>>>>>>> pretty primitive, and I have a lot of other things going on so it's
>>>>>>>>> advancing slowly. I run lein in a shim, and I create the shim's 
classpath
>>>>>>>>> using Aether so I don't actually bundle lein, I pull it in when creating
>>>>>>>>> the shim. This means that users can choose the lein version they
want, and
>>>>>>>>> the files are stored in .m2 rather than being downloaded with Cursive.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I had considered this for CCW, but chose to embed leiningen because
>>>>>>>> back in time (and still currently in practice, despite the 
growing interest
>>>>>>>> in boot) leiningen was the only option.
>>>>>>>> Having the latest leiningen downloaded at the same time as CCW is
>>>>>>>> easier for users, I think. They have to wait once for things to download,
>>>>>>>> and then they're good to go.
>>>>>>>> By downloading lein lazily, you're at risk for users going offli
>>>>>>>> ne, wiping out their ~/.m2/repo sitory (I do this fr om time to 
time). That
>>>>>>>> 's one more moving part, not to be considered too lightly, IMHO, 
but YMMV.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I'll then throw the shim away when done so memoisation won't be a
>>>>>>>>> problem. I can amortize the performance problem by eagerly creating the
>>>>>>>>> shim and loading the namespaces into it before I need it, 
similar to boot's
>>>>>>>>> pod caching. This is much less costly than eagerly starting a JVM and
>>>>>>>>> keeping that around until needed.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Have you checked the time required for the JVM itself to start
>>>>>>>> versus the time to load leiningen.main and all its dependencies? I think
>>>>>>>> I'll do it again, just to check that the ratio proves your point.
We might
>>>>>>>> be surprised discovering that the JVM startup time is not t he problem,
>>>>>>>> leading to the conclusion that reloading everything of lein in a new
>>>>>>>> classloader will n ot be that different (performance wise) than 
starting a
>>>>>>>> new JVM.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm also going to try to fiddle much less with the lein internals
>>>>>>>>> since that has caused problems when upgrading lein. Using this new
>>>>>>>>> mechanism I can call the functions representing the lein tasks 
as if they
>>>>>>>>> were being called from main, whic h greatly reduces my 
dependency on lein
>>>>>>>>> internals. I'll still have to hook for gpg/proxies etc, but I'll
be hooking
>>>>>>>>> aether rather than lein for those, which is much more stable since it's
>>>>>>>>> basically abandonware at this point - I should have done this 
ages ago. I
>>>>>>>>> can also hook into the transfer listener by hooking aether.
>>>>>>>>>
>>>>>>>>> My interface to the shim will be to pass it a list of file paths
>>>>>>>>> to the project.clj files and eventually a list of profiles, and it will
>>>>>>>>> return a map containing all the information I need about the projects.
>>>>>>>>>
>>>>>>>>> The upshot of all this is that this should fix all my current
>>>>>>>>> problems except #3, which there's no workaround for. I'll fix the
>>>>>>>>> local-repo problem by hooking lein or fiddling with the project.clj, but
>>>>>>>>> the only way to fix the slurp problem is to start a JVM per lein
project,
>>>>>>>>> or convince users not to do that. Or hook slurp, I guess.
>>>>>>>>>
>>>>>>>>> Laurent, currently I run the REPL withi n Cursive using the
>>>>>>>>> trampoline mechanism - IMO you don't have to emulate it, you can
just use
>>>>>>>>> it directly. Get lein to generate a trampoline file then parse 
that - it's
>>>>>>>>> very easy to parse since all the elements always appear in the 
same order.
>>>>>>>>> Currently I hook the lein internals to do this, but as described
above I'm
>>>>>>>>> about to start calling lein trampoline directly from the shim.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Colin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 15 August 2015 at 03:50, Laurent PETIT <laurent.petit@gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> OK understood this works as expected: lein update-in : assoc
>>>>>>>>>> :eval-in :pprint -- run -m clojure.main
>>>>>>>>>> BTW, does t hat mean that it's possible t o emulate TRAMPOLINE by
>>>>>>>>>> storing the result o f :pprint (and thus preventing the scenario where
>>>>>>>>>> there's 2 JVMs loaded, one for lein itself, and one for the project) ?
>>>>>>>>>>
>>>>>>>>>> 2015-08-15 3:04 GMT+02:00 Phil Hagelberg <phil@hagelb.org>:
>>>>>>>>>>
>>>>>>>>>>> Phil Hagelberg <phil@hagelb.org> writes:
>>>>>>>>>>>
>>>>>>>>>>> > You can try `lein assoc :eval-in :pprint run -m clojure.main`;
>>>>>>>>>>> this will
>>>>>>>>>>> > pprint the form to be run and print the classpath and JVM args
>>>>>>>>>>> instead
>>>>>>>>>>> > of actually running it.
>>>>>>>>>>>
>>>>>>>>>>> Wow, I'm rusty. Use the update-in task instead; it's built-in.
>>>>>>>>>>>
>>>>>>>>>>> -Phil
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Laurent Petit
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Laurent Petit
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Laurent Petit
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Laurent Petit
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Laurent Petit
>>
>
>


-- 
Laurent Petit