librelist archives

« back to archive

What's next for Leiningen

What's next for Leiningen

From:
Phil Hagelberg
Date:
2013-08-08 @ 04:44
Hello folks.

Things have been settling down for a while with a 2.3.0 release around
the corner containing mostly minor improvements, but I thought I'd just
give an update on my thoughts of what's coming up in Leiningen-land and
what *could* happen if interested folks wanted to take up the chase.

# Launch time

The ancient nemesis of Leiningen users, JVM boot time. We've gotten a
lot of mileage out of various tuning options, but I don't think there's
much more we can do in this direction without trying new tactics. IMO
the most promising strategy is keeping JVMs around.

We actually already have the ability to keep the project JVM resident
using `:eval-in :leiningen`, but there's no way to invoke Leiningen
tasks without paying the penalty for Leiningen's own JVM. Anyhow this
feature is pretty obscure, so it'd be great to get more users testing
it. The simplest way would be to revive the `lein interactive` task from
1.x. This could probably be done as a plugin first and then merged into
Leiningen once it matures.

A fancier way to do it would be to write an nrepl client in a
quick-launching language and have it send tasks to Leiningen and shuttle
input/output to the user. This is part of what Jark does using OCaml,
(http://icylisper.github.io/jark) but Jark also does a number of other
things that don't strike me as necessary, and Jark implements the old
nrepl protocol from before it was completely rewritten, so it can't
communicate with Leiningen as is. Even so, I think that approach is
promising.

# OS Packaging

We have a GSoC student helping out with packaging Leiningen's own
dependencies for Debian, and it looks like it's coming along nicely!
Reports indicate we're only a few weeks away from a package for
Leiningen 2. However, the .rpm packaging for Leiningen is still on
1.7.1, so there's a chance for Fedora users to step up.

# Native dependencies

Leiningen supports using native dependencies, but the documentation for
it isn't that great. Also it doesn't have any support for *making*
native dependencies. None of the core Leiningen team use native
dependencies, so this hasn't gotten much attention, but it's definitely
a big missing feature.

# 3.0

There are a bunch of backwards-incompatible changes we've been saving up
for the next major release bump. I don't have any concrete timeline on
these, but I'm grouping them together because they're all things we
can't implement in 2.x:

## Profile merge

The profile merge logic is currently really hairy. It's very difficult
to understand fully in order to make simple changes, and it needs to be
drastically simpler. A re-write is probably in order. The requirements
here right now are fairly specific and exacting, (in particular the need
to be able to "un-merge" a profile leads to all sorts of headaches) but
for a 3.0 release we can revisit what's strictly required and balance it
against the confusing behaviour of the current implementation to see if
a better compromise could be reached.

## Test selectors

Right now the test task implements test selectors basically by
monkeypatching clojure.test, which results in a really hairy
codebase. Test selectors belong as part of the testing library, so for
Leiningen 3 I would much rather delegate this elsewhere.

## Jar :filespecs

The syntax for declaring jar filespecs right now is really ugly and
complicated for no good reason. It should get totally replaced for 3.0.
This actually could be done as part of 2.x, but we can't remove the
existing version yet, so they'd have to be implemented side-by-side.

# Isolated dependency trees

This one is a bit crazier, but I thought I'd throw it out
there. Conflicting dependency sets are the bane of large projects in any
runtime, but the JVM provides the necessary mechanisms by which "branches"
of your dependency trees could be isolated into their own world via
classloaders, allowing multiple versions of the same libraries to
co-exist in the same JVM. It might get a bit hairy, but it open doors
for large projects that currently might not be able to pull in certain
libraries they want.


Anyway, those are just a few things on my mind right now. If any of
these sound like something you might like to help out with, I can go
into more detail and provide starting points.

cheers,
Phil

Re: [leiningen] What's next for Leiningen

From:
Colin Yates
Date:
2013-08-08 @ 06:03
Unfortunately I don't have anytime to contribute,  but I just wanted to say
a huge thanks to you and everyone who has made lein such an amazingly
powerful yet easily accessible tool.

Now that's buttered you up :), for my team the "jark-esque" solution is the
highest priority by a mile.

Regarding the platform specific build; I find that less useful as typically
I need to use a different version of the jdk that is available in the
repository, so we have a number of jdks in our local ~/Apps.  If the lein
package specified a jdk then we couldn't use it and if it didn't thenthat
would be weird.  I really like the flexibility of the self-downloading and
updating script approach do please don't let that go away.

But again, having used the other major build tools, for me lein was a
breath of fresh air.

Thanks,

Coo
On 8 Aug 2013 05:49, "Phil Hagelberg" <phil@hagelb.org> wrote:

>
> Hello folks.
>
> Things have been settling down for a while with a 2.3.0 release around
> the corner containing mostly minor improvements, but I thought I'd just
> give an update on my thoughts of what's coming up in Leiningen-land and
> what *could* happen if interested folks wanted to take up the chase.
>
> # Launch time
>
> The ancient nemesis of Leiningen users, JVM boot time. We've gotten a
> lot of mileage out of various tuning options, but I don't think there's
> much more we can do in this direction without trying new tactics. IMO
> the most promising strategy is keeping JVMs around.
>
> We actually already have the ability to keep the project JVM resident
> using `:eval-in :leiningen`, but there's no way to invoke Leiningen
> tasks without paying the penalty for Leiningen's own JVM. Anyhow this
> feature is pretty obscure, so it'd be great to get more users testing
> it. The simplest way would be to revive the `lein interactive` task from
> 1.x. This could probably be done as a plugin first and then merged into
> Leiningen once it matures.
>
> A fancier way to do it would be to write an nrepl client in a
> quick-launching language and have it send tasks to Leiningen and shuttle
> input/output to the user. This is part of what Jark does using OCaml,
> (http://icylisper.github.io/jark) but Jark also does a number of other
> things that don't strike me as necessary, and Jark implements the old
> nrepl protocol from before it was completely rewritten, so it can't
> communicate with Leiningen as is. Even so, I think that approach is
> promising.
>
> # OS Packaging
>
> We have a GSoC student helping out with packaging Leiningen's own
> dependencies for Debian, and it looks like it's coming along nicely!
> Reports indicate we're only a few weeks away from a package for
> Leiningen 2. However, the .rpm packaging for Leiningen is still on
> 1.7.1, so there's a chance for Fedora users to step up.
>
> # Native dependencies
>
> Leiningen supports using native dependencies, but the documentation for
> it isn't that great. Also it doesn't have any support for *making*
> native dependencies. None of the core Leiningen team use native
> dependencies, so this hasn't gotten much attention, but it's definitely
> a big missing feature.
>
> # 3.0
>
> There are a bunch of backwards-incompatible changes we've been saving up
> for the next major release bump. I don't have any concrete timeline on
> these, but I'm grouping them together because they're all things we
> can't implement in 2.x:
>
> ## Profile merge
>
> The profile merge logic is currently really hairy. It's very difficult
> to understand fully in order to make simple changes, and it needs to be
> drastically simpler. A re-write is probably in order. The requirements
> here right now are fairly specific and exacting, (in particular the need
> to be able to "un-merge" a profile leads to all sorts of headaches) but
> for a 3.0 release we can revisit what's strictly required and balance it
> against the confusing behaviour of the current implementation to see if
> a better compromise could be reached.
>
> ## Test selectors
>
> Right now the test task implements test selectors basically by
> monkeypatching clojure.test, which results in a really hairy
> codebase. Test selectors belong as part of the testing library, so for
> Leiningen 3 I would much rather delegate this elsewhere.
>
> ## Jar :filespecs
>
> The syntax for declaring jar filespecs right now is really ugly and
> complicated for no good reason. It should get totally replaced for 3.0.
> This actually could be done as part of 2.x, but we can't remove the
> existing version yet, so they'd have to be implemented side-by-side.
>
> # Isolated dependency trees
>
> This one is a bit crazier, but I thought I'd throw it out
> there. Conflicting dependency sets are the bane of large projects in any
> runtime, but the JVM provides the necessary mechanisms by which "branches"
> of your dependency trees could be isolated into their own world via
> classloaders, allowing multiple versions of the same libraries to
> co-exist in the same JVM. It might get a bit hairy, but it open doors
> for large projects that currently might not be able to pull in certain
> libraries they want.
>
>
> Anyway, those are just a few things on my mind right now. If any of
> these sound like something you might like to help out with, I can go
> into more detail and provide starting points.
>
> cheers,
> Phil
>

Re: [leiningen] What's next for Leiningen

From:
Phil Hagelberg
Date:
2013-08-08 @ 17:14
Laurent PETIT writes:

> I too find this fancier idea promising and exciting. If I understand
> it well of course! So if I try to rephrase it, it would be have a
> client/server Leiningen. This is very interesting since, then, having
> Lein as a server would mean equal plugins quality with regards to this
> feature, when it was hard to get with just "Leiningen core as a
> library".

Well I don't see this necessarily becoming the default; a fast-launching
CLI client would be a separate project which might be linked from
Leiningen's readme but wouldn't necessarily be distributed with
Leiningen itself.

But yeah, given that startup time is a big problem, I wouldn't be too
surprised if it became fairly widely used, which would mean more users
who don't have the current "one task run per JVM" model, and bugs in
tasks that rely on that assumption would be more likely to be addressed.

> In your mind, would that be one Leiningen server/JVM per project, or per
> machine?

What I've been doing with lein.el (same idea, but for eshell inside
Emacs) is launching `lein repl :headless` inside ~/.lein and having the
client connect to that. Then `:eval-in :nrepl` can be used as a further
optimization on top of that on a per-project basis, but it should be
orthogonal.

Initially I think it'd be wise to let users handle the lifecycle of the
daemons though. From what I've heard from the Cake guys, having the tool
implicitly launch daemons in the background led to some confusing
situations. We might tackle this later, but I'd rather start with the
basics.

-Phil

Re: [leiningen] What's next for Leiningen

From:
Laurent Petit
Date:
2013-08-10 @ 14:44
2013/8/8 Phil Hagelberg <phil@hagelb.org>:
>
> Laurent PETIT writes:
>
>> I too find this fancier idea promising and exciting. If I understand
>> it well of course! So if I try to rephrase it, it would be have a
>> client/server Leiningen. This is very interesting since, then, having
>> Lein as a server would mean equal plugins quality with regards to this
>> feature, when it was hard to get with just "Leiningen core as a
>> library".
>
> Well I don't see this necessarily becoming the default; a fast-launching
> CLI client would be a separate project which might be linked from
> Leiningen's readme but wouldn't necessarily be distributed with
> Leiningen itself.
>
> But yeah, given that startup time is a big problem, I wouldn't be too
> surprised if it became fairly widely used, which would mean more users
> who don't have the current "one task run per JVM" model, and bugs in
> tasks that rely on that assumption would be more likely to be addressed.

Yeah, that's the bet. The more widely used, the more debugged, that's
that simple.

>> In your mind, would that be one Leiningen server/JVM per project, or per
>> machine?
>
> What I've been doing with lein.el (same idea, but for eshell inside
> Emacs) is launching `lein repl :headless` inside ~/.lein and having the
> client connect to that. Then `:eval-in :nrepl` can be used as a further
> optimization on top of that on a per-project basis, but it should be
> orthogonal.
>
> Initially I think it'd be wise to let users handle the lifecycle of the
> daemons though. From what I've heard from the Cake guys, having the tool
> implicitly launch daemons in the background led to some confusing
> situations. We might tackle this later, but I'd rather start with the
> basics.

Ok, so I'm a little bit lost, here. Presumably I don't have enough
background with how Cake worked, how Jark works, and also don't
totally understand what you mean by "let users handle the lifecycle of
the daemons". If you plan on writing some "white paper / rationale /
specs" whatever on that, I'd be more than happy to help review it.
Right now, we might as well be talking about different things that we
may not notice ! :-D

Re: [leiningen] What's next for Leiningen

From:
Phil Hagelberg
Date:
2013-08-08 @ 16:25
Colin Yates writes:

> Regarding the platform specific build; I find that less useful as typically
> I need to use a different version of the jdk that is available in the
> repository, so we have a number of jdks in our local ~/Apps.  If the lein
> package specified a jdk then we couldn't use it and if it didn't thenthat
> would be weird.  I really like the flexibility of the self-downloading and
> updating script approach do please don't let that go away.

Yeah, the manual install isn't going anywhere; don't worry. =)

Even so, I don't think the OS-packaged version would prevent you from
using your own JDK, either by adjusting $PATH or setting $JAVA_CMD.

-Phil

Re: [leiningen] What's next for Leiningen

From:
Wolodja Wentland
Date:
2013-08-08 @ 17:08
On Thu, Aug 08, 2013 at 09:25 -0700, Phil Hagelberg wrote:
> Colin Yates writes:

> > Regarding the platform specific build; I find that less useful as typically
> > I need to use a different version of the jdk that is available in the
> > repository, so we have a number of jdks in our local ~/Apps.  If the lein
> > package specified a jdk then we couldn't use it and if it didn't thenthat
> > would be weird.  I really like the flexibility of the self-downloading and
> > updating script approach do please don't let that go away.

> Yeah, the manual install isn't going anywhere; don't worry. =)

> Even so, I don't think the OS-packaged version would prevent you from
> using your own JDK, either by adjusting $PATH or setting $JAVA_CMD.

No, and why should it? It simply uses $JAVA_CMD with a fallback on "java". On
Debian one can easily adjust the JDK in use when "java" is called via
update-java-alternatives, but nothing prevents the user from setting JAVA_CMD
to something that has been manually installed either.

The package will, however, pull in the default JRE on Debian (currently
openjdk 7) so users have to have at least that installed.
-- 
Wolodja <babilen@gmail.com>

4096R/CAF14EFC
081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC

Re: [leiningen] What's next for Leiningen

From:
Phil Hagelberg
Date:
2013-08-30 @ 03:13
Phil Hagelberg writes:

> A fancier way to do it would be to write an nrepl client in a
> quick-launching language and have it send tasks to Leiningen and shuttle
> input/output to the user. This is part of what Jark does using OCaml,

Last week I hacked this together:

https://github.com/technomancy/grenchman

It's still pretty rudimentary, but once you spin up a Leiningen JVM and
a project JVM, it can allow you to eval Clojure code from the command
line in just a couple milliseconds.

There are plenty of gotchas. The repl experience it provides is still
very primitive, and there are still a few bugs in the I/O logic. But I
think it's gotten to the point where it could be useful to folks.

You can build from source by following the instructions in the readme,
and I've compiled a native Linux binary for download here:

    http://p.hagelb.org/grench-linux-0.1.0
    http://p.hagelb.org/grench-linux-0.1.0.asc (signature)

Please let me know how it works if you get a chance to try it out. If
you find any new bugs please report them here,
https://github.com/technomancy/grenchman/issues and if you know OCaml
(or are interested in learning it) I would be thrilled to get some help.

thanks,
Phil

Re: [leiningen] What's next for Leiningen

From:
Phil Hagelberg
Date:
2013-08-30 @ 05:03
Phil Hagelberg writes:
> It's still pretty rudimentary, but once you spin up a Leiningen JVM and
> a project JVM, it can allow you to eval Clojure code from the command
> line in just a couple milliseconds.

Did I say a couple milliseconds? I meant a couple hundred
milliseconds. Hope that's not too disappointing.

-Phil

Re: [leiningen] What's next for Leiningen

From:
Laurent Petit
Date:
2013-08-08 @ 14:01
Le jeudi 8 août 2013, Phil Hagelberg a écrit :

>
> Hello folks.
>
> Things have been settling down for a while with a 2.3.0 release around
> the corner containing mostly minor improvements, but I thought I'd just
> give an update on my thoughts of what's coming up in Leiningen-land and
> what *could* happen if interested folks wanted to take up the chase.
>
> # Launch time
>
> The ancient nemesis of Leiningen users, JVM boot time. We've gotten a
> lot of mileage out of various tuning options, but I don't think there's
> much more we can do in this direction without trying new tactics. IMO
> the most promising strategy is keeping JVMs around.
>
> We actually already have the ability to keep the project JVM resident
> using `:eval-in :leiningen`, but there's no way to invoke Leiningen
> tasks without paying the penalty for Leiningen's own JVM. Anyhow this
> feature is pretty obscure, so it'd be great to get more users testing
> it. The simplest way would be to revive the `lein interactive` task from
> 1.x. This could probably be done as a plugin first and then merged into
> Leiningen once it matures.
>
> A fancier way to do it would be to write an nrepl client in a
> quick-launching language and have it send tasks to Leiningen and shuttle
> input/output to the user. This is part of what Jark does using OCaml,
> (http://icylisper.github.io/jark) but Jark also does a number of other
> things that don't strike me as necessary, and Jark implements the old
> nrepl protocol from before it was completely rewritten, so it can't
> communicate with Leiningen as is. Even so, I think that approach is
> promising.


I too find this fancier idea promising and exciting. If I understand it
well of course! So if I try to rephrase it, it would be have a
client/server Leiningen. This is very interesting since, then, having Lein
as a server would mean equal plugins quality with regards to this feature,
when it was hard to get with just "Leiningen core as a library". And I'd be
more than willing to make the required changes to counterclockwise to
support this new way of doing things.
Reusing nrepl stuff also seems interesting, since existing tools have it in
place and working right now.

In your mind, would that be one Leiningen server/JVM per project, or per
machine?

>
> # OS Packaging
>
> We have a GSoC student helping out with packaging Leiningen's own
> dependencies for Debian, and it looks like it's coming along nicely!
> Reports indicate we're only a few weeks away from a package for
> Leiningen 2. However, the .rpm packaging for Leiningen is still on
> 1.7.1, so there's a chance for Fedora users to step up.
>
> # Native dependencies
>
> Leiningen supports using native dependencies, but the documentation for
> it isn't that great. Also it doesn't have any support for *making*
> native dependencies. None of the core Leiningen team use native
> dependencies, so this hasn't gotten much attention, but it's definitely
> a big missing feature.
>
> # 3.0
>
> There are a bunch of backwards-incompatible changes we've been saving up
> for the next major release bump. I don't have any concrete timeline on
> these, but I'm grouping them together because they're all things we
> can't implement in 2.x:
>
> ## Profile merge
>
> The profile merge logic is currently really hairy. It's very difficult
> to understand fully in order to make simple changes, and it needs to be
> drastically simpler. A re-write is probably in order. The requirements
> here right now are fairly specific and exacting, (in particular the need
> to be able to "un-merge" a profile leads to all sorts of headaches) but
> for a 3.0 release we can revisit what's strictly required and balance it
> against the confusing behaviour of the current implementation to see if
> a better compromise could be reached.
>
> ## Test selectors
>
> Right now the test task implements test selectors basically by
> monkeypatching clojure.test, which results in a really hairy
> codebase. Test selectors belong as part of the testing library, so for
> Leiningen 3 I would much rather delegate this elsewhere.
>
> ## Jar :filespecs
>
> The syntax for declaring jar filespecs right now is really ugly and
> complicated for no good reason. It should get totally replaced for 3.0.
> This actually could be done as part of 2.x, but we can't remove the
> existing version yet, so they'd have to be implemented side-by-side.
>
> # Isolated dependency trees
>
> This one is a bit crazier, but I thought I'd throw it out
> there. Conflicting dependency sets are the bane of large projects in any
> runtime, but the JVM provides the necessary mechanisms by which "branches"
> of your dependency trees could be isolated into their own world via
> classloaders, allowing multiple versions of the same libraries to
> co-exist in the same JVM. It might get a bit hairy, but it open doors
> for large projects that currently might not be able to pull in certain
> libraries they want.
>
>
> Anyway, those are just a few things on my mind right now. If any of
> these sound like something you might like to help out with, I can go
> into more detail and provide starting points.
>
> cheers,
> Phil
>