librelist archives

« back to archive

a couple of "lein run" related questions

a couple of "lein run" related questions

From:
Lee Spector
Date:
2013-05-29 @ 21:20
I'm a leninigen non-power-user and new to the leiningen list... & I hope 
these aren't FAQs (I did look and didn't see them anywhere)...

- Is there a good reason that "lein new" in the default, just-installed 
state doesn't put a definition of -main in core.clj and a :main 
specification in project.clj so that "lein run" will work without having 
to figure those things out? I've found myself having to explain these 
things to several people lately, and it seems to me that they should just 
be in the default new project template. I realize that one *might* want 
*lots* of things in project.clj, and I do see the rather complete 
(overwhelming) project.clj on the website, but it seems to me like 
including the minimal stuff to get  "lein run" to work should be in the 
default.

- I've heard that "lein run" uses command-line arguments to its java call 
that attempt to minimize startup time, but that this may have substantial 
performance costs once the process begins running (e.g. because JIT 
compilation is delayed or avoided entirely). I realize that I could have 
more complete control by calling java myself, presumably after "lein 
uberjar" or something like that, but as a non-java/lein-reliant person I'd
love to have a lein task that was biased in the opposite way: run the 
project with settings likely to have the highest performance. Does this 
already exist? If not, then might it? I don't have an opinion about 
whether this would be an option to "lein run" or a separate command, or 
whatever, but I'd love to be able to do high performance runs from a "lein
run"-like call.

Sorry again if these are naive or FAQs, and I'd appreciate any pointers.

Thanks,

 -Lee

Re: [leiningen] a couple of "lein run" related questions

From:
Jean Niklas L'orange
Date:
2013-05-29 @ 23:49
Hello Lee,

On 29 May 2013 23:20, Lee Spector <lspector@hampshire.edu> wrote:

> - Is there a good reason that "lein new" in the default, [...]
>

Well, `lein new` is using the default template, which is equivalent to the
library template as far as I can remember. As the most common thing to do
is to create a library, you would usually not want a main class
(:gen-class) nor an entrypoint to the library (:main) by default.

The `lein new app appname` takes care of that and makes a new app with the
name "appname" for you, with all the preliminaries setup correctly. `lein
run` runs out of the box there.


> - I've heard that "lein run" uses command-line arguments to its java call
> that attempt to minimize startup time, but that this may have substantial
> performance costs [...]


If you want faster performance for `lein run`, then the best idea would
probably be to run `lein trampoline run` instead.

That being said, I would like to emphasize that lein is not designed as a
"program starter": If you're running a server, system or a program, you
should (uber)jar or (uber)war the system before using it in production.


We should perhaps make it easier to find this kind of information? An
introduction with a small understanding of how lein works plus a small set
of beginner tips should probably cover these cases? `lein help new` and
`lein help run` does mention these things though.

-- 
Regards,
Jean Niklas L'orange

Re: [leiningen] a couple of "lein run" related questions

From:
Lee Spector
Date:
2013-05-30 @ 00:50
Thanks so much Andy and Jean!

"lein new app myproj" does exactly what I was asking for in my first 
question, so that is completely solved. 

tl;dr FWIW (just illustrating a cultural difference I guess; I'm an 
academic researcher/teacher with a Lispy background): In my universe the 
most common thing is not to create a library, it's to write a program that
will be run by itself (possibly using other libraries). So I would have 
expected "lein new" to do what "lein new app" does, and something like 
"lein new lib" to do what "lein new" does. I will adjust :-). Also, just 
FWIW, I read "app" as meaning something like "create a double-clickable 
desktop application with a GUI and all necessary libraries built in," 
which isn't what "lein new app" does (or at least I don't think so!).

On the second question, "lein trampoline run" sounds like the solution, 
and if so then that's fantastic. Will that eliminate any performance 
issues that would stem from a "lein new" launch?

I'm confused by this:

> That being said, I would like to emphasize that lein is not designed as 
a "program starter": If you're running a server, system or a program, you 
should (uber)jar or (uber)war the system before using it in production.

I thought that the point of "lein run" was to start programs... isn't it? 
I'm not running a server, and I'm not sure what "in production" means in 
my context... I'm writing programs mostly for academic research purposes 
and I want to launch them and have them run fast.

Thanks!!

 -Lee



On May 29, 2013, at 7:49 PM, Jean Niklas L'orange wrote:

> Hello Lee,
> 
> On 29 May 2013 23:20, Lee Spector <lspector@hampshire.edu> wrote:
> - Is there a good reason that "lein new" in the default, [...]
> 
> Well, `lein new` is using the default template, which is equivalent to 
the library template as far as I can remember. As the most common thing to
do is to create a library, you would usually not want a main class 
(:gen-class) nor an entrypoint to the library (:main) by default. 
> 
> The `lein new app appname` takes care of that and makes a new app with 
the name "appname" for you, with all the preliminaries setup correctly. 
`lein run` runs out of the box there.
>  
> - I've heard that "lein run" uses command-line arguments to its java 
call that attempt to minimize startup time, but that this may have 
substantial performance costs [...]
> 
> If you want faster performance for `lein run`, then the best idea would 
probably be to run `lein trampoline run` instead.
> 
> That being said, I would like to emphasize that lein is not designed as 
a "program starter": If you're running a server, system or a program, you 
should (uber)jar or (uber)war the system before using it in production.
> 
> 
> We should perhaps make it easier to find this kind of information? An 
introduction with a small understanding of how lein works plus a small set
of beginner tips should probably cover these cases? `lein help new` and 
`lein help run` does mention these things though.
> 
> -- 
> Regards,
> Jean Niklas L'orange

--
Lee Spector, Professor of Computer Science
Cognitive Science, Hampshire College
893 West Street, Amherst, MA 01002-3359
lspector@hampshire.edu, http://hampshire.edu/lspector/
Phone: 413-559-5352, Fax: 413-559-5438

Re: [leiningen] a couple of "lein run" related questions

From:
Andy Fingerhut
Date:
2013-05-29 @ 23:40
I don't know the history of lein new, but try "lein new app myproj", and
"lein help new" for some details of other options.  "lein new app myproj"
should create a new project named "myproj" in a directory with that name,
and it has :main defined in project.clj, and a -main defined in the initial
core.clj file.

Andy


On Wed, May 29, 2013 at 2:20 PM, Lee Spector <lspector@hampshire.edu> wrote:

>
> I'm a leninigen non-power-user and new to the leiningen list... & I hope
> these aren't FAQs (I did look and didn't see them anywhere)...
>
> - Is there a good reason that "lein new" in the default, just-installed
> state doesn't put a definition of -main in core.clj and a :main
> specification in project.clj so that "lein run" will work without having to
> figure those things out? I've found myself having to explain these things
> to several people lately, and it seems to me that they should just be in
> the default new project template. I realize that one *might* want *lots* of
> things in project.clj, and I do see the rather complete (overwhelming)
> project.clj on the website, but it seems to me like including the minimal
> stuff to get  "lein run" to work should be in the default.
>
> - I've heard that "lein run" uses command-line arguments to its java call
> that attempt to minimize startup time, but that this may have substantial
> performance costs once the process begins running (e.g. because JIT
> compilation is delayed or avoided entirely). I realize that I could have
> more complete control by calling java myself, presumably after "lein
> uberjar" or something like that, but as a non-java/lein-reliant person I'd
> love to have a lein task that was biased in the opposite way: run the
> project with settings likely to have the highest performance. Does this
> already exist? If not, then might it? I don't have an opinion about whether
> this would be an option to "lein run" or a separate command, or whatever,
> but I'd love to be able to do high performance runs from a "lein run"-like
> call.
>
> Sorry again if these are naive or FAQs, and I'd appreciate any pointers.
>
> Thanks,
>
>  -Lee
>

Re: [leiningen] a couple of "lein run" related questions

From:
Phil Hagelberg
Date:
2013-05-31 @ 23:58
Lee Spector writes:

> How about something like this as a revision?:
>
> Q: Can I get improved long-term performance if I'm willing to put up 
with slower boot time?
> A: Yes. By default, "lein run" minimizes startup time via Tiered
> Compilation, but this can adversely affect long-term performance. The
> simplest way to reverse this preference is to switch profiles to
> prevent the Tiered Compilation :jvm-opts setting from being used. The
> "production" profile will do this in a minimal way: lein with-profile
> production run ...

Sure; that's much clearer, thanks.

I've just pushed it to master.

-Phil

Re: [leiningen] a couple of "lein run" related questions

From:
Lee Spector
Date:
2013-06-01 @ 00:18
On May 31, 2013, at 7:58 PM, Phil Hagelberg wrote:
> 
> Sure; that's much clearer, thanks.
> 
> I've just pushed it to master.

Great -- thanks (and of course much more so for leiningen itself!).

 -Lee

Re: [leiningen] a couple of "lein run" related questions

From:
Phil Hagelberg
Date:
2013-05-30 @ 01:59
Lee Spector writes:

> And would this be the right way to combine full compilation with other 
JVM options?:
>
> :jvm-opts ^:replace ["-Xmx58g" "-Xms58g" "-XX:+UseParallelGC"]

Sure; that works.

> I now see this whole facility... nice... but I don't see an existing
> "production" profile so I gather that you're suggesting creating
> one. Which makes sense, but it looks like it will take some study.

No need; Leiningen treats non-existent profiles as empty. Usually it
will warn when you try to invoke ones that don't exist, but production
is special-cased not to.

> If I can indeed get good long-term performance from a lein launch by
> using trampoline and ^:replace in the :jvm-opts then that's great. And
> if that's the way to do this then I'd agree that it'd be good to
> document this, as I don't think I'm alone in wanting this.

These are two separate concerns. Trampoline saves memory, and turning
off tiered compilation improves performance. You probably want both, but
they're not really related.

-Phil

Re: [leiningen] a couple of "lein run" related questions

From:
Lee Spector
Date:
2013-05-30 @ 02:24
On May 29, 2013, at 9:59 PM, Phil Hagelberg wrote:
> 
>> And would this be the right way to combine full compilation with other 
JVM options?:
>> 
>> :jvm-opts ^:replace ["-Xmx58g" "-Xms58g" "-XX:+UseParallelGC"]
> 
> Sure; that works.

Great. Just wanted to make sure I understood that correctly.

> 
>> I now see this whole facility... nice... but I don't see an existing
>> "production" profile so I gather that you're suggesting creating
>> one. Which makes sense, but it looks like it will take some study.
> 
> No need; Leiningen treats non-existent profiles as empty. Usually it
> will warn when you try to invoke ones that don't exist, but production
> is special-cased not to.

I don't understand this. Is "production" non-existent or a special case? I
don't see any mention of it in "lein help profile" (well, I do see it in 
an example, but that looks like it's just an example).

> 
>> If I can indeed get good long-term performance from a lein launch by
>> using trampoline and ^:replace in the :jvm-opts then that's great. And
>> if that's the way to do this then I'd agree that it'd be good to
>> document this, as I don't think I'm alone in wanting this.
> 
> These are two separate concerns. Trampoline saves memory, and turning
> off tiered compilation improves performance. You probably want both, but
> they're not really related.

I get that they're different, but for me at least "run without sacrificing
performance" would probably include both. In any event, it'd be great if 
the way(s) to allow runs launched with "lein run" to perform well were 
prominently documented. 

 Thanks,

 -Lee

Re: [leiningen] a couple of "lein run" related questions

From:
Phil Hagelberg
Date:
2013-05-30 @ 00:25
Jean Niklas L'orange writes:

> Well, `lein new` is using the default template, which is equivalent to the
> library template as far as I can remember. As the most common thing to do
> is to create a library, you would usually not want a main class
> (:gen-class) nor an entrypoint to the library (:main) by default.

Yeah, you really want to avoid having AOT applied for a library, which
is why the default template doesn't include :main.

> If you want faster performance for `lein run`, then the best idea would
> probably be to run `lein trampoline run` instead.

This doesn't actually help long-term performance, just memory usage.
What you need to do is turn off the :jvm-opts which limit tiered
compilation.

    :jvm-opts ^:replace []

Using a different profile will also avoid this; so `lein with-profiles
production run ...` or the like.

> That being said, I would like to emphasize that lein is not designed as a
> "program starter": If you're running a server, system or a program, you
> should (uber)jar or (uber)war the system before using it in production.

If you're deploying a production service, I'd recommend starting with an
uberjar. Still, there are places it makes sense to use Leiningen for
convenience where you'd still want to emphasize long-term performance
over startup time, so this should be documented somewhere. I'm not sure
what the right place for it is though.

-Phil

Re: [leiningen] a couple of "lein run" related questions

From:
Lee Spector
Date:
2013-05-30 @ 01:11
Thanks for the additional information Phil.

> This doesn't actually help long-term performance, just memory usage.
> What you need to do is turn off the :jvm-opts which limit tiered
> compilation.
> 
>    :jvm-opts ^:replace []

Ah, so I should "lein run trampoline" and also put something like the 
above in my defproject?

And would this be the right way to combine full compilation with other JVM
options?:

:jvm-opts ^:replace ["-Xmx58g" "-Xms58g" "-XX:+UseParallelGC"]

> 
> Using a different profile will also avoid this; so `lein with-profiles
> production run ...` or the like.

I now see this whole facility... nice... but I don't see an existing 
"production" profile so I gather that you're suggesting creating one. 
Which makes sense, but it looks like it will take some study.

> If you're deploying a production service, I'd recommend starting with an
> uberjar. Still, there are places it makes sense to use Leiningen for
> convenience where you'd still want to emphasize long-term performance
> over startup time, so this should be documented somewhere. I'm not sure
> what the right place for it is though.

I think this describes my situation. At least I don't think of what I'm 
doing as "deploying a production service." And I definitely want to 
emphasize long-term performance over startup time. 

If I can indeed get good long-term performance from a lein launch by using
trampoline and ^:replace in the :jvm-opts then that's great. And if that's
the way to do this then I'd agree that it'd be good to document this, as I
don't think I'm alone in wanting this.

Thanks so much again,

 -Lee

Re: [leiningen] a couple of "lein run" related questions

From:
Phil Hagelberg
Date:
2013-05-30 @ 16:36
Lee Spector writes:

> I don't understand this. Is "production" non-existent or a special
> case? I don't see any mention of it in "lein help profile" (well, I do
> see it in an example, but that looks like it's just an example).

It's both; there is no production profile by default, but the "warn when
you ask for a profile that doesn't exist" logic has a special case for
production for this precise reason. It'd be clearer if we just shipped
with an empty production profile, (and it would show in `show-profiles`)
so maybe we can do that in the future.

> I get that they're different, but for me at least "run without
> sacrificing performance" would probably include both. In any event,
> it'd be great if the way(s) to allow runs launched with "lein run" to
> perform well were prominently documented.

Sure; I've added this to the FAQ.

-Phil

Re: [leiningen] a couple of "lein run" related questions

From:
Lee Spector
Date:
2013-05-30 @ 17:14
On May 30, 2013, at 12:36 PM, Phil Hagelberg wrote:
> 
>> I get that they're different, but for me at least "run without
>> sacrificing performance" would probably include both. In any event,
>> it'd be great if the way(s) to allow runs launched with "lein run" to
>> perform well were prominently documented.
> 
> Sure; I've added this to the FAQ.

Sounds great... Although I don't actually see it there. You mean here?: 
https://github.com/technomancy/leiningen/blob/stable/doc/FAQ.md

 -Lee

Re: [leiningen] a couple of "lein run" related questions

From:
Jean Niklas L'orange
Date:
2013-05-30 @ 18:04
On 30 May 2013 19:14, Lee Spector <lspector@hampshire.edu> wrote:

> Sounds great... Although I don't actually see it there. You mean here?:
> https://github.com/technomancy/leiningen/blob/stable/doc/FAQ.md
>

It is here, on master:
https://github.com/technomancy/leiningen/blob/master/doc/FAQ.md

stable is only updated when we release a new version.

-- 
Regards,
Jean Niklas L'orange

Re: [leiningen] a couple of "lein run" related questions

From:
Jamie Orchard-Hays
Date:
2013-05-31 @ 10:04
Phil, there's a typo at the bottom of that page:

lein with-profiles production run ...

should be "with-profile" (singular)

Cheers,
Jamie

On May 30, 2013, at 2:04 PM, Jean Niklas L'orange wrote:

> 
> On 30 May 2013 19:14, Lee Spector <lspector@hampshire.edu> wrote:
> Sounds great... Although I don't actually see it there. You mean here?: 
https://github.com/technomancy/leiningen/blob/stable/doc/FAQ.md
> 
> It is here, on master: 
https://github.com/technomancy/leiningen/blob/master/doc/FAQ.md
> 
> stable is only updated when we release a new version.
> 
> -- 
> Regards,
> Jean Niklas L'orange

Re: [leiningen] a couple of "lein run" related questions

From:
Lee Spector
Date:
2013-05-31 @ 18:48
On May 31, 2013, at 6:04 AM, Jamie Orchard-Hays wrote:

> Phil, there's a typo at the bottom of that page:
> 
> lein with-profiles production run ...
> 
> should be "with-profile" (singular)
> 

Actually, it seems to work with or without the "s".

Although it's hard for me to be 100% sure (because my application involves
a lot of stochasticity), from my initial tests it looks like "with-profile
production" is indeed improving my long-term performance, maybe even by a 
double-digit percentage. Since I often do runs that crunch intensively for
hours or days, any improvement here is a big deal.

FWIW I don't see why it would be helpful to assume or say that "Most uses 
of Leiningen are in a context where fast boot is more important". And I 
don't think that the current question for this item -- "What are the 
downsides of Tiered Compilation?" -- is the most helpful here either. 
Until this discussion I didn't even know that I was using tiered 
compilation when I used "lein run", and I wouldn't have guessed that this 
was relevant to me from the question. Neither the leiningen readme nor the
other FAQs discuss tiered compilation, except for the one talking about an
error which I've never seen and so also wouldn't think relevant. 

How about something like this as a revision?:

---
Q: Can I get improved long-term performance if I'm willing to put up with 
slower boot time?
A: Yes. By default, "lein run" minimizes startup time via Tiered 
Compilation, but this can adversely affect long-term performance. The 
simplest way to reverse this preference is to switch profiles to prevent 
the Tiered Compilation :jvm-opts setting from being used. The "production"
profile will do this in a minimal way:  lein with-profile production run 
...
---

 -Lee

Re: [leiningen] a couple of "lein run" related questions

From:
Phil Hagelberg
Date:
2013-05-30 @ 17:26
Lee Spector writes:

> Sounds great... Although I don't actually see it there. You mean
> here?: https://github.com/technomancy/leiningen/blob/stable/doc/FAQ.md

Yeah; I added it yesterday, so it's on master, not stable.

-Phil

Re: [leiningen] a couple of "lein run" related questions

From:
Lee Spector
Date:
2013-05-30 @ 18:03
On May 30, 2013, at 1:26 PM, Phil Hagelberg wrote:
> 
> Yeah; I added it yesterday, so it's on master, not stable.

Ah -- beautiful. Very clear. Thanks, -Lee