librelist archives

« back to archive

Plugins can't be on S3

Plugins can't be on S3

From:
Alan Dipert
Date:
2012-08-29 @ 23:37
Howdy all,
I noticed today that because of the way wagon factories are loaded, it's 
not currently possible to have plugins in your project that are deployed 
to S3.

In my project, I include two plugins: s3-wagon-private and my own 
leiningen plugin for doing awesome stuff that is deployed to S3.

Leiningen doesn't try to pull my plugin from S3, and bails with "Could 
not find artifact". I think this is because wagon factories are loaded 
after all plugins are loaded instead of after each individual plugin 
has. [1]

This issue appears to bottom out in Leiningen at aether/dependency-files 
over in resolve-dependencies [2].

My naive approach to fix would be to add some kind of 'after-hook' 
capability to pomegranate/add-dependencies [3] and 
pomegranate/add-classpath [4], and to pass a function from Leiningen 
that performed the wagon loading.

Is there a workaround for this conundrum?  Is my naive approach sensible?

Thanks in advance for any feedback.

Best,
Alan

1. 

https://github.com/technomancy/leiningen/blob/b5ea5a0f09958e9e22d00a49b0ceaf559acca819/leiningen-core/src/leiningen/core/project.clj#L263
2. 

https://github.com/technomancy/leiningen/blob/b5ea5a0f09958e9e22d00a49b0ceaf559acca819/leiningen-core/src/leiningen/core/classpath.clj#L145
3. 

https://github.com/cemerick/pomegranate/blob/19fd03c039b9a0ac699bc7c0c94736bf768fa274/src/main/clojure/cemerick/pomegranate.clj#L85
4. 

https://github.com/cemerick/pomegranate/blob/19fd03c039b9a0ac699bc7c0c94736bf768fa274/src/main/clojure/cemerick/pomegranate.clj#L72

Re: [leiningen] Plugins can't be on S3

From:
Phil Hagelberg
Date:
2012-08-30 @ 16:35
Alan Dipert <alan@dipert.org> writes:

> Leiningen doesn't try to pull my plugin from S3, and bails with "Could 
> not find artifact". I think this is because wagon factories are loaded 
> after all plugins are loaded instead of after each individual plugin 
> has. [1]

Right; we don't have any way to run code in between plugin resolutions;
it's just a single call to pomegranate.

> Is there a workaround for this conundrum?  Is my naive approach sensible?

One workaround that you should be able to use immediately would be to
place your custom plugin in a profile. The base :plugins are loaded
first, so you'd have access to S3 at the time the profile's plugins were
loaded. You could use an alias so that you wouldn't have to manually
invoke with-profile:

    :aliases {"hello" ["with-profile" "hello-plugin" "hello"]}

A bit ugly under the covers, but you don't need to be exposed to the
weirdness upon every invocation.

Longer-term, it may just be a better idea to ship the S3 wagon with
Leiningen. I'm a bit reluctant to give preferential treatment to
something so vendor-specific, but it would address some serious pain
points. If S3 only supported HTTP Basic Auth this would be much less of
an issue, but unfortunately they force you to go through their
mechanisms for private access.

-Phil

Re: [leiningen] Plugins can't be on S3

From:
Alan Dipert
Date:
2012-08-31 @ 20:20
Thanks everybody for the help.  Despite a lot of tinkering, I still 
couldn't get it to work.

It would appear that top-level plugins are not loaded before profile 
plugins, at least not reliably, but I haven't dug into the code around it.

Anyway, I'm a big fan of shipping Leiningen with the S3 wagon - would 
this be a welcome addition to lein2?

Alan

On 08/30/2012 12:35 PM, Phil Hagelberg wrote:
> Alan Dipert<alan@dipert.org>  writes:
>
>> Leiningen doesn't try to pull my plugin from S3, and bails with "Could
>> not find artifact". I think this is because wagon factories are loaded
>> after all plugins are loaded instead of after each individual plugin
>> has. [1]
>
> Right; we don't have any way to run code in between plugin resolutions;
> it's just a single call to pomegranate.
>
>> Is there a workaround for this conundrum?  Is my naive approach sensible?
>
> One workaround that you should be able to use immediately would be to
> place your custom plugin in a profile. The base :plugins are loaded
> first, so you'd have access to S3 at the time the profile's plugins were
> loaded. You could use an alias so that you wouldn't have to manually
> invoke with-profile:
>
>      :aliases {"hello" ["with-profile" "hello-plugin" "hello"]}
>
> A bit ugly under the covers, but you don't need to be exposed to the
> weirdness upon every invocation.
>
> Longer-term, it may just be a better idea to ship the S3 wagon with
> Leiningen. I'm a bit reluctant to give preferential treatment to
> something so vendor-specific, but it would address some serious pain
> points. If S3 only supported HTTP Basic Auth this would be much less of
> an issue, but unfortunately they force you to go through their
> mechanisms for private access.
>
> -Phil

Re: [leiningen] Plugins can't be on S3

From:
Chas Emerick
Date:
2012-08-30 @ 13:41
The automagical loading of wagon factories is a very recent invention; 
prior to that, you had to add a snippet like this to your project.clj 
(after the defproject form):

(cemerick.pomegranate.aether/register-wagon-factory!
  "s3p" #(eval '(org.springframework.aws.maven.PrivateS3Wagon.)))

Tweak as necessary for the scheme(s) and wagon type(s) you're using.

That should make it so that plugin resolution benefits from the registered
factories.  Not as pretty as magic, but there's definitely a 
chicken-and-egg problem here.  Perhaps Leiningen should go ahead and just 
register the necessary bits for s3 and s3p.

Cheers,

- Chas

On Aug 29, 2012, at 7:37 PM, Alan Dipert wrote:

> Howdy all,
> I noticed today that because of the way wagon factories are loaded, it's 
> not currently possible to have plugins in your project that are deployed 
> to S3.
> 
> In my project, I include two plugins: s3-wagon-private and my own 
> leiningen plugin for doing awesome stuff that is deployed to S3.
> 
> Leiningen doesn't try to pull my plugin from S3, and bails with "Could 
> not find artifact". I think this is because wagon factories are loaded 
> after all plugins are loaded instead of after each individual plugin 
> has. [1]
> 
> This issue appears to bottom out in Leiningen at aether/dependency-files 
> over in resolve-dependencies [2].
> 
> My naive approach to fix would be to add some kind of 'after-hook' 
> capability to pomegranate/add-dependencies [3] and 
> pomegranate/add-classpath [4], and to pass a function from Leiningen 
> that performed the wagon loading.
> 
> Is there a workaround for this conundrum?  Is my naive approach sensible?
> 
> Thanks in advance for any feedback.
> 
> Best,
> Alan
> 
> 1. 
> 
https://github.com/technomancy/leiningen/blob/b5ea5a0f09958e9e22d00a49b0ceaf559acca819/leiningen-core/src/leiningen/core/project.clj#L263
> 2. 
> 
https://github.com/technomancy/leiningen/blob/b5ea5a0f09958e9e22d00a49b0ceaf559acca819/leiningen-core/src/leiningen/core/classpath.clj#L145
> 3. 
> 
https://github.com/cemerick/pomegranate/blob/19fd03c039b9a0ac699bc7c0c94736bf768fa274/src/main/clojure/cemerick/pomegranate.clj#L85
> 4. 
> 
https://github.com/cemerick/pomegranate/blob/19fd03c039b9a0ac699bc7c0c94736bf768fa274/src/main/clojure/cemerick/pomegranate.clj#L72