librelist archives

haproxy and Maven repository

haproxy and Maven repository by Kohsuke Kawaguchi
- Re: [hudson.infra] haproxy and Maven repository by R. Tyler Croy

haproxy and Maven repository

From:: Kohsuke Kawaguchi
Date:: 2011-01-03 @ 20:48

I was debugging a Maven deployment problem with mindless and
eventually found out what seems to be the root cause.

And while I think I understand what's causing this, I need Tyler's
help to figure out how to fix this.


When Maven deploys to Nexus, it sends out two PUT requests. The first
PUT requests fails with 401 unauthorized, and Maven then sends out the
2nd PUT response with the proper basic authentication header to make
the whole thing work. This double transfer is silly, but that's how it
works.

When Maven directly talks to Nexus, this first PUT request terminates
normally. The server responds back with 401, but the server lets the
client send the rest of the payload anyway, and even wait for the
client to initiate the close. The server then close its half of the
TCP connection and the session ends normally. IOW, at the TCP level,
there's nothing abnormal about the communication.

OTOH, when Maven talks to Nexus through haproxy, right after the
server sends back 401, haproxy issues TCP RESET packet to cut the
connection in the middle. Maven (or to be more precise it's the JRE
code) fails to notice this, it tries to reuse this connection for
sending another request, then fails to write to the socket (because
it's already terminated by the server.)

Given that Nexus waits for the client to initiate the connection tear
down (for a direct case), it must be that haproxy is killing the
connection eagerly.


With that, here's my question:

1. is there any way to verify that this is what's going on?
2. how can we change this behaviour in haproxy?

-- 
Kohsuke Kawaguchi

Re: [hudson.infra] haproxy and Maven repository

From:: R. Tyler Croy
Date:: 2011-01-05 @ 01:19

On Mon, 03 Jan 2011, Kohsuke Kawaguchi wrote:

> I was debugging a Maven deployment problem with mindless and
> eventually found out what seems to be the root cause.
> 
> And while I think I understand what's causing this, I need Tyler's
> help to figure out how to fix this.
> 
> 
> When Maven deploys to Nexus, it sends out two PUT requests. The first
> PUT requests fails with 401 unauthorized, and Maven then sends out the
> 2nd PUT response with the proper basic authentication header to make
> the whole thing work. This double transfer is silly, but that's how it
> works.
> 
> When Maven directly talks to Nexus, this first PUT request terminates
> normally. The server responds back with 401, but the server lets the
> client send the rest of the payload anyway, and even wait for the
> client to initiate the close. The server then close its half of the
> TCP connection and the session ends normally. IOW, at the TCP level,
> there's nothing abnormal about the communication.
> 
> OTOH, when Maven talks to Nexus through haproxy, right after the
> server sends back 401, haproxy issues TCP RESET packet to cut the
> connection in the middle. Maven (or to be more precise it's the JRE
> code) fails to notice this, it tries to reuse this connection for
> sending another request, then fails to write to the socket (because
> it's already terminated by the server.)
> 
> Given that Nexus waits for the client to initiate the connection tear
> down (for a direct case), it must be that haproxy is killing the
> connection eagerly.
> 
> 
> With that, here's my question:
> 
> 1. is there any way to verify that this is what's going on?
> 2. how can we change this behaviour in haproxy?

Try removing "option httpclose" from the maven backend in the haproxy config
and maybe that will help?


If that doesn't work it might be easier to move maven off to another IP and
then use haproxy as a TCP load balancer instead of an HTTP one


- R. Tyler Croy
--------------------------------------
    Code: http://github.com/rtyler
 Chatter: http://twitter.com/agentdero
          http://identi.ca/dero