librelist archives

« back to archive

Struggling a bit with procer...

Struggling a bit with procer...

From:
Eric Patey
Date:
2012-01-19 @ 22:30
Hi all.

I'm new to mongrel2, and I've really enjoyed it so far. I've just started
trying to introduce procer into my servers, and I'm having trouble. In
fact, I'm having trouble just trying to recreate the joyous experience
suggested by Chapter 4 in "The Manual".

To debug, I've reduced the complexity from the example in the manual to
just single profile - mongrel2, and AFAIK, my run script and mongrel2
configuration are the same as the example.

The symptom is that procer launches mongrel2 properly, but doesn't think
mongrel2 created a pidfile. Consequently, procer terminates because of that
error.  Here's the profiles/error.log.

DEBUG procer.c:233: Loading 3 actions.
[ERROR] (profile.c:12: errno: No such file or directory) Failed to open
/home/cocomo/dep/profiles/mongrel2/depends file that's required.
DEBUG procer.c:83: STARTED mongrel2
[ERROR] (src/unixy.c:99: errno: No such file or directory) Failed to open
PID file /home/cocomo/dep/run/mongrel2.pid for reading.
[ERROR] (src/unixy.c:99: errno: No such file or directory) Failed to open
PID file /home/cocomo/dep/run/mongrel2.pid for reading.
[INFO] (src/unixy.c:138) No previous Mongrel2 running, continuing on.
DEBUG procer.c:37: ACTION: command=/home/cocomo/dep/profiles/mongrel2/run,
pid_file=/home/cocomo/dep/run/mongrel2.pid, restart=0, depends=(null)
DEBUG procer.c:56: WAITING FOR CHILD.
[INFO] (src/unixy.c:83) Now running as UID:0, GID:0
DEBUG procer.c:60: Command ran and exited successfully, now looking for the
PID file.
[ERROR] (procer.c:96: errno: No such file or directory) mongrel2 didn't
make pidfile /home/cocomo/dep/run/mongrel2.pid.

The reality, though, is that mongrel2 was launched just fine AND it created
the pidfile just fine. My suspicion is that mongrel2 did create the
pidfile, but not quickly enough. In other words, run ran m2sh and then
returned right away. As soon as run returned, procer looked for the pid
file. I suspect that mongrel2 just hadn't created it yet. Here's my run
script.

#!/bin/sh
cd /home/cocomo/dep
m2sh start -db run/config.sqlite -host localhost

If I run procer a second time right away, everything works as expected.
Procer sees the mongrel2 pid file, and sees a process running with the
appropriate pid. Procer continues running.

To test my theory, I threw a sleep 10 at the end of my run script thinking
it would give mongrel2 enough time to create the pid file before run
returned to procer. This hack corrected the problem (but is obviously
bogus).

Does this make sense?

Thanks for any help you can offer.

-Eric

Re: [mongrel2] Struggling a bit with procer...

From:
Zed A. Shaw
Date:
2012-02-08 @ 03:02
On Thu, Jan 19, 2012 at 05:30:49PM -0500, Eric Patey wrote:
> Hi all.
> The reality, though, is that mongrel2 was launched just fine AND it created
> the pidfile just fine. My suspicion is that mongrel2 did create the
> pidfile, but not quickly enough. In other words, run ran m2sh and then
> returned right away. As soon as run returned, procer looked for the pid
> file. I suspect that mongrel2 just hadn't created it yet. Here's my run
> script.
>...
> To test my theory, I threw a sleep 10 at the end of my run script thinking
> it would give mongrel2 enough time to create the pid file before run
> returned to procer. This hack corrected the problem (but is obviously
> bogus).

That sounds like a bug in procer not trying hard enough to look for the
pid file.  Can you file a bug in github and I'll take a look at fixing
it (or someone can)?

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] Struggling a bit with procer...

From:
Eric Patey
Date:
2012-02-10 @ 02:39
Thanks for the reply, Zed.

This issue fell down on my priority list for a few weeks, but I'll be happy
to file a bug (at least). If I can come up with some free cycles, I'll
attempt a fix.

On Tue, Feb 7, 2012 at 10:02 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:

> On Thu, Jan 19, 2012 at 05:30:49PM -0500, Eric Patey wrote:
> > Hi all.
> > The reality, though, is that mongrel2 was launched just fine AND it
> created
> > the pidfile just fine. My suspicion is that mongrel2 did create the
> > pidfile, but not quickly enough. In other words, run ran m2sh and then
> > returned right away. As soon as run returned, procer looked for the pid
> > file. I suspect that mongrel2 just hadn't created it yet. Here's my run
> > script.
> >...
> > To test my theory, I threw a sleep 10 at the end of my run script
> thinking
> > it would give mongrel2 enough time to create the pid file before run
> > returned to procer. This hack corrected the problem (but is obviously
> > bogus).
>
> That sounds like a bug in procer not trying hard enough to look for the
> pid file.  Can you file a bug in github and I'll take a look at fixing
> it (or someone can)?
>
> --
> Zed A. Shaw
> http://zedshaw.com/
>

Re: [mongrel2] Struggling a bit with procer...

From:
Justin Karneges
Date:
2012-03-02 @ 21:14
I just wanted to mention that today I was following the Deploying section of 
the Mongrel2 manual and set up procer, and ran into this exact same issue.  By 
the logs it looked like a race to check the pid file.  Adding a delay at the 
end of the run script did the trick.

On Thursday, February 09, 2012 06:39:33 PM Eric Patey wrote:
> Thanks for the reply, Zed.
> 
> This issue fell down on my priority list for a few weeks, but I'll be happy
> to file a bug (at least). If I can come up with some free cycles, I'll
> attempt a fix.
> 
> On Tue, Feb 7, 2012 at 10:02 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> > On Thu, Jan 19, 2012 at 05:30:49PM -0500, Eric Patey wrote:
> > > Hi all.
> > > The reality, though, is that mongrel2 was launched just fine AND it
> > 
> > created
> > 
> > > the pidfile just fine. My suspicion is that mongrel2 did create the
> > > pidfile, but not quickly enough. In other words, run ran m2sh and then
> > > returned right away. As soon as run returned, procer looked for the pid
> > > file. I suspect that mongrel2 just hadn't created it yet. Here's my run
> > > script.
> > >
> > >...
> > >
> > > To test my theory, I threw a sleep 10 at the end of my run script
> > 
> > thinking
> > 
> > > it would give mongrel2 enough time to create the pid file before run
> > > returned to procer. This hack corrected the problem (but is obviously
> > > bogus).
> > 
> > That sounds like a bug in procer not trying hard enough to look for the
> > pid file.  Can you file a bug in github and I'll take a look at fixing
> > it (or someone can)?
> > 
> > --
> > Zed A. Shaw
> > http://zedshaw.com/

Re: [mongrel2] Struggling a bit with procer...

From:
Anthon van der Neut
Date:
2012-03-02 @ 21:44
I had the same problem with procer. In the mongrel2 + uwsgi + web2py 
howto that I am writing I describe how to solve it in the source of procer:
--- a/procer.c    Tue Feb 21 14:49:50 2012 +0100
+++ b/procer.c    Fri Mar 02 22:40:07 2012 +0100
@@ -90,6 +96,14 @@
          } else {
              Unixy_remove_dead_pidfile(prof->pid_file);
              Action_exec(action, prof);
+            if (access(bdata(prof->pid_file), R_OK) != 0) {
+                log_info("waiting for pid file for %s to appear", 
bdata(action->name));
+                hardsleep(1);
+            }
+            if (access(bdata(prof->pid_file), R_OK) != 0) {
+                log_info("still waiting for pid file for %s to appear", 
bdata(action->name));
+                hardsleep(1);
+            }
          }

          check(access(bdata(prof->pid_file), R_OK) == 0, "%s didn't 
make pidfile %s.",


uwsgi is not the quickest to write its pid file either. I have still to 
see a pidfile take two seconds (if the second
message shows in the log, there is usually something wrong and the 
pidfile does not come up at all).
If you put a sleep at the end of the run script, you might sleep 
although the .pid file is already there.



On 2012-03-02 22:14, Justin Karneges wrote:
> I just wanted to mention that today I was following the Deploying section of
> the Mongrel2 manual and set up procer, and ran into this exact same issue.  By
> the logs it looked like a race to check the pid file.  Adding a delay at the
> end of the run script did the trick.
>
> On Thursday, February 09, 2012 06:39:33 PM Eric Patey wrote:
>> Thanks for the reply, Zed.
>>
>> This issue fell down on my priority list for a few weeks, but I'll be happy
>> to file a bug (at least). If I can come up with some free cycles, I'll
>> attempt a fix.
>>
>> On Tue, Feb 7, 2012 at 10:02 PM, Zed A. Shaw<zedshaw@zedshaw.com>  wrote:
>>> On Thu, Jan 19, 2012 at 05:30:49PM -0500, Eric Patey wrote:
>>>> Hi all.
>>>> The reality, though, is that mongrel2 was launched just fine AND it
>>> created
>>>
>>>> the pidfile just fine. My suspicion is that mongrel2 did create the
>>>> pidfile, but not quickly enough. In other words, run ran m2sh and then
>>>> returned right away. As soon as run returned, procer looked for the pid
>>>> file. I suspect that mongrel2 just hadn't created it yet. Here's my run
>>>> script.
>>>>
>>>> ...
>>>>
>>>> To test my theory, I threw a sleep 10 at the end of my run script
>>> thinking
>>>
>>>> it would give mongrel2 enough time to create the pid file before run
>>>> returned to procer. This hack corrected the problem (but is obviously
>>>> bogus).
>>> That sounds like a bug in procer not trying hard enough to look for the
>>> pid file.  Can you file a bug in github and I'll take a look at fixing
>>> it (or someone can)?
>>>
>>> --
>>> Zed A. Shaw
>>> http://zedshaw.com/

Re: [mongrel2] Struggling a bit with procer...

From:
Thomas Guyot-Sionnest
Date:
2012-03-03 @ 04:24
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12-03-02 04:44 PM, Anthon van der Neut wrote:
> I had the same problem with procer. In the mongrel2 + uwsgi +
> web2py howto that I am writing I describe how to solve it in the
> source of procer: --- a/procer.c Tue Feb 21 14:49:50 2012 +0100
> +++ b/procer.c Fri Mar 02 22:40:07 2012 +0100 @@ -90,6 +96,14
> @@ } else { Unixy_remove_dead_pidfile(prof->pid_file);
> Action_exec(action, prof); + if
> (access(bdata(prof->pid_file), R_OK) != 0) { +
> log_info("waiting for pid file for %s to appear",
> bdata(action->name)); + hardsleep(1); +
> } + if (access(bdata(prof->pid_file), R_OK) != 0) { +
> log_info("still waiting for pid file for %s to appear",
> bdata(action->name)); + hardsleep(1); +
> } }
>
> check(access(bdata(prof->pid_file), R_OK) == 0, "%s didn't make
> pidfile %s.",
>
>
> uwsgi is not the quickest to write its pid file either. I have
> still to see a pidfile take two seconds (if the second message
> shows in the log, there is usually something wrong and the pidfile
> does not come up at all). If you put a sleep at the end of the run
> script, you might sleep although the .pid file is already there.
>

Why not adding an environment variable that lets you define the
startup timeout?  I'm pretty sure no mater how long you sleep here,
someone out there will figure out a way to bust it, so why not just
giving control to the user?

I'm still not convinced how is this better than daemontool's use of a
waitpid() to detect the death of the process... and after years of
using it I most likely have a biased view anyway :)

Thanks

- -- 
Thomas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9RnOMACgkQ6dZ+Kt5BchbJ6QCfcyTBws5yqVDTqaVGDGaubqG7
6DUAn2ZUEbKgCBMgZ0USz30gFqtvQxfT
=QlUb
-----END PGP SIGNATURE-----

Re: [mongrel2] Struggling a bit with procer...

From:
Anthon van der Neut
Date:
2012-03-03 @ 04:40
an environment variable sounds like a  good idea. I briefly looked at 
commandline options but procer (and mongrel2) do not seem to be 
developed with those in mind.
I will put that in and test it.
The only reason for me to use procer is that I came across it in the 
documentation and that I saw I needed to start multiple instance of 
mongrel2 and uwsgi with different configurations. I might try out 
something else after getting this to work.

I am currently struggling with mongrel2 loading ssl with the 
certificates I use with apache. And with my server running out of memory 
once a day or so since I started investigating mongrel2. Since the 
machine is at my provider and does no longer respond I need to do a hard 
reset remotely :-(
Setting things up takes way more time than than I had hoped.



On 2012-03-03 05:24, Thomas Guyot-Sionnest wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 12-03-02 04:44 PM, Anthon van der Neut wrote:
> > I had the same problem with
> procer. In the mongrel2 + uwsgi +
>
> > web2py howto that I am writing I describe how to solve it
> in the
>
> > source of procer: --- a/procer.c Tue Feb 21 14:49:50 2012
> +0100
>
> > +++ b/procer.c Fri Mar 02 22:40:07 2012 +0100 @@ -90,6
> +96,14
>
> > @@ } else { Unixy_remove_dead_pidfile(prof->pid_file);
>
> > Action_exec(action, prof); + if
>
> > (access(bdata(prof->pid_file), R_OK) != 0) { +
>
> > log_info("waiting for pid file for %s to appear",
>
> > bdata(action->name)); + hardsleep(1); +
>
> > } + if (access(bdata(prof->pid_file), R_OK) != 0) { +
>
> > log_info("still waiting for pid file for %s to appear",
>
> > bdata(action->name)); + hardsleep(1); +
>
> > } }
>
> >
>
> > check(access(bdata(prof->pid_file), R_OK) == 0, "%s
> didn't make
>
> > pidfile %s.",
>
> >
>
> >
>
> > uwsgi is not the quickest to write its pid file either. I
> have
>
> > still to see a pidfile take two seconds (if the second
> message
>
> > shows in the log, there is usually something wrong and the
> pidfile
>
> > does not come up at all). If you put a sleep at the end of
> the run
>
> > script, you might sleep although the .pid file is already
> there.
>
> >
>
> Why not adding an environment variable that lets you define the
> startup timeout?  I'm pretty sure no mater how long you sleep here,
> someone out there will figure out a way to bust it, so why not just
> giving control to the user?
>
> I'm still not convinced how is this better than daemontool's use of a
> waitpid() to detect the death of the process... and after years of
> using it I most likely have a biased view anyway :)
>
> Thanks
>
> - -- 
> Thomas
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk9RnOMACgkQ6dZ+Kt5BchbJ6QCfcyTBws5yqVDTqaVGDGaubqG7
> 6DUAn2ZUEbKgCBMgZ0USz30gFqtvQxfT
> =QlUb
> -----END PGP SIGNATURE-----
>