librelist archives

Re: [mongrel2] m2 crash on signaled config reload

From:

Matt Towers

Date:

2011-02-25 @ 16:51

I was assuming that the problems is related to a malformed config, 
particularly given that I'm hacking on the sqlite database directly and 
then redeploying.  However, I've been able to reproduce it with a default 
empty database as well.

To generate the empty database used to repro the problem, the config file 
contained only the following and was generated using m2sh:

servers = []



In the case of the databases I've modified, I've added only records to the
route and handler tables using the following procedure:

1.  Create a new empty sqlite3 database using PHP PDO
2.  Execute the config.sql file located in the mongrel source tree 
(src/config/config.sql)
3.  Execute the mimetypes.sql file located in the mongrel source tree 
(src/config/mimetypes.sql)
4.  Add a default server using the following statements:

-- ----------------------------
--  Records of "host"
-- ----------------------------
BEGIN;
INSERT INTO "host" VALUES (NULL, 1, 0, '(.+)', '(.+)');
COMMIT;

-- ----------------------------
--  Records of "server"
-- ----------------------------
BEGIN;
INSERT INTO "server" VALUES (NULL, '505417b8-1de4-454f-98b6-07eb9225cca1',
'/logs/access.log', '/logs/error.log', '/var/www/moai', 
'/run/mongrel2.pid', '(.+)', 'main', '0.0.0.0', 6767);
COMMIT;

-- ----------------------------
--  Records of "setting"
-- ----------------------------
BEGIN;
INSERT INTO "setting" VALUES (NULL, 'zeromq.threads', 1);
COMMIT;


5.  Insert a new handler record using the following:

"INSERT INTO handler VALUES (NULL, [send_spec, send_ident, recv_spec, 
recv_ident, raw_payload]);"

6.  Insert a new route record using the following:

INSERT INTO route VALUES (NULL, [route], 0, 1, [new handler_id], "handler");


That database is then copied to conf/config.sqlite and reloaded using 
SIGHUP or m2sh.


✈ Matt



On Feb 24, 2011, at 20:58 , Zed A. Shaw wrote:

> On Thu, Feb 24, 2011 at 02:24:17PM -0800, Matt Towers wrote:
>> I've been getting a weird crash today that seems to occur when I
>> deploy a new config.sqlite database and then request a reload using
>> SIGHUP (log output below).  The logs indicate that m2 fails to load
>> the config database that contains the updated handler descriptions.
>> From there m2 dies.  However, when I restart m2, the same config
>> database loads as expected and messages are successfully routed to all
>> handlers.
>> 
> 
> Does it only happen when the config has an error and it can't reload?
> That might be a code path we haven't tested.
> 
> -- 
> Zed A. Shaw
> http://zedshaw.com/

Narrowed this one down to some concrete repro steps:

Here's the mongrel2.conf file I"m using which describes only a single 
static Dir host.  In my static/ directory there is an index.html file 
containing only "hello world"

main = Server( 
    uuid="old-uuid", 
    access_log="/logs/access.log", 
    error_log="/logs/error.log", 
    chroot="/var/www/moai", 
    default_host="localhost", 
    name="test", 
    pid_file="/run/mongrel2.pid", 
    port=6767, 
    hosts = [ 
        Host(name="localhost", routes={ 
            '/': Dir(base='static/', index_file='index.html', 
                             default_ctype='text/plain') 
        }) 
    ] 
)

servers=[main]


1.  sudo m2sh load -db conf/config.sqlite -config conf/mongrel2.conf -servers test
2.  sudo m2sh start -db conf/config.sqlite -every
3.  curl http://localhost:6767       #returns "hello world"
4.  #In mongrel2.conf change the server UUID to "NEW-uuid"
5.  sudo m2sh load -db conf/config.sqlite -config conf/mongrel2.conf -servers test
6.  sudo m2sh reload -db conf/config.sqlite -every

Log output at step 6:   [INFO] (src/mongrel2.c:75) RELOAD REQUESTED, I'll 
do it on the next request.

7.  curl http://localhost:6767
8.  m2 crashes with the following output:

[INFO] (src/mongrel2.c:304) Reload requested, will load old-uuid from 
conf/config.sqlite
[INFO] (src/config/config.c:163) Loaded dir 1 with base=static/ 
index=index.html def_ctype=text/plain
[ERROR] (src/mongrel2.c:117: errno: No such file or directory) Failed to 
load server old-uuid from conf/config.sqlite
[ERROR] (src/mongrel2.c:232: errno: None) Failed to load new server config.
[ERROR] (src/mongrel2.c:306: errno: None) Failed to load the new 
configuration, will keep the old one.
[ERROR] (src/mongrel2.c:321: errno: None) Exiting due to error.


Note that in the first line of output that m2 is attempting to load the 
server "old-uuid", where "NEW-uuid" is expected.



✈ Matt

On Mon, Feb 28, 2011 at 03:15:57PM -0800, Matt Towers wrote:
> Narrowed this one down to some concrete repro steps:
> 
> [ERROR] (src/mongrel2.c:321: errno: None) Exiting due to error.
> 
> 
> Note that in the first line of output that m2 is attempting to load the 
server "old-uuid", where "NEW-uuid" is expected.

Awesome, thanks for tracking that down.  I think I know where that is
actually.

-- 
Zed A. Shaw
http://zedshaw.com/