Last question for today... :) If I have 3 handlers up, handler2 might send a message to clientID 12, which disconnected, so mongrel2 pushes a disconnect msg out, but because of zmq's push sockets being load-balanced, handler1 might catch it, while handler2 needed it to clean up its internal state. So the current disconnect handling is broken when having more than 1 handler, unless the handlers share state or communicate with each other, which complicates things. Also, I noticed that handlers only get notified about disconnected clients, when they try to send a message to them. It would be nice if handlers could receive disconnect notifications earlier, when the disconnect actually happens. The current behaviour is odd for applications dealing with "presence". Currently in (ws)chat, when the chat has been silent for a few minutes and most people have left, I only see them leave all at once when I ask "anybody there?". Ofcourse I can work around it by broadcasting pings from the server, but it would be nicer to get instant feedback. I think both these issues can be addressed by using an additional pub/sub socket where mongrel2 PUBs to its handlers. I agree this complicates things (zmq_poll or other event loop in handlers), but most apps won't need this anyway, so let's make it optional for handlers to use. Any thoughts on this? Thanks, Mathijs
On Sun, Aug 14, 2011 at 01:54:46AM +0200, Mathijs Kwik wrote: > Last question for today... :) > > If I have 3 handlers up, handler2 might send a message to clientID 12, > which disconnected, so mongrel2 pushes a disconnect msg out, but > because of zmq's push sockets being load-balanced, handler1 might > catch it, while handler2 needed it to clean up its internal state. > So the current disconnect handling is broken when having more than 1 > handler, unless the handlers share state or communicate with each > other, which complicates things. Yep, that's true, which is why I either make the handlers stateless (like HTTP is), or they use a backend like Redis to keep track of their temporary shared state, or I use a ZeroMQ server that they all talk to if it's something more complex. > Also, I noticed that handlers only get notified about disconnected > clients, when they try to send a message to them. > It would be nice if handlers could receive disconnect notifications > earlier, when the disconnect actually happens. No, this causes huge problems because you'll get "disconnect storms" whenever anything disconnects and those will dominate the traffic. It's better to only tell the handlers that "screw up" and try to talk to a dead client in a lazy way. > The current behaviour is odd for applications dealing with "presence". > Currently in (ws)chat, when the chat has been silent for a few minutes > and most people have left, I only see them leave all at once when I > ask "anybody there?". The problem is, you're trying to store user state in all the handlers and the browser, rather than where you should store it which is in a common location. It'll get too difficult to have handler X,Y,Z keep track of users themselves independantly just from user events. Instead, you need to have a common server that each of your handlers talks to for user's state, and make them more stateless. Think of your handlers as the "controller", rather than "controller+model" and it works great. > Ofcourse I can work around it by broadcasting pings from the server, > but it would be nicer to get instant feedback. > > I think both these issues can be addressed by using an additional > pub/sub socket where mongrel2 PUBs to its handlers. If you think that then it's easy for you to try. Create a single handler that reads off the PULL and routes it out to your handlers on a PUB. This is another thing that you'll do. Rather than overload Mongrel2 with all this work, Mongrel2's job is just getting the message "somewhere" as quickly as possible. Since ZeroMQ is so easily composible and flexible you can then take that and do whatever you want. This is *much* easier to work with and extend than trying to get Mongrel2 to cover all the possible ways users might want to use a zeromq socket. If you want "messages are duplicated to all handlers all the time" semantics, then you can crank that out in ZeroMQ in 10 minutes. If you then want crazy user state tracking, pings, etc. you can add that too. Try it out. -- Zed A. Shaw http://zedshaw.com/
On Sun, Aug 14, 2011 at 9:24 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote: > On Sun, Aug 14, 2011 at 01:54:46AM +0200, Mathijs Kwik wrote: >> Last question for today... :) >> >> If I have 3 handlers up, handler2 might send a message to clientID 12, >> which disconnected, so mongrel2 pushes a disconnect msg out, but >> because of zmq's push sockets being load-balanced, handler1 might >> catch it, while handler2 needed it to clean up its internal state. >> So the current disconnect handling is broken when having more than 1 >> handler, unless the handlers share state or communicate with each >> other, which complicates things. > > Yep, that's true, which is why I either make the handlers stateless > (like HTTP is), or they use a backend like Redis to keep track of their > temporary shared state, or I use a ZeroMQ server that they all talk to > if it's something more complex. > >> Also, I noticed that handlers only get notified about disconnected >> clients, when they try to send a message to them. >> It would be nice if handlers could receive disconnect notifications >> earlier, when the disconnect actually happens. > > No, this causes huge problems because you'll get "disconnect storms" > whenever anything disconnects and those will dominate the traffic. It's > better to only tell the handlers that "screw up" and try to talk to a > dead client in a lazy way. > >> The current behaviour is odd for applications dealing with "presence". >> Currently in (ws)chat, when the chat has been silent for a few minutes >> and most people have left, I only see them leave all at once when I >> ask "anybody there?". > > The problem is, you're trying to store user state in all the handlers > and the browser, rather than where you should store it which is in a > common location. It'll get too difficult to have handler X,Y,Z keep > track of users themselves independantly just from user events. > > Instead, you need to have a common server that each of your handlers > talks to for user's state, and make them more stateless. Think of your > handlers as the "controller", rather than "controller+model" and it > works great. > >> Ofcourse I can work around it by broadcasting pings from the server, >> but it would be nicer to get instant feedback. >> >> I think both these issues can be addressed by using an additional >> pub/sub socket where mongrel2 PUBs to its handlers. > > If you think that then it's easy for you to try. Create a single > handler that reads off the PULL and routes it out to your handlers on a > PUB. This is another thing that you'll do. Rather than overload > Mongrel2 with all this work, Mongrel2's job is just getting the message > "somewhere" as quickly as possible. Since ZeroMQ is so easily > composible and flexible you can then take that and do whatever you want. > > This is *much* easier to work with and extend than trying to get > Mongrel2 to cover all the possible ways users might want to use a zeromq > socket. If you want "messages are duplicated to all handlers all the > time" semantics, then you can crank that out in ZeroMQ in 10 minutes. > If you then want crazy user state tracking, pings, etc. you can add that > too. > > Try it out. I agree. I'm still getting used to the many-little-processes mindset. Probably because I still need to have a look at procer and find out that (keeping them) running is not as big an issue as it seems. > > -- > Zed A. Shaw > http://zedshaw.com/ >