Does anyone have any thoughts on how to deal with runaway (Tir) handler
processes? Ideally I'd like to have monit-like features by way of which I
can limit memory usage, CPU % use, etc. and kill off misbehaving
processes.
The problem I have with using monit is that my handlers are dynamically
created and deployed which means that I don't have a priori knowledge of
the process details. Which in turn implies that I would need to
dynamically update monit's conifg as handlers are created and destroyed
and then reload monit. Doable but it seems like a hack.
Put differently, how can I protect myself from a newly-deployed handler
whose source contains something like the following?
while(true) { allocate_memory() }
✈ Matt
On 04/05/2011 10:12 PM, Matt Towers wrote: > Does anyone have any thoughts on how to deal with runaway (Tir) > handler processes? Ideally I'd like to have monit-like features by > way of which I can limit memory usage, CPU % use, etc. and kill off > misbehaving processes. > > The problem I have with using monit is that my handlers are > dynamically created and deployed which means that I don't have a > priori knowledge of the process details. Which in turn implies that I > would need to dynamically update monit's conifg as handlers are > created and destroyed and then reload monit. Doable but it seems like > a hack. > > Put differently, how can I protect myself from a newly-deployed > handler whose source contains something like the following? > > while(true) { allocate_memory() } it looks like Mr. DJB's softlimit[1] is supposed to do this [1] http://cr.yp.to/daemontools/softlimit.html
That one looks to be pretty much the same as ulimit? Are there specific advantages to softlimit over ulimit? ✈ Matt On Apr 8, 2011, at 07:47 , Sabin Iacob wrote: > On 04/05/2011 10:12 PM, Matt Towers wrote: >> Does anyone have any thoughts on how to deal with runaway (Tir) >> handler processes? Ideally I'd like to have monit-like features by >> way of which I can limit memory usage, CPU % use, etc. and kill off >> misbehaving processes. >> >> The problem I have with using monit is that my handlers are >> dynamically created and deployed which means that I don't have a >> priori knowledge of the process details. Which in turn implies that I >> would need to dynamically update monit's conifg as handlers are >> created and destroyed and then reload monit. Doable but it seems like >> a hack. >> >> Put differently, how can I protect myself from a newly-deployed >> handler whose source contains something like the following? >> >> while(true) { allocate_memory() } > > it looks like Mr. DJB's softlimit[1] is supposed to do this > > [1] http://cr.yp.to/daemontools/softlimit.html >
On Fri, Apr 08, 2011 at 09:38:56AM -0700, Matt Towers wrote: > That one looks to be pretty much the same as ulimit? > > Are there specific advantages to softlimit over ulimit? I haven't looked but I don't think ulimit can do this on a per-process basis, and doesn't do the CPU limitation like softlimit can. Someone correct me if I'm wrong. -- Zed A. Shaw http://zedshaw.com/
It looks like softlimit will only allow you to specify a total running time for a given process, which ulimit also does. As best I can tell, neither allow you to cap CPU usage as a percentage of total. I ended up just setting handler priorities lower using nice. With ulimit, I'm pretty sure the following syntax will work for setting limits on a per-process basis. (ulimit -v 204800; exec nice -n 15 nohup lua app/myapp.lua & echo $! > run/myapp.pid) ✈ Matt On Apr 8, 2011, at 15:32 , Zed A. Shaw wrote: > On Fri, Apr 08, 2011 at 09:38:56AM -0700, Matt Towers wrote: >> That one looks to be pretty much the same as ulimit? >> >> Are there specific advantages to softlimit over ulimit? > > I haven't looked but I don't think ulimit can do this on a per-process > basis, and doesn't do the CPU limitation like softlimit can. > > Someone correct me if I'm wrong. > > -- > Zed A. Shaw > http://zedshaw.com/
On 2011-04-08 16:47, Sabin Iacob wrote: > On 04/05/2011 10:12 PM, Matt Towers wrote: >> Does anyone have any thoughts on how to deal with runaway (Tir) >> handler processes? Ideally I'd like to have monit-like features by >> way of which I can limit memory usage, CPU % use, etc. and kill off >> misbehaving processes. >> >> The problem I have with using monit is that my handlers are >> dynamically created and deployed which means that I don't have a >> priori knowledge of the process details. Which in turn implies that I >> would need to dynamically update monit's conifg as handlers are >> created and destroyed and then reload monit. Doable but it seems like >> a hack. >> >> Put differently, how can I protect myself from a newly-deployed >> handler whose source contains something like the following? >> >> while(true) { allocate_memory() } > > it looks like Mr. DJB's softlimit[1] is supposed to do this > > [1] http://cr.yp.to/daemontools/softlimit.html You also have lxc, just found it again today. http://lxc.sourceforge.net/ These are the kernel level containers, a bit like openvz, but included in the main line and available with libvirt. The interesting part is that it looks like you can have volatile containers (aka chroot environment) started from a template and trashed when stopping it. You could start one container per customer. I think I would go this way, allocating one /28 per customer on my lan (easy to put iptables rules to drop all the packets trying to reach outside of the subnet) with services available on a given ip (for example proxy, smtp, mongodb, mongrel2). The more than one ip approach is to spread the load on several VMs for one customer and allow one customer to have its own zeromq based queue, task or something like that. Still a lot to experiment... loïc
On Fri, Apr 8, 2011 at 10:56 AM, Loic d'Anterroches <loic@ceondo.com> wrote: > These are the kernel level containers, a bit like openvz, but included > in the main line and available with libvirt. The interesting part is > that it looks like you can have volatile containers (aka chroot > environment) started from a template and trashed when stopping it. You > could start one container per customer. no need for even a template. LXC can impose all kinds of limits to a fork()ed subprocess, accessible from C API. when the subprocess dies, there's nothing else to trash. > I think I would go this way, allocating one /28 per customer on my lan > (easy to put iptables rules to drop all the packets trying to reach > outside of the subnet) with services available on a given ip (for > example proxy, smtp, mongodb, mongrel2). The more than one ip approach > is to spread the load on several VMs for one customer and allow one > customer to have its own zeromq based queue, task or something like that. even if an LXC 'VM' is quite lightweight, you don't have to use a 'full VM'; just limit what you want to limit: CPU priority, RAM, maybe filesystems. with a little preparation, you can strip network access from the handlers, or restrict to just localhost, or to an 'alias' (eth0:1) where all such subprocess are a just processes from the same machine, so no need to declare new IP numbers. the idea is that LXC lets you span the whole (continuous) spectrum, from plain fork() up to a complete VM -- Javier
On 2011-04-08 18:33, Javier Guerra Giraldez wrote: > On Fri, Apr 8, 2011 at 10:56 AM, Loic d'Anterroches <loic@ceondo.com> wrote: >> These are the kernel level containers, a bit like openvz, but included >> in the main line and available with libvirt. The interesting part is >> that it looks like you can have volatile containers (aka chroot >> environment) started from a template and trashed when stopping it. You >> could start one container per customer. > > > no need for even a template. LXC can impose all kinds of limits to a > fork()ed subprocess, accessible from C API. when the subprocess dies, > there's nothing else to trash. > > >> I think I would go this way, allocating one /28 per customer on my lan >> (easy to put iptables rules to drop all the packets trying to reach >> outside of the subnet) with services available on a given ip (for >> example proxy, smtp, mongodb, mongrel2). The more than one ip approach >> is to spread the load on several VMs for one customer and allow one >> customer to have its own zeromq based queue, task or something like that. > > > even if an LXC 'VM' is quite lightweight, you don't have to use a > 'full VM'; just limit what you want to limit: CPU priority, RAM, maybe > filesystems. with a little preparation, you can strip network access > from the handlers, or restrict to just localhost, or to an 'alias' > (eth0:1) where all such subprocess are a just processes from the same > machine, so no need to declare new IP numbers. Just spent the last 2h searching without success how to mark a packet from cgroup (tc.classid) and use it with an iptables rule. Basically doing that: http://lkml.org/lkml/2008/1/23/50 Quoted from link above: > An approach that we've been experimenting with at Google is much simpler: > > - add a "network class id" subsystem, that lets you associated an id > with each cgroup > > - propagate this id to sockets created by that cgroup, and from there > to packets sent/received on that socket > > - add a new traffic filter that can select based on a packet's cgroup class id Is it possible to do it now? Basically, I just want to do that. This way, I can group my handlers in a given cgroup per customer and have the power of iptables to drop/accept connections. If you have any pointers, I would be more than happy to have them :) loïc
On 2011-04-05 21:12, Matt Towers wrote: > Does anyone have any thoughts on how to deal with runaway (Tir) handler > processes? Ideally I'd like to have monit-like features by way of which > I can limit memory usage, CPU % use, etc. and kill off misbehaving > processes. > > The problem I have with using monit is that my handlers are dynamically > created and deployed which means that I don't have a priori knowledge of > the process details. Which in turn implies that I would need to > dynamically update monit's conifg as handlers are created and destroyed > and then reload monit. Doable but it seems like a hack. > > Put differently, how can I protect myself from a newly-deployed handler > whose source contains something like the following? I start all my handlers by forking and then doing a pcntl_exec to replace the current process with the handler code. This way, my master keep track of the childs pids and can kill them, get notified when they die to restart them but also can get memory/cpu usage from the pid (this requires some system calls). You can then use ulimits for the memory limit and cpulimit for the CPU usage limit (which can be set per PID from their doc, not tested): http://cpulimit.sourceforge.net/ Again, not done any control at the moment. loïc