librelist archives

« back to archive

Configuring number of workers on a per-queue basis

Configuring number of workers on a per-queue basis

From:
Hongli Lai
Date:
2014-02-20 @ 12:12
I see that Sidekiq supports configuring the number of workers
globally. But does it also support configuring the number of workers
on a per-queue basis?

My use case is as follows. I'm writing a continuous integration
system. I have two queues, "jobs" and "emails". A job can take 15 to
30 minutes to process. Because processing a job is CPU and memory
intensive, I only want to process at most 4 jobs at the same time. But
even when I'm running the maximum number of concurrent jobs, I still
want to be able to send emails. It should not happen that emails get
blocked for 1 hour because 4 jobs are running.

Re: [sidekiq] Configuring number of workers on a per-queue basis

From:
beaver henry
Date:
2014-02-20 @ 12:23
Hongli,

I'm not sure if this is the best practice way, mike or someone else can 
chime in, but here is how I accomplish your goal. I run separate workers 
dedicated to specific queues (via monit). So worker set A is dedicated to 
run queues 'foo,bar,baz' and worker set B is configured to consume queue 
'cpu_intensive_work'. I have servers dedicated to this CPU intensive work.

In fact I run the workers for set A on a few servers with queues 
'foo,bar,baz' at 10 priority and 'cpu_intensive_work' at 1. 
This means if I have no work running for 'foo,bar,baz' I can utilize the 
other "normal" servers, but of normal work is added to queue, those 
servers will stop consuming those jobs.

I hope this helps and makes sense.

H

> On Feb 20, 2014, at 7:12 AM, Hongli Lai <hongli@phusion.nl> wrote:
> 
> I see that Sidekiq supports configuring the number of workers
> globally. But does it also support configuring the number of workers
> on a per-queue basis?
> 
> My use case is as follows. I'm writing a continuous integration
> system. I have two queues, "jobs" and "emails". A job can take 15 to
> 30 minutes to process. Because processing a job is CPU and memory
> intensive, I only want to process at most 4 jobs at the same time. But
> even when I'm running the maximum number of concurrent jobs, I still
> want to be able to send emails. It should not happen that emails get
> blocked for 1 hour because 4 jobs are running.

Re: [sidekiq] Configuring number of workers on a per-queue basis

From:
Hongli Lai
Date:
2014-02-20 @ 12:28
Hm, running two processes... Yes I suppose that will work, although
it'll use more memory, which I hope to avoid. The more memory I spend
on the workers, the less CI jobs I can run. I hope Mike can shed
some lights on whether it's possible to achieve this with just one process.

On Thu, Feb 20, 2014 at 1:23 PM, beaver henry <hbeaver@gmail.com> wrote:
> Hongli,
>
> I'm not sure if this is the best practice way, mike or someone else can 
chime in, but here is how I accomplish your goal. I run separate workers 
dedicated to specific queues (via monit). So worker set A is dedicated to 
run queues 'foo,bar,baz' and worker set B is configured to consume queue 
'cpu_intensive_work'. I have servers dedicated to this CPU intensive work.
> In fact I run the workers for set A on a few servers with queues 
'foo,bar,baz' at 10 priority and 'cpu_intensive_work' at 1.
> This means if I have no work running for 'foo,bar,baz' I can utilize the
other "normal" servers, but of normal work is added to queue, those 
servers will stop consuming those jobs.
>
> I hope this helps and makes sense.
>
> H
>
>> On Feb 20, 2014, at 7:12 AM, Hongli Lai <hongli@phusion.nl> wrote:
>>
>> I see that Sidekiq supports configuring the number of workers
>> globally. But does it also support configuring the number of workers
>> on a per-queue basis?
>>
>> My use case is as follows. I'm writing a continuous integration
>> system. I have two queues, "jobs" and "emails". A job can take 15 to
>> 30 minutes to process. Because processing a job is CPU and memory
>> intensive, I only want to process at most 4 jobs at the same time. But
>> even when I'm running the maximum number of concurrent jobs, I still
>> want to be able to send emails. It should not happen that emails get
>> blocked for 1 hour because 4 jobs are running.



-- 
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: info@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

Re: [sidekiq] Configuring number of workers on a per-queue basis

From:
Mike Perham
Date:
2014-02-20 @ 16:49
Check this page out, especially the sidekiq-limit_fetch gem:

https://github.com/mperham/sidekiq/wiki/Related-Projects


On Thu, Feb 20, 2014 at 4:28 AM, Hongli Lai <hongli@phusion.nl> wrote:

> Hm, running two processes... Yes I suppose that will work, although
> it'll use more memory, which I hope to avoid. The more memory I spend
> on the workers, the less CI jobs I can run. I hope Mike can shed
> some lights on whether it's possible to achieve this with just one process.
>
> On Thu, Feb 20, 2014 at 1:23 PM, beaver henry <hbeaver@gmail.com> wrote:
> > Hongli,
> >
> > I'm not sure if this is the best practice way, mike or someone else can
> chime in, but here is how I accomplish your goal. I run separate workers
> dedicated to specific queues (via monit). So worker set A is dedicated to
> run queues 'foo,bar,baz' and worker set B is configured to consume queue
> 'cpu_intensive_work'. I have servers dedicated to this CPU intensive work.
> > In fact I run the workers for set A on a few servers with queues
> 'foo,bar,baz' at 10 priority and 'cpu_intensive_work' at 1.
> > This means if I have no work running for 'foo,bar,baz' I can utilize the
> other "normal" servers, but of normal work is added to queue, those servers
> will stop consuming those jobs.
> >
> > I hope this helps and makes sense.
> >
> > H
> >
> >> On Feb 20, 2014, at 7:12 AM, Hongli Lai <hongli@phusion.nl> wrote:
> >>
> >> I see that Sidekiq supports configuring the number of workers
> >> globally. But does it also support configuring the number of workers
> >> on a per-queue basis?
> >>
> >> My use case is as follows. I'm writing a continuous integration
> >> system. I have two queues, "jobs" and "emails". A job can take 15 to
> >> 30 minutes to process. Because processing a job is CPU and memory
> >> intensive, I only want to process at most 4 jobs at the same time. But
> >> even when I'm running the maximum number of concurrent jobs, I still
> >> want to be able to send emails. It should not happen that emails get
> >> blocked for 1 hour because 4 jobs are running.
>
>
>
> --
> Phusion | Ruby & Rails deployment, scaling and tuning solutions
>
> Web: http://www.phusion.nl/
> E-mail: info@phusion.nl
> Chamber of commerce no: 08173483 (The Netherlands)
>

Re: [sidekiq] Configuring number of workers on a per-queue basis

From:
Hongli Lai
Date:
2014-02-24 @ 19:59
Perfect, thanks for the suggestion.

On Thu, Feb 20, 2014 at 5:49 PM, Mike Perham <mperham@gmail.com> wrote:
> Check this page out, especially the sidekiq-limit_fetch gem:
>
> https://github.com/mperham/sidekiq/wiki/Related-Projects
>
>
> On Thu, Feb 20, 2014 at 4:28 AM, Hongli Lai <hongli@phusion.nl> wrote:
>>
>> Hm, running two processes... Yes I suppose that will work, although
>> it'll use more memory, which I hope to avoid. The more memory I spend
>> on the workers, the less CI jobs I can run. I hope Mike can shed
>> some lights on whether it's possible to achieve this with just one
>> process.
>>
>> On Thu, Feb 20, 2014 at 1:23 PM, beaver henry <hbeaver@gmail.com> wrote:
>> > Hongli,
>> >
>> > I'm not sure if this is the best practice way, mike or someone else can
>> > chime in, but here is how I accomplish your goal. I run separate workers
>> > dedicated to specific queues (via monit). So worker set A is dedicated to
>> > run queues 'foo,bar,baz' and worker set B is configured to consume queue
>> > 'cpu_intensive_work'. I have servers dedicated to this CPU intensive work.
>> > In fact I run the workers for set A on a few servers with queues
>> > 'foo,bar,baz' at 10 priority and 'cpu_intensive_work' at 1.
>> > This means if I have no work running for 'foo,bar,baz' I can utilize the
>> > other "normal" servers, but of normal work is added to queue, those servers
>> > will stop consuming those jobs.
>> >
>> > I hope this helps and makes sense.
>> >
>> > H
>> >
>> >> On Feb 20, 2014, at 7:12 AM, Hongli Lai <hongli@phusion.nl> wrote:
>> >>
>> >> I see that Sidekiq supports configuring the number of workers
>> >> globally. But does it also support configuring the number of workers
>> >> on a per-queue basis?
>> >>
>> >> My use case is as follows. I'm writing a continuous integration
>> >> system. I have two queues, "jobs" and "emails". A job can take 15 to
>> >> 30 minutes to process. Because processing a job is CPU and memory
>> >> intensive, I only want to process at most 4 jobs at the same time. But
>> >> even when I'm running the maximum number of concurrent jobs, I still
>> >> want to be able to send emails. It should not happen that emails get
>> >> blocked for 1 hour because 4 jobs are running.
>>
>>
>>
>> --
>> Phusion | Ruby & Rails deployment, scaling and tuning solutions
>>
>> Web: http://www.phusion.nl/
>> E-mail: info@phusion.nl
>> Chamber of commerce no: 08173483 (The Netherlands)
>
>



-- 
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: info@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)