librelist archives

« back to archive

High Latency for calls to perform_async on RedisToGo instance

High Latency for calls to perform_async on RedisToGo instance

From:
Jack Royal-Gordon
Date:
2015-10-20 @ 18:21
A couple days ago I started experiencing long run times in certain jobs. 
It turns out that suddenly perform_async calls are taking 10 seconds to 
complete. Since I’m queueing 75K jobs, you can imagine the long run time. 
I’m running on Heroku with my Redis database on RedisToGo. I’ve contacted 
them, they’ve looked and see nothing in particular out of whack with my 
instance (which I have tried restarting), but they asked the following 
question which I don’t feel I can answer. Can one of you answer this 
question?

Also, I’d appreciate If someone could provide specifics as to which Redis 
calls are used when queueing a job in a specific queue (not the Scheduled 
queue), so I can pass that info along to RedisToGo Support.

However, I did notice SMEMBERS calls with significant time spent per call 
as we are fetching larger content per key.

cmdstat_smembers:calls=634,usec=1094004,usec_per_call=1725.56
Can you inform me why a SCAN could not be used (HSCAN, SSCAN, ZSCAN) to 
reduce overall latency with these fetches? I will continue to review and 
see if we can tune anything else in this instance.

Re: [sidekiq] High Latency for calls to perform_async on RedisToGo instance

From:
Jack Royal-Gordon
Date:
2015-10-20 @ 18:22
I should add that other Sidekiq calls, such as 
Sidekiq::RetrySet.new.count, complete acceptably fast (1-2 seconds), so 
this problem seems to be restricted to adding jobs to a queue.

> On Oct 20, 2015, at 11:21 AM, Jack Royal-Gordon <jackrg@pobox.com> wrote:
> 
> A couple days ago I started experiencing long run times in certain jobs.
It turns out that suddenly perform_async calls are taking 10 seconds to 
complete. Since I’m queueing 75K jobs, you can imagine the long run time. 
I’m running on Heroku with my Redis database on RedisToGo. I’ve contacted 
them, they’ve looked and see nothing in particular out of whack with my 
instance (which I have tried restarting), but they asked the following 
question which I don’t feel I can answer. Can one of you answer this 
question?
> 
> Also, I’d appreciate If someone could provide specifics as to which 
Redis calls are used when queueing a job in a specific queue (not the 
Scheduled queue), so I can pass that info along to RedisToGo Support.
> 
> However, I did notice SMEMBERS calls with significant time spent per 
call as we are fetching larger content per key.
> 
> cmdstat_smembers:calls=634,usec=1094004,usec_per_call=1725.56
> Can you inform me why a SCAN could not be used (HSCAN, SSCAN, ZSCAN) to 
reduce overall latency with these fetches? I will continue to review and 
see if we can tune anything else in this instance.
>