I'm just looking for a little guidance on the connection pool usage. Some background on our setup: We're currently running 20 sidekiq servers on Heroku with concurrency set to 20 each (we were going a bit over the memory limit at 50 and haven't had time to test values in between yet for full optimization). From what I understand so far, this should set us up as requiring a maximum of 440 connections to Redis. We have a RedisToGo subscription with 512 connections. The client connection count was not set in the configure block (though I am testing setting it to 1 today). With this setup, we are consistently getting "Redis::CommandError: ERR max number of clients reached." First of all, would setting the connection pool help with this issue? It seems to me that we're not getting the number of connections were supposed to be getting from RedisToGo, but perhaps I'm missing some bit of info about how many connections this setup requires. The third possibility would be that either the threads aren't releasing their connection or RedisToGo isn't freeing up said connection before the thread tries to grab a new one. Regardless of the answer to the last question, the documentation shows the connection pool configuration residing in the worker class. Does this imply that connection pools are on a per-worker-class basis? We have 15-20 different workers and I'd rather not manage them all separately. If this is the case, I'd set up a base worker class that all of our workers inherit from and which defines the redis connection pool setup. Thanks for any help/comments you can give! Already enjoying sidekiq a ton despite little issues like this and I look forward to helping make sidekiq better and faster. Already we've reduced our background processing dyno count by five dynos and decreased our processing time of ~1.3 million nightly jobs from ~10 hours to ~3 hours. Jake
On Fri, Aug 3, 2012 at 4:51 PM, Jake Mack <jakemack@gmail.com> wrote: > configure block (though I am testing setting it to 1 today). With this > setup, we are consistently getting "Redis::CommandError: ERR max number > of clients reached." Are you positive your only Redis usage is via the connection pool? You aren't using Redis.new anywhere else? > Regardless of the answer to the last question, the documentation shows > the connection pool configuration residing in the worker class. Does > this imply that connection pools are on a per-worker-class basis? We > have 15-20 different workers and I'd rather not manage them all > separately. If this is the case, I'd set up a base worker class that all > of our workers inherit from and which defines the redis connection pool > setup. It's not per-Worker, it's shared process-wide. It will be sized to whatever :size you specify in the associated configure_{client,server} block and that should be the max # of connections used by that process by Sidekiq. There's nothing stopping you from creating Redis connections yourself of course. > Thanks for any help/comments you can give! Already enjoying sidekiq a > ton despite little issues like this and I look forward to helping make > sidekiq better and faster. Already we've reduced our background > processing dyno count by five dynos and decreased our processing time of > ~1.3 million nightly jobs from ~10 hours to ~3 hours. I love hearing success stories like this. Thanks! mike
Thanks for the response. I'll have to check for all of the other places we create Redis connections ourself, but I'm relatively certain we're not creating 112 Redis connections to go over our connection limit. I just wanted to make sure that there wasn't some other source of Redis connections in Sidekiq that I was missing that could be contributing to the error we're seeing before I spent a ton of time investigating it. If the solution has something to do with Sidekiq, I'll be sure to post the answer here in case anyone else runs across a similar issue in the future. Mike Perham wrote: > > On Fri, Aug 3, 2012 at 4:51 PM, Jake Mack<jakemack@gmail.com> wrote: > >> >> configure block (though I am testing setting it to 1 today). With this >> setup, we are consistently getting "Redis::CommandError: ERR max number >> of clients reached." > > > Are you positive your only Redis usage is via the connection pool? > You aren't using Redis.new anywhere else? > >> >> Regardless of the answer to the last question, the documentation shows >> the connection pool configuration residing in the worker class. Does >> this imply that connection pools are on a per-worker-class basis? We >> have 15-20 different workers and I'd rather not manage them all >> separately. If this is the case, I'd set up a base worker class that all >> of our workers inherit from and which defines the redis connection pool >> setup. > > > It's not per-Worker, it's shared process-wide. It will be sized to > whatever :size you specify in the associated configure_{client,server} > block and that should be the max # of connections used by that process > by Sidekiq. There's nothing stopping you from creating Redis > connections yourself of course. > >> >> Thanks for any help/comments you can give! Already enjoying sidekiq a >> ton despite little issues like this and I look forward to helping make >> sidekiq better and faster. Already we've reduced our background >> processing dyno count by five dynos and decreased our processing time of >> ~1.3 million nightly jobs from ~10 hours to ~3 hours. > > > I love hearing success stories like this. Thanks! > > mike