librelist archives

« back to archive

Handling failed jobs

Handling failed jobs

From:
Tony Arcieri
Date:
2010-03-20 @ 01:00
The "failed" jobs list is fairly nifty.  However, it only includes the
exception message and not the backtrace, as far as I can tell.

Do people generally use Resque's built-in error reporting, or do they use it
in combination with SNMP/e-mail based error reporting?  If so, how are
people debugging failed jobs without the backtraces?  Is there some way to
get the backtrace I'm not seeing?

--

And as an aside, how are people encountering jobs which are experiencing a
"temporary failure" and should be retried?

-- 
Tony Arcieri
Medioh! A Kudelski Brand

Re: [resque] Handling failed jobs

From:
Andrew Nordman
Date:
2010-03-20 @ 01:10
Click the error message in the failed section to make the backtrace show up
in resque-web, or configure it to use Hoptoad or one of the other backends
if you'd like a different method of handling failures.

As far as temporary failures, I check exception errors in failed code and
depending upon the type (a common one I have is timeouts on API calls)
I re-queue the job in the rescue block.

Andrew Nordman

On Fri, Mar 19, 2010 at 8:00 PM, Tony Arcieri <tony@medioh.com> wrote:

> The "failed" jobs list is fairly nifty.  However, it only includes the
> exception message and not the backtrace, as far as I can tell.
>
> Do people generally use Resque's built-in error reporting, or do they use
> it in combination with SNMP/e-mail based error reporting?  If so, how are
> people debugging failed jobs without the backtraces?  Is there some way to
> get the backtrace I'm not seeing?
>
> --
>
> And as an aside, how are people encountering jobs which are experiencing a
> "temporary failure" and should be retried?
>
> --
> Tony Arcieri
> Medioh! A Kudelski Brand
>

Re: [resque] Handling failed jobs

From:
Tony Arcieri
Date:
2010-03-20 @ 01:18
On Fri, Mar 19, 2010 at 7:10 PM, Andrew Nordman <cadwallion@gmail.com>wrote:

> Click the error message in the failed section to make the backtrace show up
> in resque-web, or configure it to use Hoptoad or one of the other backends
> if you'd like a different method of handling failures.


Okay, I swear I tried that before and it wasn't working, but it magically
seems to be working now.  Weird.



>  As far as temporary failures, I check exception errors in failed code and
> depending upon the type (a common one I have is timeouts on API calls)
> I re-queue the job in the rescue block.
>

Do you actually walk the "failed" queue to do this, or do you rescue
"temporary" exceptions within the job and schedule them to be retried?

-- 
Tony Arcieri
Medioh! A Kudelski Brand

Re: [resque] Handling failed jobs

From:
Andrew Nordman
Date:
2010-03-20 @ 01:52
I rescue temporary exceptions within the job itself based on the exception
raised.  I figure that if the reason for the failure can't be determined by
the particular exception, it probably shouldn't be blindly re-queued,
although I still log the exception error regardless and a message that It's
being re-queued.  I believe someone was working on a plugin or fork to allow
you to re-queue a failed job from resque-web.

On Fri, Mar 19, 2010 at 8:18 PM, Tony Arcieri <tony@medioh.com> wrote:

> On Fri, Mar 19, 2010 at 7:10 PM, Andrew Nordman <cadwallion@gmail.com>wrote:
>
>> Click the error message in the failed section to make the backtrace show
>> up in resque-web, or configure it to use Hoptoad or one of the other
>> backends if you'd like a different method of handling failures.
>
>
> Okay, I swear I tried that before and it wasn't working, but it magically
> seems to be working now.  Weird.
>
>
>
>>  As far as temporary failures, I check exception errors in failed code
>> and depending upon the type (a common one I have is timeouts on API calls)
>> I re-queue the job in the rescue block.
>>
>
> Do you actually walk the "failed" queue to do this, or do you rescue
> "temporary" exceptions within the job and schedule them to be retried?
>
> --
> Tony Arcieri
> Medioh! A Kudelski Brand
>