BatchJob rerun failed batches


#1

I’m looking to catch all of the failed batches of a BatchJob and rerun them (up to 5 times) before accepting that they wont succeed. The idea is that I will check for any failures in the allComplete function and then use BatchQueue.recoverFailed to rerun. What is happening is that if there are any failures the allComplete function is never called. Is that expected? If so is there any good way to automatically rerun failed batches?

As a note, to simulate a failed batch I am throwing an error on the first batch to run.


#2

For those wondering about an answer:

allComplete will only run if all batches are successful. So to rerun batches I added a int field to the BatchJobBatch type that will keep track of the number of times it reran. Here is what the code looks like:

ExampleBatchJobBatch.c3Typ

type ExampleBatchJobBatch  {
  // Offset of the batch
  offset: int

  // Limit of the batch
  limit: int

  // Number of times the batch was retried
  retryCount: int
}

ExampleBatchJob.c3Typ

extendable entity type ExampleBatchJob extends BatchJob<ExampleBatchJob, ExampleBatchJobOptions, ExampleBatchJobBatch> type key "EX_BTCH_JOB" {

  // Instantiate the doStart method, which will be the first thing run by the start() api.
  doStart: ~ js server
  
  // this is the process that will happen on each task on a worker node. 
  processBatch: ~ js server
}

ExampleBatchJob.js

function doStart(job, options) {
    var batchTargetType = c3Type(options.targetType),
        totalRowsInType = batchTargetType.fetchCount({ filter: options.filter }),
        numBatches = Math.ceil(totalRowsInType / options.batchSize);

    for (var i = 0; i < numBatches; i++) {
        var batch = ExampleBatchJobBatch.make({
            offset: i * options.batchSize,
            limit: options.batchSize,
            retryCount: 0,
        });
        ExampleBatchJob.scheduleBatch(job, batch);
    }
}

function processBatch(batch, job, options) {
    try {
        runBatch(batch, job, options);
    } catch (error) {
        batch.retryCount++;
        if (batch.retryCount < 5) {
            processBatch(batch, job, options);
        }
    }
}

function runBatch(batch, job, options) {
    // Do batch processing
}