How to obtain total number of data files processed for indexing

#1

What command can be used to obtain the total number of data files processed in the system? and if possible records as well?

0 Likes

#2

Entries in the DataLoadUploadLog represents processed files, you can have the number with something like:

c3Count(DataLoadUploadLog)

For counting only successful uploads;

DataLoadUploadLog.fetch({filter: 'status=="SUCCESS" && processLogStatus	=="SUCCESS"'})

For counting total number of objects successfully uploaded:

var result = DataLoadUploadLog.fetch({filter: 'status=="SUCCESS" && processLogStatus	=="SUCCESS"'})
var total = result.objs.reduce(function(sum, obj){
  return sum + obj.stats.numRecords
}, 0)
0 Likes

#3

Thanks Bachr.

Should the following command also generate the same number for the DataLoadProcessLog?

var result2 = DataLoadProcessLog.fetch({filter: ‘status==“SUCCESS” && targetTypeName==“AType”’})
var totalProcessed = result2.objs.reduce(function(sum, obj){
return sum + obj.stats.successfulObjCount
}, 0)

c3Grid(DataLoadProcessLog.evaluate({projection:“sum(stats.successfulObjCount)”,filter:“targetTypeName=='AType”}))

The results i get don’t return the same results.

0 Likes

#4

I would say DataLoadUploadLog may have higher entries than DataLoadProcessLog, as when the upload fails the entry will not be passed to the processing queue. Large files can be split though.

1 Like

#5

The DataLoadUploadLog is linked to the files that have been uploaded. Each file is parsed, and is split in a number of chunks to be processed. How many can be found by checking the noOfChunksCreated in the stats block of the Upload log.
For each of those chunks, a DataLoadProcessLog entry is created to group the details for processing these records.

As such, the number of records in the ProcessLog is equal to or greater than the number of UploadLog entries.

1 Like