Invalidation Queue flow


#1

What is the data flow from Source data through each queues and lists in C3? for example Source -> sourceList -> Dataloadqueue -> ??


#2

from 7.7, the way it works is that you need to define a S3 location for each entry in FileSourceSystem, then ensure that you have an entry for each canonical in FileSourceCollection, which references which FileSourceSystem the canonical will use.

You then fire up SourceFile.syncAll(‘S3 location/inbox/CanonicalName’,{process:true}, This will synch the files found to the entries in SourceFile. If you added the {process:true} as in the above, the syncAll does more than just sync the files, it will also process them (as if you had provided SourceFile.processAll).

As the file entry in sourceFile moves form status ‘initiated’ --> ‘validated’ --> scheduled --> processing, once it hits processing, it will generate an entry in DataLoadUploadLog (DLUL). then the file gets chunked and each chunk will create an entry in DataLoadProcessLog (DLPL). Once the file has finished processing all chunks, it will update its status in SourceFile to one of three statuses (there may be more, but these are the 3 I came across):
‘completed’ or ‘rejected’ or ‘failed’.

what you may find in the non-completed entries is there will be a different level of detail as regards the error message between Sourcefile and DLUL and DLPL.

Hope this helps.