AsyncProcessingDisabled for only one canonical or type

#1

Currently when loading a large file 2GB (55 million rows) to postgres, we use this:

TenantConfig.upsert({id:‘AsyncProcessingDisabled’, value:‘true’})

Then afterwards, we change it back to false, and run refresh calc fields on the types we know to have changed.

The reason this needs to be done is because two other types have reference array fields doing a sum, and when rows are added, it starts backing up the changelog and calcfield queues massively and doesn’t perform well. Whereas running these calcs once at the end performs very well.

So the question is. Can we disable async processing for one canonical or type? So we dont have to do this switch.

1 Like

#2

I’ll let Products comment on the feature request, but if you know the types referencing this type, you can do the following as a workaround:

  1. Pause ChangeLogQueue
  2. While data are loading, you can repeatedly issue ChangeLogQueue.clear({targetTypeId:id_of_the_type_you_are_loading}) to prevent massive backlog
  3. Once you’re done loading, you can issue DependentType.refreshCalcFields({calcFieldsToRefresh:['field1','field2']})
0 Likes

#3

That is what I attempted to do, after realizing that I had forgotten to disable the async processing flag for the tenant when uploading large file, however the createBatch for change log queue was clogging the cluster. It made the c3Grid(InvalidationQueue.countAll()) take 16 minutes one time and not return after that. The clear then never finished (most likely I assume due to resource contention). I was hoping this could be selective, as most data loads are fine, just the one load needs special attention.

0 Likes