Data load workflow


When loading data through JMSDataLoadQueue was it the workflow/succession of steps to follow?

Could you correct/edit/add the following steps:

  • Pause queues
  • Unpause JMSDataLoadQueue
  • Curl files to load data
  • Execute Tag.rebuild()
  • Unpause all queues

Actual steps are dependent on the size and scale of the data load. For small(er) incremental loads, none of the above steps may be required. All you will need to do is post the data (curl file, invoke API, etc.) to the server for processing. Downstream tasks will be handled asynchronously via the invalidation queues.

For initial data load (large data loads), you will want to do the following:

  1. Disable asynchronous processing
  2. Pause all queues with the exception of the JMSDataLoadQueue and CronQueue. This step is optional if async processing is disabled.
  3. Post data to the platform (import/1, api/1 APIs)
  4. Wait for data load to complete.
  5. Unpause queues
  6. Execute Tag.rebuild(). When running Tag.rebuild() specify which tasks you need performed.
1 Like