Cronjobs with Dynamic Parameters to start analytics job


#1

I have to manually start below jobs everyday for Analytics and ETL. However, we would like to automate these tasks using cron jobs, the issue is that some variables described below need to be dynamic and updated every day before the cron jobs are executed. I also need to execute the tasks in the following order and I cannot start the next task until the previous one is completed, we have already defined the time slots to start the jobs.

  1. Analytics Job1.

  2. ETL.

  3. Anaytics Job2.

  4. FirstAnalytics job - It executes some map reduce tasks.

The first job is executed with below command.
job1 = MeterAssetAnalyticProcessor.runAnalyticsJob(“amiMeterFlag == true”, Date.deserialize(“2018-08-12T00:00:00”), Date.deserialize(“2018-08-13T00:00:00”), C3.typesys.Arry.fromPrimitive(“string”, []), 10, “analytics_8-12_8-12_UTDAug_12_2018_v1”)

I need something like this with dynamic variables/

job1 = MeterAssetAnalyticProcessor.runAnalyticsJob(“amiMeterFlag == true”, Date.deserialize(%START_DATE%), Date.deserialize(%START_DATE%+1), C3.typesys.Arry.fromPrimitive(“string”, []), 10, “analytics_%MONTH_NUM_SD%-%DAY_NUM_SD%_%MONTH_NUM_SD%-%DAY_NUM_SD%_UTD%FORMATTED_SD%_v%VER_NUM%”)

where %START_DATE% is actually the start date that analytics will process data, the %MONTH_NUM_SD% is the month number of the start date, %DAY_NUM_SD% is the day numner fo the start date, the %FORMATTED_SD% is Start Date in a formatted string and %VER_NUM%" is the version of the job.

  1. ETL

For ETL the following steps are required to start the job.

//get the id of the latest job
job = ServicePointMetricResultExportJob.get(’%ID_TOP_ENTRY%’)
//then set the endDate, startDate, new id, and path fields on the job to correspond to what range you want to run the job for:
job.endDate = ‘%END_DATE%’
job.path = ‘prod/ServicePointMetricResultExportJob/%START_DATE%to%END_DATE%/’
job.id = ‘%NEW_ID%’
job.startDate = ‘%START_DATE%’
job.merge()

job = ServicePointMetricResultExportJob.get(’%NEW_ID%’)
//then
job.start()

where %ID_TOP_ENTRY% was the latest entry of ETL job executed, then %START_DATE% and %END_DATE% define the period of the data used in the ETL process
and the %NEW_ID% is calculated based on the start and end dates.

  1. For the Analyticsjob after ETL, it is basicsally same as the First Analytics job, we just need to update the version number %VER_NUM%.

job2 = MeterAssetAnalyticProcessor.runAnalyticsJob(“amiMeterFlag == true”, Date.deserialize(%START_DATE%), Date.deserialize(%START_DATE%+1), C3.typesys.Arry.fromPrimitive(“string”, []), 10, “analytics_%MONTH_NUM_SD%-%DAY_NUM_SD%_%MONTH_NUM_SD%-%DAY_NUM_SD%_UTD%FORMATTED_SD%_v%VER_NUM%”)

Does any one has used dynamic variables on Cron jobs?


#2

Sounds like you’re talking about a Workflow. Instead of manually synchronizing steps that need to happen one after another, use WorkflowSteps to manage this. The type is called ‘Workflow’ and it supports dynamic arguments natively.


#3

@rileysiebel this sounds awesome. Does this functionality work in 7.7?


#4

It exists in 7.7, whether it “works” I’ll let you be the judge.


#5

Is there any example in confluence about how to use workflows?

Also what means the following in the steps field of the workflow?
All steps contained within the workflow. This is not used for orchestration. It is for informational purposes only.

And from what I understand I need to define all the jobs that I want to run as WorkflowSteps and the Workflow will coordinate the WorklfowRun. Is this correct?


#6
  1. If there are examples in confluence, they might as well not exist. Customers don’t have access to confluence. If you find the “workflows” documentattion (c3doc + type docs) insufficient please let us know what specific questions you have

  2. The ‘steps’ field is just used for you to see all the steps in the workflow but is NOT used when trying to execute the steps in the correct order

  3. exactly


#7

I found the c3doc description about Workflows and it really describes what we need. However, there are no examples on how to use it. I will try to figure it out, otherwise I will come with more questions, thanks.