Could not resolve concurrency scenario for normalization after 3 attempts

#1

I’ve a MapReduce job that does an evalMetric on map phase, but it crashes with the following error:

c3.love.exceptions.C3RuntimeException: MetricEngine error : c3.love.exceptions.C3RuntimeException: Error c3.love.exceptions.C3RuntimeException: DataLoaderImpl Error : Call to fetchNormalizedData failed : Unable to normalize timeseries action=fetchNormalizedData,type=RegisterMeasurementSeries,parentId=AS_TELENET-POWER_3877C7_AE_DC,error=Internal Error : Could not resolve concurrency scenario for normalization after 3 attempts.
  at c3.love.expr.DataLoaderImpl.throwError(DataLoaderImpl.java:103)
  at c3.love.expr.DataLoaderImpl.fetchNormalizedData(DataLoaderImpl.java:135)
  at c3.love.expr.DataLoaderImpl.getNormalizedData(DataLoaderImpl.java:368)
  at c3.love.expr.eval.EvalVisitor.processTimeseriesNodes(EvalVisitor.java:813)
  at c3.love.expr.eval.EvalVisitor.processTimeseriesContext(EvalVisitor.java:901)
  at c3.love.expr.eval.EvalVisitor.visitNameOrKeywordNode(EvalVisitor.java:1016)
  at c3.love.expr.graph.AstNodeVisitor.visitNameNode(AstNodeVisitor.java:171)
  at c3.love.expr.graph.AstNodeVisitor.visitNode(AstNodeVisitor.java:72)
  at c3.love.expr.eval.EvalVisitor.visitGetPropNode(EvalVisitor.java:1098)
  at c3.love.expr.graph.AstNodeVisitor.visitNode(AstNodeVisitor.java:80)
  at c3.love.expr.eval.EvalVisitor.evalArgs(EvalVisitor.java:602)
  at c3.love.expr.eval.EvalVisitor.evaluateFunctionCall(EvalVisitor.java:411)
  at c3.love.expr.eval.EvalVisitor.visitCallNode(EvalVisitor.java:351)
  at c3.love.expr.graph.AstNodeVisitor.visitNode(AstNodeVisitor.java:96)
  at c3.love.expr.eval.EvalVisitor.evalArgs(EvalVisitor.java:602)
  at c3.love.expr.eval.EvalVisitor.evaluateFunctionCall(EvalVisitor.java:411)
  at c3.love.expr.eval.EvalVisitor.visitCallNode(EvalVisitor.java:351)
  at c3.love.expr.graph.AstNodeVisitor.visitNode(AstNodeVisitor.java:96)
  at c3.love.expr.eval.EvalVisitor.evaluateExpression(EvalVisitor.java:122)
  at c3.love.expr.eval.EvalFacade.evalExprWithBindings(EvalFacade.java:188)
  at c3.service.metric.SimpleMetricEvaluator.evaluateExpressionBasedMetric(SimpleMetricEvaluator.java:577)
  at c3.service.metric.SimpleMetricEvaluator.evaluateMetric(SimpleMetricEvaluator.java:528)
  at c3.service.metric.MetricEvaluatable.compute(MetricEvaluatable.java:296)
  at . . .

Any idea what could fix this metric evaluation?

0 Likes

DataLoadQueue and AnalyticsQueue errors using FileTimedData
#2

@bachr This means that there is high concurrency in how the time series is getting invalidated & reads. What it is trying to do is, when the normalized time series state is marked as invalid, it tries to re-normalize and read the series again, but before reading some other thread has marked this same series as invalid again (may be due to data load happening?)

If after 3 attempts we’re not able to resolve this, we throw this error without infinitely trying to read the series. The simplest approach will be to retry reading the series / normalize the series incrementally so that the map reduce job does not have to do the work on normalizing on the fly

1 Like

#3

There were no data loading, whenever I try these metric evals the error is thrown:

FixedAsset.evalMetrics({
  start: "2018-05-19",
  end: "2018-05-20",
  expressions: ["MeasuredConsumptionNaturalGasTotal", "ExternalTemperature", "HDD", "MeasuredElectricityActivePowerPeak", "MeasuredElectricityActiveEnergyConsumption", "MeasuredElectricityReactiveEnergyConsumption", "MeasuredElectricityDailyActiveEnergyConsumption", "MeasuredElectricityDailyReactiveEnergyConsumption",  "MeasuredConsumptionThermalEnergyTotal"],
  interval: "DAY",
  filter: "facilityLevel=='Building' || (!exists(facilityLevel) && !exists(parent))"
})

I will try to configure incremental normalization and see if it fixes the problem. thanks

0 Likes

#4

Please open a support ticket. Its possible there is an issue in the register normalizer resulting in possible normalization issues. Please refer to EXC-14025 for additional insights and potential workarounds.

Additional concurrency checks were added in 7.7 to prevent similar issues in future releases.

1 Like

#5

@bachr I would also suggest trying to evaluate those metrics one at a time ans see if they succeed. You are firing 10 compound metrics at once; it is possible that some of them reference same SimpleMetrics, so you get concurrent evaluation (and concurrent normalization). It is possible that if you run them all one by one they will all succeed.

(This is likely not a viable solution long-term but would be a useful data point.)

0 Likes

#6

@yaroslav this is what I thought but I’m having same error when trying to evaluate one metric at a time

var metrics = ["MeasuredConsumptionNaturalGasTotal", "ExternalTemperature", "HDD", "MeasuredElectricityActivePowerPeak", "MeasuredElectricityActiveEnergyConsumption", "MeasuredElectricityReactiveEnergyConsumption", "MeasuredElectricityDailyActiveEnergyConsumption", "MeasuredElectricityDailyReactiveEnergyConsumption",  "MeasuredConsumptionThermalEnergyTotal"]
var result = metrics.map(function(m) {
  console.log('evaluating', m);
  return FixedAsset.evalMetrics({
    start: "2018-05-19",
    end: "2018-05-20",
    expressions: [m],
    interval: "DAY",
    filter: "facilityLevel=='Building' || (!exists(facilityLevel) && !exists(parent))"
})})

10:43:58.605 VM643:1 evaluating MeasuredConsumptionNaturalGasTotal
10:44:02.099 VM643:1 evaluating ExternalTemperature
10:44:20.345 VM643:1 evaluating HDD
10:44:50.672 VM643:1 evaluating MeasuredElectricityActivePowerPeak
10:44:54.641 VM643:1 evaluating MeasuredElectricityActiveEnergyConsumption
10:44:58.201 VM643:1 evaluating MeasuredElectricityReactiveEnergyConsumption
10:45:01.644 VM643:1 evaluating MeasuredElectricityDailyActiveEnergyConsumption

c3.love.exceptions.C3RuntimeException: c3.love.exceptions.C3RuntimeException: 
MetricEngine error : c3.love.exceptions.C3RuntimeException: Error c3.love.exceptions.C3RuntimeException: 
DataLoaderImpl Error : Call to fetchNormalizedData failed : Unable to normalize timeseries action=fetchNormalizedData,type=RegisterMeasurementSeries,parentId=AS_TELENET-POWER_387684_AE_DC,
error=Internal Error : Could not resolve concurrency scenario for normalization after 3 attempts.

It seems that MeasuredElectricityDailyActiveEnergyConsumption is the one failing!
This is the definition of the metric, nothing fancy!

{
      "id": "MeasuredElectricityDailyActiveEnergyConsumption_FixedAsset",
      "name": "MeasuredElectricityDailyActiveEnergyConsumption",
      "srcType": {
        "moduleName": "structure",
        "typeName": "FixedAsset"
      },
      "expression": "sum(sum(normalized.data.quantity))",
      "path": "denormChildren.to.servicePoints.measurements.(measurementType == 'MeasuredElectricityDailyActiveEnergyConsumption')",
      "description": "Measured Electricity Active Peak Power"
    }
0 Likes