[MetricEngine] missing value seems wrong when doing subtract operation


#1

For example, A - B

Below is metric evaluation result of expression A

>> res_A=source_type.evalMetric({id:source_id, expression: "A", interval:"DAY", start:"2017-10-18", end: "2017-10-21"});
>> res_A.data();
> [0, 0, 0]
>> res_A.missing();
> [100, 100, 100]

Below is metric evaluation result of expression B

>> res_B=source_type.evalMetric({id:source_id, expression: "B", interval:"DAY", start:"2017-10-18", end: "2017-10-21"});
>> res_B.data();
> [4, 9, 0]
>> res_B.missing();
> [58, 0, 100]

When evaluating A+B, missing values are calculated as expected,

>> res_A_plus_B=source_type.evalMetric({id:source_id, expression: "A + B", interval:"DAY", start:"2017-10-18", end: "2017-10-21"});
>> res_A_plus_B.data();
> [4, 9, 0]
>> res_A_plus_B.missing();
> [58, 0, 100]

However, when evaluating A - B, missing values are all 100. See below

>> res_A_subtract_B=source_type.evalMetric({id:source_id, expression: "A - B", interval:"DAY", start:"2017-10-18", end: "2017-10-21"});
>> res_A_subtract_B.data();
> [-4, -9, 0]
>> res_A_subtract_B.missing();
> [100, 100, 100]

when evaluating A + (-1 * B), missing values are calculated as expected

>> res_A_subtract_B_2=source_type.evalMetric({id:source_id, expression: "A + (-1*B)", interval:"DAY", start:"2017-10-18", end: "2017-10-21"});
>> res_A_subtract_B_2.data();
> [-4, -9, 0]
>> res_A_subtract_B_2.missing();
> [58, 0, 100]

#2

Here’s a table of binary operators and the choices we have made for computing the resultant missing values when those operators are applied on two timeseries:

Operands: X% missing with Y% missing
-------------------------------------------------------------------------------
Binary Operator                                     | Resulting missing values
-------------------------------------------------------------------------------
SUM,AVG,MIN,MAX,OR                                  | MIN(X, Y) % missing
MULTIPLY,DIVIDE,SUBTRACT,MODULO,AND,>,>=,<,<=,==,!= | MAX(X, Y) % missing

Here are these rules applied to some examples:

Operands: 100% missing with 0% missing
------------------------------------------------------------------------------
SUM,AVG,MIN,MAX,OR                                  | 0% missing
MULTIPLY,DIVIDE,SUBTRACT,MODULO,AND,>,>=,<,<=,==,!= | 100% missing

Operands: 100% missing with 100% missing
------------------------------------------------------------------------------
SUM,AVG,MIN,MAX,OR                                  | 100% missing
MULTIPLY,DIVIDE,SUBTRACT,MODULO,AND,>,>=,<,<=,==,!= | 100% missing

Operands: 50% missing with 0% missing
------------------------------------------------------------------------------
SUM,AVG,MIN,MAX,OR                                  | 0% missing
MULTIPLY,DIVIDE,SUBTRACT,MODULO,AND,>,>=,<,<=,==,!= | 50% missing

Operands: 50% missing with 25% missing
------------------------------------------------------------------------------
SUM,AVG,MIN,MAX,OR                                  | 25% missing
MULTIPLY,DIVIDE,SUBTRACT,MODULO,AND,>,>=,<,<=,==,!= | 50% missing

And finally, here are the rules correctly applied to the expressions in your example:

Timeseries Expression    | Missing values
---------------------------------------------------
A                        | [100, 100, 100]
B                        | [58,  0,   100]
A + B                    | [58,  0,   100]
A - B                    | [100, 100, 100]
(-1 * B)                 | [58,  0,   100]
A + (-1 * B)             | [58,  0,   100]

The case of subtraction is understandably unintuitive, but here is an explanation from one of our platform engineers:

In the algebraic world, any subtraction can be expressed as addition and vice-versa. However, in the real world, the completeness state of subtraction relies on both participating entities equally. Therefore, absence of one entity is as severe as the absence of one in case of multiplication. Hence the grouping of SUBTRACT with MULTIPLY and DIVIDE.

Here’s an example that should help reason the real world implications of SUBTRACT:
Consumption of ServicePointMINUSConsumption of one of its smart meters

Hope this helps.


Replace negative values with 0 (thresholding) in a metric