Actual normalized points for incremental normalization


#1

I have read (in internal documentation) that we do normalize buckets of months duration when doing incremental normalization.
For example, if we have an already normalized timeseries that has measurements from January, 1st to May, 31th and we receive new measurements for period [June, 10th to June, 15th], the incremental normalization will fetch data from May, 1st to June 30th and normalize it.
It this correct?
Are we able to modify the bucket duration? (align to one week, one day…)


#2

@NabilKoroghli yes the normalizer tries to get the bare minimum amount of data required in order to correctly normalize the buckets. For e.g. since there is a gap from May 31st to June 10th, we need to fetch data for the month of may as well.

The bucket interval is dynamically decided based on the interval of normalization. So I recommend do provide the interval of normalization on the series header. Anything less than FIVE_MINUTE (exclusive of itself) will be bucketed into DAY and everything else will be bucketed into MONTH

The overhead for intervals equal to or greater than FIVE_MINUTE and being bucketed into MONTH is not super significant and hence the month bucketing.


#3

@rohit.sureka, we plan to do data integration each hour (for a grain of FIVE_MINUTE) and we would like to be more efficient when doing normalization by taking only days buckets (not months).
Is it possible to make it configurable?