Recommanded way for saving a Timeseries


#1

I need to precompute multiple timeseries resulting from rollupMetrics in a cron job and save them somewhere in the platform to reuse them afterward in a chart (rollupMetrics may take more that 1mn to complete).
Is there an elegant way or existing type for saving those timeseries ?
My current solution is:

  • Declare a TimedDataHeader type RollupCache having a metricName property to distinguish between several timeseries (sufficient for my case).
  • Declare a TimedDataPoint RollupCachePoint with a double value property to save the timeseries data points (I assume that all my timeseries are returning double values)
  • Manage the upsert of the RollupCache and its data points with a custom static method on RollupCache upsertFromTimeseries that handles the data point persistance and the removal of the previous data points.
    The function looks like this (not final):
function upsertTimeseries(metricName, timeseries)  {

    if (timeseries) {
        var fetchRes = RollupCache.fetch({ filter: Filter.eq("metricName", metricName) });
        var header;

        if (fetchRes.count === 0) {
            header = RollupCache.create({ metricName: metricName });
        }  else {
            header = fetchRes.objs[0];
            // remove the existing points
            _.each(header.data, function (pt) {
                RollupCachePoint.remove(pt.id);
            });
        }

        var data = timeseries.data();
        var dates = timeseries.dates();

        for (var i = 0; i < data.length; i++) {
            RollupCachePoint.create({ parent: header, value: data[i], start: dates[i] });
        }

        return header;
    }}
  • mix RollupCache with MetricEvaluatable and create a SimpleMetric on RollupCache
  • evaluate the created SimpleMetric on a RollupCache for a specific metricName filter in order to get my original timeseries.

It there a simpler solution to achieve this?


#2

You can specify an id for RollupCachePoint and RollupCache to avoid checking for existence and removing existing data.

metricName and timeseries can be required parameters so you don’t need to check whether timeseries exists.

the code can be simplified to

var header = RollupCache.merge({id: metricName, metricName: metricName});
var data = timeseries.data();
var dates = timeseries.dates();

for (var i = 0; i < data.length; i++) {
   RollupCachePoint.merge({
      id: Cassandra.c3Id(header.id, dates[i].toString('yyyy-MM-dd')),
      parent: header,
      value: data[i],
      start: dates[i],
   });
}

You will have to create new metric to read the cached data.


#3

It is also possible to store the whole timeseries into Cassandra at once, e.g., field data being of type [double]. There is a tradeoff: storing a single point fits Cassandra nicely but has higher space overhead (id and parent per point); storing a timeseries at once seems to work fine and that might even match your use case better.