C3CSV on large dataset causes browser OOM

#1

When trying to export a dataset to CSV using c3CSV, chrome responded with ‘paused before potential out of memory’ error.

Essentially, I want to take a dataset from one env/tag and be able to import it into another, since we have corrected all the data. Ideally a curl command that invokes something like Objectname.fetch() would be ideal.

Anyone got any pointers?

1 Like

#2

You can leverage Export API to export records of a certain type into a file on S3. That file can then be downloaded via curl.

1 Like

#3

This worked a charm for me, thanks @yaroslav

for others wanting to copy this, please be mindful of quotes and double quotes in the text
var TARGET_BUCKET = “s3://c3–sbox-origin2-origin-test”;
var BUCKET_PATH = “exports”;

var typeName = "WellStreamMeasurementSeries";
var csvHeader = "parent,unitConstraint,measurementType,interval,treatment";
var prefix = TARGET_BUCKET + "/" + BUCKET_PATH + "/" + [typeName, DateTime.now().toString("yyyyMMddhhmmss")].join("_");
var spec = BatchExportSpec.make({
  numFiles: 1,
  limit: -1,
  contentType: "text/csv",
  csvHeader: csvHeader,
  targetType: {typeName: typeName},
  targetPath: prefix
});
var job = Export.startExport(spec);

I then used S3.listFiles to find the exact file name with full path,
S3.listFiles('s3://c3--sbox-origin2-origin-test/exports/')

then issued a curl command to download the file to my local terminal, as follows:
curl -o "CanonicalWellMeasurementSeriesConfig.csv" https://origin-qa.c3iot.com/file/1/origin/prod/s3://c3--sbox-origin2-origin-test/exports/WellStreamMeasurementSeries_20180606091505-4ZP81RCQHL-0.csv -u ferenc.mantfeld@c3iot.com:***password****

Please be mindful that to specify which fields to export, these are listed in the the csvHeader variable above as a comma-separated list of field names. For the resultant file to be used as an input, the column header names need to match the Canonical definition, case sensitive!!!

2 Likes