Export to Parquet/CSV Error

#1

I’m using the Export API to export some data to S3. I’m getting an error when trying to export into Parquet or CSV format. But first, exporting to JSON works just fine, as follows:

var targetPath = "s3://some-bucket/C3Export";
var typeName = "RiskScore";
var historyName = “SomeAsset_RiskType”;
var prefix = targetPath + "/" + ["test_", DateTime.now().toString("yyyyMMddhhmmss"), typeName, historyName].join("_");
var spec = BatchExportSpec.make({
  numFiles: 1,
  deleteExisting: true,
  limit: -1,
  contentType: "application/json”,
  targetType: {typeName: typeName},
  filter: "parent.id=='" + historyName + "'",
  targetPath: prefix
});
var job = Export.startExport(spec);

However if I change contentType to vnd.apache.parquet+binary or text/csv then I get the following error:

job.status()
=> “carbon parameter is required for export to any content type other then xml."

First issue here is that BatchExportSpec doesn’t have a carbon parameter. Nonetheless, if I go ahead and include it as follows I get an error:
"Extraneous property "carbon" on object type "BatchExportSpec". [MakeValidationError]”.

(Second issue is that I have no idea what a carbon transform property is, in case I do need one…)

What do I need to change to successfully export into CSV/Parquet?

0 Likes

#2

When exporting to parquet, you need to specify a csvHeader. This should resolve the error.

var csvHeader = "field1, field2, field3, field4";
targetType = Ref.make({typeName:"MyType"});
var bes = BatchExportSpec.make({
			numFiles: 1, 
			limit:-1, 
			deleteExisting: true,
			csvHeader:csvHeader, 
			fileUrlOrEncodedPathPrefix: "test_", 
			targetType:targetType, 
			contentType:"application/vnd.apache.parquet+binary"});

var result = Export.startExport(bes);

When exporting to application/json, please specify the jsonHeader parameter.

0 Likes

#3

Thanks Scott, this resolved the issue.

0 Likes