BatchExportSpec creating pipe ('|') delimited files


#1

I am looking to create an export in S3 off of a type, but I want it to be pipe (’|’) separated, as opposed to a JSON file that is created. Or is it possible to set it a to “text/csv”, and then using the csvHeader & csvFormat fields on ExportDataCommonSpec I map the values to a pipe formatting?

My attempt goes as follows:
var spec = BatchExportSpec.make({
numFiles: 1,
limit: -1,
contentType: ‘application/json’, // To be modified
fileUrlOrEncodedPathPrefix: categoryName + “" + clientName + "” + time,
targetType: {typeName: typeName},
targetPath: folderStructure
// csvHeader: ##
// csvFormat: map<"", ">
});
var job = Export.startExport(spec);


#2

When you export a CSV file, you need to provide in the BatchExportSpec:

  • csvHeader which is a comma separated of the fields in your type that needs to be exported.
  • csvFormat is optional, it’s for formatting the values, where each field in csvHeader is a key and the value is the formatting.

For example,

var spec = BatchExportSpec.make({
  contentType: 'text/csv',
  csvHeader: '..., timestamp, ...',
  csvFormat: {
    "timestamp": DateTimeFormat.fromString("yyyy-MM-dd'T'HH:mm z")
  },
...
}

I don’t think you can specify a custom separator, the csv file will be using the default , separator. But you can customize your exported data by having a targetAction but I doubt this will considerably slow down the export.


#3

@tedmister @bachr I think content type should have the option where we pass delimiter in the content type
could you try

contentType: "text/csv; delimiter = '|'"

If this does not work we should create a ticket to do so.


#4

Adding the delimiter of ‘|’ didn’t modify the actual delimiter produced in the output file. I will be going down the route of adding a targetAction to modify the delimiter. It would be nice functionality to add if we could modify the delimiter from the contentType declaration.


#5

Yes, we do support it in import in DI, I don’t see why we should not support here. Lets create a platform ticket and then add a TODO to your code.


#6

That’s good to know, I will do so. Thanks for all your help.


closed #7

#8

@garrynigel Hey Garry, Do you have any thoughts to share on the targetAction that Bachir suggested to accomplish this same behavior we are looking for?


#9

Here is an example on how to use targetAction and have | delimiter Customizing exported data with targetAction


#10

This feature is available in later versions of 7.8. You can specify delimiter in the content-type as shown below:

    Export.startExport(BatchExportSpec.make({ 
    targetType: "TypeName", 
    contentType: "text/csv; delimiter=|", 
    limit:5, 
    csvHeader: "id|name|number", 
    targetPath: "test", 
    numFiles:1
 }))