copyFile, moveFile does not retain metadata values


#1

when using AzureFileSystem.copyFiles, I see that the copied files do not retain the original file’s meta data, specifically contentType (“text/csv”) and contentEncoding (“gzip”).

Is there a platform based solution to this?

Need to move about 50K files between environments as the customer expects C3 to have utilities to transfer files without them having to reload between environments.


#2

you should use File.copy and File.move and not use specific cloud apis.


#3

Negative, unless there is an option to preserve the metadata with File.copy. This first one is the original file

Then I moved it with the following:

File.make(“azure://sbox-qgc/fs/shellqgc/prod/inbox/CanonicalWell/inbox/Well_01022019.csv.gz”).copy(“azure://sbox-qgc/fs/shellqgc/historical/inbox/CanonicalWell/Well_01022019.csv.gz”);

Then look what happens to the target:


#5

I fixed it as follows (but this approach is only ok for one or two files):

File.make(‘azure://sbox-qgc/fs/shellqgc/historical/inbox/CanonicalWell/Well_01022019.csv.gz’).replaceContentTypeAndEncoding(“text/csv”,“gzip”);


#6

@FerencM hope you created a ticket for platform to fix this. should be specific issue for Azure only and a simple fix.
But again I will say that use File.fromString File.make and so on. Platform should figure underlying file system for you.