Curl to download datafile from S3 to local drive


#1

This is just a quick tip to anyone that wants to download datafiles that were originally loaded to any S3 bucket visible to a tag and now wants to download that file to a local directory.

First, locate the exact file in the S3 bucket, using S3.listFiles (example below):
S3.listFiles(‘s3://igbuild-digital-data-staging/Pi_Priority_Data/Csv/’);

Be sure to expand the file list to capture the entire file name (if the name is too long, a sneaky ‘…’ appears in the file name, which can be difficult to spot, unless you are aware of the possibility).

Then once you have that, use curl in terminal and the tag url to download the file (-o option of curl saves the file):
curl -o “GasThrouput1.csv” https://tenant-stage.c3iot.com/file/1/tenant/ferenc/s3://igbuild-digital-data-staging/Pi_Priority_Data/Csv/C3GasThroughPumpModel.csv/part-00000-e53d8736-9015-456c-aa2e-3b562760bf5d-c000.csv -u username.lastname@c3iot.com:*****

The above will save a file called GasThroughput1.csv to your current directory. As it runs, you will see the progress bar showing you how much of the file has been downloaded, as the screenshot below:
image


#2

What if I want to download that file through an API?


Downloading S3 File through UI Action
#3

If the file is not too large, you can use something like:

var data = S3File.make({url: "s3://bucket/path/to/file"}).readString();
var filename = "data.txt"
console.save(data, filename)

Before that, make sure to run this snippet that basically create an html element then dispatch a click event to trigger the export:

(function(console){

    console.save = function(data, filename){

        if(!data) {
            console.error('Console.save: No data')
            return;
        }

        if(!filename) filename = 'console.json'

        if(typeof data === "object"){
            data = JSON.stringify(data, undefined, 4)
        }

        var blob = new Blob([data], {type: 'text/json'}),
            e    = document.createEvent('MouseEvents'),
            a    = document.createElement('a')

        a.download = filename
        a.href = window.URL.createObjectURL(blob)
        a.dataset.downloadurl =  ['text/json', a.download, a.href].join(':')
        e.initMouseEvent('click', true, false, window, 0, 0, 0, 0, 0, false, false, false, false, 0, null)
        a.dispatchEvent(e)
    }
})(console)

#4

It’s also possible to use from console the c3CSV command:

c3CSV
Download a comma-separated values (spreadsheet) file of the argument object.

c3CSV(data)

#7

An alternative if you are using Mac or Unix based operating system is to run xargs n -1 curl -u -O <

where includes the C3 url and then the S3 path direct to that file.


#8

Did you ever implement this through any sort of UI action? I’m able to do this successfully through console, but am struggling with a UI action implementation.


#9

Hi there, my request seems related to this thread.
How can I download a csv file created in Jupyter? I fetched some types, did some manipulations/transformations, now I want to save a dump of data to my local machine. How do I do it?

I tried to follow FerencM tip but I could not locate the exact file using the S3 bucket using S3.listFiles. Plus curl doesn’t work from

First, locate the exact file in the S3 bucket, using S3.listFiles (example below):
S3.listFiles(‘s3://igbuild-digital-data-staging/Pi_Priority_Data/Csv/’);

Thanks
Alessandro