SourceFIle.processAll() NullPointerException


#1

Hi there. I am trying to integrate data to a package stored on a local server. I curled with the following command:

curl -u BA:BA -v -H Content-Type:text/csv -X PUT --data-binary @./SmartBulb.csv http://localhost:8080/import/1/lightbulb/kasdenbunkerc3iotcom/CanonicalSmartBulb/SmartBulb.csv

And it said the data uploaded successfully. However, no data appeared in the console (Empty SmartBulb.fetch() and DataLoadProcessLog.fetch()).

FileSourceSystem exists, FileSourceCollection includes CanonicalSmartBulb, and SourceFile seems to be good with path matching the rootUrl of the FileSourceSystem (screenshots attached).


I am unsure if I am supposed to SourceFile.syncAll() and SourceFile.processAll(), but when I run processAll(): I get the following error:

all.js?env=browser&compat:19965 POST http://localhost:8080/api/1/lightbulb/kasdenbunkerc3iotcom/SourceFile?action=processAll 500 (Server Error)
C3.client.sendRequest @ all.js?env=browser&compat:19965
request @ all.js?env=browser&compat:860
call @ all.js?env=browser&compat:566
c3Call @ all.js?env=browser&compat:99
_call @ all.js?env=browser&compat:2704
(anonymous) @ VM4166:5
(anonymous) @ VM4354:1
all.js?env=browser&compat:569 Uncaught C3.client.ActionError {name: "ActionError", message: "wrapped NullPointerException", error: C3.t…s.Obj, stack: "Error: wrapped NullPointerException↵    at new C3.…114:20), <anonymous>:5:8)↵    at <anonymous>:1:12"}

Any help either with this specific error or if I’ve taken a wrong step would be much appreciated!


Error when importing file with CURL: default value doesn't exist!
#2

I’m not sure about the NullPointerException you got from calling processAll, but make sure to have the TenantConfig JMSEnabled set to true:

TenantConfig.configStr('JMSEnabled')
// if false run:
TenantConfig.putConfig('JMSEnabled', true)

#3

I checked it and mine is set to true, but thanks for the suggestion


#4

@kasden when you curled file should have been loaded. are any of your queues paused? run InvalidationQueue.resumeAll()
Also could you also try

SourceFile.get('<id>').process()

#5

I ran that resume command, but to no avail. Calling process() on the SourceFile instance returns the same wrapped NullPointerException error. Is there more documentation on the process() function? I’m inclined to believe my csv, canonical, transform, or type file is wrong, though they should all 4 be copied exactly from the lightbulb tutorials.


#6

Update. Thanks to Garry for helping me figure everything out! The problem was that even though it looked like the csv file was successfully transferred to the server, it was essentially just an empty file placeholder that causes process() to error. We still have not figured out if there is a correct curl command that will transfer the csv file contents correctly, but it is possible to upload the files by re-defining the FileSourceSystem rootUrl to the local machine like they do here:

https://c3energy.atlassian.net/wiki/spaces/C3DS/pages/674824263/Running+c3-server+locally+on+docker+and+provisioning+data+locally

And then using docker cp to manually transfer the files to the local server running on Docker.

Still, if anyone knows why the curl command wasn’t uploading the file contents correctly, that would be valuable knowledge for this thread for the future.


#7

Okay, after much digging, I fixed the problem and I think I actually understand how it works now. The volume on my docker was not linked to the correct directory on my local machine. If it’s linked correctly, the data files will be synced automatically and you won’t have to use docker cp to transfer them.

To conclude, if you ever get this error, you might be having the same problem that I did: The rootUrl of your FileSourceSystem is pointing to a wrong directory. This could be because it points to an s3 bucket that is invalid or that you don’t have access to, or because it points to some directory that does not contain the csv files. Just because a curl command says “upload fine”, that does not mean the files were actually transferred to the location where your FileSourceSystem will look for them.

To upload data files locally and without S3, manually point your FileSourceSystem to a directory on your docker that is linked to your local data directory, as defined in this tutorial https://c3energy.atlassian.net/wiki/spaces/C3DS/pages/674824263/Running+c3-server+locally+on+docker+and+provisioning+data+locally
If you continue to get the same error, make sure that your data folder on the docker really is linked to your local data folder, such as by bashing into the docker console and checking the folder contents.