errorSummary: Queue for this Canonical type: ... doesn't exists


#1

I’m trying to load some data using a series of steps that worked just fine on v7.7.6.

This error that I’m hitting is on 7.8.1

  1. I curl my files (*.csv.gz) to S3 and verify that the files can be seen with:

FileSourceCollection.get(FSC).listFiles()._a)

  1. used SourceFile.syncFile()

  2. Attempt to process the file using:

SourceFile.get(<filename>).process()

In SourceFile.fetch(), I see that the file is almost immediately placed into rejected status.

The error is:

errorSummary: Queue for this canonical type: <mySourceType>, tenant: <myTenant>, tag: prod doesn't exists

There is no further information in DataLoadProcessLog.

Some questions to potential responders of this thread:

  1. Should the Canonical Queue not be automatically created when I provision?
  2. How can I troubleshoot this?
  3. How can I look in the type system to know which canonical queues do exist?

#2

@paulyip Yes you are right, looks like the canonical jms queues haven’t been created. Did we set the

TenantConfig.putConfig('JMSEnabled', true) 

Also after doing the above it is possible to verify if the jms queues exist using

JMS.listQueues()

Another thing is if you want to avoid jms you can set the jmsDisabled flag on the FileSourceCollection for who you are loading data to be true, to bypass jms and load the file


#3

Thanks Garry for your reply.

I ran: TenantConfig.putConfig('JMSEnabled', true)
And then, when I tried : JMS.listQueues()

I’m getting the error:

Failed to connect to Tibco Server tcp://stage-presales-tibco-01:17232

It seems like Tibco may be down or communications is blocked?


#4

Yes. This is where Operations team can help.


#5

Just an update.

I was able to use the 2nd suggestion from Garry:

x = FileSourceCollection.get(<myFileSourceCollection>)
x.jmsDisabled=true
x.upsert()

Now, file processing works.

However, now here are some follow-on questions:

By setting jmsDisabled=true, if not JMS, what is the data processing framework/method that is used? Do I forego/bypass invalidation queues all together?

Should I be setting jmsDisabled=true for all my FSC’s? or is the right solution to get that connection issue with Tibco resolved?

I heard about changes to Queues in 7.8.x – so wondering if there is more to this story.

Thanks

Paul


#6

jmsDisabled=true is to avoid using jms for chunking in data-loading due to various issues observed(connectivity issues, tibco service issues, chunks being stuck, cache issues). The data is streamed from the file and processed in a single thread.

However this is not the solution to the connectivity issue but a workaround. As Yaro mentioned earlier, kindly check with OPs about the connectivity to the tibco service.

Once you have the service up and running, you might not need jmsDisabled=true to be set on all FSCs.