Cron job firing ahead of time

#1

We noticed that a daily cron job sporadically kicks off way ahead of time. Any ideas why this may be happening?

#2

@yaroslav Not sure. I’ve never seen it before

#3

@yaroslav I do seem to recall when we had this before that the servers times had drifter apart and one server thought it was later than it was and pushed the queue entry to “pending” and it got executed on another server and that set the time to what it thought it was. I’m suspecting that here. Can you check the time on the various servers?

#4

@trothwein is there a way to do that from console? Tunneling and JS.exec() ?

#5

I think you can just tunnel and do DateTime.now() (or something like that)

#6

@trothwein This is what I did and the server time is in sync:

var ch = Cluster.hosts();
function t() { return Date.now().toString(); }
ch.forEach(function(f){
	var nodeId = f.nodeId + ':8080';
	c3Tunnel(nodeId);
	console.log("node: "+ nodeId + ": " + JS.exec(t));
	c3Tunnel();
})

Here is the output:

#7

@yaroslav Are you positive there aren’t any that were cutoff in that screen shot? We’ll have to dig further if not.

#8

No. Here is the full screenshot:

40 workers and 2 masters. Let me know if you want a ticket and I can create one.

#9

@yaroslav Yes, please file a ticket and include the url (canonical is preferable). I’m wondering if the “recoverStuck” operation is incorrectly thinking the queue entry is stuck and changing it to pending.

#10

Update: this turned out to be an issue with “concurrent” configuration option. There was a server bug that is fixed in 7.9.