Limitations on the InterValType for map reduce jobs

#1

Is this a valid type for the InterValType in a map reduce job:

type ExampleIntermediate {
Key1:double
Key2:double
Key3:double
}

Does it need to be entity or persistable or anything, or is this fine?
I have used double without problem before, and the example has an int, are general c3 types allowed like this?

#2

This should be fine, have you tried it? Here’s an example that works for me:

entity type ProtectionIslandEventAssignerJob extends MapReduce<ProtectionIsland, string, PIEAssignmentWeight, int>
type PIEAssignmentWeight mixes Double {  
  /**
   * The new protection island that the event might be localized to.
   */
  protectionIsland: !ProtectionIsland
   
  /**
   * The probability of that the event should be localized to the island.
   */
  value: !~
}
#3

I am in the process of debugging a job that produces no output where I expect it, whereas a similar job that uses double as the intermediate type works, so I just wanted to check that there wasn’t an issue with the intermediate type being a non primitive or something else.

Does a MapReduce job need a type key at the end? The lightbulb example has AGG_CNS_MFG for example. Your example doesn’t have one.

See below.
entity type MyAggregateRollupJob mixes MapReduce<MyType1, string, AggregateRollupIntermediate, AggregateRollup> type key 'AGG_ROLLUP_JOB' {

Is there any way to see a count of how many map records got output? Or other metrics for a job so I can see what is going on?

map:

function map(batch, objs, job) {

    var intermediaryGroup = {};
    ...
    foreach (thing in things) { 
      if(! (myDynamicKeyHere in intermediaryGroup)){
           intermediaryGroup[myDynamicKeyHere] = AggregateRollupIntermediate.make();
          intermediaryGroup[myDynamicKeyHere].Key1 += 1 ;
      }
    }
    return intermediaryGroup 
}

Do I need to .create() the type or is the definition of the dictionary wrong?

Edit: I got one record to output now, so I just need to debug more.

1 Like
closed #4