Map, Reduce, MapCompleted, Completed Dependencies

It seems that there are some dependencies among these function in MapReduce. For instance, reduce does not run unless all maps are successful.

What are the dependencies and requirements for these functions to run? For instance, I noticed I can’t run reduce unless the map returns results and reduce references these results? However, I want to run a function only if all the maps are successful, and I’m not sure if mapCompleted() or completed() would accomplish this.

Additionally, if mapCompleted() or completed() fail, it would appear that the MapReduceRun returns completed regardless, indicating nothing went wrong?

1 Like

mapCompleted() or completed() are essentially just callbacks and should not have any bearing on the status of a map or reduce step. If map is completed without errors and there are results from the map step, then reduce step should have run automatically.

Can you verify the the following:

  1. MR job has no errors
  2. fetch MapResults with filter “runIdAndOutKey==”
  3. does this show up in logs “Processing reduce of mr_job=”