Hi, In practice, if the consumer vertex parallelism = 1 then scatter-gather and broadcast will behave the same since a 1-partition scatter-gather is equivalent to a broadcast ☺
So if your consumer vertex parallelism = 1, then you can use OrderedPartitionedKVEdgeConfig and it will work like a broadcast even though it is using scatter-gather. You don’t need to specify broadcast property. Bikas From: Azuryy Yu [mailto:[email protected]] Sent: Thursday, March 26, 2015 11:59 PM To: [email protected] Subject: Re: Why broadcast Edge property? Sorry I didn't make my question clear, I mean if I use OrderedPartitionedKVEdgeConfig , and use broadcast edge property with parallism 1, can I get a correct result or Tez will complain exception? On Fri, Mar 27, 2015 at 2:52 PM, Jianfeng (Jeff) Zhang <[email protected]<mailto:[email protected]>> wrote: OrderedPartitionedKVEdgeConfig is only for ScatterGather. UnorderedKVEdgeConfig is for broadcast and 1-1 edge. Usually broadcast output is not sorted OrderedPartitionedKVEdgeConfig.java public EdgeProperty createDefaultEdgeProperty() { EdgeProperty edgeProperty = EdgeProperty.create(EdgeProperty.DataMovementType.SCATTER_GATHER, EdgeProperty.DataSourceType.PERSISTED, EdgeProperty.SchedulingType.SEQUENTIAL, OutputDescriptor.create( getOutputClassName()).setUserPayload(getOutputPayload()), InputDescriptor.create( getInputClassName()).setUserPayload(getInputPayload())); Utils.setEdgePropertyHistoryText(this, edgeProperty); return edgeProperty; } Best Regard, Jeff Zhang From: Azuryy Yu <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Friday, March 27, 2015 at 2:45 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Why broadcast Edge property? OrderedPartitionedKVEdgeConfig
