rahulLiving commented on code in PR #2623:
URL: https://github.com/apache/phoenix/pull/2623#discussion_r4259072460
##########
phoenix-core-server/src/main/java/org/apache/phoenix/mapreduce/PhoenixInputFormat.java:
##########
@@ -82,6 +94,34 @@ public RecordReader<NullWritable, T>
createRecordReader(InputSplit split,
@Override
public List<InputSplit> getSplits(JobContext context) throws IOException,
InterruptedException {
+ boolean coalescingEnabled =
context.getConfiguration().getBoolean(SPLIT_COALESCING_ENABLED,
+ DEFAULT_SPLIT_COALESCING_ENABLED);
+ if (coalescingEnabled) {
+ // Coalescing merges all of a RegionServer's splits into a single
mapper, so stats-based
+ // splitting (which fragments each region into many sub-scan splits to
gain parallelism) would
+ // just be undone here, wasting the guidepost work. Force
region-granular splits before
+ // coalescing, mirroring PhoenixSyncTableTool.
+ if
(PhoenixConfigurationUtil.getSplitByStats(context.getConfiguration())) {
+ LOGGER.info(
+ "Split coalescing is enabled ({}); disabling stats-based splitting
({}) so "
+ + "region-granular splits are coalesced per RegionServer",
+ SPLIT_COALESCING_ENABLED,
PhoenixConfigurationUtil.MAPREDUCE_SPLIT_BY_STATS);
+ }
+ PhoenixConfigurationUtil.setSplitByStats(context.getConfiguration(),
false);
+ }
+ List<InputSplit> baseSplits = getBaseSplits(context);
+ return coalescingEnabled
+ ? RegionServerSplitCoalescer.coalesceWithGuard(baseSplits)
+ : baseSplits;
+ }
+
+ /**
+ * Generates the raw region-boundary splits (one {@link PhoenixInputSplit}
per region, or per scan
+ * when splitting by stats) without any coalescing. Exposed as a seam so
subclasses that need to
+ * post-process the region-granular splits (e.g. filter already-processed
regions) can do so
+ * before opting into coalescing themselves.
+ */
+ protected List<InputSplit> getBaseSplits(JobContext context) throws
IOException {
Review Comment:
I don't think we need a seperate method here and avoid updating any existing
usage ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
