[
https://issues.apache.org/jira/browse/GOBBLIN-1126?focusedWorklogId=430435&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430435
]
ASF GitHub Bot logged work on GOBBLIN-1126:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 04/May/20 21:46
Start Date: 04/May/20 21:46
Worklog Time Spent: 10m
Work Description: autumnust commented on a change in pull request #2966:
URL: https://github.com/apache/incubator-gobblin/pull/2966#discussion_r419748400
##########
File path:
gobblin-compaction/src/main/java/org/apache/gobblin/compaction/mapreduce/orc/OrcKeyDedupReducer.java
##########
@@ -37,6 +53,30 @@ protected void setOutKey(OrcValue valueToRetain) {
// do nothing since initReusableObject has assigned value for outKey.
}
+ @Override
+ protected void reduce(OrcKey key, Iterable<OrcValue> values, Context context)
+ throws IOException, InterruptedException {
+
+ /* Map from hash of value(Typed in OrcStruct) object to its times of
duplication*/
+ Map<Integer, Integer> valuesToRetain = new HashMap<>();
+ int valueHash = 0;
+
+ for (OrcValue value : values) {
Review comment:
The benefit is during the sorting phase and MR is going to compare key,
the comparison is down to a much shortened object instead of the whole object
(and we get rid of complex column like array , map, etc.). This is improving
map throughput quite a bit, and on the reducer side we are not doing much more
if the shuffle key is selected properly: We chose uuid + header.time and
chances for two different records to collide on this pair is pretty low.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 430435)
Time Spent: 1h 10m (was: 1h)
> Supporting Column Projection in OrcStruct in compaction for making shuffle
> key slimmer
> --------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1126
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1126
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Lei Sun
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
