[ https://issues.apache.org/jira/browse/HIVE-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934980#action_12934980 ]
Siying Dong commented on HIVE-1802: ----------------------------------- For any Group by, we needed 2 mem-copies. One from Text objects to buffer, one add an extra tag to the end of the buffer. Now, the case with single Text takes no mem-copy (except the first byte is 0) and for multiple keys it needs one (from Text object to buffer). For join, we needed 2 mem-copies. One from Text to buffer, one add tag. Now one single Text needs one copy from buffer to add a tag. Other cases we still need two copies. > Encode MapReduce Shuffling Keys Differently for Single string/bigint Key > ------------------------------------------------------------------------- > > Key: HIVE-1802 > URL: https://issues.apache.org/jira/browse/HIVE-1802 > Project: Hive > Issue Type: Improvement > Reporter: Siying Dong > Assignee: Siying Dong > Attachments: HIVE-1802.1.patch, HIVE-1802.2.patch > > > Delimiters are not needed if we only have one shuffling key, and in the same > time escaping delimiters are not needed. We can save some CPU time on > serializing and shuffle slightly less amount of data to save memory footprint > and network traffic. > Also there is a bug that for group-by, we by mistake add a -1 to the end of > the key and pay one more unnecessary mem-copy. Can be easily fixed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.