[
https://issues.apache.org/jira/browse/PHOENIX-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133597#comment-15133597
]
James Taylor commented on PHOENIX-1973:
---------------------------------------
Nice work, [~sergey.soldatov]. Reduce shuffle bytes went from 1027596399 to
113138799 (9x lower). What's the time difference between before and after?
> Improve CsvBulkLoadTool performance by moving keyvalue construction from map
> phase to reduce phase
> --------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-1973
> URL: https://issues.apache.org/jira/browse/PHOENIX-1973
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Rajeshbabu Chintaguntla
> Assignee: Sergey Soldatov
> Fix For: 4.4.1
>
> Attachments: PHOENIX-1973-1.patch, PHOENIX-1973-2.patch
>
>
> It's similar to HBASE-8768. Only thing is we need to write custom mapper and
> reducer in Phoenix. In Map phase we just need to get row key from primary key
> columns and write the full text of a line as usual(to ensure sorting). In
> reducer we need to get actual key values by running upsert query.
> It's basically reduces lot of map output to write to disk and data need to be
> transferred through network.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
