[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated PIG-984: --- Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed. Thanks Richard. > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch, PIG-984_1.patch, PIG-984_1.patch, > PIG-984_1.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Status: Patch Available (was: Open) > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch, PIG-984_1.patch, PIG-984_1.patch, > PIG-984_1.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Attachment: PIG-984_1.patch Fix the compile errors. > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch, PIG-984_1.patch, PIG-984_1.patch, > PIG-984_1.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Status: Open (was: Patch Available) > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch, PIG-984_1.patch, PIG-984_1.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Attachment: PIG-984_1.patch This patch removed the debug (info) message. > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch, PIG-984_1.patch, PIG-984_1.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Status: Patch Available (was: Open) > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch, PIG-984_1.patch, PIG-984_1.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Status: Open (was: Patch Available) > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch, PIG-984_1.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Status: Patch Available (was: Open) > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch, PIG-984_1.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Attachment: PIG-984_1.patch This patch fixed the above QA errors. > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch, PIG-984_1.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Status: Open (was: Patch Available) > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Attachment: PIG-984.patch Based on the feedbacks, the hint is changed to 'using "collected". It's now implemented as a map-side group operation. This patch contains the implementation. > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Status: Patch Available (was: Open) > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > Attachments: PIG-984.patch > > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-984) PERFORMANCE: Implement a map-side group operator to speed up processing of ordered data
[ https://issues.apache.org/jira/browse/PIG-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-984: - Assignee: Richard Ding > PERFORMANCE: Implement a map-side group operator to speed up processing of > ordered data > > > Key: PIG-984 > URL: https://issues.apache.org/jira/browse/PIG-984 > Project: Pig > Issue Type: New Feature >Reporter: Richard Ding >Assignee: Richard Ding > > The general group by operation in Pig needs both mappers and reducers (the > aggregation is done in reducers). This incurs disk writes/reads between > mappers and reducers. > However, in the cases where the input data has the following properties >1. The records with the same key are grouped together (such as the data is > sorted by the keys). >2. The records with the same key are in the same mapper input. > the group by operation can be performed in the mappers only and thus remove > the overhead of disk writes/reads. > Alan proposed adding a hint to the group by clause like this one: > {code} > A = load 'input' using SomeLoader(...); > B = group A by $0 using "mapside"; > C = foreach B generate ... > {code} > The proposed addition of using "mapside" to group will be a mapside group > operator that collects all records for a given key into a buffer. When it > sees a key change it will emit the key and bag for records it had buffered. > It will assume that all keys for a given record are collected together and > thus there is not need to buffer across keys. > It is expected that "SomeLoader" will be implemented by data systems such as > Zebra to ensure the data emitted by the loader satisfies the above properties > (1) and (2). > It will be the responsibility of the user (or the loader) to guarantee these > properties (1) & (2) before invoking the mapside hint for the group by > clause. The Pig runtime can't check for the errors in the input data. > For the group by clauses with mapside hint, Pig Latin will only support group > by columns (including *), not group by expressions nor group all. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.