jeongyooneo opened a new pull request #30: [NEMO-63] On-demand dynamic optimization metric aggregation URL: https://github.com/apache/incubator-nemo/pull/30 JIRA: [NEMO-63: On-demand dynamic optimization metric aggregation](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-63) **Major changes:** - As metric data used in DataSkewRuntimePass are sent via NCSMessage and piled up until stage completion, it caused OOM in 100~200GB-scale skew experiment. This PR aggregates them on-demand to avoid this. **Minor changes to note:** - Added MapReduceSkew beam example, which generates skewed key-value tuple from skewed input in map stage and calculates median of the per-key values in reduce stage. **Tests for the changes:** - Adapted DataSkewRuntimePassTest according to the change **Other comments:** - N/A resolves [NEMO-63](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-63)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
