Gera Shegalov created MAPREDUCE-5932: ----------------------------------------
Summary: Provide an option to use a dedicated reduce-side shuffle log Key: MAPREDUCE-5932 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5932 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.4.0 Reporter: Gera Shegalov Assignee: Gera Shegalov For reducers in large jobs our users cannot easily spot portions of the log associated with problems with their code. An example reducer with INFO-level logging generates ~3500 lines / ~700KiB lines per second. 95% of the log is the client-side of the shuffle {{org.apache.hadoop.mapreduce.task.reduce.*}} {code} $ wc syslog 3642 48192 691013 syslog $ grep task.reduce syslog | wc 3424 46534 659038 $ grep task.reduce.ShuffleScheduler syslog | wc 1521 17745 251458 $ grep task.reduce.Fetcher syslog | wc 1045 15340 223683 $ grep task.reduce.InMemoryMapOutput syslog | wc 400 4800 72060 $ grep task.reduce.MergeManagerImpl syslog | wc 432 8200 106555 {code} Byte percentage breakdown: {code} Shuffle total: 95% ShuffleScheduler: 36% Fetcher: 32% InMemoryMapOutput: 10% MergeManagerImpl: 15% {code} While this is information is actually often useful for devops debugging shuffle performance issues, the job users are often lost. We propose to have a dedicated syslog.shuffle file. -- This message was sent by Atlassian JIRA (v6.2#6252)