okumin opened a new pull request, #320: URL: https://github.com/apache/tez/pull/320
This is a sample application that uses FairShuffleVertexManager. We can use it like this. ``` $ hadoop jar /opt/tez/tez-examples-*.jar demuxerdatagen hdfs:///tmp/in 30 200 $ hadoop jar /opt/tez/tez-examples-*.jar demuxer hdfs:///tmp/in hdfs:///tmp/out 100 $ hadoop jar /opt/tez/tez-examples-*.jar demuxer hdfs:///tmp/in hdfs:///tmp/out_precise 100 true ``` The following are the results on my machine. We can see skewed keys are written by multiple reducers. ``` zookage@client-node-0:~$ hdfs dfs -ls /tmp/out Found 33 items -rw-r--r-- 3 hdfs supergroup 0 2023-12-11 09:20 /tmp/out/_SUCCESS -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:18 /tmp/out/category-00000-r-00007 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:18 /tmp/out/category-00001-r-00007 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:18 /tmp/out/category-00002-r-00007 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:18 /tmp/out/category-00003-r-00007 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:18 /tmp/out/category-00004-r-00007 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:18 /tmp/out/category-00005-r-00007 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:18 /tmp/out/category-00006-r-00008 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:18 /tmp/out/category-00007-r-00008 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:18 /tmp/out/category-00008-r-00008 -rw-r--r-- 3 hdfs supergroup 6000 2023-12-11 09:18 /tmp/out/category-00009-r-00008 -rw-r--r-- 3 hdfs supergroup 15000 2023-12-11 09:17 /tmp/out/category-00010-r-00000 -rw-r--r-- 3 hdfs supergroup 30000 2023-12-11 09:17 /tmp/out/category-00011-r-00000 -rw-r--r-- 3 hdfs supergroup 60000 2023-12-11 09:17 /tmp/out/category-00012-r-00000 -rw-r--r-- 3 hdfs supergroup 120000 2023-12-11 09:17 /tmp/out/category-00013-r-00000 -rw-r--r-- 3 hdfs supergroup 243000 2023-12-11 09:17 /tmp/out/category-00014-r-00000 -rw-r--r-- 3 hdfs supergroup 489000 2023-12-11 09:17 /tmp/out/category-00015-r-00000 -rw-r--r-- 3 hdfs supergroup 981000 2023-12-11 09:17 /tmp/out/category-00016-r-00001 -rw-r--r-- 3 hdfs supergroup 1965000 2023-12-11 09:17 /tmp/out/category-00017-r-00001 -rw-r--r-- 3 hdfs supergroup 3930000 2023-12-11 09:17 /tmp/out/category-00018-r-00001 -rw-r--r-- 3 hdfs supergroup 7863000 2023-12-11 09:17 /tmp/out/category-00019-r-00001 -rw-r--r-- 3 hdfs supergroup 15726000 2023-12-11 09:17 /tmp/out/category-00020-r-00001 -rw-r--r-- 3 hdfs supergroup 31455000 2023-12-11 09:17 /tmp/out/category-00021-r-00001 -rw-r--r-- 3 hdfs supergroup 62913000 2023-12-11 09:20 /tmp/out/category-00022-r-00002 -rw-r--r-- 3 hdfs supergroup 125829000 2023-12-11 09:20 /tmp/out/category-00023-r-00002 -rw-r--r-- 3 hdfs supergroup 251658000 2023-12-11 09:20 /tmp/out/category-00024-r-00002 -rw-r--r-- 3 hdfs supergroup 503316000 2023-12-11 09:20 /tmp/out/category-00025-r-00002 -rw-r--r-- 3 hdfs supergroup 1006632000 2023-12-11 09:20 /tmp/out/category-00026-r-00002 -rw-r--r-- 3 hdfs supergroup 2013264000 2023-12-11 09:20 /tmp/out/category-00027-r-00002 -rw-r--r-- 3 hdfs supergroup 2093796120 2023-12-11 09:18 /tmp/out/category-00028-r-00003 -rw-r--r-- 3 hdfs supergroup 1932734880 2023-12-11 09:18 /tmp/out/category-00028-r-00004 -rw-r--r-- 3 hdfs supergroup 4187592240 2023-12-11 09:20 /tmp/out/category-00029-r-00005 -rw-r--r-- 3 hdfs supergroup 3865469760 2023-12-11 09:20 /tmp/out/category-00029-r-00006 ``` ``` zookage@client-node-0:~$ hdfs dfs -ls /tmp/out_precise Found 39 items -rw-r--r-- 3 hdfs supergroup 0 2023-12-11 09:26 /tmp/out_precise/_SUCCESS -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:24 /tmp/out_precise/category-00000-r-00016 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:24 /tmp/out_precise/category-00001-r-00016 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:24 /tmp/out_precise/category-00002-r-00016 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:24 /tmp/out_precise/category-00003-r-00016 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:24 /tmp/out_precise/category-00004-r-00016 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:24 /tmp/out_precise/category-00005-r-00016 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:24 /tmp/out_precise/category-00006-r-00017 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:24 /tmp/out_precise/category-00007-r-00017 -rw-r--r-- 3 hdfs supergroup 3000 2023-12-11 09:24 /tmp/out_precise/category-00008-r-00017 -rw-r--r-- 3 hdfs supergroup 6000 2023-12-11 09:24 /tmp/out_precise/category-00009-r-00017 -rw-r--r-- 3 hdfs supergroup 15000 2023-12-11 09:23 /tmp/out_precise/category-00010-r-00000 -rw-r--r-- 3 hdfs supergroup 30000 2023-12-11 09:23 /tmp/out_precise/category-00011-r-00000 -rw-r--r-- 3 hdfs supergroup 60000 2023-12-11 09:23 /tmp/out_precise/category-00012-r-00000 -rw-r--r-- 3 hdfs supergroup 120000 2023-12-11 09:23 /tmp/out_precise/category-00013-r-00000 -rw-r--r-- 3 hdfs supergroup 243000 2023-12-11 09:23 /tmp/out_precise/category-00014-r-00000 -rw-r--r-- 3 hdfs supergroup 489000 2023-12-11 09:23 /tmp/out_precise/category-00015-r-00000 -rw-r--r-- 3 hdfs supergroup 981000 2023-12-11 09:24 /tmp/out_precise/category-00016-r-00001 -rw-r--r-- 3 hdfs supergroup 1965000 2023-12-11 09:24 /tmp/out_precise/category-00017-r-00001 -rw-r--r-- 3 hdfs supergroup 3930000 2023-12-11 09:24 /tmp/out_precise/category-00018-r-00001 -rw-r--r-- 3 hdfs supergroup 7863000 2023-12-11 09:24 /tmp/out_precise/category-00019-r-00001 -rw-r--r-- 3 hdfs supergroup 15726000 2023-12-11 09:24 /tmp/out_precise/category-00020-r-00001 -rw-r--r-- 3 hdfs supergroup 31455000 2023-12-11 09:24 /tmp/out_precise/category-00021-r-00001 -rw-r--r-- 3 hdfs supergroup 62913000 2023-12-11 09:24 /tmp/out_precise/category-00022-r-00002 -rw-r--r-- 3 hdfs supergroup 125829000 2023-12-11 09:24 /tmp/out_precise/category-00023-r-00002 -rw-r--r-- 3 hdfs supergroup 251658000 2023-12-11 09:24 /tmp/out_precise/category-00024-r-00002 -rw-r--r-- 3 hdfs supergroup 503316000 2023-12-11 09:24 /tmp/out_precise/category-00025-r-00003 -rw-r--r-- 3 hdfs supergroup 1006632000 2023-12-11 09:24 /tmp/out_precise/category-00026-r-00004 -rw-r--r-- 3 hdfs supergroup 1046897280 2023-12-11 09:24 /tmp/out_precise/category-00027-r-00005 -rw-r--r-- 3 hdfs supergroup 966366720 2023-12-11 09:24 /tmp/out_precise/category-00027-r-00006 -rw-r--r-- 3 hdfs supergroup 1308622575 2023-12-11 09:24 /tmp/out_precise/category-00028-r-00007 -rw-r--r-- 3 hdfs supergroup 1308622575 2023-12-11 09:24 /tmp/out_precise/category-00028-r-00008 -rw-r--r-- 3 hdfs supergroup 1409285850 2023-12-11 09:24 /tmp/out_precise/category-00028-r-00009 -rw-r--r-- 3 hdfs supergroup 1046898060 2023-12-11 09:24 /tmp/out_precise/category-00029-r-00010 -rw-r--r-- 3 hdfs supergroup 1046898060 2023-12-11 09:24 /tmp/out_precise/category-00029-r-00011 -rw-r--r-- 3 hdfs supergroup 1570347090 2023-12-11 09:26 /tmp/out_precise/category-00029-r-00012 -rw-r--r-- 3 hdfs supergroup 1570347090 2023-12-11 09:26 /tmp/out_precise/category-00029-r-00013 -rw-r--r-- 3 hdfs supergroup 1570347090 2023-12-11 09:26 /tmp/out_precise/category-00029-r-00014 -rw-r--r-- 3 hdfs supergroup 1248224610 2023-12-11 09:26 /tmp/out_precise/category-00029-r-00015 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
