thanks bikas. i will have a try
发自网易邮箱手机版 On 2015-04-11 14:13 , Bikas Saha Wrote: It will try to produce 1200 splits. 1200 is not guaranteed but it will be close. If 1200 splits are generated then Tez will ask YARN for 1200 containers. Tez will start using and re-using containers as soon as it can. However the following can happen. Tez asks YARN for 1200. YARN provides 600 quickly but for some reason takes a long time to give next 600. Tez could end up completing the job by reusing the original 600 containers before getting the full 1200. However, this is unlikely in your case. It looks like the cluster is idle and waiting for the job to use all resources. So your job should get full capacity and Tez should use full capacity. However this hand-tuning is not recommended. Gopal’s suggestion of decreasing the min split size is more generic since it will adapt to different cluster/queue sizes. Bikas From:[email protected] [mailto:[email protected]] Sent: Friday, April 10, 2015 10:57 PM To:[email protected] Subject: Re: RE: Re:Re: How to use all yarn cluster resources for one tez job does “tez.grouping.split-count=1200” generates 1200 splits and also use 1200 yarn container to run? thanks On 2015-04-11 13:45 , Bikas Saha Wrote: Hi, Alternatively, if you know that you want about 1200 splits then you can set “tez.grouping.split-count=1200” using any of the methods shown by Gopal. The grouping logic will try to produce about 1200 splits. Prewarming containers helps in scenarios where the Tez AM (session) is started much before the actual DAG is ready to run. E.g. starting a Hive session starts and pre-warms a Tez Session. But the actual DAG is submitted after the user has type the SQL statement. So there is time in between to launch the pre-warm containers. If your scenario matches this pattern then pre-warming will help. Otherwise, pre-warming may not help that much. Bikas From: skaterQiang [mailto:[email protected]] Sent: Friday, April 10, 2015 10:11 PM To:[email protected] Subject: Re:Re: How to use all yarn cluster resources for one tez job Thanks Gopal: I have checked the tez 0.6.0 code From the code, there is a locationHint there(I am not sure when the location hint is set), But it looks like task will reuse the container before going to ask new container. Also I have seen there is a warm up function in the client side. So I think if I can warm up 1000 container for one tez job, then that tez should be run fast. And my network is 10GE, so I think network io is not a bottle neck for my case, so whether make the data aware hint is not that important. But if I can make all tasks to be run at the same time, then it will be good. >set tez.grouping.split-waves=1.7 the split-waves measures current queue capacity *1.7x to go wider than the actual available capacity [skater]But my env use default queue and root user, so *1.7 is useless to this case >set tez.grouping.min-size=16777216 [skater] What does this parameter mean? Do we have a wiki to trace the options? THanks in advance, Following is the code I found for make new container: if (locationHint != null) { TaskBasedLocationAffinity taskAffinity = locationHint.getAffinitizedTask(); if (taskAffinity != null) { Vertex vertex = appContext.getCurrentDAG().getVertex(taskAffinity.getVertexName()); Preconditions.checkNotNull(vertex, "Invalid vertex in task based affinity " + taskAffinity + " for attempt: " + taskAttempt.getID()); int taskIndex = taskAffinity.getTaskIndex(); Preconditions.checkState(taskIndex >=0 && taskIndex < vertex.getTotalTasks(), "Invalid taskIndex in task based affinity " + taskAffinity + " for attempt: " + taskAttempt.getID()); TaskAttempt affinityAttempt = vertex.getTask(taskIndex).getSuccessfulAttempt(); if (affinityAttempt != null) { Preconditions.checkNotNull(affinityAttempt.getAssignedContainerID(), affinityAttempt.getID()); taskScheduler.allocateTask(taskAttempt, event.getCapability(), affinityAttempt.getAssignedContainerID(), Priority.newInstance(event.getPriority()), event.getContainerContext(), event); return; } LOG.info("Attempt: " + taskAttempt.getID() + " has task based affinity to " + taskAffinity + " but no locality information exists for it. Ignoring hint."); // fall through with null hosts/racks } else { hosts = (locationHint.getHosts() != null) ? locationHint .getHosts().toArray( new String[locationHint.getHosts().size()]) : null; racks = (locationHint.getRacks() != null) ? locationHint.getRacks() .toArray(new String[locationHint.getRacks().size()]) : null; } } taskScheduler.allocateTask(taskAttempt, event.getCapability(), hosts, racks, Priority.newInstance(event.getPriority()), event.getContainerContext(), event); At 2015-04-11 12:58:06, "Gopal Vijayaraghavan" <[email protected]> wrote: > >> I have a hive full scan job , with hive on mr I can fully use >>the whole cluster's 1000 cpu vcores(I use the split size to make mapper >>tasks to be 1200), >> But in tez, tez only use around 700 vcores, I have also set the same >>hive split size. So how do I configure tez? to make tez fully use all the >>cluster resources? > >If you¹re on hive-1.0/later, the option to go wide is called >tez.grouping.split-waves. > >With ORC, the regular MRv2 splits generates empty tasks (so that not all >map-tasks process valid ranges). > >But to get it as wide as possible > >set mapred.max.split.size=33554432 >set tez.grouping.split-waves=1.7 >set tez.grouping.min-size=16777216 > >should do the trick, the split-waves measures current queue capacity * >1.7x to go wider than the actual available capacity. > >In previous versions (0.13/0.14), ³set² commands don¹t work, so the >options are prefixed by the tez.am.* - you have to do > >hive -hiveconf tez.am.grouping.split-waves=1.7 -hiveconf >tez.grouping.min-size=16777216 -hiveconf mapred.max.split.size=33554432 > > >We hope to throw away these hacks in hive-1.2 & for this Prasanth checked >in a couple of different split strategies for ORC in hive-1.2.0 >(ETL/BI/HYBRID) etc. > >I will probably send out my slides about ORC (incl. new split gen) after >Hadoop Summit Europe, if you want more details. > >Ideally, any tests with the latest code would help me fix anything that¹s >specific to your use-cases. > > >Cheers, >Gopal > > > > > > >
