[jira] [Commented] (TEZ-1488) Implement HashComparator in TezBytesComparator
[ https://issues.apache.org/jira/browse/TEZ-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114954#comment-14114954 ] Siddharth Seth commented on TEZ-1488: - +1, renames etc look good. (the patch doesn't apply though) I think you missed the unit test file in the last patch. > Implement HashComparator in TezBytesComparator > - > > Key: TEZ-1488 > URL: https://issues.apache.org/jira/browse/TEZ-1488 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: TEZ-1488.1.patch, TEZ-1488.2.patch > > > Speed up TezBytesComparator by ~20% when used in PipelinedSorter. > This moves part of the key comparator into the partition comparator, which is > a single register operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1516) Log transfer rate for Broadcast Fetch
[ https://issues.apache.org/jira/browse/TEZ-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114987#comment-14114987 ] Rajesh Balamohan commented on TEZ-1516: --- +1 very minor comments - HttpConnection.connect(connecionTimeout) already logs the time taken to connect to specific URL. Should the connect time logging in broadcast/scatter-gather fetcher be removed? - In ShuffleScheduler, I believe you meant to set "double mbs = (double) totalBytesShuffledTillNow / (1024 * 1024);" - In ShuffleScheduler, average transfer time could be slightly misleading as there would be time when fetchers are not pulling data (could be due to lack of events or blocked on merge). > Log transfer rate for Broadcast Fetch > - > > Key: TEZ-1516 > URL: https://issues.apache.org/jira/browse/TEZ-1516 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-1516.1.txt > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1508) Log a warning if Xmx is configured incorrectly.
[ https://issues.apache.org/jira/browse/TEZ-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-1508: - Attachment: TEZ-1508-v1.patch > Log a warning if Xmx is configured incorrectly. > > > Key: TEZ-1508 > URL: https://issues.apache.org/jira/browse/TEZ-1508 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Jonathan Eagles > Labels: newbie > Attachments: TEZ-1508-v1.patch > > > Users may incorrectly configure Xmx for tasks. Xmx should in most cases be > less than the container/task size given that YARN will kill containers that > exceed the memory limits. > Given that we already parse the java opts to detect Xmx in the java opts, it > should be trivial to add a check if the value is valid and log a warning if > not ( for both the Tez AM and the vertices in a DAG). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1516) Log transfer rate for Broadcast Fetch
[ https://issues.apache.org/jira/browse/TEZ-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-1516: Attachment: TEZ-1516.2.txt Thanks for the review. - Removed the fetcher changes. - Fixed to double - Have changed the log message to indicate what is being printed (CumulativeDataFetched/TimeSinceInputStarted) > Log transfer rate for Broadcast Fetch > - > > Key: TEZ-1516 > URL: https://issues.apache.org/jira/browse/TEZ-1516 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-1516.1.txt, TEZ-1516.2.txt > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (TEZ-1522) Enhance natural order scheduler to prevent downstream vertex from monopolizing the cluster resources
Rajesh Balamohan created TEZ-1522: - Summary: Enhance natural order scheduler to prevent downstream vertex from monopolizing the cluster resources Key: TEZ-1522 URL: https://issues.apache.org/jira/browse/TEZ-1522 Project: Apache Tez Issue Type: Bug Reporter: Rajesh Balamohan M2 M7 \ / (sg) \/ R3/ (b) \ / (b) \ / \ / M5 | R6 Plz refer to the attachment (task runtime SVG). In this case, M5 got scheduled much earlier than R3 (green color in the diagram) and retained lots of containers. R3 got less containers to work with. Attaching the output from the status monitor when the job ran; Map_5 has taken up almost all of cluster resource, whereas Reducer_3 got fraction of the capacity. Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 Reducer_6: 0/1 Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 Reducer_6: 0/1 Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 Reducer_6: 0/1 Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 14(+7)/8000 Reducer_6: 0/1 Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 63(+14)/8000 Reducer_6: 0/1 Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 159(+22)/8000Reducer_6: 0/1 Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 308(+29)/8000Reducer_6: 0/1 ... Creating this JIRA as a placeholder for scheduler enhancement. One possibililty could be to schedule lesser number of tasks in downstream vertices, based on the information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (TEZ-1523) Enhance natural order scheduler to prevent downstream vertex from monopolizing the cluster resources
[ https://issues.apache.org/jira/browse/TEZ-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan moved HIVE-7910 to TEZ-1523: - Key: TEZ-1523 (was: HIVE-7910) Project: Apache Tez (was: Hive) > Enhance natural order scheduler to prevent downstream vertex from > monopolizing the cluster resources > > > Key: TEZ-1523 > URL: https://issues.apache.org/jira/browse/TEZ-1523 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan > Labels: performance > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (R3 is mentioned as green color in the > diagram) and retained lots of containers. R3 got less containers to work > with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all containers, whereas Reducer_3 got fraction of the > capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (TEZ-1523) Enhance natural order scheduler to prevent downstream vertex from monopolizing the cluster resources
[ https://issues.apache.org/jira/browse/TEZ-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned TEZ-1523: - Assignee: Rajesh Balamohan > Enhance natural order scheduler to prevent downstream vertex from > monopolizing the cluster resources > > > Key: TEZ-1523 > URL: https://issues.apache.org/jira/browse/TEZ-1523 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Labels: performance > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (R3 is mentioned as green color in the > diagram) and retained lots of containers. R3 got less containers to work > with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all containers, whereas Reducer_3 got fraction of the > capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1523) Enhance natural order scheduler to prevent downstream vertex from monopolizing the cluster resources
[ https://issues.apache.org/jira/browse/TEZ-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1523: -- Assignee: (was: Rajesh Balamohan) > Enhance natural order scheduler to prevent downstream vertex from > monopolizing the cluster resources > > > Key: TEZ-1523 > URL: https://issues.apache.org/jira/browse/TEZ-1523 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan > Labels: performance > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (R3 is mentioned as green color in the > diagram) and retained lots of containers. R3 got less containers to work > with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all containers, whereas Reducer_3 got fraction of the > capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1522) Enhance natural order scheduler to prevent downstream vertex from monopolizing the cluster resources
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1522: -- Attachment: task_runtime.svg > Enhance natural order scheduler to prevent downstream vertex from > monopolizing the cluster resources > > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan > Labels: performance > Attachments: task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1521) VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log
[ https://issues.apache.org/jira/browse/TEZ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1521: Description: The TezEvents may be added to pendingTaskEvents when task is not scheduled. In this case, > VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log > --- > > Key: TEZ-1521 > URL: https://issues.apache.org/jira/browse/TEZ-1521 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang >Assignee: Jeff Zhang > > The TezEvents may be added to pendingTaskEvents when task is not scheduled. > In this case, -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1521) VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log
[ https://issues.apache.org/jira/browse/TEZ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1521: Description: The TezEvents may be added to pendingTaskEvents when task is not scheduled. In this case, VertexDataMovementEventsGeneratedEvent will been logged twice. (was: The TezEvents may be added to pendingTaskEvents when task is not scheduled. In this case, ) > VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log > --- > > Key: TEZ-1521 > URL: https://issues.apache.org/jira/browse/TEZ-1521 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang >Assignee: Jeff Zhang > > The TezEvents may be added to pendingTaskEvents when task is not scheduled. > In this case, VertexDataMovementEventsGeneratedEvent will been logged twice. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1521) VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log
[ https://issues.apache.org/jira/browse/TEZ-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1521: Description: The TezEvents may be added to pendingTaskEvents and route again later when task is not scheduled. In this case, VertexDataMovementEventsGeneratedEvent will been logged twice. (was: The TezEvents may be added to pendingTaskEvents when task is not scheduled. In this case, VertexDataMovementEventsGeneratedEvent will been logged twice.) > VertexDataMovementEventsGeneratedEvent may be logged twice in recovery log > --- > > Key: TEZ-1521 > URL: https://issues.apache.org/jira/browse/TEZ-1521 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang >Assignee: Jeff Zhang > > The TezEvents may be added to pendingTaskEvents and route again later when > task is not scheduled. In this case, VertexDataMovementEventsGeneratedEvent > will been logged twice. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1345) Add checks to guarantee all init events are written to recovery to consider vertex initialized
[ https://issues.apache.org/jira/browse/TEZ-1345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115230#comment-14115230 ] Jeff Zhang commented on TEZ-1345: - [~bikassaha][~hitesh] I think maybe we could add the inputName to the init events then when we recover we could know which initializer need to call again if we don't find its corresponding init event in recovery log. > Add checks to guarantee all init events are written to recovery to consider > vertex initialized > -- > > Key: TEZ-1345 > URL: https://issues.apache.org/jira/browse/TEZ-1345 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Jeff Zhang > Attachments: Tez-1345-2.patch, Tez-1345.patch > > > Related to issue discovered in TEZ-1033 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (TEZ-1345) Add checks to guarantee all init events are written to recovery to consider vertex initialized
[ https://issues.apache.org/jira/browse/TEZ-1345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115230#comment-14115230 ] Jeff Zhang edited comment on TEZ-1345 at 8/29/14 1:10 PM: -- [~bikassaha], [~hitesh] I think maybe we could add the inputName to the init events then when we recover we could know which initializer need to call again if we don't find its corresponding init event in recovery log. was (Author: zjffdu): [~bikassaha][~hitesh] I think maybe we could add the inputName to the init events then when we recover we could know which initializer need to call again if we don't find its corresponding init event in recovery log. > Add checks to guarantee all init events are written to recovery to consider > vertex initialized > -- > > Key: TEZ-1345 > URL: https://issues.apache.org/jira/browse/TEZ-1345 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Jeff Zhang > Attachments: Tez-1345-2.patch, Tez-1345.patch > > > Related to issue discovered in TEZ-1033 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1476) DAGClient waitForCompletion output is confusing
[ https://issues.apache.org/jira/browse/TEZ-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115300#comment-14115300 ] Jeff Zhang commented on TEZ-1476: - [~seth.siddha...@gmail.com], [~jeagles] I notice that after this patch the vertex status is printed. 2 thoughts: * Should we add GetVertexStatus to StatusGetOpts as an option for users to decide whether to print VertexStatus. Because it will print lots of info if dag has many vertices. * If VertexStatus is printed, could we add indention in front of VertexStatus to differentiate DAGStatus. > DAGClient waitForCompletion output is confusing > --- > > Key: TEZ-1476 > URL: https://issues.apache.org/jira/browse/TEZ-1476 > Project: Apache Tez > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.5.1 > > Attachments: TEZ-1476-v1.patch, TEZ-1476-v2.patch > > > When a DAG is submitted - "2014-08-21 16:38:06,153 INFO [main] > rpc.DAGClientRPCImpl (DAGClientRPCImpl.java:log(428)) - Waiting for DAG to > start running" is logged. > After this, nothing seems to get logged till the first task completes. > It would be useful to log when the state changes to RUNNING - as well as at > least one line stating the number of tasks, etc (0% progress line). Also, > progress could be logged every few seconds irrespective of whether it has > changed or not to give the impression that the job has not just gotten stuck. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1522) Enhance natural order scheduler to prevent downstream vertex from monopolizing the cluster resources
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-1522: Priority: Blocker (was: Major) Target Version/s: 0.5.1 > Enhance natural order scheduler to prevent downstream vertex from > monopolizing the cluster resources > > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Blocker > Labels: performance > Attachments: task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1522) Enhance natural order scheduler to prevent downstream vertex from monopolizing the cluster resources
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115360#comment-14115360 ] Siddharth Seth commented on TEZ-1522: - This is also critical for local mode, especially since it does not support pre-emption. The semantics for min/max fraction used in ShuffleScheduler would be a good start, but keeping in mind all source tasks rather than just the ones from ShuffleEdges. At a later point, this could make use of available cluster capacity to make better decisions. > Enhance natural order scheduler to prevent downstream vertex from > monopolizing the cluster resources > > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Blocker > Labels: performance > Attachments: task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1522) Slow start settings can result in out of order execution and slowdown of upstream work
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1522: Summary: Slow start settings can result in out of order execution and slowdown of upstream work (was: Enhance natural order scheduler to prevent downstream vertex from monopolizing the cluster resources) > Slow start settings can result in out of order execution and slowdown of > upstream work > -- > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Blocker > Labels: performance > Attachments: task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1508) Log a warning if Xmx is configured incorrectly.
[ https://issues.apache.org/jira/browse/TEZ-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115541#comment-14115541 ] Hitesh Shah commented on TEZ-1508: -- Comments: - >= container.size should be a good initial check. Folks may end up setting -Xmx7G for a 8g container which would work but will end up logging a lot of warns. Maybe, we can just augment the log to say recommended value is heap fraction times container.size when we find the Xmx to be set to larger than the container size. - TezClientUtils seems fine. - Pattern - would be good to optimize its usage. - The only other nit I have is regarding where this warning is logged for vertices that are configured badly? With this patch, this would be logged somewhere in the AM where the container launch context is constructed. Would this be more useful if it was caught and logged at DAG submission? > Log a warning if Xmx is configured incorrectly. > > > Key: TEZ-1508 > URL: https://issues.apache.org/jira/browse/TEZ-1508 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Jonathan Eagles > Labels: newbie > Attachments: TEZ-1508-v1.patch > > > Users may incorrectly configure Xmx for tasks. Xmx should in most cases be > less than the container/task size given that YARN will kill containers that > exceed the memory limits. > Given that we already parse the java opts to detect Xmx in the java opts, it > should be trivial to add a check if the value is valid and log a warning if > not ( for both the Tez AM and the vertices in a DAG). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1488) Implement HashComparator in TezBytesComparator
[
https://issues.apache.org/jira/browse/TEZ-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated TEZ-1488:
-
Attachment: TEZ-1488.3.patch
Patch includes renames - will only apply with {{git apply -v -p0
TEZ-1488.3.patch}}
> Implement HashComparator in TezBytesComparator
> -
>
> Key: TEZ-1488
> URL: https://issues.apache.org/jira/browse/TEZ-1488
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: TEZ-1488.1.patch, TEZ-1488.2.patch, TEZ-1488.3.patch
>
>
> Speed up TezBytesComparator by ~20% when used in PipelinedSorter.
> This moves part of the key comparator into the partition comparator, which is
> a single register operation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (TEZ-1488) Rename HashComparator to ProxyComparator and implement in TezBytesComparator
[ https://issues.apache.org/jira/browse/TEZ-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V resolved TEZ-1488. -- Resolution: Fixed commit bfba46724d043dce22e0b313782b0c9e30bc2094 Author: Gopal V Date: Fri Aug 29 11:26:36 2014 -0700 TEZ-1488. Rename HashComparator to ProxyComparator and implement in TezBytesComparator. (gopalv) > Rename HashComparator to ProxyComparator and implement in TezBytesComparator > > > Key: TEZ-1488 > URL: https://issues.apache.org/jira/browse/TEZ-1488 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: TEZ-1488.1.patch, TEZ-1488.2.patch, TEZ-1488.3.patch > > > Speed up TezBytesComparator by ~20% when used in PipelinedSorter. > This moves part of the key comparator into the partition comparator, which is > a single register operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1488) Rename HashComparator to ProxyComparator and implement in TezBytesComparator
[ https://issues.apache.org/jira/browse/TEZ-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-1488: - Summary: Rename HashComparator to ProxyComparator and implement in TezBytesComparator (was: Implement HashComparator in TezBytesComparator) > Rename HashComparator to ProxyComparator and implement in TezBytesComparator > > > Key: TEZ-1488 > URL: https://issues.apache.org/jira/browse/TEZ-1488 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 0.6.0 > > Attachments: TEZ-1488.1.patch, TEZ-1488.2.patch, TEZ-1488.3.patch > > > Speed up TezBytesComparator by ~20% when used in PipelinedSorter. > This moves part of the key comparator into the partition comparator, which is > a single register operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1488) Rename HashComparator to ProxyComparator and implement in TezBytesComparator
[ https://issues.apache.org/jira/browse/TEZ-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-1488: - Fix Version/s: 0.6.0 > Rename HashComparator to ProxyComparator and implement in TezBytesComparator > > > Key: TEZ-1488 > URL: https://issues.apache.org/jira/browse/TEZ-1488 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 0.6.0 > > Attachments: TEZ-1488.1.patch, TEZ-1488.2.patch, TEZ-1488.3.patch > > > Speed up TezBytesComparator by ~20% when used in PipelinedSorter. > This moves part of the key comparator into the partition comparator, which is > a single register operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1488) Rename HashComparator to ProxyComparator and implement in TezBytesComparator
[ https://issues.apache.org/jira/browse/TEZ-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-1488: - Attachment: TEZ-1488.4.patch > Rename HashComparator to ProxyComparator and implement in TezBytesComparator > > > Key: TEZ-1488 > URL: https://issues.apache.org/jira/browse/TEZ-1488 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 0.6.0 > > Attachments: TEZ-1488.1.patch, TEZ-1488.2.patch, TEZ-1488.3.patch, > TEZ-1488.4.patch > > > Speed up TezBytesComparator by ~20% when used in PipelinedSorter. > This moves part of the key comparator into the partition comparator, which is > a single register operation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1519) TezTaskRunner should not initialize TezConfiguration in TezChild
[ https://issues.apache.org/jira/browse/TEZ-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115690#comment-14115690 ] Chen He commented on TEZ-1519: -- Hi [~hitesh], Local Mode needs some configurations to be passed from DAGAppMaster's AppContext to TezTaskRunner, such as TezConfiguration.TEZ_LOCAL_MODE. > TezTaskRunner should not initialize TezConfiguration in TezChild > > > Key: TEZ-1519 > URL: https://issues.apache.org/jira/browse/TEZ-1519 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Priority: Blocker > > Should be doing a new Configuration and augmenting with the config data from > tez-conf.pb. > Need confirmation on tez-conf.pb being localized for all containers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
Gopal V created TEZ-1524:
Summary: getDAGStatus seems to fork out the entire JVM
Key: TEZ-1524
URL: https://issues.apache.org/jira/browse/TEZ-1524
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Gopal V
Fix For: 0.6.0
Tracked down a consistent fork() call to
{code}
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
at
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
at
org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
{code}
[~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TEZ-1519) TezTaskRunner should not initialize TezConfiguration in TezChild
[ https://issues.apache.org/jira/browse/TEZ-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115759#comment-14115759 ] Hitesh Shah commented on TEZ-1519: -- For local mode, I would assume that regardless of using TezConfiguration or not, the configs cannot be passed to the task without some special code to add the new settings. Doing a new TezConfiguation() loads tez-site.xml which I am assuming is not modified in local mode. > TezTaskRunner should not initialize TezConfiguration in TezChild > > > Key: TEZ-1519 > URL: https://issues.apache.org/jira/browse/TEZ-1519 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Priority: Blocker > > Should be doing a new Configuration and augmenting with the config data from > tez-conf.pb. > Need confirmation on tez-conf.pb being localized for all containers. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115769#comment-14115769
]
Hitesh Shah commented on TEZ-1524:
--
UGI should be using the Groups class that does caching internally.
Can you look for these logs in debug mode ( looking at 2.6.0-SNAPSHOT code
though ):
- "Returning cached groups for" ( implies obtained from cache )
- "Returning fetched groups"
> getDAGStatus seems to fork out the entire JVM
> -
>
> Key: TEZ-1524
> URL: https://issues.apache.org/jira/browse/TEZ-1524
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Gopal V
> Fix For: 0.6.0
>
>
> Tracked down a consistent fork() call to
> {code}
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
> {code}
> [~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115774#comment-14115774
]
Hitesh Shah commented on TEZ-1524:
--
Nevermind, ran a simple job on a single node cluster. The AM logs show "fetched
groups" logged once followed by "cached groups".
> getDAGStatus seems to fork out the entire JVM
> -
>
> Key: TEZ-1524
> URL: https://issues.apache.org/jira/browse/TEZ-1524
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Gopal V
> Fix For: 0.6.0
>
>
> Tracked down a consistent fork() call to
> {code}
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
> {code}
> [~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115781#comment-14115781
]
Hitesh Shah commented on TEZ-1524:
--
[~gopalv] Could you add more details as this points to a bug in UGI if it is
indeed forking on each ask for groupNames.
> getDAGStatus seems to fork out the entire JVM
> -
>
> Key: TEZ-1524
> URL: https://issues.apache.org/jira/browse/TEZ-1524
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Gopal V
> Fix For: 0.6.0
>
>
> Tracked down a consistent fork() call to
> {code}
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
> {code}
> [~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-1524:
-
Affects Version/s: (was: 0.6.0)
0.5.0
> getDAGStatus seems to fork out the entire JVM
> -
>
> Key: TEZ-1524
> URL: https://issues.apache.org/jira/browse/TEZ-1524
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Gopal V
>
> Tracked down a consistent fork() call to
> {code}
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
> {code}
> [~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115806#comment-14115806
]
Gopal V commented on TEZ-1524:
--
The cache does not cache misses.
> getDAGStatus seems to fork out the entire JVM
> -
>
> Key: TEZ-1524
> URL: https://issues.apache.org/jira/browse/TEZ-1524
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Gopal V
>
> Tracked down a consistent fork() call to
> {code}
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
> {code}
> [~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-1524:
-
Fix Version/s: (was: 0.6.0)
> getDAGStatus seems to fork out the entire JVM
> -
>
> Key: TEZ-1524
> URL: https://issues.apache.org/jira/browse/TEZ-1524
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Gopal V
>
> Tracked down a consistent fork() call to
> {code}
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
> {code}
> [~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115818#comment-14115818
]
Gopal V commented on TEZ-1524:
--
{code}
2014-08-29 09:27:12,602 WARN [IPC Server handler 0 on 59734]
org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to
get groups for user foo
org.apache.hadoop.util.Shell$ExitCodeException: id: foo: No such user
...
2014-08-29 09:27:12,602 WARN [IPC Server handler 0 on 59734]
org.apache.hadoop.security.UserGroupInformation: No groups available for user
{code}
I will submit a patch.
> getDAGStatus seems to fork out the entire JVM
> -
>
> Key: TEZ-1524
> URL: https://issues.apache.org/jira/browse/TEZ-1524
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Gopal V
>
> Tracked down a consistent fork() call to
> {code}
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
> {code}
> [~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated TEZ-1524:
-
Attachment: TEZ-1524.1.patch
> getDAGStatus seems to fork out the entire JVM
> -
>
> Key: TEZ-1524
> URL: https://issues.apache.org/jira/browse/TEZ-1524
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Gopal V
> Attachments: TEZ-1524.1.patch
>
>
> Tracked down a consistent fork() call to
> {code}
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
> {code}
> [~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TEZ-1524) getDAGStatus seems to fork out the entire JVM
[
https://issues.apache.org/jira/browse/TEZ-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115877#comment-14115877
]
Hitesh Shah commented on TEZ-1524:
--
Is this in a non-secure cluster with acls disabled? We might be able to work
around this to invoking getGroups only when needed. It wont prevent calls for
non-existent users but should help a bit.
> getDAGStatus seems to fork out the entire JVM
> -
>
> Key: TEZ-1524
> URL: https://issues.apache.org/jira/browse/TEZ-1524
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Gopal V
> Attachments: TEZ-1524.1.patch
>
>
> Tracked down a consistent fork() call to
> {code}
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1409)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getRPCUserGroups(DAGClientAMProtocolBlockingPBServerImpl.java:75)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:102)
> at
> org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7375)
> {code}
> [~hitesh] - would it make sense to cache this at all?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (TEZ-1522) Slow start settings can result in out of order execution and slowdown of upstream work
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115927#comment-14115927 ] Siddharth Seth commented on TEZ-1522: - [~bikassaha] - I don't think slow start settings are the problem here. Scheduling the ShuffleVertexManager does not account for other incoming edges, which may still have pending tasks. > Slow start settings can result in out of order execution and slowdown of > upstream work > -- > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Blocker > Labels: performance > Attachments: task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (TEZ-1525) BroadcastLoadGen testcase
Gopal V created TEZ-1525: Summary: BroadcastLoadGen testcase Key: TEZ-1525 URL: https://issues.apache.org/jira/browse/TEZ-1525 Project: Apache Tez Issue Type: Test Affects Versions: 0.6.0 Reporter: Gopal V Assignee: Gopal V Broadcast load generator test example -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1525) BroadcastLoadGen testcase
[ https://issues.apache.org/jira/browse/TEZ-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-1525: - Attachment: TEZ-1525.1.patch > BroadcastLoadGen testcase > - > > Key: TEZ-1525 > URL: https://issues.apache.org/jira/browse/TEZ-1525 > Project: Apache Tez > Issue Type: Test >Affects Versions: 0.6.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: TEZ-1525.1.patch > > > Broadcast load generator test example -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1525) BroadcastLoadGen testcase
[ https://issues.apache.org/jira/browse/TEZ-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116036#comment-14116036 ] Gopal V commented on TEZ-1525: -- [~sseth]: can you take a quick look? > BroadcastLoadGen testcase > - > > Key: TEZ-1525 > URL: https://issues.apache.org/jira/browse/TEZ-1525 > Project: Apache Tez > Issue Type: Test >Affects Versions: 0.6.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: TEZ-1525.1.patch > > > Broadcast load generator test example -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1508) Log a warning if Xmx is configured incorrectly.
[ https://issues.apache.org/jira/browse/TEZ-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-1508: - Attachment: TEZ-1508-v2.patch [~hitesh], moved the AM and Vertex java opts verification to the time of dag submission. Improved the message to give the user a hint on setting the Xmx setting. Optimized the pattern to be static and only compile once. > Log a warning if Xmx is configured incorrectly. > > > Key: TEZ-1508 > URL: https://issues.apache.org/jira/browse/TEZ-1508 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Jonathan Eagles > Labels: newbie > Attachments: TEZ-1508-v1.patch, TEZ-1508-v2.patch > > > Users may incorrectly configure Xmx for tasks. Xmx should in most cases be > less than the container/task size given that YARN will kill containers that > exceed the memory limits. > Given that we already parse the java opts to detect Xmx in the java opts, it > should be trivial to add a check if the value is valid and log a warning if > not ( for both the Tez AM and the vertices in a DAG). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1525) BroadcastLoadGen testcase
[ https://issues.apache.org/jira/browse/TEZ-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116229#comment-14116229 ] Bikas Saha commented on TEZ-1525: - lgtm. the division by 6 is counting some overhead? > BroadcastLoadGen testcase > - > > Key: TEZ-1525 > URL: https://issues.apache.org/jira/browse/TEZ-1525 > Project: Apache Tez > Issue Type: Test >Affects Versions: 0.6.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: TEZ-1525.1.patch > > > Broadcast load generator test example -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1522) ShuffleVertexManager scheduling can result in out of order execution and slowdown of upstream work
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1522: Summary: ShuffleVertexManager scheduling can result in out of order execution and slowdown of upstream work (was: Slow start settings can result in out of order execution and slowdown of upstream work) > ShuffleVertexManager scheduling can result in out of order execution and > slowdown of upstream work > -- > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Blocker > Labels: performance > Attachments: task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (TEZ-1522) ShuffleVertexManager scheduling can result in out of order execution and slowdown of upstream work
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116230#comment-14116230 ] Bikas Saha edited comment on TEZ-1522 at 8/30/14 4:27 AM: -- I was trying to say the blindly following the slow start setting can result in downstream tasks getting scheduled ahead of upstream tasks. Also given that this is not a regression, I have made this critical but not a blocker. Unless this was made a blocker for local mode? was (Author: bikassaha): I was trying to say the blindly following the slow start setting can result in downstream tasks getting scheduled ahead of upstream tasks. Also given that this is not a regression, I have made this critical but not a blocker. Unless this was made a blocker for local mode. > ShuffleVertexManager scheduling can result in out of order execution and > slowdown of upstream work > -- > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Critical > Labels: performance > Attachments: task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (TEZ-1522) ShuffleVertexManager scheduling can result in out of order execution and slowdown of upstream work
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1522: Priority: Critical (was: Blocker) > ShuffleVertexManager scheduling can result in out of order execution and > slowdown of upstream work > -- > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Critical > Labels: performance > Attachments: task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1522) ShuffleVertexManager scheduling can result in out of order execution and slowdown of upstream work
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116230#comment-14116230 ] Bikas Saha commented on TEZ-1522: - I was trying to say the blindly following the slow start setting can result in downstream tasks getting scheduled ahead of upstream tasks. Also given that this is not a regression, I have made this critical but not a blocker. Unless this was made a blocker for local mode. > ShuffleVertexManager scheduling can result in out of order execution and > slowdown of upstream work > -- > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Critical > Labels: performance > Attachments: task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1522) ShuffleVertexManager scheduling can result in out of order execution and slowdown of upstream work
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116234#comment-14116234 ] Bikas Saha commented on TEZ-1522: - Summarizing a bunch of separate offline discussions on this. The main issue is that downstream vertex managers (in this case the shufflevertexmanager) can end up scheduling their tasks ahead of upstream vertex tasks that they depend on. This may be because the upstream vertex manager hasnt yet scheduled its own tasks due to its policy. There are multiple potential solutions at hand with pros and cons 1) Make all scheduling happen via the InputReadyVertexManager like logic that ensures that tasks are ready to schedule only when their inputs are ready. This could be extended to consider tasks ready when inputs are ready or input tasks are ready to be scheduled. Pros: Never cause deadlocks (barring retries which will get handled via preemption if needed). Cons: Can never start a task when its input tasks arent even ready to run. Performance is no worse in general than today but there may be corner cases where a task could start out of order and fetch data from a completed larger input while waiting for a smaller input to get scheduled. 2) Change the VertexImpl state machine to allow downstream vertices to start only when some tasks of its own vertex have been scheduled. This does not work in general because after a few tasks have been scheduled the remaining could again be deadlocked by downstream tasks. 3) Improve preemption to act faster in these situation. [~rajesh.balamohan] In the example job, did R3 get scheduled after some M5s got preempted by the task scheduler to create space? It would be useful to have the AM logs to find out why it took so long for this to happen and show any room for improvement. Can you please attach the logs. Pros: The scheduling is optimistic and runs anything as soon as available. However, if there is resource contention then we can end up wasting work by having to preempt the optimistically started work. If the cluster has capacity then this may result in the best performance. Cons: wasted work. Latency in preemption kicking in. 4) Enhance the DAG scheduler to arbitrate between different vertex tasks that are ready to run. The DAG scheduler could look at the all the tasks that are ready to run, the available resources and estimate how much can be optimistically started and hold of the remaining. Pros: Tries to achieve a balance between optimistic execution of tasks out of order with potential deadlocks. Cons: Cannot guarantee that deadlocks will not occur. In free clusters may end up slowing down the execution. This was the reason the DAG scheduler abstraction was put in place. However, for a large and complex DAG with multiple concurrent branches of execution, it may be quite difficult to solve for an arbitration decision between all possible dependencies that want to get scheduled. > ShuffleVertexManager scheduling can result in out of order execution and > slowdown of upstream work > -- > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Critical > Labels: performance > Attachments: task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number o
