[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341473#comment-16341473 ] Young Chen commented on YARN-7732: -- The existing code does set up a class map for different simulator types, but the set up is very rigid in only supporting mapreduce. Yes, the additional params in the AMSimulator init are for describing AMSimulator implementation specific parameters. The eventual goal is to support many different types of AMSimulators to better model a diverse workload. An AMSimulator that executes an arbitrary DAG could also be introduced in the Synth generator. The SLS and Rumen formats are still mapreduce specific - future work could possibly extend those to be generic as well, but that's out of the scope of this patch. Thanks for taking a look [~leftnoteasy]! Any suggestions/improvements are very welcome! > Support Generic AM Simulator from SynthGenerator > > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:00286 EndHTML:12564 StartFragment:03634 > EndFragment:12474 StartSelection:03700 EndSelection:12464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add( > new ContainerSimulator(containerResource, containerLifeTime, > hostname, DEFAULT_REDUCER_PRIORITY, "reduce")); > } > {code} > > In addition, the syn.json format supported only mapreduce (the parameters > were very specific: mtime, rtime, mtasks, rtasks, etc..). > This patch aims to introduce a new syn.json format that can describe generic > jobs, and the SLS setup required to support the synth generation of generic > jobs. > See syn_generic.json for an equivalent of the previous syn.json in the new > format. > Using the new generic format, we describe a StreamAMSimulator simulates a > long running streaming service that maintains N number of containers for the > lifetime of the AM. See syn_stream.json. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350853#comment-16350853 ] Young Chen commented on YARN-7732: -- Added back compatibility with JobStory and JobStoryProducer interfaces for gridmix integration in [^YARN-7732.04.patch] > Support Generic AM Simulator from SynthGenerator > > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch, YARN-7732.04.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:00286 EndHTML:12564 StartFragment:03634 > EndFragment:12474 StartSelection:03700 EndSelection:12464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add( > new ContainerSimulator(containerResource, containerLifeTime, > hostname, DEFAULT_REDUCER_PRIORITY, "reduce")); > } > {code} > > In addition, the syn.json format supported only mapreduce (the parameters > were very specific: mtime, rtime, mtasks, rtasks, etc..). > This patch aims to introduce a new syn.json format that can describe generic > jobs, and the SLS setup required to support the synth generation of generic > jobs. > See syn_generic.json for an equivalent of the previous syn.json in the new > format. > Using the new generic format, we describe a StreamAMSimulator simulates a > long running streaming service that maintains N number of containers for the > lifetime of the AM. See syn_stream.json. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351129#comment-16351129 ] genericqa commented on YARN-7732: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 2s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 9s{color} | {color:orange} hadoop-tools/hadoop-sls: The patch generated 1 new + 50 unchanged - 1 fixed = 51 total (was 51) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 40s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 37s{color} | {color:red} hadoop-tools/hadoop-sls generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 7s{color} | {color:green} hadoop-sls in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 48m 53s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-tools/hadoop-sls | | | int value cast to float and then passed to Math.round in org.apache.hadoop.yarn.sls.synthetic.SynthJob.getTaskAttemptInfo(TaskType, int, int) At SynthJob.java:and then passed to Math.round in org.apache.hadoop.yarn.sls.synthetic.SynthJob.getTaskAttemptInfo(TaskType, int, int) At SynthJob.java:[line 345] | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-7732 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12909038/YARN-7732.04.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 7223e1d20bd1 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 5072388 | | maven | version: Apache Maven 3.3
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366548#comment-16366548 ] genericqa commented on YARN-7732: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 35s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 41s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} hadoop-tools/hadoop-sls: The patch generated 0 new + 50 unchanged - 1 fixed = 50 total (was 51) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 42s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 9s{color} | {color:green} hadoop-sls in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 17s{color} | {color:red} The patch generated 2 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 49m 44s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-7732 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910844/YARN-7732.05.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 361d6a34d163 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 8013475 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19718/testReport/ | | asflicense | https://builds.apache.org/job/PreCommit-YARN-Build/19718/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 467 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls | | Console output | https://builds.apache.org/job/PreCommit-Y
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366683#comment-16366683 ] Wangda Tan commented on YARN-7732: -- Thanks [~youchen], in general the patch looks good, I'd like to do some basic validations using old traces in the next Monday. If I don't get back by next Monday, please feel free to commit the patch to trunk. And is there any compatibility issue for syn.json? I saw the description says: {quote}See syn_generic.json for an equivalent of the previous syn.json in the new format. {quote} cc: [~curino] > Support Generic AM Simulator from SynthGenerator > > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch, YARN-7732.04.patch, YARN-7732.05.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:00286 EndHTML:12564 StartFragment:03634 > EndFragment:12474 StartSelection:03700 EndSelection:12464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add( > new ContainerSimulator(containerResource, containerLifeTime, > hostname, DEFAULT_REDUCER_PRIORITY, "reduce")); > } > {code} > > In addition, the syn.json format supported only mapreduce (the parameters > were very specific: mtime, rtime, mtasks, rtasks, etc..). > This patch aims to introduce a new syn.json format that can describe generic > jobs, and the SLS setup required to support the synth generation of generic > jobs. > See syn_generic.json for an equivalent of the previous syn.json in the new > format. > Using the new generic format, we describe a StreamAMSimulator simulates a > long running streaming service that maintains N number of containers for the > lifetime of the AM. See syn_stream.json. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366707#comment-16366707 ] Young Chen commented on YARN-7732: -- Right now this is only for the synthetic generator, so the old traces shouldn't be affected at all. Let me know if you run into any issues though! There should be no backwards compatibility issues - I made sure to explicitly check if an old syn.json was being used, and if so extract the parameters and convert it internally to the new format. This is done in SynthTraceJobProducer#validateJobDef and tested in TestSynthJobGeneration#testMapReduce. > Support Generic AM Simulator from SynthGenerator > > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch, YARN-7732.04.patch, YARN-7732.05.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:00286 EndHTML:12564 StartFragment:03634 > EndFragment:12474 StartSelection:03700 EndSelection:12464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add( > new ContainerSimulator(containerResource, containerLifeTime, > hostname, DEFAULT_REDUCER_PRIORITY, "reduce")); > } > {code} > > In addition, the syn.json format supported only mapreduce (the parameters > were very specific: mtime, rtime, mtasks, rtasks, etc..). > This patch aims to introduce a new syn.json format that can describe generic > jobs, and the SLS setup required to support the synth generation of generic > jobs. > See syn_generic.json for an equivalent of the previous syn.json in the new > format. > Using the new generic format, we describe a StreamAMSimulator simulates a > long running streaming service that maintains N number of containers for the > lifetime of the AM. See syn_stream.json. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370610#comment-16370610 ] Carlo Curino commented on YARN-7732: Thanks [~leftnoteasy] for the review. [~youchen] please fix the ASF license issue (by adding an exclusion in pom.xml), and I will commit to trunk based on Wangda's review (and a quick skim from me). > Support Generic AM Simulator from SynthGenerator > > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch, YARN-7732.04.patch, YARN-7732.05.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:00286 EndHTML:12564 StartFragment:03634 > EndFragment:12474 StartSelection:03700 EndSelection:12464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add( > new ContainerSimulator(containerResource, containerLifeTime, > hostname, DEFAULT_REDUCER_PRIORITY, "reduce")); > } > {code} > > In addition, the syn.json format supported only mapreduce (the parameters > were very specific: mtime, rtime, mtasks, rtasks, etc..). > This patch aims to introduce a new syn.json format that can describe generic > jobs, and the SLS setup required to support the synth generation of generic > jobs. > See syn_generic.json for an equivalent of the previous syn.json in the new > format. > Using the new generic format, we describe a StreamAMSimulator simulates a > long running streaming service that maintains N number of containers for the > lifetime of the AM. See syn_stream.json. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370648#comment-16370648 ] Young Chen commented on YARN-7732: -- ASF license exclusions added. > Support Generic AM Simulator from SynthGenerator > > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch, YARN-7732.04.patch, YARN-7732.05.patch, YARN-7732.06.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:00286 EndHTML:12564 StartFragment:03634 > EndFragment:12474 StartSelection:03700 EndSelection:12464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add( > new ContainerSimulator(containerResource, containerLifeTime, > hostname, DEFAULT_REDUCER_PRIORITY, "reduce")); > } > {code} > > In addition, the syn.json format supported only mapreduce (the parameters > were very specific: mtime, rtime, mtasks, rtasks, etc..). > This patch aims to introduce a new syn.json format that can describe generic > jobs, and the SLS setup required to support the synth generation of generic > jobs. > See syn_generic.json for an equivalent of the previous syn.json in the new > format. > Using the new generic format, we describe a StreamAMSimulator simulates a > long running streaming service that maintains N number of containers for the > lifetime of the AM. See syn_stream.json. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370726#comment-16370726 ] genericqa commented on YARN-7732: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 9s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} hadoop-tools/hadoop-sls: The patch generated 0 new + 49 unchanged - 1 fixed = 49 total (was 50) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 36s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 16s{color} | {color:green} hadoop-sls in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 51m 52s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-7732 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12911283/YARN-7732.06.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux 2d05a4642d12 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 9028cca | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19748/testReport/ | | Max. process+thread count | 467 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19748/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370821#comment-16370821 ] Carlo Curino commented on YARN-7732: Thanks [~youchen] for the contribution, and [~leftnoteasy] for reviewing. I committed this to trunk, and cherry picked back this patch (and YARN-7798) to branch-3, since it was a clean cherry-pick and spot runs of SLS tests look good. [~leftnoteasy] and [~yufeigu], if you see issue with this cherry-pick let me know we can easily revert, I would like as much as possible to have all the SLS newer magic available in all branches, as it is very useful for regression/integration/performance testing. [~youchen] can you see why YARN-7798 does not apply to branch-2, it might be a very simple fix, in which case, please provide a patch for both YARN-7798 and YARN-7732 that works in branch-2, so we an backport there as well. > Support Generic AM Simulator from SynthGenerator > > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch, YARN-7732.04.patch, YARN-7732.05.patch, YARN-7732.06.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:00286 EndHTML:12564 StartFragment:03634 > EndFragment:12474 StartSelection:03700 EndSelection:12464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add( > new ContainerSimulator(containerResource, containerLifeTime, > hostname, DEFAULT_REDUCER_PRIORITY, "reduce")); > } > {code} > > In addition, the syn.json format supported only mapreduce (the parameters > were very specific: mtime, rtime, mtasks, rtasks, etc..). > This patch aims to introduce a new syn.json format that can describe generic > jobs, and the SLS setup required to support the synth generation of generic > jobs. > See syn_generic.json for an equivalent of the previous syn.json in the new > format. > Using the new generic format, we describe a StreamAMSimulator simulates a > long running streaming service that maintains N number of containers for the > lifetime of the AM. See syn_stream.json. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370825#comment-16370825 ] Hudson commented on YARN-7732: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13689 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13689/]) YARN-7732. Support Generic AM Simulator from SynthGenerator. (carlo curino: rev 84cea0011ffe510d24cf9f2952944f7a6fe622cf) * (edit) hadoop-tools/hadoop-sls/pom.xml * (add) hadoop-tools/hadoop-sls/src/test/resources/syn_generic.json * (edit) hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/TestSynthJobGeneration.java * (edit) hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/BaseSLSRunnerTest.java * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/synthetic/SynthTraceJobProducer.java * (add) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/StreamAMSimulator.java * (delete) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/synthetic/SynthJobClass.java * (add) hadoop-tools/hadoop-sls/src/test/resources/syn_stream.json * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/synthetic/SynthJob.java * (edit) hadoop-tools/hadoop-sls/src/test/resources/syn.json * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java * (delete) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/synthetic/SynthWorkload.java * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/MRAMSimulator.java * (edit) hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/appmaster/TestAMSimulator.java * (add) hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/TestSLSStreamAMSynth.java * (add) hadoop-tools/hadoop-sls/src/test/java/org/apache/hadoop/yarn/sls/TestSLSGenericSynth.java * (add) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/package-info.java * (edit) hadoop-tools/hadoop-sls/src/test/resources/sls-runner.xml > Support Generic AM Simulator from SynthGenerator > > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch, YARN-7732.04.patch, YARN-7732.05.patch, YARN-7732.06.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:00286 EndHTML:12564 StartFragment:03634 > EndFragment:12474 StartSelection:03700 EndSelection:12464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((i
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370881#comment-16370881 ] Wangda Tan commented on YARN-7732: -- [~curino], according to https://issues.apache.org/jira/browse/INFRA-15859, I think we don't need to commit changes to branch-3. The branch-3 should be removed. > Support Generic AM Simulator from SynthGenerator > > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch, YARN-7732.04.patch, YARN-7732.05.patch, YARN-7732.06.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:00286 EndHTML:12564 StartFragment:03634 > EndFragment:12474 StartSelection:03700 EndSelection:12464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add( > new ContainerSimulator(containerResource, containerLifeTime, > hostname, DEFAULT_REDUCER_PRIORITY, "reduce")); > } > {code} > > In addition, the syn.json format supported only mapreduce (the parameters > were very specific: mtime, rtime, mtasks, rtasks, etc..). > This patch aims to introduce a new syn.json format that can describe generic > jobs, and the SLS setup required to support the synth generation of generic > jobs. > See syn_generic.json for an equivalent of the previous syn.json in the new > format. > Using the new generic format, we describe a StreamAMSimulator simulates a > long running streaming service that maintains N number of containers for the > lifetime of the AM. See syn_stream.json. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7732) Support Generic AM Simulator from SynthGenerator
[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371824#comment-16371824 ] Carlo Curino commented on YARN-7732: Thanks [~leftnoteasy], so should we then push to branch-3.0 (for all 3.x future branches?) > Support Generic AM Simulator from SynthGenerator > > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator >Reporter: Young Chen >Assignee: Young Chen >Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch, YARN-7732.04.patch, YARN-7732.05.patch, YARN-7732.06.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:00286 EndHTML:12564 StartFragment:03634 > EndFragment:12474 StartSelection:03700 EndSelection:12464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size( > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add( > new ContainerSimulator(containerResource, containerLifeTime, > hostname, DEFAULT_REDUCER_PRIORITY, "reduce")); > } > {code} > > In addition, the syn.json format supported only mapreduce (the parameters > were very specific: mtime, rtime, mtasks, rtasks, etc..). > This patch aims to introduce a new syn.json format that can describe generic > jobs, and the SLS setup required to support the synth generation of generic > jobs. > See syn_generic.json for an equivalent of the previous syn.json in the new > format. > Using the new generic format, we describe a StreamAMSimulator simulates a > long running streaming service that maintains N number of containers for the > lifetime of the AM. See syn_stream.json. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org