[ https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15989320#comment-15989320 ]
Yufei Gu edited comment on YARN-6522 at 4/28/17 7:17 PM: --------------------------------------------------------- Uploaded patch v1. It largely simplifies the job configuration. Here is a job configuration for FS minshare preemption test. What it looks like before patch: {code} { "am.type" : "mapreduce", "job.start.ms" : 0, "job.end.ms" : 60000, "job.queue.name" : "default", "job.id" : "job_1", "job.user" : "default", "job.tasks" : [ { "container.host" : "/default-rack/node1", "container.start.ms" : 6664, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" } ] } { "am.type" : "mapreduce", "job.start.ms" : 0, "job.end.ms" : 60000, "job.queue.name" : "default", "job.id" : "job_11", "job.user" : "default", "job.tasks" : [ { "container.host" : "/default-rack/node1", "container.start.ms" : 6664, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" } ] } { "am.type" : "mapreduce", "job.start.ms" : 10000, "job.end.ms" : 60000, "job.queue.name" : "min-wait", "job.id" : "job_2", "job.user" : "default", "job.tasks" : [ { "container.host" : "/default-rack/node1", "container.start.ms" : 10000, "container.end.ms" : 40000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 10000, "container.end.ms" : 40000, "container.priority" : 20, "container.type" : "map" } ] } {code} and after patch: {code} { "num_nodes":3} { "job.start.ms" : 0, "job.queue.name": "default", "job.tasks": [ {"count":29, "duration.ms": 50000}]} { "job.start.ms" : 10000, "job.queue.name": "min-wait", "job.tasks":[{"count":2, "duration.ms": 30000}]} {code} was (Author: yufeigu): Uploaded patch v1. It largely simplifies the job configuration. Here is a job configuration for minshare preemption test. What it looks like before patch: {code} { "am.type" : "mapreduce", "job.start.ms" : 0, "job.end.ms" : 60000, "job.queue.name" : "default", "job.id" : "job_1", "job.user" : "default", "job.tasks" : [ { "container.host" : "/default-rack/node1", "container.start.ms" : 6664, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" } ] } { "am.type" : "mapreduce", "job.start.ms" : 0, "job.end.ms" : 60000, "job.queue.name" : "default", "job.id" : "job_11", "job.user" : "default", "job.tasks" : [ { "container.host" : "/default-rack/node1", "container.start.ms" : 6664, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node1", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node3", "container.start.ms" : 6665, "container.end.ms" : 60000, "container.priority" : 20, "container.type" : "map" } ] } { "am.type" : "mapreduce", "job.start.ms" : 10000, "job.end.ms" : 60000, "job.queue.name" : "min-wait", "job.id" : "job_2", "job.user" : "default", "job.tasks" : [ { "container.host" : "/default-rack/node1", "container.start.ms" : 10000, "container.end.ms" : 40000, "container.priority" : 20, "container.type" : "map" }, { "container.host" : "/default-rack/node2", "container.start.ms" : 10000, "container.end.ms" : 40000, "container.priority" : 20, "container.type" : "map" } ] } {code} and after patch: {code} { "num_nodes":3} { "job.start.ms" : 0, "job.queue.name": "default", "job.tasks": [ {"count":29, "duration.ms": 50000}]} { "job.start.ms" : 10000, "job.queue.name": "min-wait", "job.tasks":[{"count":2, "duration.ms": 30000}]} {code} > Make SLS JSON input file format simple and scalable > --------------------------------------------------- > > Key: YARN-6522 > URL: https://issues.apache.org/jira/browse/YARN-6522 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator > Affects Versions: 3.0.0-alpha2 > Reporter: Yufei Gu > Assignee: Yufei Gu > Attachments: YARN-6522.001.patch > > > SLS input format is verbose, and it doesn't scale out. We can improve it in > these ways: > # We need to configure tasks one by one if there are more than one task in a > job, which means the job configuration usually includes lots of redundant > items. Specify the number task for a task configuration will solve this issue. > # Container host is useful for locality testing. It is obnoxious to specify > container host for each task for tests unrelated to locality. We would like > to make it optional. > # For most tests, we don't care about job.id. Make it optional and generated > automatically by default. > # job.finish.ms doesn't make sense, just remove it. > # container type and container priority should be optional as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org