Re: [jira] [Created] (YARN-7672) hadoop-sls can not simulate huge scale of YARN

[Yufei Gu]

+1 for this idea. It will provide better isolation for scheduler simulating
to split NM simulators or/and AM simulators to different daemons, which can
be placed to different hosts.


On Wed, Dec 20, 2017 at 2:18 AM Jasson Chenwei 
wrote:

> It is an interesting point. So you mean you will set up like two daemons,
> one for RM(acts for resource allocation) and the other for NMs(each thread
> represents for a NM ?).  I am also working on a large scale cluster
> simulation project now. I am interested in your configuration for your
> simulation(machine capacity, machine load) and your bottleneck (like if the
> performance is bottlenecked by CPU or the lock-contention or the other
> resources)???
>
>
> Wei Chen
>
> On Mon, Dec 18, 2017 at 11:54 PM, zhangshilong (JIRA) 
> wrote:
>
> > zhangshilong created YARN-7672:
> > --
> >
> >  Summary: hadoop-sls can not simulate huge scale of YARN
> >  Key: YARN-7672
> >  URL: https://issues.apache.org/jira/browse/YARN-7672
> >  Project: Hadoop YARN
> >   Issue Type: Improvement
> > Reporter: zhangshilong
> > Assignee: zhangshilong
> >
> >
> > Our YARN cluster scale to nearly 10 thousands nodes.
> > We need to do scheduler pressure test.
> > we start  2000+ threads to simulate NM and AM. So  cpu.load very high to
> > 100+. I thought that will affect  performance evaluation of scheduler.
> > So I thought to separate the scheduler from the simulator.
> > I start a real RM. Then SLS will register nodes to RM,And submit apps to
> > RM using RM RPC.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.4.14#64029)
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>
-- 
Best,

Yufei

Re: [jira] [Created] (YARN-7672) hadoop-sls can not simulate huge scale of YARN

[Jasson Chenwei]

It is an interesting point. So you mean you will set up like two daemons,
one for RM(acts for resource allocation) and the other for NMs(each thread
represents for a NM ?).  I am also working on a large scale cluster
simulation project now. I am interested in your configuration for your
simulation(machine capacity, machine load) and your bottleneck (like if the
performance is bottlenecked by CPU or the lock-contention or the other
resources)???


Wei Chen

On Mon, Dec 18, 2017 at 11:54 PM, zhangshilong (JIRA) 
wrote:

> zhangshilong created YARN-7672:
> --
>
>  Summary: hadoop-sls can not simulate huge scale of YARN
>  Key: YARN-7672
>  URL: https://issues.apache.org/jira/browse/YARN-7672
>  Project: Hadoop YARN
>   Issue Type: Improvement
> Reporter: zhangshilong
> Assignee: zhangshilong
>
>
> Our YARN cluster scale to nearly 10 thousands nodes.
> We need to do scheduler pressure test.
> we start  2000+ threads to simulate NM and AM. So  cpu.load very high to
> 100+. I thought that will affect  performance evaluation of scheduler.
> So I thought to separate the scheduler from the simulator.
> I start a real RM. Then SLS will register nodes to RM,And submit apps to
> RM using RM RPC.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (YARN-7672) hadoop-sls can not simulate huge scale of YARN

[zhangshilong (JIRA)]

zhangshilong created YARN-7672:
--

 Summary: hadoop-sls can not simulate huge scale of YARN
 Key: YARN-7672
 URL: https://issues.apache.org/jira/browse/YARN-7672
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: zhangshilong
Assignee: zhangshilong


Our YARN cluster scale to nearly 10 thousands nodes.
We need to do scheduler pressure test.
we start  2000+ threads to simulate NM and AM. So  cpu.load very high to 100+. 
I thought that will affect  performance evaluation of scheduler. 
So I thought to separate the scheduler from the simulator.
I start a real RM. Then SLS will register nodes to RM,And submit apps to RM 
using RM RPC.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org