Re: [VOTE] Merge YARN-3368 (new web UI) to trunk

[Vrushali C]

Thanks everyone!

Subru: Yes, I will back-port YARN-7170 as well.

Will do some more sanity tests and check QA for YARN-3368_branch2 and then
will proceed with the merge to branch2.

thanks
Vrushali


On Thu, Oct 19, 2017 at 3:24 PM, Subramaniam V K  wrote:

> I think we should get in YARN-7170 before we merge the UI to branch-2
> especially since it looks like it is ready for commit?
>
> Thanks,
> Subru
>
> On Thu, Oct 19, 2017 at 3:18 PM, Wangda Tan  wrote:
>
>> Eric, thanks for updating your vote! I just committed YARN-7338
>>
>> Vrushali:
>>
>> Since the YARN-7364 is an issue needs to be fixed in trunk/branch-3.0, I'm
>> preferred to merge all existing JIRAs to branch-2 first and backport
>> YARN-7364 separately.
>>
>> Thanks,
>> Wangda
>>
>>
>> On Thu, Oct 19, 2017 at 10:04 AM, Eric Yang 
>> wrote:
>>
>> > +1 on merge when YARN-7338 is resolved.
>> >
>> > On 10/16/17, 3:33 PM, "Eric Yang"  wrote:
>> >
>> > Thank you
>> >
>> > Regards,
>> > Eric
>> >
>> > From: Vrushali C 
>> > Date: Monday, October 16, 2017 at 3:26 PM
>> > To: Eric Yang 
>> > Cc: Rohith Sharma K S , Sunil G <
>> > sun...@apache.org>, "yarn-dev@hadoop.apache.org" <
>> > yarn-dev@hadoop.apache.org>, Wangda Tan 
>> > Subject: Re: [VOTE] Merge YARN-3368 (new web UI) to trunk
>> >
>> > Thanks Eric, I have filed https://issues.apache.org/
>> > jira/browse/YARN-7338 to track this.
>> >
>> > On Mon, Oct 16, 2017 at 1:30 PM, Eric Yang > > > wrote:
>> > -1 (binding).
>> >
>> > Ui2 does not seem to support same origin policy for cross site
>> > scripting prevention.
>> > The following parameters has no effect for /ui2:
>> >
>> > hadoop.http.cross-origin.enabled = true
>> > yarn.resourcemanager.webapp.cross-origin.enabled = true
>> >
>> > This is because ui2 is designed as a separate web application.
>> > WebFilters setup for existing resource manager doesn’t apply to the new
>> web
>> > application.
>> > Please open JIRA to track the security issue and resolve the problem
>> > prior to backporting this to branch-2.
>> > This would minimize the risk to open up security hole in branch-2.
>> >
>> > Thank you
>> >
>> > Regards,
>> > Eric
>> >
>> > On 10/16/17, 1:03 PM, "Vrushali C"  > vrushalic2...@gmail.com>> wrote:
>> >
>> > We are planning to include the new YARN UI as part of 2.9
>> > release.  There
>> > is work in progress for back-porting the UI to branch2.
>> >
>> > Details at https://issues.apache.org/jira/browse/YARN-7169.
>> >
>> >
>> >
>> > On Mon, Nov 7, 2016 at 7:22 AM, Rohith Sharma K S <
>> > rohithsharm...@apache.org
>> > > wrote:
>> >
>> > > I just noticed that my voted mail has been sent to Wangda and
>> > forgotten to
>> > > keep yarn-dev in cc. Its my bad:-(  I am forwarding my voted
>> > mail to
>> > > yarn-dev.
>> > >
>> > > Thanks & Regards
>> > > Rohith Sharma K S
>> > >
>> > >
>> > > -- Forwarded message --
>> > > From: Rohith Sharma K S  > rohithsharm...@apache.org>>
>> > > Date: 3 November 2016 at 12:06
>> > > Subject: Re: [VOTE] Merge YARN-3368 (new web UI) to trunk
>> > > To: Wangda Tan > eele...@gmail.com>>
>> > >
>> > >
>> > > +1
>> > >
>> > > Built from YARN-3368 branch and hosted in cluster. It is
>> pretty
>> > much good
>> > > user experience UI.
>> > > I hosted new web UI in same port as existing UI. I was able to
>> > experience
>> > > Queue, Application and Nodes pages.
>> > >
>> > >
>> > > Thanks & Regards
>> > > Rohith Sharma K S
>> > >
>> > > On 1 November 2016 at 04:23, Wangda Tan > > > wrote:
>> > >
>> > > > YARN Devs,
>> > > >
>> > > > We propose to merge YARN-3368 (YARN next generation web UI)
>> > development
>> > > > branch into trunk for better development, would like to hear
>> > your
>> > > thoughts
>> > > > before sending out vote mail.
>> > > >
>> > > > The new UI will co-exist with the old YARN UI, by default it
>> > is disabled.
>> > > > Please refer to User documentation of the new YARN UI
>> > > > > oop/blob/YARN-3368/hadoop-yarn
>> > > > -project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/
>> > YarnUI2.md>
>> > > > for
>> > > > more details.
>> > > >
>> > > > In addition, 

[VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Junping Du]

Hi folks,
 I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.

 Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
will be the latest stable/production release for Apache Hadoop - it includes 
315 new fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical 
issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC1

  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
66c47f2a01ad9637879e95f80c41f798373828fb

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1064

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 10/24/2017 6pm PST time.

Thanks,

Junping

Re: [VOTE] Merge YARN-3368 (new web UI) to trunk

[Subramaniam V K]

I think we should get in YARN-7170 before we merge the UI to branch-2
especially since it looks like it is ready for commit?

Thanks,
Subru

On Thu, Oct 19, 2017 at 3:18 PM, Wangda Tan  wrote:

> Eric, thanks for updating your vote! I just committed YARN-7338
>
> Vrushali:
>
> Since the YARN-7364 is an issue needs to be fixed in trunk/branch-3.0, I'm
> preferred to merge all existing JIRAs to branch-2 first and backport
> YARN-7364 separately.
>
> Thanks,
> Wangda
>
>
> On Thu, Oct 19, 2017 at 10:04 AM, Eric Yang  wrote:
>
> > +1 on merge when YARN-7338 is resolved.
> >
> > On 10/16/17, 3:33 PM, "Eric Yang"  wrote:
> >
> > Thank you
> >
> > Regards,
> > Eric
> >
> > From: Vrushali C 
> > Date: Monday, October 16, 2017 at 3:26 PM
> > To: Eric Yang 
> > Cc: Rohith Sharma K S , Sunil G <
> > sun...@apache.org>, "yarn-dev@hadoop.apache.org" <
> > yarn-dev@hadoop.apache.org>, Wangda Tan 
> > Subject: Re: [VOTE] Merge YARN-3368 (new web UI) to trunk
> >
> > Thanks Eric, I have filed https://issues.apache.org/
> > jira/browse/YARN-7338 to track this.
> >
> > On Mon, Oct 16, 2017 at 1:30 PM, Eric Yang  > > wrote:
> > -1 (binding).
> >
> > Ui2 does not seem to support same origin policy for cross site
> > scripting prevention.
> > The following parameters has no effect for /ui2:
> >
> > hadoop.http.cross-origin.enabled = true
> > yarn.resourcemanager.webapp.cross-origin.enabled = true
> >
> > This is because ui2 is designed as a separate web application.
> > WebFilters setup for existing resource manager doesn’t apply to the new
> web
> > application.
> > Please open JIRA to track the security issue and resolve the problem
> > prior to backporting this to branch-2.
> > This would minimize the risk to open up security hole in branch-2.
> >
> > Thank you
> >
> > Regards,
> > Eric
> >
> > On 10/16/17, 1:03 PM, "Vrushali C"  vrushalic2...@gmail.com>> wrote:
> >
> > We are planning to include the new YARN UI as part of 2.9
> > release.  There
> > is work in progress for back-porting the UI to branch2.
> >
> > Details at https://issues.apache.org/jira/browse/YARN-7169.
> >
> >
> >
> > On Mon, Nov 7, 2016 at 7:22 AM, Rohith Sharma K S <
> > rohithsharm...@apache.org
> > > wrote:
> >
> > > I just noticed that my voted mail has been sent to Wangda and
> > forgotten to
> > > keep yarn-dev in cc. Its my bad:-(  I am forwarding my voted
> > mail to
> > > yarn-dev.
> > >
> > > Thanks & Regards
> > > Rohith Sharma K S
> > >
> > >
> > > -- Forwarded message --
> > > From: Rohith Sharma K S  rohithsharm...@apache.org>>
> > > Date: 3 November 2016 at 12:06
> > > Subject: Re: [VOTE] Merge YARN-3368 (new web UI) to trunk
> > > To: Wangda Tan  >>
> > >
> > >
> > > +1
> > >
> > > Built from YARN-3368 branch and hosted in cluster. It is pretty
> > much good
> > > user experience UI.
> > > I hosted new web UI in same port as existing UI. I was able to
> > experience
> > > Queue, Application and Nodes pages.
> > >
> > >
> > > Thanks & Regards
> > > Rohith Sharma K S
> > >
> > > On 1 November 2016 at 04:23, Wangda Tan  > > wrote:
> > >
> > > > YARN Devs,
> > > >
> > > > We propose to merge YARN-3368 (YARN next generation web UI)
> > development
> > > > branch into trunk for better development, would like to hear
> > your
> > > thoughts
> > > > before sending out vote mail.
> > > >
> > > > The new UI will co-exist with the old YARN UI, by default it
> > is disabled.
> > > > Please refer to User documentation of the new YARN UI
> > > >  > > > -project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/
> > YarnUI2.md>
> > > > for
> > > > more details.
> > > >
> > > > In addition, There’re two working-in-progress features need
> > the new UI to
> > > > be merged to trunk for further development.
> > > >
> > > >   1) UI of YARN Timeline Server v2 (YARN-2928)
> > > >   2) UI of YARN ResourceManager Federation (YARN-2915).
> > > >
> > > > *Status of YARN next generation web UI*
> > > >
> > > > Completed features
> > > >
> > > >- 

Re: [VOTE] Merge YARN-3368 (new web UI) to trunk

[Wangda Tan]

Eric, thanks for updating your vote! I just committed YARN-7338

Vrushali:

Since the YARN-7364 is an issue needs to be fixed in trunk/branch-3.0, I'm
preferred to merge all existing JIRAs to branch-2 first and backport
YARN-7364 separately.

Thanks,
Wangda


On Thu, Oct 19, 2017 at 10:04 AM, Eric Yang  wrote:

> +1 on merge when YARN-7338 is resolved.
>
> On 10/16/17, 3:33 PM, "Eric Yang"  wrote:
>
> Thank you
>
> Regards,
> Eric
>
> From: Vrushali C 
> Date: Monday, October 16, 2017 at 3:26 PM
> To: Eric Yang 
> Cc: Rohith Sharma K S , Sunil G <
> sun...@apache.org>, "yarn-dev@hadoop.apache.org" <
> yarn-dev@hadoop.apache.org>, Wangda Tan 
> Subject: Re: [VOTE] Merge YARN-3368 (new web UI) to trunk
>
> Thanks Eric, I have filed https://issues.apache.org/
> jira/browse/YARN-7338 to track this.
>
> On Mon, Oct 16, 2017 at 1:30 PM, Eric Yang  > wrote:
> -1 (binding).
>
> Ui2 does not seem to support same origin policy for cross site
> scripting prevention.
> The following parameters has no effect for /ui2:
>
> hadoop.http.cross-origin.enabled = true
> yarn.resourcemanager.webapp.cross-origin.enabled = true
>
> This is because ui2 is designed as a separate web application.
> WebFilters setup for existing resource manager doesn’t apply to the new web
> application.
> Please open JIRA to track the security issue and resolve the problem
> prior to backporting this to branch-2.
> This would minimize the risk to open up security hole in branch-2.
>
> Thank you
>
> Regards,
> Eric
>
> On 10/16/17, 1:03 PM, "Vrushali C" > wrote:
>
> We are planning to include the new YARN UI as part of 2.9
> release.  There
> is work in progress for back-porting the UI to branch2.
>
> Details at https://issues.apache.org/jira/browse/YARN-7169.
>
>
>
> On Mon, Nov 7, 2016 at 7:22 AM, Rohith Sharma K S <
> rohithsharm...@apache.org
> > wrote:
>
> > I just noticed that my voted mail has been sent to Wangda and
> forgotten to
> > keep yarn-dev in cc. Its my bad:-(  I am forwarding my voted
> mail to
> > yarn-dev.
> >
> > Thanks & Regards
> > Rohith Sharma K S
> >
> >
> > -- Forwarded message --
> > From: Rohith Sharma K S >
> > Date: 3 November 2016 at 12:06
> > Subject: Re: [VOTE] Merge YARN-3368 (new web UI) to trunk
> > To: Wangda Tan >
> >
> >
> > +1
> >
> > Built from YARN-3368 branch and hosted in cluster. It is pretty
> much good
> > user experience UI.
> > I hosted new web UI in same port as existing UI. I was able to
> experience
> > Queue, Application and Nodes pages.
> >
> >
> > Thanks & Regards
> > Rohith Sharma K S
> >
> > On 1 November 2016 at 04:23, Wangda Tan  > wrote:
> >
> > > YARN Devs,
> > >
> > > We propose to merge YARN-3368 (YARN next generation web UI)
> development
> > > branch into trunk for better development, would like to hear
> your
> > thoughts
> > > before sending out vote mail.
> > >
> > > The new UI will co-exist with the old YARN UI, by default it
> is disabled.
> > > Please refer to User documentation of the new YARN UI
> > >  > > -project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/
> YarnUI2.md>
> > > for
> > > more details.
> > >
> > > In addition, There’re two working-in-progress features need
> the new UI to
> > > be merged to trunk for further development.
> > >
> > >   1) UI of YARN Timeline Server v2 (YARN-2928)
> > >   2) UI of YARN ResourceManager Federation (YARN-2915).
> > >
> > > *Status of YARN next generation web UI*
> > >
> > > Completed features
> > >
> > >- Cluster Overview Page
> > >- Scheduler page
> > >- Applications / Application / Application-attempts pages
> > >- Nodes / Node page
> > >
> > > Integration to YARN
> > >
> > >- Hosts new web UI in RM
> > >- Integrates to maven build / package
> > >
> > > Miscs:
> > >
> > >- Added dependencies to LICENSE.txt/NOTICE.txt
> > >- Documented how to use it. 

[jira] [Created] (YARN-7371) NPE in ServiceMaster after RM is restarted and then the ServiceMaster is killed

[Chandni Singh (JIRA)]

Chandni Singh created YARN-7371:
---

 Summary: NPE in ServiceMaster after RM is restarted and then the 
ServiceMaster is killed
 Key: YARN-7371
 URL: https://issues.apache.org/jira/browse/YARN-7371
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Chandni Singh


java.lang.NullPointerException
at 
org.apache.hadoop.yarn.service.ServiceScheduler.recoverComponents(ServiceScheduler.java:313)
at 
org.apache.hadoop.yarn.service.ServiceScheduler.serviceStart(ServiceScheduler.java:265)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at 
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.yarn.service.ServiceMaster.main(ServiceMaster.java:150)

Steps:
1. Stopped RM and then started it
2. Application was still running
3. Killed the ServiceMaster to check if it recovers
4. Next attempt failed with the above exception



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merge Resource Types (YARN-3926) to branch-3.0

[Wangda Tan]

Hi Daniel,

Thanks for starting the thread and working on branch-3.0 merge efforts.

I'm in favor of bringing resource types in branch-3.0.

Could you please share test you have done and performance numbers to
compare branch-3.0 and branch-3.0 + resource types patches? I will +1 to
the merge if we see similar performance after applying resource types
patches comparing to trunk

- Wangda


On Thu, Oct 19, 2017 at 1:47 PM, Andrew Wang 
wrote:

> +0, as Daniel said we discussed this a lot off-list.
>
> Let's make sure the docs are up to snuff, and we update the site release
> notes to have a blurb on resource types.
>
> Hoping we can get a merge VOTE kicked off ASAP (tomorrow?) since we're down
> to the wire for the proposed RC0 schedule.
>
> On Thu, Oct 19, 2017 at 12:53 PM, Daniel Templeton 
> wrote:
>
> > After much offline discussion with Wangda, Sunil, Varun V., and Andrew
> > we've agreed that it would make sense to pull resource types into
> > branch-3.0 ahead of the Hadoop 3.0 RC0.  Resource types has already been
> > merged into trunk/3.1.  Now I'd like open a discussion about getting it
> > into 3.0 GA.  Here's the run-down:
> >
> > Feature Details
> > ---
> > Resource types replaces the two primitives that tracked CPU and memory
> > with an array of objects to track an arbitrary set of resources (that
> must
> > always include CPU and memory).  The resource manager reads the master
> list
> > of supported resources from its configs.  The node managers read their
> > resource values from their configs and report them to the resource
> manager
> > in their heartbeats.  The clients read the supported resource types from
> > their configs (or an RM service) and specify them in the application
> > submission.  At a high level, nothing else changes.
> >
> > The Resource object is a core construct in the resource manager and
> > scheduler.  All application operations end up touching Resource objects
> as
> > we determine fit or share-based priority for applications, queues, and
> > nodes.  As this feature replaces the core of how Resource objects work,
> > resource types impacts almost every aspect of the resource manager's
> > operation.  The change is pervasive, but not radical.
> >
> > The resource types patches as merged into trunk/3.1 include an additional
> > feature called resource profiles.  Resource profiles are actually
> > independent of resource types, and either is useful without the other.
> The
> > resource profiles code is still in a bit of flux, so the current plan is
> to
> > pull only the resource types code into branch-3.0.  I have backported
> only
> > the resource types patches into the resource-types branch.  Unit tests
> are
> > passing, and I don't see any significant risk from the split.  The diff
> > between the resource-types branch and branch-3.0 is available as a
> > branch-3.0 patch on YARN-7013[1].
> >
> > Justification for 3.0
> > -
> > Resource types (leaving out resource profiles) is in a stable state and
> is
> > well tested with unit tests, performance tests, and functional tests with
> > both the fair scheduler and the capacity scheduler.  Tests were run on
> both
> > the resource-types branch and the original YARN-3926 branch. There is
> some
> > additional work to do, but none of it's critical (except maybe improving
> > the docs).  Our confidence level in the feature is good.
> >
> > Resource types doesn't introduce incompatible changes to any Public and
> > Stable APIs.  The are some incompatible changes to Public and Unstable
> > APIs, but that's what a major release is for.  The Resource object proto
> > retains the CPU and memory fields and adds a new field for any additional
> > resource types to retain wire compatibility.  Other proto changes are all
> > additive.
> >
> > While it's not possible to turn resource types off per se, if the user
> > does not activate the feature, the operation of YARN will be unchanged.
> > Getting this feature into Hadoop 3.0 gives us the required groundwork to
> > make progress on tidying up the usage details without having to drag in a
> > large set of invasive changes into 3.1.
> >
> > If we don't pull resource types into 3.0, it will open a persistent
> > channel through which failures can be introduced through backporting.
> The
> > differences introduced by resource types are significant enough that it
> > will be an issue for scheduler and resource manager patches between 3.1
> and
> > 3.0.
> >
> > From the other side, resource types is a pervasive change, and there's no
> > turning it off.  Users will be impacted by it regardless of whether they
> > choose to use it or not.  While we've tested it, the feature represents a
> > large number of changes to core code that's critical to the resource
> > manager's operation.  If we're going to introduce a large change like
> this,
> > no matter how well tested, we should do it in 3.0 where users 

[jira] [Created] (YARN-7370) Intra-queue preemption properties should be refreshable

[Eric Payne (JIRA)]

Eric Payne created YARN-7370:


 Summary: Intra-queue preemption properties should be refreshable
 Key: YARN-7370
 URL: https://issues.apache.org/jira/browse/YARN-7370
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacity scheduler, scheduler preemption
Affects Versions: 3.0.0-alpha3, 2.8.0
Reporter: Eric Payne


At least the properties for {{max-allowable-limit}} and {{minimum-threshold}} 
should be refreshable. It would also be nice to make 
{{intra-queue-preemption.enabled}} and {{preemption-order-policy}} refreshable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merge Resource Types (YARN-3926) to branch-3.0

[Andrew Wang]

+0, as Daniel said we discussed this a lot off-list.

Let's make sure the docs are up to snuff, and we update the site release
notes to have a blurb on resource types.

Hoping we can get a merge VOTE kicked off ASAP (tomorrow?) since we're down
to the wire for the proposed RC0 schedule.

On Thu, Oct 19, 2017 at 12:53 PM, Daniel Templeton 
wrote:

> After much offline discussion with Wangda, Sunil, Varun V., and Andrew
> we've agreed that it would make sense to pull resource types into
> branch-3.0 ahead of the Hadoop 3.0 RC0.  Resource types has already been
> merged into trunk/3.1.  Now I'd like open a discussion about getting it
> into 3.0 GA.  Here's the run-down:
>
> Feature Details
> ---
> Resource types replaces the two primitives that tracked CPU and memory
> with an array of objects to track an arbitrary set of resources (that must
> always include CPU and memory).  The resource manager reads the master list
> of supported resources from its configs.  The node managers read their
> resource values from their configs and report them to the resource manager
> in their heartbeats.  The clients read the supported resource types from
> their configs (or an RM service) and specify them in the application
> submission.  At a high level, nothing else changes.
>
> The Resource object is a core construct in the resource manager and
> scheduler.  All application operations end up touching Resource objects as
> we determine fit or share-based priority for applications, queues, and
> nodes.  As this feature replaces the core of how Resource objects work,
> resource types impacts almost every aspect of the resource manager's
> operation.  The change is pervasive, but not radical.
>
> The resource types patches as merged into trunk/3.1 include an additional
> feature called resource profiles.  Resource profiles are actually
> independent of resource types, and either is useful without the other.  The
> resource profiles code is still in a bit of flux, so the current plan is to
> pull only the resource types code into branch-3.0.  I have backported only
> the resource types patches into the resource-types branch.  Unit tests are
> passing, and I don't see any significant risk from the split.  The diff
> between the resource-types branch and branch-3.0 is available as a
> branch-3.0 patch on YARN-7013[1].
>
> Justification for 3.0
> -
> Resource types (leaving out resource profiles) is in a stable state and is
> well tested with unit tests, performance tests, and functional tests with
> both the fair scheduler and the capacity scheduler.  Tests were run on both
> the resource-types branch and the original YARN-3926 branch. There is some
> additional work to do, but none of it's critical (except maybe improving
> the docs).  Our confidence level in the feature is good.
>
> Resource types doesn't introduce incompatible changes to any Public and
> Stable APIs.  The are some incompatible changes to Public and Unstable
> APIs, but that's what a major release is for.  The Resource object proto
> retains the CPU and memory fields and adds a new field for any additional
> resource types to retain wire compatibility.  Other proto changes are all
> additive.
>
> While it's not possible to turn resource types off per se, if the user
> does not activate the feature, the operation of YARN will be unchanged.
> Getting this feature into Hadoop 3.0 gives us the required groundwork to
> make progress on tidying up the usage details without having to drag in a
> large set of invasive changes into 3.1.
>
> If we don't pull resource types into 3.0, it will open a persistent
> channel through which failures can be introduced through backporting.  The
> differences introduced by resource types are significant enough that it
> will be an issue for scheduler and resource manager patches between 3.1 and
> 3.0.
>
> From the other side, resource types is a pervasive change, and there's no
> turning it off.  Users will be impacted by it regardless of whether they
> choose to use it or not.  While we've tested it, the feature represents a
> large number of changes to core code that's critical to the resource
> manager's operation.  If we're going to introduce a large change like this,
> no matter how well tested, we should do it in 3.0 where users already
> expect some bumps in the road.  Bringing in a large change like this in a
> 3.1 release, when users expect the release to have stabilized, sounds like
> a bad idea.
>
>
> What do folks think about pulling resource types back into branch-3.0 in
> time for RC0?  Any concerns?
>
> Thanks to Varun Vasudev, Sunil Govind, Wangda Tan, Yufei Gu, Grant Sohn,
> Jason Lowe, Arun Suresh, Karthik Kambatla, Vinod Vavilapalli, and Andrew
> Wang for their work on getting the resource types work done, backported,
> tested, and on track for 3.0.
>
> [1]: https://issues.apache.org/jira/secure/attachment/12892456/
> YARN-7013.branch-3.0.002.patch
>

[jira] [Created] (YARN-7369) Improve the docs

[Daniel Templeton (JIRA)]

Daniel Templeton created YARN-7369:
--

 Summary: Improve the docs
 Key: YARN-7369
 URL: https://issues.apache.org/jira/browse/YARN-7369
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: docs
Affects Versions: 3.1.0
Reporter: Daniel Templeton
Assignee: Daniel Templeton






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7368) Yarn Work-Preserving Better Handling Failed Disk

[BELUGA BEHR (JIRA)]

BELUGA BEHR created YARN-7368:
-

 Summary: Yarn Work-Preserving Better Handling Failed Disk
 Key: YARN-7368
 URL: https://issues.apache.org/jira/browse/YARN-7368
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager, yarn
Affects Versions: 2.8.1
Reporter: BELUGA BEHR


If the drive that hosts the {{yarn.nodemanager.recovery.dir}} is broken then 
the entire NodeManager will not start.  Please improve this so that if the 
directory is not able to be created/accessed then the recovery portion of the 
NM is simply skipped and the NM continues to operate as normal.

It may also be beneficial to be able to define multiple directories, like YARN 
logging directories, so that if one drive fails, not all of the recovery data 
is lost.


https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/NodeManagerRestart.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[DISCUSS] Merge Resource Types (YARN-3926) to branch-3.0

[Daniel Templeton]

After much offline discussion with Wangda, Sunil, Varun V., and Andrew 
we've agreed that it would make sense to pull resource types into 
branch-3.0 ahead of the Hadoop 3.0 RC0.  Resource types has already been 
merged into trunk/3.1.  Now I'd like open a discussion about getting it 
into 3.0 GA.  Here's the run-down:


Feature Details
---
Resource types replaces the two primitives that tracked CPU and memory 
with an array of objects to track an arbitrary set of resources (that 
must always include CPU and memory).  The resource manager reads the 
master list of supported resources from its configs.  The node managers 
read their resource values from their configs and report them to the 
resource manager in their heartbeats.  The clients read the supported 
resource types from their configs (or an RM service) and specify them in 
the application submission.  At a high level, nothing else changes.


The Resource object is a core construct in the resource manager and 
scheduler.  All application operations end up touching Resource objects 
as we determine fit or share-based priority for applications, queues, 
and nodes.  As this feature replaces the core of how Resource objects 
work, resource types impacts almost every aspect of the resource 
manager's operation.  The change is pervasive, but not radical.


The resource types patches as merged into trunk/3.1 include an 
additional feature called resource profiles.  Resource profiles are 
actually independent of resource types, and either is useful without the 
other.  The resource profiles code is still in a bit of flux, so the 
current plan is to pull only the resource types code into branch-3.0.  I 
have backported only the resource types patches into the resource-types 
branch.  Unit tests are passing, and I don't see any significant risk 
from the split.  The diff between the resource-types branch and 
branch-3.0 is available as a branch-3.0 patch on YARN-7013[1].


Justification for 3.0
-
Resource types (leaving out resource profiles) is in a stable state and 
is well tested with unit tests, performance tests, and functional tests 
with both the fair scheduler and the capacity scheduler.  Tests were run 
on both the resource-types branch and the original YARN-3926 branch. 
There is some additional work to do, but none of it's critical (except 
maybe improving the docs).  Our confidence level in the feature is good.


Resource types doesn't introduce incompatible changes to any Public and 
Stable APIs.  The are some incompatible changes to Public and Unstable 
APIs, but that's what a major release is for.  The Resource object proto 
retains the CPU and memory fields and adds a new field for any 
additional resource types to retain wire compatibility.  Other proto 
changes are all additive.


While it's not possible to turn resource types off per se, if the user 
does not activate the feature, the operation of YARN will be unchanged.  
Getting this feature into Hadoop 3.0 gives us the required groundwork to 
make progress on tidying up the usage details without having to drag in 
a large set of invasive changes into 3.1.


If we don't pull resource types into 3.0, it will open a persistent 
channel through which failures can be introduced through backporting.  
The differences introduced by resource types are significant enough that 
it will be an issue for scheduler and resource manager patches between 
3.1 and 3.0.


From the other side, resource types is a pervasive change, and there's 
no turning it off.  Users will be impacted by it regardless of whether 
they choose to use it or not.  While we've tested it, the feature 
represents a large number of changes to core code that's critical to the 
resource manager's operation.  If we're going to introduce a large 
change like this, no matter how well tested, we should do it in 3.0 
where users already expect some bumps in the road.  Bringing in a large 
change like this in a 3.1 release, when users expect the release to have 
stabilized, sounds like a bad idea.



What do folks think about pulling resource types back into branch-3.0 in 
time for RC0?  Any concerns?


Thanks to Varun Vasudev, Sunil Govind, Wangda Tan, Yufei Gu, Grant Sohn, 
Jason Lowe, Arun Suresh, Karthik Kambatla, Vinod Vavilapalli, and Andrew 
Wang for their work on getting the resource types work done, backported, 
tested, and on track for 3.0.


[1]: 
https://issues.apache.org/jira/secure/attachment/12892456/YARN-7013.branch-3.0.002.patch


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7367) ResourceInformation lacks stability and audience annotations

[Daniel Templeton (JIRA)]

Daniel Templeton created YARN-7367:
--

 Summary: ResourceInformation lacks stability and audience 
annotations
 Key: YARN-7367
 URL: https://issues.apache.org/jira/browse/YARN-7367
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: yarn
Affects Versions: 3.1.0
Reporter: Daniel Templeton






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Do we still have nightly (or even weekly) unit test run for Hadoop projects?

[Wangda Tan]

Gotcha, thanks!

- Wangda

On Thu, Oct 19, 2017 at 7:25 AM, Sean Busbey  wrote:

> Here's the email from last night to common-dev@hadoop:
>
> https://s.apache.org/ARe1
>
> On Wed, Oct 18, 2017 at 10:42 PM, Akira Ajisaka 
> wrote:
>
>> Yes, qbt runs nightly and it sends e-mail to dev lists.
>> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/
>>
>> Regards,
>> Akira
>>
>>
>> On 2017/10/19 7:54, Wangda Tan wrote:
>>
>>> Hi,
>>>
>>> Do we still have nightly (or even weekly) unit test run for Hadoop
>>> projects? I couldn't find it on Jenkins dashboard and I haven't seen
>>> reports set to dev lists for a while.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
>
> --
> busbey
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/

[Oct 18, 2017 10:06:30 PM] (junping_du) HADOOP-14958. Fix source-level 
compatibility after HADOOP-11252.




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.net.TestDNS 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler 

Timed out junit tests :

   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-compile-javac-root.txt
  [284K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [148K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [380K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [40K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Merge YARN-3368 (new web UI) to trunk

[Eric Yang]

+1 on merge when YARN-7338 is resolved.

On 10/16/17, 3:33 PM, "Eric Yang"  wrote:

Thank you

Regards,
Eric

From: Vrushali C 
Date: Monday, October 16, 2017 at 3:26 PM
To: Eric Yang 
Cc: Rohith Sharma K S , Sunil G 
, "yarn-dev@hadoop.apache.org" , 
Wangda Tan 
Subject: Re: [VOTE] Merge YARN-3368 (new web UI) to trunk

Thanks Eric, I have filed https://issues.apache.org/jira/browse/YARN-7338 
to track this.

On Mon, Oct 16, 2017 at 1:30 PM, Eric Yang 
> wrote:
-1 (binding).

Ui2 does not seem to support same origin policy for cross site scripting 
prevention.
The following parameters has no effect for /ui2:

hadoop.http.cross-origin.enabled = true
yarn.resourcemanager.webapp.cross-origin.enabled = true

This is because ui2 is designed as a separate web application.  WebFilters 
setup for existing resource manager doesn’t apply to the new web application.
Please open JIRA to track the security issue and resolve the problem prior 
to backporting this to branch-2.
This would minimize the risk to open up security hole in branch-2.

Thank you

Regards,
Eric

On 10/16/17, 1:03 PM, "Vrushali C" 
> wrote:

We are planning to include the new YARN UI as part of 2.9 release.  
There
is work in progress for back-porting the UI to branch2.

Details at https://issues.apache.org/jira/browse/YARN-7169.



On Mon, Nov 7, 2016 at 7:22 AM, Rohith Sharma K S 

> wrote:

> I just noticed that my voted mail has been sent to Wangda and 
forgotten to
> keep yarn-dev in cc. Its my bad:-(  I am forwarding my voted mail to
> yarn-dev.
>
> Thanks & Regards
> Rohith Sharma K S
>
>
> -- Forwarded message --
> From: Rohith Sharma K S 
>
> Date: 3 November 2016 at 12:06
> Subject: Re: [VOTE] Merge YARN-3368 (new web UI) to trunk
> To: Wangda Tan >
>
>
> +1
>
> Built from YARN-3368 branch and hosted in cluster. It is pretty much 
good
> user experience UI.
> I hosted new web UI in same port as existing UI. I was able to 
experience
> Queue, Application and Nodes pages.
>
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 1 November 2016 at 04:23, Wangda Tan 
> wrote:
>
> > YARN Devs,
> >
> > We propose to merge YARN-3368 (YARN next generation web UI) 
development
> > branch into trunk for better development, would like to hear your
> thoughts
> > before sending out vote mail.
> >
> > The new UI will co-exist with the old YARN UI, by default it is 
disabled.
> > Please refer to User documentation of the new YARN UI
> >  > -project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md>
> > for
> > more details.
> >
> > In addition, There’re two working-in-progress features need the new 
UI to
> > be merged to trunk for further development.
> >
> >   1) UI of YARN Timeline Server v2 (YARN-2928)
> >   2) UI of YARN ResourceManager Federation (YARN-2915).
> >
> > *Status of YARN next generation web UI*
> >
> > Completed features
> >
> >- Cluster Overview Page
> >- Scheduler page
> >- Applications / Application / Application-attempts pages
> >- Nodes / Node page
> >
> > Integration to YARN
> >
> >- Hosts new web UI in RM
> >- Integrates to maven build / package
> >
> > Miscs:
> >
> >- Added dependencies to LICENSE.txt/NOTICE.txt
> >- Documented how to use it. (In hadoop-yarn-project/hadoop-
> yarn/hadoop-
> >yarn-site/src/site/markdown/YarnUI2.md)
> >
> > Major items will finish on trunk:
> >
> >- Security support
> >
> > We have run the new UI in our internal cluster for more than 3 
months,
> lots
> > of people have tried the new UI and gave lots of valuable feedbacks 
and
> > reported suggestions / issues to us. We fixed many of them so now we
 

Re: Do we still have nightly (or even weekly) unit test run for Hadoop projects?

[Sean Busbey]

Here's the email from last night to common-dev@hadoop:

https://s.apache.org/ARe1

On Wed, Oct 18, 2017 at 10:42 PM, Akira Ajisaka  wrote:

> Yes, qbt runs nightly and it sends e-mail to dev lists.
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/
>
> Regards,
> Akira
>
>
> On 2017/10/19 7:54, Wangda Tan wrote:
>
>> Hi,
>>
>> Do we still have nightly (or even weekly) unit test run for Hadoop
>> projects? I couldn't find it on Jenkins dashboard and I haven't seen
>> reports set to dev lists for a while.
>>
>> Thanks,
>> Wangda
>>
>>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>


-- 
busbey

[jira] [Created] (YARN-7365) ResourceLocalization cache cleanup thread stuck

[Bibin A Chundatt (JIRA)]

Bibin A Chundatt created YARN-7365:
--

 Summary: ResourceLocalization cache cleanup thread stuck
 Key: YARN-7365
 URL: https://issues.apache.org/jira/browse/YARN-7365
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.8.0, 3.0.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Critical


{code}
"ResourceLocalizationService Cache Cleanup" #36 prio=5 os_prio=0 
tid=0x7f943562a000 nid=0x1017 waiting on condition [0x7f9419bd7000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xc21103f8> (a 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at 
org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:47)
at 
org.apache.hadoop.util.concurrent.HadoopScheduledThreadPoolExecutor.afterExecute(HadoopScheduledThreadPoolExecutor.java:69)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}

ResourceLocalization Cache Clean Up thread waiting on {{FutureTask.get()}} for 
infinite time after first execution



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7364) Queue dash board in new YARN UI has incorrect values

[Sunil G (JIRA)]

Sunil G created YARN-7364:
-

 Summary: Queue dash board in new YARN UI has incorrect values
 Key: YARN-7364
 URL: https://issues.apache.org/jira/browse/YARN-7364
 Project: Hadoop YARN
  Issue Type: Bug
  Components: webapp
Reporter: Sunil G
Assignee: Sunil G


Queue dashboard in cluster over page does not show queue metrics.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org