date:20160705

Re: [VOTE] merge feature branch YARN-2928

2016-07-05 Thread Sangjin Lee

Thanks for taking a look at it and providing your comments Allen.

As for the hadoop-aws pom change, it was necessary at one point because
hadoop-aws pulls in joda-time 2.8.1 via the AWS dependency and
hadoop-yarn-server-tests (on which hadoop-aws depends) was pulling in
joda-time 1.6 via hadoop-yarn-server-timelineservice via phoenix. Now that
the phoenix dependency has been refactored out into
hadoop-yarn-server-timelineservice-hbase-tests, we no longer need this
change in the hadoop-aws pom. I'll remove this change before we merge (hope
that's fine with you all).

Valid points on some of the documentation issues. As Naga mentioned, we
should open a new JIRA post-merge, and clean up some of these points.

Regards,
Sangjin


On Mon, Jul 4, 2016 at 11:10 PM, Allen Wittenauer  wrote:

>
> -0:
>
> I haven't looked at the code changes (although the pom change in
> hadoop-aws seems pretty oddball).  But I did do a quick pass through the
> documentation. It needs some formatting, cleanup, and consistency work done
> (e.g., not using toc macro, lack of language specific code formatting,
> usage of version numbers are gonna be awkward in the future, etc) . It's
> also using the wrong command for trunk, highlighting a problem of
> developing on branch-2 and then forward porting.  (hint: daemon stuff has
> been handled by bin/yarn in trunk for almost 2 years now...)
>
> Not enough to stop a merge, IMHO, but definitely should be followed up on
> post-merge.
>
> On 2016-07-01 09:00 (-0700), Sangjin Lee  wrote:
> > Hi folks,>
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge>
> > feature branch YARN-2928 [2] (Timeline Service v.2) to trunk. The vote
> will>
> > run for 7 days, and will end July 8 9:00 AM PDT.>
> >
> > We have been developing the feature in a feature branch (YARN-2928 [3])
> for>
> > a while, and we are reasonably confident that the state of the feature>
> > meets the criteria to be merged onto trunk and we'd love folks to get
> their>
> > hands on it in a test capacity and provide valuable feedback so that we
> can>
> > make it production-ready.>
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and>
> > usability improvements based on a new architecture. You can browse the>
> > requirements/design doc, the storage schema doc, the new entity/data
> model,>
> > the YARN documentation, and also discussions on subsequent milestones on>
> > YARN-2928 [2].>
> >
> > What we would like to merge to trunk is termed "alpha 1" (milestone 1).
> The>
> > feature has a complete end-to-end read/write flow, and you should be
> able>
> > to start setting it up and testing it. At a high level, the following
> are>
> > the key features that have been implemented:>
> >
> > - distributed writers (collectors) as NM aux services>
> > - HBase storage>
> > - new entity model that includes flows>
> > - setting the flow context via YARN app tags>
> > - real time metrics aggregation to the application level and the flow
> level>
> > - rich REST API that supports filters, complex conditionals, limits,>
> > content selection, etc.>
> > - YARN generic events and system metrics>
> > - integration with Distributed Shell and MapReduce>
> >
> > There are a total of 139 subtasks that were completed as part of this>
> > effort.>
> >
> > We paid close attention to ensure that once disabled Timeline Service
> v.2>
> > does not impact existing functionality when disabled (by default).>
> >
> > Special thanks go to all the folks who worked real hard on this feature:
> Li>
> > Lu, Junping Du, Vinod Kumar Vavilapalli, Naga G. R., Varun Saxena,
> Vrushali>
> > C, and Joep Rottinghuis!>
> >
> > Regards,>
> > Sangjin>
> >
> > [1]>
> >
> https://lists.apache.org/thread.html/314288f399f5ef96b2f4f1fc055ac5d440310c8d98ace9a0045e8729@%3Cyarn-dev.hadoop.apache.org%3E
> >
> > [2] https://issues.apache.org/jira/browse/YARN-2928>
> > [3] https://github.com/apache/hadoop/commits/YARN-2928>
> >
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (YARN-5313) TestLogsCLI.testFetchApplictionLogsAsAnotherUser fails in trunk

2016-07-05 Thread Xuan Gong (JIRA)

Xuan Gong created YARN-5313:
---

 Summary: TestLogsCLI.testFetchApplictionLogsAsAnotherUser fails in 
trunk
 Key: YARN-5313
 URL: https://issues.apache.org/jira/browse/YARN-5313
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Xuan Gong


We have reverted HADOOP-12718 recently which caused this failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-3561) Non-AM Containers continue to run even after AM is stopped

2016-07-05 Thread Vinod Kumar Vavilapalli (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-3561.
---
Resolution: Duplicate

Duplicate of HADOOP-12317.

> Non-AM Containers continue to run even after AM is stopped
> --
>
> Key: YARN-3561
> URL: https://issues.apache.org/jira/browse/YARN-3561
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, yarn
>Affects Versions: 2.6.0
> Environment: debian 7
>Reporter: Gour Saha
>Priority: Critical
> Attachments: app0001.zip, application_1431771946377_0001.zip
>
>
> Non-AM containers continue to run even after application is stopped. This 
> occurred while deploying Storm 0.9.3 using Slider (0.60.0 and 0.70.1) in a 
> Hadoop 2.6 deployment. 
> Following are the NM logs from 2 different nodes:
> *host-07* - where Slider AM was running
> *host-03* - where Storm NIMBUS container was running.
> *Note:* The logs are partial, starting with the time when the relevant Slider 
> AM and NIMBUS containers were allocated, till the time when the Slider AM was 
> stopped. Also, the large number of "Memory usage" log lines were removed 
> keeping only a few starts and ends of every segment.
> *NM log from host-07 where Slider AM container was running:*
> {noformat}
> 2015-04-29 00:39:24,614 INFO  monitor.ContainersMonitorImpl 
> (ContainersMonitorImpl.java:run(356)) - Stopping resource-monitoring for 
> container_1428575950531_0020_02_01
> 2015-04-29 00:41:10,310 INFO  ipc.Server (Server.java:saslProcess(1306)) - 
> Auth successful for appattempt_1428575950531_0021_01 (auth:SIMPLE)
> 2015-04-29 00:41:10,322 INFO  containermanager.ContainerManagerImpl 
> (ContainerManagerImpl.java:startContainerInternal(803)) - Start request for 
> container_1428575950531_0021_01_01 by user yarn
> 2015-04-29 00:41:10,322 INFO  containermanager.ContainerManagerImpl 
> (ContainerManagerImpl.java:startContainerInternal(843)) - Creating a new 
> application reference for app application_1428575950531_0021
> 2015-04-29 00:41:10,323 INFO  application.Application 
> (ApplicationImpl.java:handle(464)) - Application 
> application_1428575950531_0021 transitioned from NEW to INITING
> 2015-04-29 00:41:10,325 INFO  nodemanager.NMAuditLogger 
> (NMAuditLogger.java:logSuccess(89)) - USER=yarn   IP=10.84.105.162
> OPERATION=Start Container Request   TARGET=ContainerManageImpl  
> RESULT=SUCCESS  APPID=application_1428575950531_0021
> CONTAINERID=container_1428575950531_0021_01_01
> 2015-04-29 00:41:10,328 WARN  logaggregation.LogAggregationService 
> (LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root 
> Log Dir [/app-logs] already exist, but with incorrect permissions. Expected: 
> [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple 
> users.
> 2015-04-29 00:41:10,328 WARN  logaggregation.AppLogAggregatorImpl 
> (AppLogAggregatorImpl.java:(182)) - rollingMonitorInterval is set as 
> -1. The log rolling mornitoring interval is disabled. The logs will be 
> aggregated after this application is finished.
> 2015-04-29 00:41:10,351 INFO  application.Application 
> (ApplicationImpl.java:transition(304)) - Adding 
> container_1428575950531_0021_01_01 to application 
> application_1428575950531_0021
> 2015-04-29 00:41:10,352 INFO  application.Application 
> (ApplicationImpl.java:handle(464)) - Application 
> application_1428575950531_0021 transitioned from INITING to RUNNING
> 2015-04-29 00:41:10,356 INFO  container.Container 
> (ContainerImpl.java:handle(999)) - Container 
> container_1428575950531_0021_01_01 transitioned from NEW to LOCALIZING
> 2015-04-29 00:41:10,357 INFO  containermanager.AuxServices 
> (AuxServices.java:handle(196)) - Got event CONTAINER_INIT for appId 
> application_1428575950531_0021
> 2015-04-29 00:41:10,357 INFO  localizer.LocalizedResource 
> (LocalizedResource.java:handle(203)) - Resource 
> hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/htrace-core-3.0.4.jar
>  transitioned from INIT to DOWNLOADING
> 2015-04-29 00:41:10,357 INFO  localizer.LocalizedResource 
> (LocalizedResource.java:handle(203)) - Resource 
> hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/jettison-1.1.jar
>  transitioned from INIT to DOWNLOADING
> 2015-04-29 00:41:10,358 INFO  localizer.LocalizedResource 
> (LocalizedResource.java:handle(203)) - Resource 
> hdfs://zsexp/user/yarn/.slider/cluster/storm1/tmp/application_1428575950531_0021/am/lib/api-util-1.0.0-M20.jar
>  transitioned from INIT to DOWNLOADING
> 2015-04-29 00:41:10,358 INFO  localizer.LocalizedResource 
> (LocalizedResource.java:handle(203)) - Resource 
>

Re: [VOTE] merge feature branch YARN-2928

2016-07-05 Thread 俊平堵

+1 (binding). Thanks for pushing it out, Sangjin!

Thanks,

Junping

2016-07-01 17:00 GMT+01:00 Sangjin Lee :

> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-2928 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end July 8 9:00 AM PDT.
>
> We have been developing the feature in a feature branch (YARN-2928 [3]) for
> a while, and we are reasonably confident that the state of the feature
> meets the criteria to be merged onto trunk and we'd love folks to get their
> hands on it in a test capacity and provide valuable feedback so that we can
> make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. You can browse the
> requirements/design doc, the storage schema doc, the new entity/data model,
> the YARN documentation, and also discussions on subsequent milestones on
> YARN-2928 [2].
>
> What we would like to merge to trunk is termed "alpha 1" (milestone 1). The
> feature has a complete end-to-end read/write flow, and you should be able
> to start setting it up and testing it. At a high level, the following are
> the key features that have been implemented:
>
> - distributed writers (collectors) as NM aux services
> - HBase storage
> - new entity model that includes flows
> - setting the flow context via YARN app tags
> - real time metrics aggregation to the application level and the flow level
> - rich REST API that supports filters, complex conditionals, limits,
> content selection, etc.
> - YARN generic events and system metrics
> - integration with Distributed Shell and MapReduce
>
> There are a total of 139 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks go to all the folks who worked real hard on this feature: Li
> Lu, Junping Du, Vinod Kumar Vavilapalli, Naga G. R., Varun Saxena, Vrushali
> C, and Joep Rottinghuis!
>
> Regards,
> Sangjin
>
> [1]
>
> https://lists.apache.org/thread.html/314288f399f5ef96b2f4f1fc055ac5d440310c8d98ace9a0045e8729@%3Cyarn-dev.hadoop.apache.org%3E
> [2] https://issues.apache.org/jira/browse/YARN-2928
> [3] https://github.com/apache/hadoop/commits/YARN-2928
>

Re: [DICUSS] Upgrading Guice to 4.0(HADOOP-12064)

2016-07-05 Thread Ravi Prakash

Go Go Go! Thanks for all the upgrade work Tsuyoshi!

On Thu, Jun 30, 2016 at 12:03 PM, Tsuyoshi Ozawa  wrote:

> Thanks, Andrew.
>
> Based on discussion here, I would like to merge it into *trunk* if
> there are no objection tomorrow.
>
> Thanks,
> - Tsuyoshi
>
> On Wed, Jun 29, 2016 at 12:28 PM, Andrew Wang 
> wrote:
> > I think it's okay to merge. We've already bumped other deps in trunk.
> >
> > On Wed, Jun 29, 2016 at 12:27 PM, Tsuyoshi Ozawa 
> wrote:
> >>
> >> I forgot to mention about importance point: it's a blocker issue to
> >> compile Hadoop with JDK8. Hence, we need to merge it on both client
> >> side and server slide anyway.
> >>
> >> Thanks,
> >> - Tsuyoshi
> >>
> >> On Wed, Jun 29, 2016 at 12:24 PM, Tsuyoshi Ozawa 
> wrote:
> >> > Thanks Vinod, Sangjin, Sean for your comment.
> >> >
> >> > Okay, I will take a look at the class path isolation.
> >> > Should I postpone to merge Guice upgrade to trunk? IMHO, it works with
> >> > tests, so it's okay to merge to runk. Thoughts?
> >> >
> >> > - Tsuyoshi
> >> >
> >> > On Wed, Jun 29, 2016 at 12:10 PM, Sangjin Lee 
> wrote:
> >> >> Yeah it would be awesome if we can get feedback and/or suggestions on
> >> >> these
> >> >> JIRAs (HADOOP-11804 and HADOOP-13070).
> >> >>
> >> >> Thanks,
> >> >> Sangjin
> >> >>
> >> >> On Wed, Jun 29, 2016 at 10:55 AM, Sean Busbey 
> >> >> wrote:
> >> >>>
> >> >>> At the very least, I'm running through an updated shaded hadoop
> client
> >> >>> this week[1] (HBase is my test application and it wandered onto some
> >> >>> private things that broke in branch-2). And Sangjin has a good lead
> on
> >> >>> an lower-short-term-cost incremental improvement for runtime
> isolation
> >> >>> of apps built on yarn/mapreduce[2]. He's been patiently waiting for
> >> >>> more review feedback.
> >> >>>
> >> >>>
> >> >>> [1]: https://issues.apache.org/jira/browse/HADOOP-11804
> >> >>> [2]: https://issues.apache.org/jira/browse/HADOOP-13070
> >> >>>
> >> >>> On Wed, Jun 29, 2016 at 12:33 PM, Vinod Kumar Vavilapalli
> >> >>>  wrote:
> >> >>> > My strong expectation is that we’ll have a version of classpath
> >> >>> > isolation in our first release of 3.x. I’m planning to spending
> some
> >> >>> > cycles
> >> >>> > right away on this.
> >> >>> >
> >> >>> > Assuming classpath isolation gets in, it is reasonable to bump up
> >> >>> > our
> >> >>> > dependencies like Jetty / Guice to the latest stable versions.
> >> >>> >
> >> >>> > Thanks
> >> >>> > +Vinod
> >> >>> >
> >> >>> >> On Jun 27, 2016, at 6:01 AM, Tsuyoshi Ozawa 
> >> >>> >> wrote:
> >> >>> >>
> >> >>> >> Hi developers,
> >> >>> >>
> >> >>> >> I will plan to upgrade Google Guice dependency on trunk. The
> change
> >> >>> >> also includes asm and cglib upgrade.
> >> >>> >> I checked following points:
> >> >>> >>
> >> >>> >> * Both HDFS and YARN UIs work well.
> >> >>> >> * All webIU-related tests pass as described on HADOOP-12064.
> >> >>> >> * Ran mapreduce job, and it works well.
> >> >>> >>
> >> >>> >> https://issues.apache.org/jira/browse/HADOOP-12064
> >> >>> >>
> >> >>> >> Do you have any concern or opinion?  I would like to merge it to
> >> >>> >> trunk
> >> >>> >> on this Friday if you have no objections.
> >> >>> >>
> >> >>> >> Best,
> >> >>> >> - Tsuyoshi
> >> >>> >>
> >> >>> >>
> >> >>> >>
> -
> >> >>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> >>> >> For additional commands, e-mail:
> common-dev-h...@hadoop.apache.org
> >> >>> >>
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> -
> >> >>> > To unsubscribe, e-mail:
> mapreduce-dev-unsubscr...@hadoop.apache.org
> >> >>> > For additional commands, e-mail:
> >> >>> > mapreduce-dev-h...@hadoop.apache.org
> >> >>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> busbey
> >> >>>
> >> >>>
> -
> >> >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >> >>>
> >> >>
> >>
> >> -
> >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >>
> >
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] merge feature branch YARN-2928

2016-07-05 Thread Li Lu

+1 (binding). 

Let’s look into all concerns raised during the merge vote in post-merge phase. 

Li Lu

> On Jul 5, 2016, at 10:17, Wangda Tan  wrote:
> 
> +1 (Binding),
> Looking forward to merge this into trunk!
> 
> - Wangda
> 
> On Tue, Jul 5, 2016 at 4:18 AM, Naganarasimha G R (Naga) <
> garlanaganarasi...@huawei.com> wrote:
> 
>> +1 (Binding),
>> 
>> May be we need to further look into the points mentioned by Allen !
>> 
>> Regards,
>> + Naga
>> 
>> From: Allen Wittenauer [a...@apache.org]
>> Sent: Tuesday, July 05, 2016 11:40
>> To: yarn-dev@hadoop.apache.org
>> Subject: Re: [VOTE] merge feature branch YARN-2928
>> 
>> -0:
>> 
>> I haven't looked at the code changes (although the pom change in
>> hadoop-aws seems pretty oddball).  But I did do a quick pass through the
>> documentation. It needs some formatting, cleanup, and consistency work done
>> (e.g., not using toc macro, lack of language specific code formatting,
>> usage of version numbers are gonna be awkward in the future, etc) . It's
>> also using the wrong command for trunk, highlighting a problem of
>> developing on branch-2 and then forward porting.  (hint: daemon stuff has
>> been handled by bin/yarn in trunk for almost 2 years now...)
>> 
>> Not enough to stop a merge, IMHO, but definitely should be followed up on
>> post-merge.
>> 
>> On 2016-07-01 09:00 (-0700), Sangjin Lee  wrote:
>>> Hi folks,>
>>> 
>>> Per earlier discussion [1], I'd like to start a formal vote to merge>
>>> feature branch YARN-2928 [2] (Timeline Service v.2) to trunk. The vote
>> will>
>>> run for 7 days, and will end July 8 9:00 AM PDT.>
>>> 
>>> We have been developing the feature in a feature branch (YARN-2928 [3])
>> for>
>>> a while, and we are reasonably confident that the state of the feature>
>>> meets the criteria to be merged onto trunk and we'd love folks to get
>> their>
>>> hands on it in a test capacity and provide valuable feedback so that we
>> can>
>>> make it production-ready.>
>>> 
>>> In a nutshell, Timeline Service v.2 delivers significant scalability and>
>>> usability improvements based on a new architecture. You can browse the>
>>> requirements/design doc, the storage schema doc, the new entity/data
>> model,>
>>> the YARN documentation, and also discussions on subsequent milestones on>
>>> YARN-2928 [2].>
>>> 
>>> What we would like to merge to trunk is termed "alpha 1" (milestone 1).
>> The>
>>> feature has a complete end-to-end read/write flow, and you should be
>> able>
>>> to start setting it up and testing it. At a high level, the following
>> are>
>>> the key features that have been implemented:>
>>> 
>>> - distributed writers (collectors) as NM aux services>
>>> - HBase storage>
>>> - new entity model that includes flows>
>>> - setting the flow context via YARN app tags>
>>> - real time metrics aggregation to the application level and the flow
>> level>
>>> - rich REST API that supports filters, complex conditionals, limits,>
>>> content selection, etc.>
>>> - YARN generic events and system metrics>
>>> - integration with Distributed Shell and MapReduce>
>>> 
>>> There are a total of 139 subtasks that were completed as part of this>
>>> effort.>
>>> 
>>> We paid close attention to ensure that once disabled Timeline Service
>> v.2>
>>> does not impact existing functionality when disabled (by default).>
>>> 
>>> Special thanks go to all the folks who worked real hard on this feature:
>> Li>
>>> Lu, Junping Du, Vinod Kumar Vavilapalli, Naga G. R., Varun Saxena,
>> Vrushali>
>>> C, and Joep Rottinghuis!>
>>> 
>>> Regards,>
>>> Sangjin>
>>> 
>>> [1]>
>>> 
>> https://lists.apache.org/thread.html/314288f399f5ef96b2f4f1fc055ac5d440310c8d98ace9a0045e8729@%3Cyarn-dev.hadoop.apache.org%3E
>>> 
>>> [2] https://issues.apache.org/jira/browse/YARN-2928>
>>> [3] https://github.com/apache/hadoop/commits/YARN-2928>
>>> 
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> 
>> 
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5312) Parameter 'size' in the webservices "/containerlogs/$containerid/$filename" and in AHSWebServices is semantically confusing

2016-07-05 Thread Vinod Kumar Vavilapalli (JIRA)

Vinod Kumar Vavilapalli created YARN-5312:
-

 Summary: Parameter 'size' in the webservices 
"/containerlogs/$containerid/$filename" and in AHSWebServices is semantically 
confusing
 Key: YARN-5312
 URL: https://issues.apache.org/jira/browse/YARN-5312
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli


This got added in YARN-5088 and I found this while reviewing YARN-5224.
bq. Also, the parameter 'size' in the API 
"/containerlogs/$containerid/$filename" and similarly in AHSWebServices is 
confusing with semantics. I think we are better off with an offset and size.

An offset (in bytes, +ve to indicate from the start and -ve to indicate from 
the end) together with a size (in bytes) indicating how much to read from the 
offset are a better combination - this is how most file-system APIs look like, 
for comparison.

I can also imagine number of lines as a better unit than bytes for offset and 
size - perhaps yet another ticket.

/cc [~vvasudev].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

car service appointment morning

2016-07-05 Thread Jian He

Hi all,

Have a car service appointment this morning. I’ll come to office around before 
noon.
 I’ll be online while waiting in the car service center.

Jian
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] merge feature branch YARN-2928

2016-07-05 Thread Wangda Tan

+1 (Binding),
Looking forward to merge this into trunk!

- Wangda

On Tue, Jul 5, 2016 at 4:18 AM, Naganarasimha G R (Naga) <
garlanaganarasi...@huawei.com> wrote:

> +1 (Binding),
>
> May be we need to further look into the points mentioned by Allen !
>
> Regards,
> + Naga
> 
> From: Allen Wittenauer [a...@apache.org]
> Sent: Tuesday, July 05, 2016 11:40
> To: yarn-dev@hadoop.apache.org
> Subject: Re: [VOTE] merge feature branch YARN-2928
>
> -0:
>
> I haven't looked at the code changes (although the pom change in
> hadoop-aws seems pretty oddball).  But I did do a quick pass through the
> documentation. It needs some formatting, cleanup, and consistency work done
> (e.g., not using toc macro, lack of language specific code formatting,
> usage of version numbers are gonna be awkward in the future, etc) . It's
> also using the wrong command for trunk, highlighting a problem of
> developing on branch-2 and then forward porting.  (hint: daemon stuff has
> been handled by bin/yarn in trunk for almost 2 years now...)
>
> Not enough to stop a merge, IMHO, but definitely should be followed up on
> post-merge.
>
> On 2016-07-01 09:00 (-0700), Sangjin Lee  wrote:
> > Hi folks,>
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge>
> > feature branch YARN-2928 [2] (Timeline Service v.2) to trunk. The vote
> will>
> > run for 7 days, and will end July 8 9:00 AM PDT.>
> >
> > We have been developing the feature in a feature branch (YARN-2928 [3])
> for>
> > a while, and we are reasonably confident that the state of the feature>
> > meets the criteria to be merged onto trunk and we'd love folks to get
> their>
> > hands on it in a test capacity and provide valuable feedback so that we
> can>
> > make it production-ready.>
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and>
> > usability improvements based on a new architecture. You can browse the>
> > requirements/design doc, the storage schema doc, the new entity/data
> model,>
> > the YARN documentation, and also discussions on subsequent milestones on>
> > YARN-2928 [2].>
> >
> > What we would like to merge to trunk is termed "alpha 1" (milestone 1).
> The>
> > feature has a complete end-to-end read/write flow, and you should be
> able>
> > to start setting it up and testing it. At a high level, the following
> are>
> > the key features that have been implemented:>
> >
> > - distributed writers (collectors) as NM aux services>
> > - HBase storage>
> > - new entity model that includes flows>
> > - setting the flow context via YARN app tags>
> > - real time metrics aggregation to the application level and the flow
> level>
> > - rich REST API that supports filters, complex conditionals, limits,>
> > content selection, etc.>
> > - YARN generic events and system metrics>
> > - integration with Distributed Shell and MapReduce>
> >
> > There are a total of 139 subtasks that were completed as part of this>
> > effort.>
> >
> > We paid close attention to ensure that once disabled Timeline Service
> v.2>
> > does not impact existing functionality when disabled (by default).>
> >
> > Special thanks go to all the folks who worked real hard on this feature:
> Li>
> > Lu, Junping Du, Vinod Kumar Vavilapalli, Naga G. R., Varun Saxena,
> Vrushali>
> > C, and Joep Rottinghuis!>
> >
> > Regards,>
> > Sangjin>
> >
> > [1]>
> >
> https://lists.apache.org/thread.html/314288f399f5ef96b2f4f1fc055ac5d440310c8d98ace9a0045e8729@%3Cyarn-dev.hadoop.apache.org%3E
> >
> > [2] https://issues.apache.org/jira/browse/YARN-2928>
> > [3] https://github.com/apache/hadoop/commits/YARN-2928>
> >
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (YARN-5311) Document graceful decommission CLI and usage

2016-07-05 Thread Junping Du (JIRA)

Junping Du created YARN-5311:


 Summary: Document graceful decommission CLI and usage
 Key: YARN-5311
 URL: https://issues.apache.org/jira/browse/YARN-5311
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: documentation
Reporter: Junping Du
Assignee: Junping Du






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2016-07-05 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/

[Jul 4, 2016 2:15:04 PM] (aajisaka) HDFS-10589. Javadoc for HAState#HAState and 
HAState#setStateInternal




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes 
   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.cli.TestLogsCLI 
   hadoop.yarn.client.api.impl.TestYarnClient 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/diff-compile-javac-root.txt
  [168K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/diff-javadoc-javadoc-root.txt
  [2.3M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [144K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/85/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [124K]

Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

RE: [VOTE] merge feature branch YARN-2928

2016-07-05 Thread Naganarasimha G R (Naga)

+1 (Binding),

May be we need to further look into the points mentioned by Allen !

Regards,
+ Naga

From: Allen Wittenauer [a...@apache.org]
Sent: Tuesday, July 05, 2016 11:40
To: yarn-dev@hadoop.apache.org
Subject: Re: [VOTE] merge feature branch YARN-2928

-0:

I haven't looked at the code changes (although the pom change in hadoop-aws 
seems pretty oddball).  But I did do a quick pass through the documentation. It 
needs some formatting, cleanup, and consistency work done (e.g., not using toc 
macro, lack of language specific code formatting, usage of version numbers are 
gonna be awkward in the future, etc) . It's also using the wrong command for 
trunk, highlighting a problem of developing on branch-2 and then forward 
porting.  (hint: daemon stuff has been handled by bin/yarn in trunk for almost 
2 years now...)

Not enough to stop a merge, IMHO, but definitely should be followed up on 
post-merge.

On 2016-07-01 09:00 (-0700), Sangjin Lee  wrote:
> Hi folks,>
>
> Per earlier discussion [1], I'd like to start a formal vote to merge>
> feature branch YARN-2928 [2] (Timeline Service v.2) to trunk. The vote will>
> run for 7 days, and will end July 8 9:00 AM PDT.>
>
> We have been developing the feature in a feature branch (YARN-2928 [3]) for>
> a while, and we are reasonably confident that the state of the feature>
> meets the criteria to be merged onto trunk and we'd love folks to get their>
> hands on it in a test capacity and provide valuable feedback so that we can>
> make it production-ready.>
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and>
> usability improvements based on a new architecture. You can browse the>
> requirements/design doc, the storage schema doc, the new entity/data model,>
> the YARN documentation, and also discussions on subsequent milestones on>
> YARN-2928 [2].>
>
> What we would like to merge to trunk is termed "alpha 1" (milestone 1). The>
> feature has a complete end-to-end read/write flow, and you should be able>
> to start setting it up and testing it. At a high level, the following are>
> the key features that have been implemented:>
>
> - distributed writers (collectors) as NM aux services>
> - HBase storage>
> - new entity model that includes flows>
> - setting the flow context via YARN app tags>
> - real time metrics aggregation to the application level and the flow level>
> - rich REST API that supports filters, complex conditionals, limits,>
> content selection, etc.>
> - YARN generic events and system metrics>
> - integration with Distributed Shell and MapReduce>
>
> There are a total of 139 subtasks that were completed as part of this>
> effort.>
>
> We paid close attention to ensure that once disabled Timeline Service v.2>
> does not impact existing functionality when disabled (by default).>
>
> Special thanks go to all the folks who worked real hard on this feature: Li>
> Lu, Junping Du, Vinod Kumar Vavilapalli, Naga G. R., Varun Saxena, Vrushali>
> C, and Joep Rottinghuis!>
>
> Regards,>
> Sangjin>
>
> [1]>
> https://lists.apache.org/thread.html/314288f399f5ef96b2f4f1fc055ac5d440310c8d98ace9a0045e8729@%3Cyarn-dev.hadoop.apache.org%3E>
> [2] https://issues.apache.org/jira/browse/YARN-2928>
> [3] https://github.com/apache/hadoop/commits/YARN-2928>
>
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] merge feature branch YARN-2928

2016-07-05 Thread Allen Wittenauer


-0:

I haven't looked at the code changes (although the pom change in hadoop-aws 
seems pretty oddball).  But I did do a quick pass through the documentation. It 
needs some formatting, cleanup, and consistency work done (e.g., not using toc 
macro, lack of language specific code formatting, usage of version numbers are 
gonna be awkward in the future, etc) . It's also using the wrong command for 
trunk, highlighting a problem of developing on branch-2 and then forward 
porting.  (hint: daemon stuff has been handled by bin/yarn in trunk for almost 
2 years now...)

Not enough to stop a merge, IMHO, but definitely should be followed up on 
post-merge.

On 2016-07-01 09:00 (-0700), Sangjin Lee  wrote: 
> Hi folks,> 
> 
> Per earlier discussion [1], I'd like to start a formal vote to merge> 
> feature branch YARN-2928 [2] (Timeline Service v.2) to trunk. The vote will> 
> run for 7 days, and will end July 8 9:00 AM PDT.> 
> 
> We have been developing the feature in a feature branch (YARN-2928 [3]) for> 
> a while, and we are reasonably confident that the state of the feature> 
> meets the criteria to be merged onto trunk and we'd love folks to get their> 
> hands on it in a test capacity and provide valuable feedback so that we can> 
> make it production-ready.> 
> 
> In a nutshell, Timeline Service v.2 delivers significant scalability and> 
> usability improvements based on a new architecture. You can browse the> 
> requirements/design doc, the storage schema doc, the new entity/data model,> 
> the YARN documentation, and also discussions on subsequent milestones on> 
> YARN-2928 [2].> 
> 
> What we would like to merge to trunk is termed "alpha 1" (milestone 1). The> 
> feature has a complete end-to-end read/write flow, and you should be able> 
> to start setting it up and testing it. At a high level, the following are> 
> the key features that have been implemented:> 
> 
> - distributed writers (collectors) as NM aux services> 
> - HBase storage> 
> - new entity model that includes flows> 
> - setting the flow context via YARN app tags> 
> - real time metrics aggregation to the application level and the flow level> 
> - rich REST API that supports filters, complex conditionals, limits,> 
> content selection, etc.> 
> - YARN generic events and system metrics> 
> - integration with Distributed Shell and MapReduce> 
> 
> There are a total of 139 subtasks that were completed as part of this> 
> effort.> 
> 
> We paid close attention to ensure that once disabled Timeline Service v.2> 
> does not impact existing functionality when disabled (by default).> 
> 
> Special thanks go to all the folks who worked real hard on this feature: Li> 
> Lu, Junping Du, Vinod Kumar Vavilapalli, Naga G. R., Varun Saxena, Vrushali> 
> C, and Joep Rottinghuis!> 
> 
> Regards,> 
> Sangjin> 
> 
> [1]> 
> https://lists.apache.org/thread.html/314288f399f5ef96b2f4f1fc055ac5d440310c8d98ace9a0045e8729@%3Cyarn-dev.hadoop.apache.org%3E>
>  
> [2] https://issues.apache.org/jira/browse/YARN-2928> 
> [3] https://github.com/apache/hadoop/commits/YARN-2928> 
> 
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] merge feature branch YARN-2928

[jira] [Created] (YARN-5313) TestLogsCLI.testFetchApplictionLogsAsAnotherUser fails in trunk

[jira] [Resolved] (YARN-3561) Non-AM Containers continue to run even after AM is stopped

Re: [VOTE] merge feature branch YARN-2928

Re: [DICUSS] Upgrading Guice to 4.0(HADOOP-12064)

Re: [VOTE] merge feature branch YARN-2928

[jira] [Created] (YARN-5312) Parameter 'size' in the webservices "/containerlogs/$containerid/$filename" and in AHSWebServices is semantically confusing

car service appointment morning

Re: [VOTE] merge feature branch YARN-2928

[jira] [Created] (YARN-5311) Document graceful decommission CLI and usage

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

RE: [VOTE] merge feature branch YARN-2928

Re: [VOTE] merge feature branch YARN-2928

13 matches

Site Navigation

Mail list logo

Footer information