[jira] [Created] (MAPREDUCE-7047) Make HAR tool support IndexedLogAggregtionController
Xuan Gong created MAPREDUCE-7047: Summary: Make HAR tool support IndexedLogAggregtionController Key: MAPREDUCE-7047 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7047 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Xuan Gong Assignee: Xuan Gong In https://issues.apache.org/jira/browse/MAPREDUCE-6415, we have created a tool to combine aggregated logs into HAR files which currently only work for TFileLogAggregationFileController. We should make it support IndexedLogAggregtionController as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler configuration) to trunk, branch-3.0, branch-2
+1 (binding) Xuan Gong > >On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jyhung2...@gmail.com> >wrote: > >> Hi all, >> >> From discussion at [1], I'd like to start a vote to merge feature branch >> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, >>ending >> Monday Oct 9 at 11:00AM PDT. >> >> This branch adds a framework to the scheduler to allow scheduler >> configuration mutation on the fly, including a REST and CLI interface, >>and >> an interface for the scheduler configuration backing store. Currently >>the >> capacity scheduler implements this framework. >> >> Umbrella is here (YARN-5734 >> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is >>here >> ( >> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All >>required >> tasks for this feature are committed. Since this feature changes RM >>only, >> we have tested this on a local RM setup with a suite of configuration >> changes with no issue so far. >> >> Key points: >> - The feature is turned off by default, and must be explicitly >>configured >> to turn on. When turned off, the behavior reverts back to the original >>file >> based mechanism for changing scheduler configuration (i.e. yarn rmadmin >> -refreshQueues). >> - The framework was designed in a way to be extendable to other >>schedulers >> (most notably FairScheduler). >> - A pluggable ACL policy (YARN-5949 >> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins >> fine-grained control for who can change what configurations. >> - The configuration storage backend is also pluggable. Currently an >> in-memory, leveldb, and zookeeper implementation are supported. >> >> There were 15 subtasks completed for this feature. >> >> Huge thanks to everyone who helped with reviews, commits, guidance, and >> technical discussion/design, including Carlo Curino, Xuan Gong, Subru >> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, >>Vinod >> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou. >> >> [1] >> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/% >> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=pkec9dpggyx9rzf3n...@mail.gmail.com%3E >> >> Jonathan Hung >> - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.7.3 RC2
+1 Binding. Built from source code. Deployed single node cluster. Successfully ran some example jobs. Thanks Xuan Gong > On Aug 23, 2016, at 10:33 AM, Sunil Govind <sunil.gov...@gmail.com> wrote: > > Hi All > > +1 non-binding > > 1. Built from source and deployed 2 node cluster > 2. Tested node labels basic scenarios > 3. Ran sleep and wordcount. > 4. Verified few cli commands and web UI. > > Thanks > Sunil > > On Thu, Aug 18, 2016 at 7:35 AM Vinod Kumar Vavilapalli <vino...@apache.org> > wrote: > >> Hi all, >> >> I've created a new release candidate RC2 for Apache Hadoop 2.7.3. >> >> As discussed before, this is the next maintenance release to follow up >> 2.7.2. >> >> The RC is available for validation at: >> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ < >> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/> >> >> The RC tag in git is: release-2.7.3-RC2 >> >> The maven artifacts are available via repository.apache.org < >> http://repository.apache.org/> at >> https://repository.apache.org/content/repositories/orgapachehadoop-1046 < >> https://repository.apache.org/content/repositories/orgapachehadoop-1046> >> >> The release-notes are inside the tar-balls at location >> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I >> hosted this at >> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html < >> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html> for >> your quick perusal. >> >> As you may have noted, >> - few issues with RC0 forced a RC1 [1] >> - few more issues with RC1 forced a RC2 [2] >> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) >> caused 2.7.3 (along with every other Hadoop release) to slip by quite a >> bit. This release's related discussion thread is linked below: [3]. >> >> Please try the release and vote; the vote will run for the usual 5 days. >> >> Thanks, >> Vinod >> >> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: >> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 >> <https://www.mail-archive.com/hdfs-dev@hadoop.apache.org/index.html#26106> >> [2] [VOTE] Release Apache Hadoop 2.7.3 RC1: >> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html < >> https://www.mail-archive.com/hdfs-dev@hadoop.apache.org/msg26336.html> >> [3] 2.7.3 release plan: >> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html < >> http://markmail.org/thread/6yv2fyrs4jlepmmr> - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6623) TestRMNMInfo and TestNetworkedJob fails in trunk
Xuan Gong created MAPREDUCE-6623:
Summary: TestRMNMInfo and TestNetworkedJob fails in trunk
Key: MAPREDUCE-6623
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6623
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Xuan Gong
TestRMNMInfo:
{code}
Running org.apache.hadoop.mapreduce.v2.TestRMNMInfo
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.347 sec <<<
FAILURE! - in org.apache.hadoop.mapreduce.v2.TestRMNMInfo
testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 1.572
sec <<< FAILURE!
java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
{code}
TestNetworkedJob
{code}
testNetworkedJob:174 expected:<[[Thu Jan 28 22:41:20 + 2016] Application is
Activated, waiting for resources to be assigned for AM. Details : AM Partition
= ; Partition Resource = <memory:8192, vCores:16> ; Queue's
Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 0.0 % ; Queue's
Absolute max capacity = 100.0 % ; ]> but was:<[]>
TestRMNMInfo.testRMNMInfo:111 Unexpected number of live nodes: expected:<4>
but was:<0>
{code}
JDK version: JDK v1.8.0_66
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6621) Memory Link in JobClient#submitJobInternal()
Xuan Gong created MAPREDUCE-6621: Summary: Memory Link in JobClient#submitJobInternal() Key: MAPREDUCE-6621 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6618) YarnClientProtocolProvider leaking the YarnClient thread.
Xuan Gong created MAPREDUCE-6618:
Summary: YarnClientProtocolProvider leaking the YarnClient thread.
Key: MAPREDUCE-6618
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6618
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
YarnClientProtocolProvider creates YarnRunner which includes
ResourceMgrDelegate. In ResourceMgrDelegate, we would initiate and start
yarnclient. The yarnClient thread would be leaked due to
{code}
@Override
public void close(ClientProtocol clientProtocol) throws IOException {
// nothing to do
}
{code} in YarnClientProtocolProvider
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [VOTE] Release Apache Hadoop 2.7.2 RC2
+1 (binding) - Successfully built from source - Deployed on a single node cluster - Ran a simple MR job and a distributed Shell job successfully. Thanks Xuan Gong > On Jan 19, 2016, at 3:30 PM, Jason Lowe <jl...@yahoo-inc.com.INVALID> wrote: > > That's reasonable, especially if we don't take nearly as long for 2.7.3. > Note that there are almost 50 JIRAs already committed to 2.7.3, so hopefully > we'll have a plan for that soon. > +1 (binding) for 2.7.2 RC2. > Jason > > > From: Vinod Kumar Vavilapalli <vino...@apache.org> > To: mapreduce-dev@hadoop.apache.org; Jason Lowe <jl...@yahoo-inc.com> > Cc: Hadoop Common <common-...@hadoop.apache.org>; > "hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; > "yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org> > Sent: Tuesday, January 19, 2016 5:25 PM > Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC2 > > The JIRA YARN-4610 links YARN-3434 as the one causing the breakage, and > YARN-3434 already exists in 2.7.1 itself. That categorizes the new issue as > an existing bug. > If you agree with that sentiment, and given that there is a clear > work-around, in the interest of progress of 2.7.2 (we have spent > 2 months > on this now), I’d like to move forward. > Please LMK what you think. > Thanks+Vinod > > > On Jan 19, 2016, at 3:13 PM, Jason Lowe <jl...@yahoo-inc.com.INVALID> wrote: > -1 (binding) > We have been running a release derived from 2.7 on some of our clusters, and > we recently hit a bug where an application making large container requests > can drastically slow down container allocations for other users in the same > queue. See YARN-4610 for details. Since > yarn.scheduler.capacity.reservations-continue-look-all-nodes is on by > default, I think we should fix this. If we decide to ship 2.7.2 without that > fix then the release notes should call out that JIRA and mention the > workaround of setting > yarn.scheduler.capacity.reservations-continue-look-all-nodes to false. > Jason > > > From: Vinod Kumar Vavilapalli <vino...@apache.org> > To: Hadoop Common <common-...@hadoop.apache.org>; hdfs-...@hadoop.apache.org; > yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org > Sent: Thursday, January 14, 2016 10:57 PM > Subject: [VOTE] Release Apache Hadoop 2.7.2 RC2 > > Hi all, > > I've created an updated release candidate RC2 for Apache Hadoop 2.7.2. > > As discussed before, this is the next maintenance release to follow up 2.7.1. > > The RC is available for validation at: > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/ > > The RC tag in git is: release-2.7.2-RC2 > > The maven artifacts are available via repository.apache.org > <http://repository.apache.org/> at > https://repository.apache.org/content/repositories/orgapachehadoop-1027 > <https://repository.apache.org/content/repositories/orgapachehadoop-1027> > > The release-notes are inside the tar-balls at location > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted > this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html > <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for > your quick perusal. > > As you may have noted, > - I terminated the RC1 related voting thread after finding out that we didn’t > have a bunch of patches that are already in the released 2.6.3 version. After > a brief discussion, we decided to keep the parallel 2.6.x and 2.7.x releases > incremental, see [4] for this discussion. > - The RC0 related voting thread got halted due to some critical issues. It > took a while again for getting all those blockers out of the way. See the > previous voting thread [3] for details. > - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a > bit. This release's related discussion threads are linked below: [1] and [2]. > > Please try the release and vote; the vote will run for the usual 5 days. > > Thanks, > Vinod > > [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes > <http://markmail.org/message/oozq3gvd4nhzsaes> > [2]: Planning Apache Hadoop 2.7.2 > http://markmail.org/message/iktqss2qdeykgpqk > <http://markmail.org/message/iktqss2qdeykgpqk> > [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: > http://markmail.org/message/5txhvr2qdiqglrwc > <http://markmail.org/message/5txhvr2qdiqglrwc> > [4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: > http://markmail.org/thread/n7ljbsnquihn3wlw > > > > >
Re: [VOTE] Release Apache Hadoop 2.7.2 RC1
+1 (binding) Thanks Xuan Gong > On Dec 21, 2015, at 4:25 PM, Chen He <airb...@gmail.com> wrote: > > +1 (non-binding) > > deploy to a 2-node cluster > run loadgen and wc without problem > > > Kindly ask a silly question, why we separate 2.6.x from 2.7.x? > > On Mon, Dec 21, 2015 at 11:51 AM, Wangda Tan <wheele...@gmail.com> wrote: > >> +1 (binding) >> >> - Build & deploy single-node Hadoop from source code >> - Add/Remove node labels to queues/nodes >> - Run distributed shell commanding using default/specified node labels >> >> Thanks, >> Wangda >> >> >> On Mon, Dec 21, 2015 at 9:58 AM, Masatake Iwasaki < >> iwasak...@oss.nttdata.co.jp> wrote: >> >>> +1(non-binding) >>> >>> - verified mds and signature of source and binary tarball >>> - started 3 node cluster and ran example jobs such as wordcount and >>> terasort >>> - built from source tarball with -Pnative on CentOS 7 and OpenJDK 7 >>> - built site documentation and skimmed the contents >>> >>> Thanks, >>> Masatake Iwasaki >>> >>> >>> >>> On 12/17/15 11:49, Vinod Kumar Vavilapalli wrote: >>> >>>> Hi all, >>>> >>>> I've created a release candidate RC1 for Apache Hadoop 2.7.2. >>>> >>>> As discussed before, this is the next maintenance release to follow up >>>> 2.7.1. >>>> >>>> The RC is available for validation at: >>>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/ < >>>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/> >>>> >>>> The RC tag in git is: release-2.7.2-RC1 >>>> >>>> The maven artifacts are available via repository.apache.org < >>>> http://repository.apache.org/> at >>>> >> https://repository.apache.org/content/repositories/orgapachehadoop-1026/ >>>> < >> https://repository.apache.org/content/repositories/orgapachehadoop-1026/ >>>>> >>>> >>>> The release-notes are inside the tar-balls at location >>>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I >>>> hosted this at >>>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html >> for >>>> quick perusal. >>>> >>>> As you may have noted, >>>> - The RC0 related voting thread got halted due to some critical >> issues. >>>> It took a while again for getting all those blockers out of the way. See >>>> the previous voting thread [3] for details. >>>> - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by >>>> quite a bit. This release's related discussion threads are linked below: >>>> [1] and [2]. >>>> >>>> Please try the release and vote; the vote will run for the usual 5 days. >>>> >>>> Thanks, >>>> Vinod >>>> >>>> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes < >>>> http://markmail.org/message/oozq3gvd4nhzsaes> >>>> [2]: Planning Apache Hadoop 2.7.2 >>>> http://markmail.org/message/iktqss2qdeykgpqk < >>>> http://markmail.org/message/iktqss2qdeykgpqk> >>>> [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: >>>> http://markmail.org/message/5txhvr2qdiqglrwc >>>> >>>> >>>> >>> >>
[jira] [Resolved] (MAPREDUCE-6566) Add retry support to mapreduce CLI tool
[ https://issues.apache.org/jira/browse/MAPREDUCE-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved MAPREDUCE-6566. -- Resolution: Fixed Fix Version/s: 2.8.0 > Add retry support to mapreduce CLI tool > --- > > Key: MAPREDUCE-6566 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6566 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6566.001.patch, MAPREDUCE-6566.002.patch > > > MAPREDUCE-6251 added support for retries to JobClient. However the MR CLI > class doesn't use the JobClient. It would be useful to add support for > retries to the CLI class as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Release Apache Hadoop 2.6.3 RC0
+1 (binding), Build and deploy the cluster from source code. Ran a few example jobs and passed successfully. Xuan Gong > On Dec 16, 2015, at 4:07 PM, Arpit Agarwal <aagar...@hortonworks.com> wrote: > > +1 (binding) > > - Verified signatures for source and binary distributions > - Built jars from source with java 1.7.0_79 > - Deployed single-node pseudo-cluster > - Ran example map reduce jobs > - Ran hdfs admin commands, verified NN web UI shows expected usages > > > > On 12/11/15, 4:16 PM, "Junping Du" <j...@hortonworks.com> wrote: > >> >> Hi all developers in hadoop community, >> I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the next >> maintenance release to follow up 2.6.2.) according to email thread of >> release plan 2.6.3 [1]. Sorry for this RC coming a bit late as several >> blocker issues were getting committed until yesterday. Below is the details: >> >> The RC is available for validation at: >> *http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/ >> <http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/>* >> >> The RC tag in git is: release-2.6.3-RC0 >> >> The maven artifacts are staged via repository.apache.org at: >> *https://repository.apache.org/content/repositories/orgapachehadoop-1025/? >> <https://repository.apache.org/content/repositories/orgapachehadoop-1025/>* >> >> You can find my public key at: >> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS >> >> Please try the release and vote. The vote will run for the usual 5 days. >> >> Thanks and happy weekend! >> >> >> Cheers, >> >> Junping >> >> >> [1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y >>
Re: [VOTE] Release Apache Hadoop 2.6.1 RC1
+1 (non-binding) Download and compile the source code, run several MR jobs. Xuan Gong > On Sep 16, 2015, at 7:10 PM, Vinod Kumar Vavilapalli <vino...@apache.org> > wrote: > > Hi all, > > After a nearly month long [1] toil, with loads of help from Sangjin Lee and > Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release > candidate RC1 for hadoop-2.6.1. > > RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes > for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779, > YARN-3248), additional fix to avoid incompatibility (YARN-3740), other UI > bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch for > YARN-2890) that Jeff Zhang raised. > > The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/ > > The RC tag in git is: release-2.6.1-RC1 > > The maven artifacts are available via repository.apache.org at > https://repository.apache.org/content/repositories/orgapachehadoop-1021 > > Some notes from our release process > - - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] - > non-committed but desired patches. 2.6.1 is already big as is and is late > by any standard, we can definitely include them in the next release. > - The 2.6.1 wiki page [3] captures some (but not all) of the context of > the patches that we pushed in. > - Given the number of fixes pushed [4] in, we had to make a bunch of > changes to our original plan - we added a few improvements that helped us > backport patches easier (or in many cases made backports possible), and we > dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676, > HDFS-7611, HDFS-7843, HDFS-8850). > - I ran all the unit tests which (surprisingly?) passed. (Except for one, > which pointed out a missing fix HDFS-7552). > > As discussed before [5] > - This release is the first point release after 2.6.0 > - I’d like to use this as a starting release for 2.6.2 in a few weeks and > then follow up with more of these. > > Please try the release and vote; the vote will run for the usual 5 days. > > Thanks, > Vinod > > [0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy > [1] Hadoop 2.6.1 Release process thread: > http://markmail.org/thread/wkbgkxkhntx5tlux > [2] 2.6.1 Pending tickets: > https://issues.apache.org/jira/issues/?filter=12331711 > [3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1 > -Working-Notes > [4] List of 2.6.1 patches pushed: > https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1 > %20and%20labels%20%3D%20%222.6.1-candidate%22 > [5] Planning Hadoop 2.6.1 release: > http://markmail.org/thread/sbykjn5xgnksh6wg > > PS: > - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these > fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off > 2.6.1. > - The additional patches in RC1 that got into 2.6.1 all the way from 2.8 > are NOT in 2.7.2 yet, this will be done as a followup.
Re: [VOTE] Release Apache Hadoop 2.6.1 RC1
Update my vote from +1 (non-binding) to +1 binding Thanks Xuan Gong > On Sep 17, 2015, at 2:05 PM, Xuan Gong <xg...@hortonworks.com> wrote: > > +1 (non-binding) > Download and compile the source code, run several MR jobs. > > Xuan Gong > >> On Sep 16, 2015, at 7:10 PM, Vinod Kumar Vavilapalli <vino...@apache.org> >> wrote: >> >> Hi all, >> >> After a nearly month long [1] toil, with loads of help from Sangjin Lee and >> Akira Ajisaka, and 153 (RC0)+7(RC1) commits later, I've created a release >> candidate RC1 for hadoop-2.6.1. >> >> RC1 is RC0 [0] (for which I opened and closed a vote last week) + UI fixes >> for the issue Sangjin raised (YARN-3171 and the dependencies YARN-3779, >> YARN-3248), additional fix to avoid incompatibility (YARN-3740), other UI >> bugs (YARN-1884, YARN-3544) and the MiniYARNCluster issue (right patch for >> YARN-2890) that Jeff Zhang raised. >> >> The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.6.1-RC1/ >> >> The RC tag in git is: release-2.6.1-RC1 >> >> The maven artifacts are available via repository.apache.org at >> https://repository.apache.org/content/repositories/orgapachehadoop-1021 >> >> Some notes from our release process >> - - Sangjin and I moved out a bunch of items pending from 2.6.1 [2] - >> non-committed but desired patches. 2.6.1 is already big as is and is late >> by any standard, we can definitely include them in the next release. >> - The 2.6.1 wiki page [3] captures some (but not all) of the context of >> the patches that we pushed in. >> - Given the number of fixes pushed [4] in, we had to make a bunch of >> changes to our original plan - we added a few improvements that helped us >> backport patches easier (or in many cases made backports possible), and we >> dropped a few that didn't make sense (HDFS-7831, HDFS-7926, HDFS-7676, >> HDFS-7611, HDFS-7843, HDFS-8850). >> - I ran all the unit tests which (surprisingly?) passed. (Except for one, >> which pointed out a missing fix HDFS-7552). >> >> As discussed before [5] >> - This release is the first point release after 2.6.0 >> - I’d like to use this as a starting release for 2.6.2 in a few weeks and >> then follow up with more of these. >> >> Please try the release and vote; the vote will run for the usual 5 days. >> >> Thanks, >> Vinod >> >> [0] Hadoop 2.6.1 RC0 vote: http://markmail.org/thread/ubut2rn3lodc55iy >> [1] Hadoop 2.6.1 Release process thread: >> http://markmail.org/thread/wkbgkxkhntx5tlux >> [2] 2.6.1 Pending tickets: >> https://issues.apache.org/jira/issues/?filter=12331711 >> [3] 2.6.1 Wiki page: https://wiki.apache.org/hadoop/Release-2.6.1 >> -Working-Notes >> [4] List of 2.6.1 patches pushed: >> https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%202.6.1 >> %20and%20labels%20%3D%20%222.6.1-candidate%22 >> [5] Planning Hadoop 2.6.1 release: >> http://markmail.org/thread/sbykjn5xgnksh6wg >> >> PS: >> - Note that branch-2.6 which will be the base for 2.6.2 doesn't have these >> fixes yet. Once 2.6.1 goes through, I plan to rebase branch-2.6 based off >> 2.6.1. >> - The additional patches in RC1 that got into 2.6.1 all the way from 2.8 >> are NOT in 2.7.2 yet, this will be done as a followup. >
Re: [VOTE] Release Apache Hadoop 2.7.1 RC0
+1 (non-binding) Compiled and deployed a single node cluster, ran all the tests. Xuan Gong On 6/29/15, 1:03 PM, Arpit Gupta ar...@hortonworks.com wrote: +1 (non binding) We have been testing rolling upgrades and downgrades from 2.6 to this release and have had successful runs. -- Arpit Gupta Hortonworks Inc. http://hortonworks.com/ On Jun 29, 2015, at 12:45 PM, Lei Xu l...@cloudera.com wrote: +1 binding Downloaded src and bin distribution, verified md5, sha1 and sha256 checksums of both tar files. Built src using mvn package. Ran a pseudo HDFS cluster Ran dfs -put some files, and checked files on NN's web interface. On Mon, Jun 29, 2015 at 11:54 AM, Wangda Tan wheele...@gmail.com wrote: +1 (non-binding) Compiled and deployed a single node cluster, tried to change node labels and run distributed_shell with node label specified. On Mon, Jun 29, 2015 at 10:30 AM, Ted Yu yuzhih...@gmail.com wrote: +1 (non-binding) Compiled hbase branch-1 with Java 1.8.0_45 Ran unit test suite which passed. On Mon, Jun 29, 2015 at 7:22 AM, Steve Loughran ste...@hortonworks.com wrote: +1 binding from me. Tests: Rebuild slider with Hadoop.version=2.7.1; ran all the tests including against a secure cluster. Repeated for windows running Java 8. All tests passed On 29 Jun 2015, at 09:45, Vinod Kumar Vavilapalli vino...@apache.org wrote: Hi all, I've created a release candidate RC0 for Apache Hadoop 2.7.1. As discussed before, this is the next stable release to follow up 2.6.0, and the first stable one in the 2.7.x line. The RC is available for validation at: *http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/ http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/* The RC tag in git is: release-2.7.1-RC0 The maven artifacts are available via repository.apache.org at * https://repository.apache.org/content/repositories/orgapachehadoop-101 9/ https://repository.apache.org/content/repositories/orgapachehadoop-101 9/ * Please try the release and vote; the vote will run for the usual 5 days. Thanks, Vinod PS: It took 2 months instead of the planned [1] 2 weeks in getting this release out: post-mortem in a separate thread. [1]: A 2.7.1 release to follow up 2.7.0 http://markmail.org/thread/zwzze6cqqgwq4rmw -- Lei (Eddy) Xu Software Engineer, Cloudera
Re: [VOTE] Release Apache Hadoop 2.7.0 RC0
+1 (Non-binding) + Built the source with Java 7 successfully + Deployed a single node cluster, and ran MR example jobs and DS jobs successfully + RM and ATS webUI showed the apps' information correctly Thanks Xuan Gong On 4/11/15, 9:30 PM, Zhijie Shen zs...@hortonworks.com wrote: +1 (Binding) + Downloaded the source tarball + Built the source with Java 7 successfully + Deployed a single node insecure cluster, and ran MR example jobs and DS jobs successfully, MR AM didn't hang + RM and TS webUI showed the apps' information correctly Thanks, Zhijie From: Wangda Tan wheele...@gmail.com Sent: Friday, April 10, 2015 5:25 PM To: yarn-...@hadoop.apache.org Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; Vinod Kumar Vavilapalli Subject: Re: [VOTE] Release Apache Hadoop 2.7.0 RC0 +1 (Non-binding) Built from src, deployed a single node cluster, and tried to run some MR jobs. On Fri, Apr 10, 2015 at 4:44 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Hi all, I've created a release candidate RC0 for Apache Hadoop 2.7.0. The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.7.0-RC0/ The RC tag in git is: release-2.7.0-RC0 The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1017/ As discussed before - This release will only work with JDK 1.7 and above - I¹d like to use this as a starting release for 2.7.x [1], depending on how it goes, get it stabilized and potentially use a 2.7.1 in a few weeks as the stable release. Please try the release and vote; the vote will run for the usual 5 days. Thanks, Vinod [1]: A 2.7.1 release to follow up 2.7.0 http://markmail.org/thread/zwzze6cqqgwq4rmw
[jira] [Resolved] (MAPREDUCE-6019) MapReduce changes for exposing YARN/MR endpoints on multiple interfaces.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved MAPREDUCE-6019. -- Resolution: Fixed Fix Version/s: 2.6.0 Committed to trunk and branch-2 with YARN-1994 MapReduce changes for exposing YARN/MR endpoints on multiple interfaces. Key: MAPREDUCE-6019 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6019 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Xuan Gong Assignee: Craig Welch Fix For: 2.6.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: [VOTE] Release Apache Hadoop 2.4.1
+1 (non-binding) Successfully build from resource code On Sat, Jun 21, 2014 at 7:05 PM, Wangda Tan wheele...@gmail.com wrote: +1 (non-binding), Built from rc1 source code Installed single node cluster. Ran several YARN applications and get passed. Thanks, Wangda On Sun, Jun 22, 2014 at 9:07 AM, Zhijie Shen zs...@hortonworks.com wrote: +1 (non-binding) Successfully redo the steps for rc0 before for rc1 as well. On Sun, Jun 22, 2014 at 6:22 AM, Steve Loughran ste...@hortonworks.com wrote: +1 (binding) 1. rm -rf ~/.m2/repository/org/apache/hadoop/ 2. build and test of slider/incubating develop branch with profile hadoop-2.4.1, which downloaded all the new artifacts from the repository 3. -tests passed -steve On 20 June 2014 23:51, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created another release candidate (rc1) for hadoop-2.4.1 based on the feedback that I would like to push out. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (MAPREDUCE-5824) TestPipesNonJavaInputFormat.testFormat fails in windows
Xuan Gong created MAPREDUCE-5824: Summary: TestPipesNonJavaInputFormat.testFormat fails in windows Key: MAPREDUCE-5824 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5824 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.4.0 Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5795) Job should be marked as Falied if it is recovered from commit.
Xuan Gong created MAPREDUCE-5795: Summary: Job should be marked as Falied if it is recovered from commit. Key: MAPREDUCE-5795 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong If Resource manager is restarted when a job is in commit state, The job is not able to recovered after RM restart and it is marked as Killed. The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (MAPREDUCE-5795) Job should be marked as Falied if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong reopened MAPREDUCE-5795: -- Reopen the ticket. This ticket is mainly for which job status should be saved in Job History File for the case that AM was in a committing state when RM restarted Job should be marked as Falied if it is recovered from commit. -- Key: MAPREDUCE-5795 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong If Resource manager is restarted when a job is in commit state, The job is not able to recovered after RM restart and it is marked as Killed. The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5424) TestNonExistentJob failing after YARN-873
[ https://issues.apache.org/jira/browse/MAPREDUCE-5424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved MAPREDUCE-5424. -- Resolution: Duplicate is duplicated as MAPREDUCE-5421 TestNonExistentJob failing after YARN-873 - Key: MAPREDUCE-5424 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5424 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Vinod Kumar Vavilapalli Assignee: Xuan Gong Priority: Blocker -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5325) ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter---MR changes
Xuan Gong created MAPREDUCE-5325: Summary: ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter---MR changes Key: MAPREDUCE-5325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5325 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5226) Change AMRMProtocol api to throw IOException and YarnRemoteException in MRv2
Xuan Gong created MAPREDUCE-5226: Summary: Change AMRMProtocol api to throw IOException and YarnRemoteException in MRv2 Key: MAPREDUCE-5226 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5226 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5212) Change ClientRMProtocol api to throw IOException and YarnRemoteException in MR api
Xuan Gong created MAPREDUCE-5212: Summary: Change ClientRMProtocol api to throw IOException and YarnRemoteException in MR api Key: MAPREDUCE-5212 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5212 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5204) Handle YarnRemoteException separately from IOException in MR api
Xuan Gong created MAPREDUCE-5204: Summary: Handle YarnRemoteException separately from IOException in MR api Key: MAPREDUCE-5204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5204 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong YarnRemoteException is not rooted as IOException, so in MR api, we need to handle them separately from IOException -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
