Re: [E] [VOTE] Hadoop 3.1.x EOL

[Kihwal Lee]

+1 (binding)

On Thu, Jun 3, 2021 at 1:14 AM Akira Ajisaka  wrote:

> Dear Hadoop developers,
>
> Given the feedback from the discussion thread [1], I'd like to start
> an official vote
> thread for the community to vote and start the 3.1 EOL process.
>
> What this entails:
>
> (1) an official announcement that no further regular Hadoop 3.1.x releases
> will be made after 3.1.4.
> (2) resolve JIRAs that specifically target 3.1.5 as won't fix.
>
> This vote will run for 7 days and conclude by June 10th, 16:00 JST [2].
>
> Committers are eligible to cast binding votes. Non-committers are welcomed
> to cast non-binding votes.
>
> Here is my vote, +1
>
> [1]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__s.apache.org_w9ilb=DwIBaQ=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY=b6gUZYewojO-9YMJdyeI_g=XwkYgkbxoxl9GFqT_D4JQxurNTSSFm9Q2nNXBsx7kvQ=_t7jj-Mw4LwNSdTgrKrtA5aeB_8t2feBuG89IjMHRYE=
> [2]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.timeanddate.com_worldclock_fixedtime.html-3Fmsg-3D4-26iso-3D20210610T16-26p1-3D248=DwIBaQ=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY=b6gUZYewojO-9YMJdyeI_g=XwkYgkbxoxl9GFqT_D4JQxurNTSSFm9Q2nNXBsx7kvQ=Z7y-vP3_I6MdDApyRBz6c_tBnVtuvyjZB_k5q1zaR2c=
>
> Regards,
> Akira
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [E] Re: [DISCUSS] Change project style guidelines to allow line length 100

[Kihwal Lee]

+1 for the 100 char limit.
But I would have liked 132 columns more.  :)

Kihwal

On Mon, May 24, 2021 at 1:46 PM Sean Busbey 
wrote:

> Hi folks!
>
> The consensus seems pretty strongly in favor of increasing the line length
> limit. Do folks still want to see a formal VOTE thread?
>
>
> > On May 19, 2021, at 4:22 PM, Sean Busbey 
> wrote:
> >
> > Hello!
> >
> > What do folks think about changing our line length guidelines to allow
> for 100 character width?
> >
> > Currently, we tell folks to follow the sun style guide with some
> exception unrelated to line length. That guide says width of 80 is the
> standard and our current check style rules act as enforcement.
> >
> > Looking at the current trunk codebase our nightly build shows a total of
> ~15k line length violations; it’s about 18% of identified checkstyle issues.
> >
> > The vast majority of those line length violations are <= 100 characters
> long. 100 characters happens to be the length for the Google Java Style
> Guide, another commonly adopted style guide for java projects, so I suspect
> these longer lines leaking past the checkstyle precommit warning might be a
> reflection of committers working across multiple java codebases.
> >
> > I don’t feel strongly about lines being longer, but I would like to move
> towards more consistent style enforcement as a project. Updating our
> project guidance to allow for 100 character lines would reduce the
> likelihood that folks bringing in new contributions need a precommit test
> cycle to get the formatting correct.
> >
> > Does anyone feel strongly about keeping the line length limit at 80
> characters?
> >
> > Does anyone feel strongly about contributions coming in that clear up
> line length violations?
> >
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC1)

[Kihwal Lee]

Gabor,
If you want to release asap, you can simply revert HDFS-14941 in the
release branch for now. It is causing the issue and was committed after
3.1.3.  This causes failure of the automated upgrade process and namenode
memory leak.

Kihwal

On Tue, Jun 23, 2020 at 8:47 AM Akira Ajisaka  wrote:

> Hi Gabor,
>
> Thank you for your work!
>
> Kihwal reported IBR leak in standby NameNode:
> https://issues.apache.org/jira/browse/HDFS-15421.
> I think this is a blocker and this affects 3.1.4-RC1. Would you check this?
>
> Best regards,
> Akira
>
> On Mon, Jun 22, 2020 at 10:26 PM Gabor Bota  .invalid>
> wrote:
>
> > Hi folks,
> >
> > I have put together a release candidate (RC1) for Hadoop 3.1.4.
> >
> > The RC is available at:
> http://people.apache.org/~gabota/hadoop-3.1.4-RC1/
> > The RC tag in git is here:
> > https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC1
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1267/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C
> >
> > Please try the release and vote. The vote will run for 5 weekdays,
> > until June 30. 2020. 23:00 CET.
> >
> > Thanks,
> > Gabor
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Kihwal Lee]

+1

Kihwal

On Tue, Aug 20, 2019 at 10:03 PM Wangda Tan  wrote:

> Hi all,
>
> This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> and 3.0 EOL. This is based on discussions of [1]
>
> This discussion runs for 7 days and will conclude on Aug 28 Wed.
>
> Please feel free to share your thoughts.
>
> Thanks,
> Wangda
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> ,
>

[jira] [Created] (MAPREDUCE-7177) Disable speculative execution in TestDFSIO

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-7177:
-

 Summary: Disable speculative execution in TestDFSIO
 Key: MAPREDUCE-7177
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7177
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.8.5, 3.2.0
Reporter: Kihwal Lee


When TestDFSIO runs in a certain environment where a subset of mappers are 
slow, the speculative execution can start.  In the write phase, this will make 
existing mapper to fail in next addBlock() since the output files are 
overwritten.

To make the benchmark more predictable and repeatable, speculation must be 
implicitly disabled in TestDFSIO itself. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

[Kihwal Lee]

+1 (binding)

- Built from source
- Brought up a single node cluster
- Checked basic HDFS functions and checked the UIs
- Ran several simple jobs.


On Mon, Sep 10, 2018 at 7:01 AM 俊平堵  wrote:

> Hi all,
>
>  I've created the first release candidate (RC0) for Apache
> Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> includes 33 important fixes and improvements.
>
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
>
>
> The RC tag in git is: release-2.8.5-RC0
>
>
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1140
>
>
> Please try the release and vote; the vote will run for the usual 5
> working
> days, ending on 9/15/2018 PST time.
>
>
> Thanks,
>
>
> Junping
>

Re: Apache Hadoop 3.0.1 Release plan

[Kihwal Lee]

HADOOP-14060 is a blocker.  Daryn will add more detail to the jira or to
this thread.

On Thu, Feb 8, 2018 at 7:01 AM, Brahma Reddy Battula 
wrote:

> Hi Eddy,
>
> HDFS-12990 got committed to 3.0.1,can we have RC for 3.0.1 (only  YARN-5742
> blocker is open )  ?
>
>
> On Sat, Feb 3, 2018 at 12:40 AM, Chris Douglas 
> wrote:
>
> > On Fri, Feb 2, 2018 at 10:22 AM, Arpit Agarwal  >
> > wrote:
> > > Do you plan to roll an RC with an uncommitted fix? That isn't the right
> > approach.
> >
> > The fix will be committed to the release branch. We'll vote on the
> > release, and if it receives a majority of +1 votes then it becomes
> > 3.0.1. That's how the PMC decides how to move forward. In this case,
> > that will also resolve whether or not it can be committed to trunk.
> >
> > If this logic is unpersuasive, then we can require a 2/3 majority to
> > replace the codebase. Either way, the PMC will vote to define the
> > consensus view when it is not emergent.
> >
> > > This issue has good visibility and enough discussion.
> >
> > Yes, it has. We always prefer consensus to voting, but when discussion
> > reveals that complete consensus is impossible, we still need a way
> > forward. This is rare, and usually reserved for significant changes
> > (like merging YARN). Frankly, it's embarrassing to resort to it here,
> > but here we are.
> >
> > > If there is a binding veto in effect then the change must be abandoned.
> > Else you should be able to proceed with committing. However, 3.0.0 must
> be
> > called out as an abandoned release if we commit it.
> >
> > This is not accurate. A binding veto from any committer halts
> > progress, but the PMC sets the direction of the project. That includes
> > making decisions that are not universally accepted. -C
> >
> > > On 2/1/18, 3:01 PM, "Lei Xu"  wrote:
> > >
> > > Sounds good to me, ATM.
> > >
> > > On Thu, Feb 1, 2018 at 2:34 PM, Aaron T. Myers 
> > wrote:
> > > > Hey Anu,
> > > >
> > > > My feeling on HDFS-12990 is that we've discussed it quite a bit
> > already and
> > > > it doesn't seem at this point like either side is going to budge.
> > I'm
> > > > certainly happy to have a phone call about it, but I don't expect
> > that we'd
> > > > make much progress.
> > > >
> > > > My suggestion is that we simply include the patch posted to
> > HDFS-12990 in
> > > > the 3.0.1 RC and call this issue out clearly in the subsequent
> > VOTE thread
> > > > for the 3.0.1 release. Eddy, are you up for that?
> > > >
> > > > Best,
> > > > Aaron
> > > >
> > > > On Thu, Feb 1, 2018 at 1:13 PM, Lei Xu  wrote:
> > > >>
> > > >> +Xiao
> > > >>
> > > >> My understanding is that we will have this for 3.0.1.   Xiao,
> > could
> > > >> you give your inputs here?
> > > >>
> > > >> On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer <
> > aengin...@hortonworks.com>
> > > >> wrote:
> > > >> > Hi Eddy,
> > > >> >
> > > >> > Thanks for driving this release. Just a quick question, do we
> > have time
> > > >> > to close this issue?
> > > >> > https://issues.apache.org/jira/browse/HDFS-12990
> > > >> >
> > > >> > or are we abandoning it? I believe that this is the last
> window
> > for us
> > > >> > to fix this issue.
> > > >> >
> > > >> > Should we have a call and get this resolved one way or
> another?
> > > >> >
> > > >> > Thanks
> > > >> > Anu
> > > >> >
> > > >> > On 2/1/18, 10:51 AM, "Lei Xu"  wrote:
> > > >> >
> > > >> > Hi, All
> > > >> >
> > > >> > I just cut branch-3.0.1 from branch-3.0.  Please make sure
> > all
> > > >> > patches
> > > >> > targeted to 3.0.1 being checked in both branch-3.0 and
> > branch-3.0.1.
> > > >> >
> > > >> > Thanks!
> > > >> > Eddy
> > > >> >
> > > >> > On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu  >
> > wrote:
> > > >> > > Hi, All
> > > >> > >
> > > >> > > We have released Apache Hadoop 3.0.0 in December [1]. To
> > further
> > > >> > > improve the quality of release, we plan to cut
> > branch-3.0.1 branch
> > > >> > > tomorrow for the preparation of Apache Hadoop 3.0.1
> > release. The
> > > >> > focus
> > > >> > > of 3.0.1 will be fixing blockers (3), critical bugs (1)
> > and bug
> > > >> > fixes
> > > >> > > [2].  No new features and improvement should be
> included.
> > > >> > >
> > > >> > > We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote
> > for RC on
> > > >> > Feb
> > > >> > > 1st, targeting for Feb 9th release.
> > > >> > >
> > > >> > > Please feel free to share your insights.
> > > >> > >
> > > >> > > [1]
> > > >> > 

Re: Apache Hadoop 2.8.2 Release Plan

[Kihwal Lee]

Thanks for driving the next 2.8 release, Junping. While I was committing a 
blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
2.8.2.
Thanks,Kihwal

On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du  
wrote:

Hi all,
    Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
released today which is a special security release. Now, we should work towards 
2.8.2 release which aim for production deployment. The focus obviously is to 
fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We 
currently have 13 blocker/critical issues, and 10 of them are Patch Available.

  I plan to cut an RC in a month - target for releasing before end of Aug., to 
give enough time for outstanding blocker / critical issues. Will start moving 
out any tickets that are not blockers and/or won't fit the timeline. For 
progress of releasing effort, please refer our release wiki [2].

  Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
[2] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


From: Vinod Kumar Vavilapalli 
Sent: Thursday, July 20, 2017 1:05 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released

Hi all,

The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
page: http://hadoop.apache.org/releases.html#Download
This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus 
security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.

Please note that 2.8.x release line continues to be not yet ready for 
production use. Critical issues are being ironed out via testing and downstream 
adoption. Production users should wait for a subsequent release in the 2.8.x 
line.

Thanks
+Vinod


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6767) TestSlive fails after a common change

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-6767:
-

 Summary: TestSlive fails after a common change
 Key: MAPREDUCE-6767
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6767
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Kihwal Lee






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [Release thread] 2.6.5 release activities

[Kihwal Lee]

You might want the snapshot bug fix done in HDFS-7056. This bug creates 
snapshot filediffs even if you never use snapshot. For 2.6, we will have to do 
it in a separate jira to pick up the fix only. Related to this, HDFS-9696 might 
be of interest too.
Kihwal

  From: Chris Trezzo 
 To: Karthik Kambatla  
Cc: "common-...@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org" ; 
"mapreduce-dev@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org" 
 Sent: Thursday, August 11, 2016 6:42 PM
 Subject: Re: [Release thread] 2.6.5 release activities
   
Thanks everyone for the discussion! Based on what has been said in this
thread, I will move forward on the preparation efforts (working with
Sangjin and the community) for a 2.6.5 release. If there is a strong
objection to this, please let us know.

I see three initial tasks going forward:

1. Fix the pre-commit build for branch-2.6. I am not entirely sure what is
involved here, but will start looking into it using HADOOP-12800 as a
starting point.

2. Look through the unresolved issues still targeted to 2.6.5 and
resolve/re-target as necessary. There are currently 17 of them:
https://issues.apache.org/jira/issues/?jql=(project%20%
3D%20%22Hadoop%20Common%22%20OR%20project%20%3D%20%22Hadoop%20YARN%22%20OR%
20project%20%3D%20%22Hadoop%20Map%2FReduce%22%20OR%
20project%20%3D%20%22Hadoop%20HDFS%22)%20AND%20(
fixVersion%20%3D%202.6.5%20OR%20%22Target%20Version%2Fs%22%
20%3D%202.6.5)%20AND%20(status%20!%3D%20Resolved%20AND%20status%20!%3D%
20Closed)%20ORDER%20BY%20status%20ASC

3. Look through the backport candidates in the spreadsheet in more detail
and backport/drop as necessary. There are currently 15 of them:
https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC
8hTRUemHvYPPzY/edit?usp=sharing

Finally, I think it would be awesome to continue the release end-of-life
discussion. As we get better and better at release cadence it is going to
be really helpful to have an established EOL policy. It will be less
confusing for contributors and help set clear expectations for end users as
to when they need to start making reasonable migration plans, especially if
they want to stay on a well supported release line. I would be happy to
help with this effort as well. It would be great if we could leverage that
discussion and have EOL information for the 2.6 line when we release 2.6.5.

As always, please let me know if I have missed something.

Thanks!
Chris Trezzo

On Thu, Aug 11, 2016 at 1:03 PM, Karthik Kambatla 
wrote:

> Since there is sufficient interest in 2.6.5, we should probably do it. All
> the reasons Allen outlines make sense.
>
> That said, Junping brings up a very important point that we should think of
> for future releases. For a new user or a user that does not directly
> contribute to the project, more stable releases make it hard to pick from.
>
> As Chris T mentioned, the notion of EOL for our releases seems like a good
> idea. However, to come up with any EOLs, we need to have some sort of
> cadence for the releases. While this is hard for major releases (big bang,
> potentially incompatible features), it should be doable for minor releases.
>
> How do people feel about doing a minor release every 6 months, with
> follow-up maintenance releases every 2 months until the next minor and as
> needed after that? That way, we could EOL a minor release a year after its
> initial release? In the future, we could consider shrinking this window. In
> addition to the EOL, this also makes our releases a little more predictable
> for both users and vendors. Note that 2.6.0 went out almost 2 years ago and
> we haven't had a new minor release in 14 months. I am happy to start
> another DISCUSS thread around this if people think it is useful.
>
> Thanks
> Karthik
>
> On Thu, Aug 11, 2016 at 12:50 PM, Allen Wittenauer <
> a...@effectivemachines.com
> > wrote:
>
> >
> > > On Aug 11, 2016, at 8:10 AM, Junping Du  wrote:
> > >
> > > Allen, to be clear, I am not against any branch release effort here.
> > However,
> >
> >                "I'm not an X but "
> >
> > > as RM for previous releases 2.6.3 and 2.6.4, I feel to have
> > responsibility to take care branch-2.6 together with other RMs (Vinod and
> > Sangjin) on this branch and understand current gap - especially, to get
> > consensus from community on the future plan for 2.6.x.
> > > Our bylaw give us freedom for anyone to do release effort, but our
> bylaw
> > doesn't stop our rights for reasonable question/concern on any release
> > plan. As you mentioned below, people can potentially fire up branch-1
> > release effort. But if you call a release plan tomorrow for branch-1, I
> > cannot imagine nobody will question on that effort. Isn't it?
> >
> >                From previous discussions I've seen 

Re: 2.7.3 release plan

[Kihwal Lee]

Just reverted HDFS-8791 from branch-2.7.Eulogy: Although it has ascended to a 
better version, it did caught an upgrade bug while in branch-2.7.
Kihwal


  From: Vinod Kumar Vavilapalli 
 To: yarn-...@hadoop.apache.org 
Cc: Hadoop Common ; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org
 Sent: Wednesday, April 6, 2016 5:00 PM
 Subject: Re: 2.7.3 release plan
   
Down to only 10 blocker / critical tickets 
(https://issues.apache.org/jira/issues/?filter=12335343 
) now!

Thanks
+Vinod

> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli  
> wrote:
> 
> Hi all,
> 
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did 
> go out mid February). Got a little busy since.
> 
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. 
> The focus obviously is to have blocker issues [1], bug-fixes and *no* 
> features / improvements.
> 
> I hope to cut an RC in a week - giving enough time for outstanding blocker / 
> critical issues. Will start moving out any tickets that are not blockers 
> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets 
> outstanding as of now.
> 
> Thanks,
> +Vinod
> 
> [1] 2.7.3 release blockers: 
> https://issues.apache.org/jira/issues/?filter=12335343


  

Re: Looking to a Hadoop 3 release

[Kihwal Lee]

Moving Hadoop 3 forward sounds fine. If EC is one of the main motivations, are 
we getting rid of branch-2.8? 

Kihwal

  From: Andrew Wang 
 To: "common-...@hadoop.apache.org"  
Cc: "yarn-...@hadoop.apache.org" ; 
"mapreduce-dev@hadoop.apache.org" ; hdfs-dev 

 Sent: Thursday, February 18, 2016 4:35 PM
 Subject: Re: Looking to a Hadoop 3 release
   
Hi all,

Reviving this thread. I've seen renewed interest in a trunk release since
HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
shell script rewrite, and many other improvements, I think it's time to
revisit Hadoop 3.0 release plans.

My overall plan is still the same as in my original email: a series of
regular alpha releases leading up to beta and GA. Alpha releases make it
easier for downstreams to integrate with our code, and making them regular
means features can be included when they are ready.

I know there are some incompatible changes waiting in the wings
(i.e. HDFS-6984 making FileStatus a PB rather than Writable, some of
HADOOP-9991 bumping dependency versions) that would be good to get in. If
you have changes like this, please set the target version to 3.0.0 and mark
them "Incompatible". We can use this JIRA query to track:

https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hadoop%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority

There's some release-related stuff that needs to be sorted out (namely, the
new CHANGES.txt and release note generation from Yetus), but I'd
tentatively like to roll the first alpha a month out, so third week of
March.

Best,
Andrew

On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata  wrote:

> Avoiding the use of JDK8 language features (and, presumably, APIs)
> means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> source version to JDK8.
>
> Also, note that releasing from trunk is a way of achieving #3, it's
> not a way of abandoning it.
>
>
>
> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang 
> wrote:
> > Hi Raymie,
> >
> > Konst proposed just releasing off of trunk rather than cutting a
> branch-2,
> > and there was general agreement there. So, consider #3 abandoned. 1&2 can
> > be achieved at the same time, we just need to avoid using JDK8 language
> > features in trunk so things can be backported.
> >
> > Best,
> > Andrew
> >
> > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata 
> wrote:
> >
> >> In this (and the related threads), I see the following three
> requirements:
> >>
> >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> >>
> >> 2. "We'll still be releasing 2.x releases for a while, with similar
> >> feature sets as 3.x."
> >>
> >> 3. Avoid the "risk of split-brain behavior" by "minimize backporting
> >> headaches. Pulling trunk > branch-2 > branch-2.x is already tedious.
> >> Adding a branch-3, branch-3.x would be obnoxious."
> >>
> >> These three cannot be achieved at the same time.  Which do we abandon?
> >>
> >>
> >> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia 
> >> wrote:
> >> >
> >> >> On Mar 5, 2015, at 3:21 PM, Siddharth Seth  wrote:
> >> >>
> >> >> 2) Simplification of configs - potentially separating client side
> >> configs
> >> >> and those used by daemons. This is another source of perpetual
> confusion
> >> >> for users.
> >> > + 1 on this.
> >> >
> >> > sanjay
> >>
>

  

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Kihwal Lee]

+1 (binding)Checked out the source and built.Ran basic hdfs and mapred tests on 
a single node cluster

Kihwal
 

  From: Vinod Kumar Vavilapalli 
 To: Hadoop Common ; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org 
 Sent: Thursday, January 14, 2016 10:57 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.2 RC2
   
Hi all,

I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.

As discussed before, this is the next maintenance release to follow up 2.7.1.

The RC is available for validation at: 
http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/

The RC tag in git is: release-2.7.2-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1027 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - I terminated the RC1 related voting thread after finding out that we didn’t 
have a bunch of patches that are already in the released 2.6.3 version. After a 
brief discussion, we decided to keep the parallel 2.6.x and 2.7.x releases 
incremental, see [4] for this discussion.
 - The RC0 related voting thread got halted due to some critical issues. It 
took a while again for getting all those blockers out of the way. See the 
previous voting thread [3] for details.
 - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
bit. This release's related discussion threads are linked below: [1] and [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 

[2]: Planning Apache Hadoop 2.7.2 http://markmail.org/message/iktqss2qdeykgpqk 

[3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
http://markmail.org/message/5txhvr2qdiqglrwc 

[4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
http://markmail.org/thread/n7ljbsnquihn3wlw

  

Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

[Kihwal Lee]

+1 (binding)- Verified the signature/digest.
- I've built the dist tree with the native support from the source.- Brought up 
a single node cluster and ran a set of basic tests.

 

  From: Vinod Kumar Vavilapalli 
 To: Hadoop Common ; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org 
Cc: Vinod Kumar Vavilapalli 
 Sent: Wednesday, December 16, 2015 8:49 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.2 RC1
   
Hi all,

I've created a release candidate RC1 for Apache Hadoop 2.7.2.

As discussed before, this is the next maintenance release to follow up 2.7.1.

The RC is available for validation at: 
http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/ 


The RC tag in git is: release-2.7.2-RC1

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1026/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
for quick perusal.

As you may have noted,
 - The RC0 related voting thread got halted due to some critical issues. It 
took a while again for getting all those blockers out of the way. See the 
previous voting thread [3] for details.
 - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
bit. This release's related discussion threads are linked below: [1] and [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 

[2]: Planning Apache Hadoop 2.7.2 http://markmail.org/message/iktqss2qdeykgpqk 

[3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
http://markmail.org/message/5txhvr2qdiqglrwc


   

Re: [VOTE] Release Apache Hadoop 2.7.2 RC0

[Kihwal Lee]

We found HDFS-9426. The rolling upgrade finalization is not backward 
compatible.I.e. 2.7.1 or 2.6.x datanodes will ignore finalization.

So -1.
Kihwal
  From: Vinod Kumar Vavilapalli 
 To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org 
Cc: vino...@apache.org 
 Sent: Wednesday, November 11, 2015 10:31 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.2 RC0
   
Hi all,


I've created a release candidate RC0 for Apache Hadoop 2.7.2.


As discussed before, this is the next maintenance release to follow up
2.7.1.


The RC is available for validation at:

*http://people.apache.org/~vinodkv/hadoop-2.7.2-RC0/

*


The RC tag in git is: release-2.7.2-RC0


The maven artifacts are available via repository.apache.org at

*https://repository.apache.org/content/repositories/orgapachehadoop-1023/

*


As you may have noted, an unusually long 2.6.3 release caused 2.7.2 to slip
by quite a bit. This release's related discussion threads are linked below:
[1] and [2].


Please try the release and vote; the vote will run for the usual 5 days.


Thanks,

Vinod


[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes

[2]: Planning Apache Hadoop 2.7.2
http://markmail.org/message/iktqss2qdeykgpqk


   

Re: 2.7.2 release plan

[Kihwal Lee]

I will try to get them in or bug Daryn. HDFS-8498 doesn't seem a new bug, so I 
kicked it out to 2.7.3.

Kihwal
  From: Vinod Vavilapalli <vino...@hortonworks.com>
 To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>; Kihwal Lee 
<kih...@yahoo-inc.com> 
Cc: "hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; Chris Nauroth 
<cnaur...@hortonworks.com>; "yarn-...@hadoop.apache.org" 
<yarn-...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" 
<mapreduce-dev@hadoop.apache.org>; Vinod Kumar Vavilapalli 
<vino...@apache.org>; Ming Ma <min...@twitter.com> 
 Sent: Wednesday, October 28, 2015 12:16 PM
 Subject: Re: 2.7.2 release plan
   
I see you already put in both HDFS-8950 and HDFS-7725, so we are good there.

Can you please do a pass on HDFS-8871 and may be help nudge Daryn on HDFS-8893, 
HDFS-8674 and HDFS-8498?

Thanks
+Vinod
 


> On Oct 27, 2015, at 8:12 AM, Kihwal Lee <kih...@yahoo-inc.com.INVALID> wrote:
> 
> I think we need HDFS-8950 and HDFS-7725 in 2.7.2.It should be easy to 
> backport/cherry-pick HDFS-7725. For HDFS-8950, it will be nice if Ming can 
> chime in.
> Kihwal
> 
>      From: Tsuyoshi Ozawa <oz...@apache.org>
> To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org> 
> Cc: Chris Nauroth <cnaur...@hortonworks.com>; "yarn-...@hadoop.apache.org" 
> <yarn-...@hadoop.apache.org>; "hdfs-...@hadoop.apache.org" 
> <hdfs-...@hadoop.apache.org>; "mapreduce-dev@hadoop.apache.org" 
> <mapreduce-dev@hadoop.apache.org>; Vinod Kumar Vavilapalli 
> <vino...@apache.org> 
> Sent: Tuesday, October 27, 2015 2:39 AM
> Subject: Re: 2.7.2 release plan
> 
> Vinod and Chris,
> 
> Thanks for your reply. I'll do also backport not only bug fixes but
> also documentations(I think 2.7.2 includes them). It helps users a lot.
> 
> Best,
> - Tsuyoshi
> 
> On Tuesday, 27 October 2015, Vinod Vavilapalli <vino...@hortonworks.com>
> wrote:
> 
>> +1.
>> 
>> Thanks
>> +Vinod
>> 
>>> On Jul 16, 2015, at 8:18 AM, Chris Nauroth <cnaur...@hortonworks.com
>> <javascript:;>> wrote:
>>> 
>>> I'd be comfortable with inclusion of any doc-only patch in minor
>> releases.
>>> There is a lot of value to end users in pushing documentation fixes as
>>> quickly as possible, and they don't bear the same risk of regressions or
>>> incompatibilities as code changes.
>>> 
>>> --Chris Nauroth
>>> 
>>> 
>>> 
>>> 
>>> On 7/16/15, 12:38 AM, "Tsuyoshi Ozawa" <oz...@apache.org <javascript:;>>
>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> thank you for starting the discussion about 2.7.2 release.
>>>> 
>>>>> The focus obviously is to have blocker issues [2], bug-fixes and *no*
>>>> features / improvements.
>>>> 
>>>> I've committed YARN-3170, which is an improvement of documentation. I
>>>> thought documentation pages which can be fit into branch-2.7 can be
>>>> included easily. Should I revert it?
>>>> 
>>>>>> I need help from all committers in automatically
>>>> merging in any patch that fits the above criterion into 2.7.2 instead of
>>>> only on trunk or 2.8.
>>>> 
>>>> Sure, I'll try my best.
>>>> 
>>>>> That way we can include not only blocker but also critical bug fixes to
>>>>> 2.7.2 release.
>>>> 
>>>> As Vinod mentioned, we should also apply major bug fixes into
>> branch-2.7.
>>>> 
>>>> Thanks,
>>>> - Tsuyoshi
>>>> 
>>>> On Thu, Jul 16, 2015 at 3:52 PM, Akira AJISAKA
>>>> <ajisa...@oss.nttdata.co.jp <javascript:;>> wrote:
> 
> 
>>>>> Thanks Vinod for starting 2.7.2 release plan.
>>>>> 
>>>>>> The focus obviously is to have blocker issues [2], bug-fixes and *no*
>>>>>> features / improvements.
>>>>> 
>>>>> Can we adopt the plan as Karthik mentioned in "Additional maintenance
>>>>> releases for Hadoop 2.y versions" thread? That way we can include not
>>>>> only
>>>>> blocker but also critical bug fixes to 2.7.2 release.
>>>>> 
>>>>> In addition, branch-2.7 is a special case. (2.7.1 is the first stable
>>>>> release) Therefore I'm thinking we can include major bug fixes as well.
>

Re: 2.7.2 release plan

[Kihwal Lee]

I think we need HDFS-8950 and HDFS-7725 in 2.7.2.It should be easy to 
backport/cherry-pick HDFS-7725. For HDFS-8950, it will be nice if Ming can 
chime in.
Kihwal

  From: Tsuyoshi Ozawa 
 To: "common-...@hadoop.apache.org"  
Cc: Chris Nauroth ; "yarn-...@hadoop.apache.org" 
; "hdfs-...@hadoop.apache.org" 
; "mapreduce-dev@hadoop.apache.org" 
; Vinod Kumar Vavilapalli  
 Sent: Tuesday, October 27, 2015 2:39 AM
 Subject: Re: 2.7.2 release plan
   
Vinod and Chris,

Thanks for your reply. I'll do also backport not only bug fixes but
also documentations(I think 2.7.2 includes them). It helps users a lot.

Best,
- Tsuyoshi

On Tuesday, 27 October 2015, Vinod Vavilapalli 
wrote:

> +1.
>
> Thanks
> +Vinod
>
> > On Jul 16, 2015, at 8:18 AM, Chris Nauroth  > wrote:
> >
> > I'd be comfortable with inclusion of any doc-only patch in minor
> releases.
> > There is a lot of value to end users in pushing documentation fixes as
> > quickly as possible, and they don't bear the same risk of regressions or
> > incompatibilities as code changes.
> >
> > --Chris Nauroth
> >
> >
> >
> >
> > On 7/16/15, 12:38 AM, "Tsuyoshi Ozawa" >
> wrote:
> >
> >> Hi,
> >>
> >> thank you for starting the discussion about 2.7.2 release.
> >>
> >>> The focus obviously is to have blocker issues [2], bug-fixes and *no*
> >> features / improvements.
> >>
> >> I've committed YARN-3170, which is an improvement of documentation. I
> >> thought documentation pages which can be fit into branch-2.7 can be
> >> included easily. Should I revert it?
> >>
>  I need help from all committers in automatically
> >> merging in any patch that fits the above criterion into 2.7.2 instead of
> >> only on trunk or 2.8.
> >>
> >> Sure, I'll try my best.
> >>
> >>> That way we can include not only blocker but also critical bug fixes to
> >>> 2.7.2 release.
> >>
> >> As Vinod mentioned, we should also apply major bug fixes into
> branch-2.7.
> >>
> >> Thanks,
> >> - Tsuyoshi
> >>
> >> On Thu, Jul 16, 2015 at 3:52 PM, Akira AJISAKA
> >> > wrote:


> >>> Thanks Vinod for starting 2.7.2 release plan.
> >>>
>  The focus obviously is to have blocker issues [2], bug-fixes and *no*
>  features / improvements.
> >>>
> >>> Can we adopt the plan as Karthik mentioned in "Additional maintenance
> >>> releases for Hadoop 2.y versions" thread? That way we can include not
> >>> only
> >>> blocker but also critical bug fixes to 2.7.2 release.
> >>>
> >>> In addition, branch-2.7 is a special case. (2.7.1 is the first stable
> >>> release) Therefore I'm thinking we can include major bug fixes as well.
> >>>
> >>> Regards,
> >>> Akira
> >>>
> >>>
> >>> On 7/16/15 04:13, Vinod Kumar Vavilapalli wrote:
> 
>  Hi all,
> 
> 
>  Thanks everyone for the push on 2.7.1! Branch-2.7 is now open for
>  commits
>  to a 2.7.2 release. JIRA also now has a 2.7.2 version for all the
>  sub-projects.
> 
> 
>  Continuing the previous 2.7.1 thread on steady maintenance releases
>  [1],
>  we
>  should follow up 2.7.1 with a 2.7.2 within 4 weeks. Earlier I tried a
>  2-3
>  week cycle for 2.7.1, but it seems to be impractical given the
>  community
>  size. So, I propose we target a release by the end for 4 weeks from
>  now,
>  starting the release close-down within 2-3 weeks.
> 
>  The focus obviously is to have blocker issues [2], bug-fixes and *no*
>  features / improvements. I need help from all committers in
>  automatically
>  merging in any patch that fits the above criterion into 2.7.2 instead
>  of
>  only on trunk or 2.8.
> 
>  Thoughts?
> 
>  Thanks,
> 
>  +Vinod
> 
>  [1] A 2.7.1 release to follow up 2.7.0
>  http://markmail.org/message/zwzze6cqqgwq4rmw
> 
>  [2] 2.7.2 release blockers:
>  https://issues.apache.org/jira/issues/?filter=12332867
> 
> >>>
> >>
> >
> >
>
>


   

Re: [DISCUSS] About the details of JDK-8 support

[Kihwal Lee]

sun.security.krb5.KrbApReq was creating a static MD5 digest object and not 
synchronizing access.
This has been fixed in jdk8u60. 

http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89

One of the visible symptom is RPC reader thread getting 
ArrayIndexOutOfBoundsException from 
sun.security.provider.DigestBase.engineUpdate. More concerning case is a reader 
operating on a wrong digest.

Kihwal



From: Jean-Baptiste Note 
To: common-...@hadoop.apache.org 
Cc: "mapreduce-dev@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org" ; dev 
; "yarn-...@hadoop.apache.org" 
 
Sent: Tuesday, October 13, 2015 5:00 AM
Subject: Re: [DISCUSS] About the details of JDK-8 support


Hi,

As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
production with security enabled.
We reported some gripes with some specific java versions:
https://issues.cloudera.org/browse/DISTRO-732

I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
problems with SPNEGO SSO.

JB

Re: [VOTE] Release Apache Hadoop 2.7.0 RC0

[Kihwal Lee]

+1 (binding)Built the source from the tag and ran basic test on pseudo 
distributed cluster.
Kihwal

  From: Vinod Kumar Vavilapalli vino...@apache.org
 To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org 
Cc: vino...@apache.org 
 Sent: Friday, April 10, 2015 6:44 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.0 RC0
   
Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.0.

 The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.7.0-RC0/

The RC tag in git is: release-2.7.0-RC0

 The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1017/

As discussed before
 - This release will only work with JDK 1.7 and above
 - I’d like to use this as a starting release for 2.7.x [1], depending on
how it goes, get it stabilized and potentially use a 2.7.1 in a few
weeks as the stable release.

 Please try the release and vote; the vote will run for the usual 5 days.

 Thanks,
 Vinod

 [1]: A 2.7.1 release to follow up 2.7.0
http://markmail.org/thread/zwzze6cqqgwq4rmw

  

mapreduce precommit build does not show test results

[Kihwal Lee]

Ever since build #8430 in 8/29, test results are not showing up.  I see there 
was build config changes around that time. Anyone with the right permission 
care to take a look? 

Thanks,Kihwal

Re: [VOTE] Migration from subversion to git for version control

[Kihwal Lee]

+1 (binding)

Kihwal

On 8/8/14, 9:57 PM, Karthik Kambatla ka...@cloudera.com wrote:

I have put together this proposal based on recent discussion on this
topic.

Please vote on the proposal. The vote runs for 7 days.

   1. Migrate from subversion to git for version control.
   2. Force-push to be disabled on trunk and branch-* branches. Applying
   changes from any of trunk/branch-* to any of branch-* should be through
   git cherry-pick -x.
   3. Force-push on feature-branches is allowed. Before pulling in a
   feature, the feature-branch should be rebased on latest trunk and the
   changes applied to trunk through git rebase --onto or git
cherry-pick
   commit-range.
   4. Every time a feature branch is rebased on trunk, a tag that
   identifies the state before the rebase needs to be created (e.g.
   tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted
once
   the feature is pulled into trunk and the tags are no longer useful.
   5. The relevance/use of tags stay the same after the migration.

Thanks
Karthik

PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of
vote and will be Lazy 2/3 majority of PMC members.

Re: Jenkins build fails

[Kihwal Lee]

Now builds are failing because of this. Please make sure build works with
-Pnative.



[exec] CMake Error at
/usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108
(message):
 [exec]   Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
 [exec] Call Stack (most recent call first):
 [exec]   
/usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315
(_FPHSA_FAILURE_MESSAGE)
 [exec]   /usr/share/cmake-2.8/Modules/FindPkgConfig.cmake:106
(find_package_handle_standard_args)
 [exec]   main/native/fuse-dfs/CMakeLists.txt:23 (find_package)


On 7/9/14, 9:05 AM, Giridharan Kesavan gkesa...@hortonworks.com wrote:

I took care of the svn upgrade issue


-giri


On Wed, Jul 9, 2014 at 5:05 AM, Giridharan Kesavan
gkesa...@hortonworks.com
 wrote:


 I'm looking into this.

 -giri


 On Tue, Jul 8, 2014 at 8:15 PM, Akira AJISAKA
ajisa...@oss.nttdata.co.jp
 wrote:

 Filed https://issues.apache.org/jira/browse/HADOOP-10804
 Please correct me if I am wrong..

 Thanks,
 Akira

 (2014/07/09 11:24), Akira AJISAKA wrote:
  Hi Hadoop developers,
 
  Now Jenkins is failing with the below message.
  I'm thinking this is caused by the upgrade of Jenkins server.
  After the upgrade, the version of svn client was also upgraded,
  so the following errors occurred.
 
  It will be fixed by executing 'svn upgrade' before executing
  other svn commands. I'll file a JIRA and create a patch shortly.
 
  Regards,
  Akira
 
  
==
  
==
   Testing patch for HADOOP-10661.
  
==
  
==
 
 
  svn: E155036: Please see the 'svn upgrade' command
  svn: E155036: The working copy at
  '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
  is too old (format 10) to work with client version '1.8.8 (r1568071)'
  (expects format 31). You need to upgrade the working copy first.
 
  svn: E155036: Please see the 'svn upgrade' command
  svn: E155036: The working copy at
  '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
  is too old (format 10) to work with client version '1.8.8 (r1568071)'
  (expects format 31). You need to upgrade the working copy first.
 
  svn: E155036: Please see the 'svn upgrade' command
  svn: E155036: The working copy at
  '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
  is too old (format 10) to work with client version '1.8.8 (r1568071)'
  (expects format 31). You need to upgrade the working copy first.
 
  svn: E155036: Please see the 'svn upgrade' command
  svn: E155036: The working copy at
  '/home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk'
  is too old (format 10) to work with client version '1.8.8 (r1568071)'
  (expects format 31). You need to upgrade the working copy first.
 
  Build step 'Execute shell' marked build as failure
  Archiving artifacts
  Description set: HADOOP-10661
  Recording test results
  Finished: FAILURE
 




-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity
to 
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified
that 
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately 
and delete it from your system. Thank You.

Re: [VOTE] Change by-laws on release votes: 5 days instead of 7

[Kihwal Lee]

+1 (binding)

Kihwal

On 6/24/14, 3:53 AM, Arun C Murthy a...@hortonworks.com wrote:

Folks,

 As discussed, I'd like to call a vote on changing our by-laws to change
release votes from 7 days to 5.

 I've attached the change to by-laws I'm proposing.

 Please vote, the vote will the usual period of 7 days.

thanks,
Arun



[main]$ svn diff
Index: author/src/documentation/content/xdocs/bylaws.xml
===
--- author/src/documentation/content/xdocs/bylaws.xml   (revision 1605015)
+++ author/src/documentation/content/xdocs/bylaws.xml   (working copy)
@@ -344,7 +344,16 @@
 pVotes are open for a period of 7 days to allow all active
 voters time to consider the vote. Votes relating to code
 changes are not subject to a strict timetable but should be
-made as timely as possible./p/li
+made as timely as possible./p
+
+ ul
+ li strongProduct Release - Vote Timeframe/strong
+   pRelease votes, alone, run for a period of 5 days. All other
+ votes are subject to the above timeframe of 7 days./p
+ /li
+   /ul
+   /li
+
/ul
/section
 /body
-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity
to 
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified
that 
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately 
and delete it from your system. Thank You.

[jira] [Created] (MAPREDUCE-5939) StartTime showing up as the epoch time in JHS UI after upgrade

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-5939:
-

 Summary: StartTime showing up as the epoch time in JHS UI after 
upgrade
 Key: MAPREDUCE-5939
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5939
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee


After upgrading from 0.23.x to 2.5, the start time of old apps are showing up 
as the epoch time.  It looks like 2.5 expects start time to be encoded at the 
end of the jhist file name (-[timestamp].jhist). It should have been 
made backward compatible.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Release Apache Hadoop 2.4.1

[Kihwal Lee]

If we ever respin 2.4.1, I strongly suggest HDFS-6527 be included.


Kihwal

On 6/19/14, 4:56 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote:

I think we should include this issue in 2.4.1, so I uploaded a patch to
fix it. I'll appreciate your review.

Thanks,
Akira

(2014/06/18 12:13), Vinod Kumar Vavilapalli wrote:

 There is one item [MAPREDUCE-5830 HostUtil.getTaskLogUrl is not
backwards binary compatible with 2.3] marked for 2.4. Should we include
it?

 There is no patch there yet, it doesn't really help much other than
letting older clients compile - even if we put the API back in, the URL
returned is invalid.

 +Vinod

 On Jun 16, 2014, at 9:27 AM, Arun C Murthy a...@hortonworks.com wrote:

 Folks,

 I've created a release candidate (rc0) for hadoop-2.4.1 (bug-fix
release) that I would like to push out.

 The RC is available at:
http://people.apache.org/~acmurthy/hadoop-2.4.1-rc0
 The RC tag in svn is here:
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7
days.

 thanks,
 Arun



 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/hdp/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the
reader
 of this message is not the intended recipient, you are hereby notified
that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
immediately
 and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 0.23.11

[Kihwal Lee]

Checked out the source, built and started a single node cluster.
Ran a couple of sample jobs.

+1 (binding)

Kihwal

On 6/19/14, 10:14 AM, Thomas Graves tgra...@yahoo-inc.com.INVALID
wrote:

Hey Everyone,

There have been various bug fixes that have went into
branch-0.23 since the 0.23.10 release.  We think its time to do a 0.23.11.

This is also the last planned release off of branch-0.23 we plan on doing.

The RC is available at:
http://people.apache.org/~tgraves/hadoop-0.23.11-candidate-0/


The RC Tag in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.11-rc0/

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days
til June 26th.

I am +1 (binding).

thanks,
Tom Graves

Re: [VOTE] Release Apache Hadoop 2.4.1

[Kihwal Lee]

Committed HDFS-6527

On 6/20/14, 1:34 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote:

I just committed MAPREDUCE-5830 to all the needed branches.

+Vinod
Hortonworks Inc.
http://hortonworks.com/


On Fri, Jun 20, 2014 at 11:25 AM, Arun C Murthy a...@hortonworks.com
wrote:

 Thanks for the feedback Vinod, Akira  Kihwal.

 I'll re-spin rc1 with MAPREDUCE-5830  HDFS-6527.

 @Kihwal - Can you, please, merge HDFS-6527 to branch-2.4 and
branch-2.4.1?

 thanks,
 Arun

 On Jun 20, 2014, at 7:32 AM, Kihwal Lee kih...@yahoo-inc.com.INVALID
 wrote:

  If we ever respin 2.4.1, I strongly suggest HDFS-6527 be included.
 
 
  Kihwal
 
  On 6/19/14, 4:56 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp
wrote:
 
  I think we should include this issue in 2.4.1, so I uploaded a patch
to
  fix it. I'll appreciate your review.
 
  Thanks,
  Akira
 
  (2014/06/18 12:13), Vinod Kumar Vavilapalli wrote:
 
  There is one item [MAPREDUCE-5830 HostUtil.getTaskLogUrl is not
  backwards binary compatible with 2.3] marked for 2.4. Should we
include
  it?
 
  There is no patch there yet, it doesn't really help much other than
  letting older clients compile - even if we put the API back in, the
URL
  returned is invalid.
 
  +Vinod
 
  On Jun 16, 2014, at 9:27 AM, Arun C Murthy a...@hortonworks.com
 wrote:
 
  Folks,
 
  I've created a release candidate (rc0) for hadoop-2.4.1 (bug-fix
  release) that I would like to push out.
 
  The RC is available at:
  http://people.apache.org/~acmurthy/hadoop-2.4.1-rc0
  The RC tag in svn is here:
  
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc0
 
  The maven artifacts are available via repository.apache.org.
 
  Please try the release and vote; the vote will run for the usual 7
  days.
 
  thanks,
  Arun
 
 
 
  --
  Arun C. Murthy
  Hortonworks Inc.
  http://hortonworks.com/hdp/
 
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
  entity to
  which it is addressed and may contain information that is
 confidential,
  privileged and exempt from disclosure under applicable law. If the
  reader
  of this message is not the intended recipient, you are hereby
notified
  that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you
have
  received this communication in error, please contact the sender
  immediately
  and delete it from your system. Thank You.
 
 
 
 

 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/hdp/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the
reader
 of this message is not the intended recipient, you are hereby notified
that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
immediately
 and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity
to 
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified
that 
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately 
and delete it from your system. Thank You.

Re: Policy on adding timeouts to tests

[Kihwal Lee]

The most common mistake is making incorrect assumptions on test/component
run-time. We often forget how slower/faster things can be on different
platforms and how the load on the machine at the time of test execution
affect the run time.  If not sure, leave it to surefire. If things run
generally slow on certain environment and the 10 min timeout generates
many false positives, it is easier to adjust. But early detection of
failures is still preferred, if possible.

Some are using timeout for validating performance in unit tests.  This
should not be done in unit tests. Also, many test issues have been due to
incorrect assumptions on the ordering/timing of events and resulting state
updates. These test cases usually timeout waiting for certain state to be
reached.

Kihwal

On 4/16/14, 11:24 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote:

The other advantage of timeout is early failure - earlier than the uber
10 min timeout that seems to exist in the build files. Usually the
test-writer has a general idea of how long the test is supposed to run
and if that doesn't happen, we can fail early. Clearly, this involves
choosing a reasonable timeout so that the test can pass on local
machines, different OSes and/or in VMs.

+Vinod

On Apr 15, 2014, at 11:37 AM, Chris Nauroth cnaur...@hortonworks.com
wrote:

 +common-dev, hdfs-dev
 
 My understanding of the current situation is that we had a period where
we
 tried to enforce adding timeouts on all new tests in patches, but it
caused
 trouble, and now we're back to not requiring it.  Jenkins test-patch
isn't
 checking for it anymore.
 
 I don't think patches are getting rejected for using timeouts though.
 
 The difficulty is that execution time is quite sensitive to the build
 environment.  (Consider top-of-the-line server hardware used in build
 infrastructure vs. a dev running a VirtualBox VM with 1 dedicated CPU,
2 GB
 RAM and slow virtualized disk.)  When we were enforcing timeouts, it was
 quite common to see follow-up patches tuning up the timeout settings to
 make tests work reliably in a greater variety of environments.  At that
 point, the benefit of using the timeout becomes questionable, because
now
 the fast machine is running with the longer timeout too.
 
 Chris Nauroth
 Hortonworks
 http://hortonworks.com/
 
 
 
 On Mon, Apr 14, 2014 at 9:41 AM, Karthik Kambatla
ka...@cloudera.comwrote:
 
 Hi folks
 
 Just wanted to check what our policy for adding timeouts to tests is.
Do we
 encourage/discourage using timeouts for tests? If we discourage using
 timeouts for tests in general, are we okay with adding timeouts for a
few
 tests where we explicitly want the test to fail if it takes longer
than a
 particular amount of time?
 
 Thanks
 Karthik
 
 
 -- 
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
entity to 
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the
reader 
 of this message is not the intended recipient, you are hereby notified
that 
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
immediately 
 and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity
to 
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified
that 
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately 
and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 0.23.10

[Kihwal Lee]

I've built the tag and ran some basic tests on a single node cluster.

+1 (binding)

Kihwal



On Tuesday, December 3, 2013 12:24 AM, Thomas Graves tgra...@yahoo-inc.com 
wrote:
 
Hey Everyone,

There have been lots of improvements and bug fixes that have went into
branch-0.23 since the 0.23.9 release.  We think its time to do a 0.23.10
so I have created a release candidate (rc0) for a Hadoop-0.23.10 release.

The RC is available at:
http://people.apache.org/~tgraves/hadoop-0.23.10-rc0/


The RC Tag in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.10-rc0/


The maven artifacts are available via repository.apache.org.


Please try the release and vote; the vote will run for the usual 7 days
til December 9th.

I am +1 (binding).

thanks,
Tom Graves

Re: [VOTE] Release Apache Hadoop 2.2.0

[Kihwal Lee]

+1  Ran a set of tests against a single node cluster.

Kihwal



On Monday, October 7, 2013 2:02 AM, Arun C Murthy a...@hortonworks.com wrote:
 
Folks,

I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to 
get released - this release fixes a small number of bugs and some protocol/api 
issues which should ensure they are now stable and will not change in 
hadoop-2.x.

The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down 
the symlinks-related issues. I'll release note the fact that we have disabled 
it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the 
last couple of weeks.





--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Kihwal Lee]

It's your call, Arun.  I.e. as long you believe rc2 meets the expectations and 
objectives of 2.1.0-beta.

Kihwal



 From: Arun Murthy a...@hortonworks.com
To: common-...@hadoop.apache.org common-...@hadoop.apache.org 
Cc: Kihwal Lee kih...@yahoo-inc.com; mapreduce-dev@hadoop.apache.org 
mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org 
yarn-...@hadoop.apache.org 
Sent: Friday, August 16, 2013 3:44 PM
Subject: Re: [VOTE] Release Apache Hadoop 2.1.0-beta
 

That makes sense too.


On Aug 16, 2013, at 10:39 AM, Vinod Kumar Vavilapalli
vino...@hortonworks.com wrote:


 We need to make a call on what blockers will be. From my limited 
 understanding, this doesn't seem like a API or a compatibility issue. Can we 
 not fix it in subsequent bug-fix releases?

 I do see a lot of follow up releases to 2.1.0. Getting this release out will 
 help downstream projects start testing with all the API stuff that has 
 already gone in 2.1.0.

 Thanks,
 +Vinod

 On Aug 16, 2013, at 10:13 AM, Kihwal Lee wrote:

 We have found HADOOP-9880, which prevents Namenode HA from running with 
 security.

 Kihwal


 
 From: Arun C Murthy a...@hortonworks.com
 To: common-...@hadoop.apache.org common-...@hadoop.apache.org; 
 hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; 
 mapreduce-dev@hadoop.apache.org mapreduce-dev@hadoop.apache.org; 
 yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org
 Sent: Thursday, August 15, 2013 4:15 PM
 Subject: [VOTE] Release Apache Hadoop 2.1.0-beta


 Folks,

 I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would 
 like to get released - this fixes the bugs we saw since the last go-around 
 (rc1).

 The RC is available at: 
 http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/
 The RC tag in svn is here: 
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun

 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Kihwal Lee]

I've changed the target version of HADOOP-9880 to 2.1.1.  Please change it 
back, if you feel that it needs to be in 2.1.0-beta.


Kihwal



 From: Kihwal Lee kih...@yahoo-inc.com
To: Arun Murthy a...@hortonworks.com; common-...@hadoop.apache.org 
common-...@hadoop.apache.org 
Cc: mapreduce-dev@hadoop.apache.org mapreduce-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org 
Sent: Friday, August 16, 2013 4:55 PM
Subject: Re: [VOTE] Release Apache Hadoop 2.1.0-beta
 

It's your call, Arun.  I.e. as long you believe rc2 meets the expectations and 
objectives of 2.1.0-beta.

Kihwal



From: Arun Murthy a...@hortonworks.com
To: common-...@hadoop.apache.org common-...@hadoop.apache.org 
Cc: Kihwal Lee kih...@yahoo-inc.com; mapreduce-dev@hadoop.apache.org 
mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org 
yarn-...@hadoop.apache.org 
Sent: Friday, August 16, 2013 3:44 PM
Subject: Re: [VOTE] Release Apache Hadoop 2.1.0-beta


That makes sense too.


On Aug 16, 2013, at 10:39 AM, Vinod Kumar Vavilapalli
vino...@hortonworks.com wrote:


 We need to make a call on what blockers will be. From my limited 
 understanding, this doesn't seem like a API or a compatibility issue. Can we 
 not fix it in subsequent bug-fix releases?

 I do see a lot of follow up releases to 2.1.0. Getting this release out will 
 help downstream projects start testing with all the API stuff that has 
 already gone in 2.1.0.

 Thanks,
 +Vinod

 On Aug 16, 2013, at 10:13 AM, Kihwal Lee wrote:

 We have found HADOOP-9880, which prevents Namenode HA from running with 
 security.

 Kihwal


 
 From: Arun C Murthy a...@hortonworks.com
 To: common-...@hadoop.apache.org common-...@hadoop.apache.org; 
 hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; 
 mapreduce-dev@hadoop.apache.org mapreduce-dev@hadoop.apache.org; 
 yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org
 Sent: Thursday, August 15, 2013 4:15 PM
 Subject: [VOTE] Release Apache Hadoop 2.1.0-beta


 Folks,

 I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would 
 like to get released - this fixes the bugs we saw since the last go-around 
 (rc1).

 The RC is available at: 
 http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/
 The RC tag in svn is here: 
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun

 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Kihwal Lee]

Another blocker, HADOOP-9850, has been committed.


Kihwal



 From: Arun C Murthy a...@hortonworks.com
To: Daryn Sharp da...@yahoo-inc.com 
Cc: hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org mapreduce-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org; 
common-...@hadoop.apache.org common-...@hadoop.apache.org 
Sent: Thursday, August 1, 2013 1:30 PM
Subject: Re: [VOTE] Release Apache Hadoop 2.1.0-beta
 

Ok, thanks for heads up Daryn. I'll spin an RC2 once HADOOP-9816 gets in - I'd 
appreciate if you could help push the fix in ASAP.

Thanks again!

Arun

On Aug 1, 2013, at 9:38 AM, Daryn Sharp da...@yahoo-inc.com wrote:

 I broke RPC QOP for integrity and privacy options. :(  See blocker 
 HADOOP-9816.  I think I understand the problem and it shouldn't be hard to 
 fix.
 
 The bug went unnoticed because sadly there are no unit tests for the QOP 
 options, even though it just involves a conf setting.
 
 Daryn
 
 
 On Jul 29, 2013, at 5:00 PM, Arun C Murthy wrote:
 
 Ok, I think we are close to rc1 now - the last of blockers should be 
 committed later today… I'll try and spin RC1 tonight.
 
 thanks,
 Arun
 
 On Jul 21, 2013, at 12:43 AM, Devaraj Das d...@hortonworks.com wrote:
 
 I have just raised https://issues.apache.org/jira/browse/HDFS-5016 .. This
 bug can easily be reproduced by some HBase tests. I'd like this to be
 considered before we make a beta release. Have spoken about this with some
 hdfs folks offline and I am told that it is being worked on.
 
 Thanks
 Devaraj
 
 
 On Wed, Jul 17, 2013 at 4:25 PM, Alejandro Abdelnur tuc...@gmail.comwrote:
 
 As I've mentioned in my previous email, if we get YARN-701 in, we should
 also get in the fix for unmanaged AMs in an un-secure setup in 2.1.0-beta.
 Else is a regression of a functionality it is already working.
 
 Because of that, to avoid continuing delaying the release, I'm suggesting
 to mention in the release notes the API changes and behavior changes that
 YARN-918 and YARN-701 will bring into the next beta or GA release.
 
 thx
 
 
 On Wed, Jul 17, 2013 at 4:14 PM, Vinod Kumar Vavilapalli 
 vino...@hortonworks.com wrote:
 
 
 On Jul 17, 2013, at 1:04 PM, Alejandro Abdelnur wrote:
 
 * YARN-701
 
 It should be addressed before a GA release.
 
 Still, as it is this breaks unmanaged AMs and to me
 that would be a blocker for the beta.
 
 YARN-701 and the unmanaged AMs fix should be committed
 in tandem.
 
 * YARN-918
 
 It is a consequence of YARN-701 and depends on it.
 
 
 
 YARN-918 is an API change. And YARN-701 is a behaviour change. We need
 both in 2.1.0.
 
 
 
 * YARN-926
 
 It would be nice to have it addressed before GA release.
 
 
 Either ways. I'd get it in sooner than later specifically when we are
 trying to replace the old API with the new one.
 
 Thanks,
 +Vino
 
 
 
 
 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/
 
 
 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845

[Kihwal Lee]

Sorry to hijack the thread but, I also wanted to mention Avro. See HADOOP-9672.
The version we are using has memory leak and inefficiency issues. We've seen 
users running into it.

Kihwal



 From: Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com
To: common-...@hadoop.apache.org common-...@hadoop.apache.org 
Cc: hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org mapreduce-dev@hadoop.apache.org 
Sent: Thursday, August 8, 2013 1:59 AM
Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
 

Hi,

About Hadoop, Harsh is dealing with this problem in HADOOP-9346.
For more detail, please see the JIRA ticket:
https://issues.apache.org/jira/browse/HADOOP-9346

- Tsuyoshi

On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur t...@cloudera.com wrote:
 I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.

 As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
 justify the upgrade.

 Doing the upgrade now, with the first beta, will make things easier for
 downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
 we do the upgrade later, downstream projects will have to support 2
 different versions and they my get in nasty waters due to classpath issues.

 I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
 and it works fine (something is broken in trunk in the RPC layer YARN-885).

 Now, to do this it will require a few things:

 * Make sure protobuf 2.5.0 is available in the jenkins box
 * A follow up email to dev@ aliases indicating developers should install
 locally protobuf 2.5.0

 Thanks.

 --
 Alejandro

[jira] [Resolved] (MAPREDUCE-3894) 0.23 and trunk MR builds fail intermittently

[Kihwal Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee resolved MAPREDUCE-3894.
---

Resolution: Fixed

 0.23 and trunk MR builds fail intermittently
 

 Key: MAPREDUCE-3894
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3894
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.24.0, 0.23.2
Reporter: Kihwal Lee

 The builds occasionally report ABORTED or FAILURE, which is not caused by the 
 new code change included in the builds. We are not sure since when they have 
 been broken this way, but Bobby's guess is around Feb 10.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Release Apache Hadoop 0.23.9

[Kihwal Lee]

+1 Downloaded it and ran several sample tests in a pseudo-distributed
cluster.


Kihwal

On 7/1/13 12:20 PM, Thomas Graves tgra...@yahoo-inc.com wrote:

I've created a release candidate (RC0) for hadoop-0.23.9 that I would like
to release.

The RC is available at:
http://people.apache.org/~tgraves/hadoop-0.23.9-candidate-0/
The RC tag in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.9-rc0/

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days
til July 8th.

I am +1 (binding).

thanks,
Tom Graves

Re: [VOTE] Release Apache Hadoop 2.0.5-alpha (rc2)

[Kihwal Lee]

+1

Built from source and ran a couple of jobs in a pseudo-distributed cluster.
Kihwal

On 6/3/13 2:51 PM, Konstantin Boudnik c...@apache.org wrote:

I have rolled out release candidate (rc2) for hadoop-2.0.5-alpha.

The difference between rc1 and rc2 is the optimistic release date is
set for
06/06/2013 in the CHANGES.txt files.

The binary artifact is the same - there's no need to rebuild it. The maven
artifacts are the same.

The difference between the two RCs:

svn diff \
  
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc
1/ \
  
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc
2/

New RC builds are uploaded to the web.
The RC is available at:
http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc2/
The RC tag in svn is here:
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc2

I would like to extend the vote for another three days for it is such a
minor
change that doesn't affect anything but the recorded release date. Please
cast your vote before 06/06/2013 5pm PDT.

Thanks for your patience!
  Cos

On Fri, May 31, 2013 at 09:27PM, Konstantin Boudnik wrote:
 All,
 
 I have created a release candidate (rc1) for hadoop-2.0.5-alpha that I
would
 like to release.
 
 This is a stabilization release that includes fixed for a couple a of
issues
 discovered in the testing with BigTop 0.6.0 release candidate.
 
 The RC is available at:
http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc1/
 The RC tag in svn is here:
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc
1
 
 The maven artifacts will be available via repository.apache.org on Sat,
June
 1st, 2013 at 2 pm PDT as outlined here
 http://s.apache.org/WKD
 
 Please try the release bits and vote; the vote will run for the 3 days,
 because this is just a version name change. The bits are identical to
the ones
 voted on before in
 http://s.apache.org/2041move
 
 Thanks for your voting
   Cos
 

Re: [VOTE] Release hadoop-2.0.3-alpha

[Kihwal Lee]

+1

Verified checksums. Built from source, deployed and ran a couple of jobs.


Kihwal

On 2/6/13 9:59 PM, Arun C Murthy a...@hortonworks.com wrote:

Folks,

I've created a release candidate (rc0) for hadoop-2.0.3-alpha that I
would like to release.

This release contains several major enhancements such as QJM for HDFS HA,
multi-resource scheduling for YARN, YARN ResourceManager restart etc.
Also YARN has achieved significant stability at scale (more details from
Y! folks here: http://s.apache.org/VYO).

The RC is available at:
http://people.apache.org/~acmurthy/hadoop-2.0.3-alpha-rc0/
The RC tag in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.3-alpha-rc0/

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun



--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: division by zero in getLocalPathForWrite()

[Kihwal Lee]

Ted,

I couldn't reproduce it by just running the test case. When you reproduce
it, look at the stderr/stdout file somewhere under
target/org.apache.hadoop.mapred.MiniMRCluster. Look for the one under the
directory whose name containing the app id.

I did run into a similar problem and the stderr said:
/bin/bash: /bin/java: No such file or directory

It was because JAVA_HOME was not set. But in this case the exit code was
127 (shell not being able to locate the command to exec). In the hudson
job, the exit code was 1, so I think it's something else.

Kihwal

On 10/29/12 11:56 PM, Ted Yu yuzhih...@gmail.com wrote:

TestRowCounter still fails:
https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/244/testReport/j
unit/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterNoColu
mn/

but there was no 'divide by zero' exception.

Cheers

On Thu, Oct 25, 2012 at 8:04 AM, Ted Yu yuzhih...@gmail.com wrote:

 I will try 2.0.2-alpha release.

 Cheers


 On Thu, Oct 25, 2012 at 7:54 AM, Ted Yu yuzhih...@gmail.com wrote:

 Thanks for the quick response, Robert.
 Here is the hadoop version being used:
 hadoop-two.version2.0.1-alpha/hadoop-two.version

 If there is newer release, I am willing to try that before filing JIRA.


 On Thu, Oct 25, 2012 at 7:07 AM, Robert Evans
ev...@yahoo-inc.comwrote:

 It looks like you are running with an older version of 2.0, even
though
 it
 does not really make much of a difference in this case,  The issue
shows
 up when getLocalPathForWrite thinks there is no space on to write to
on
 any of the disks it has configured.  This could be because you do not
 have
 any directories configured.  I really don't know for sure exactly
what is
 happening.  It might be disk fail in place removing disks for you
because
 of other issues. Either way we should file a JIRA against Hadoop to
make
 it so we never get the / by zero error and provide a better way to
handle
 the possible causes.

 --Bobby Evans

 On 10/24/12 11:54 PM, Ted Yu yuzhih...@gmail.com wrote:

 Hi,
 HBase has Jenkins build against hadoop 2.0
 I was checking why TestRowCounter sometimes failed:
 
 
https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/231/testRepor
t/o

 
rg.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterExclusiv
eCol
 umn/
 
 I think the following could be the cause:
 
 2012-10-22 23:46:32,571 WARN  [AsyncDispatcher event handler]
 resourcemanager.RMAuditLogger(255): USER=jenkins
 OPERATION=Application
 Finished - Failed  TARGET=RMAppManager RESULT=FAILURE
  DESCRIPTION=App
 failed with state: FAILED  PERMISSIONS=Application
 application_1350949562159_0002 failed 1 times due to AM Container for
 appattempt_1350949562159_0002_01 exited with  exitCode: -1000 due
 to: java.lang.ArithmeticException: / by zero
at

 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPat
hFor
 Write(LocalDirAllocator.java:355)
at

 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAl
loca
 tor.java:150)
at

 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAl
loca
 tor.java:131)
at

 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAl
loca
 tor.java:115)
at

 
org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getL
ocal
 PathForWrite(LocalDirsHandlerService.java:257)
at

 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.R
esou

 
rceLocalizationService$LocalizerRunner.run(ResourceLocalizationService
.jav
 a:849)
 
 However, I don't seem to find where in getLocalPathForWrite()
division
 by
 zero could have arisen.
 
 Comment / hint is welcome.
 
 Thanks

[jira] [Created] (MAPREDUCE-4470) Fix TestCombineFileInputFormat.testForEmptyFile

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-4470:
-

 Summary: Fix TestCombineFileInputFormat.testForEmptyFile
 Key: MAPREDUCE-4470
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4470
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
 Fix For: 2.1.0-alpha, 3.0.0


TestCombineFileInputFormat.testForEmptyFile started failing after HADOOP-8599. 

It expects one split on an empty input file, but with HADOOP-8599 it gets zero. 
The new behavior seems correct, but is it breaking anything else?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4471) TestClientRMService.testGetQueueInfo failing after MR-4427

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-4471:
-

 Summary: TestClientRMService.testGetQueueInfo failing after MR-4427
 Key: MAPREDUCE-4471
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4471
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.1.0-alpha
Reporter: Kihwal Lee
 Fix For: 2.1.0-alpha, head


TestClientRMService.testGetQueueInfo has been consistently failing since 
MAPREDUCE-4427.

{noformat}
java.lang.NullPointerException
at 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:407)
at 
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueInfo(ClientRMService.java:393)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService.testGetQueueInfo(TestClientRMService.java:138)
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-4471) TestClientRMService.testGetQueueInfo failing after MR-4427

[Kihwal Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee resolved MAPREDUCE-4471.
---

Resolution: Invalid

 TestClientRMService.testGetQueueInfo failing after MR-4427
 --

 Key: MAPREDUCE-4471
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4471
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.1.0-alpha
Reporter: Kihwal Lee
 Fix For: 2.1.0-alpha, head


 TestClientRMService.testGetQueueInfo has been consistently failing since 
 MAPREDUCE-4427.
 {noformat}
 java.lang.NullPointerException
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:407)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueInfo(ClientRMService.java:393)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService.testGetQueueInfo(TestClientRMService.java:138)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4416) Some tests run twice or fail if Clover is enabled

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-4416:
-

 Summary: Some tests run twice or fail if Clover is enabled
 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
 Fix For: 2.0.1-alpha, 3.0.0


Some tests run twice. E.g. try mvn test -Dtest=TestJobConf. It runs under 
hadoop-mapreduce-client-core and hadoop-mapreduce-client-jobclient.

There are number of tests running under hadoop-mapreduce-client-jobclient that 
fail if Clover is enabled. Whenever a job is launched, AM doesn't start because 
it can't locate the clover jar file.

It seems this started happening after MAPREDUCE-4253.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4387) RM gets fatal error and exits during TestRM

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-4387:
-

 Summary: RM gets fatal error and exits during TestRM
 Key: MAPREDUCE-4387
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4387
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
 Fix For: 2.0.1-alpha, 3.0.0


It doesn't happen on my desktop, but it happens frequently during the builds 
with clover enabled. Surefire will report it as fork failure.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4384) Race conditions in IndexCache

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-4384:
-

 Summary: Race conditions in IndexCache
 Key: MAPREDUCE-4384
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4384
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
 Fix For: 0.23.3, 2.0.1-alpha, 3.0.0


TestIndexCache is intermittently failing due to a race condition. Up on 
inspection of IndexCache implementation, more potential issues have been 
discovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4308) Remove excessive split log messages

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-4308:
-

 Summary: Remove excessive split log messages
 Key: MAPREDUCE-4308
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4308
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobtracker
Affects Versions: 1.0.3
Reporter: Kihwal Lee
 Fix For: 1.1.0


Job tracker currently prints out information on every split.

{noformat}
2012-05-20 00:06:01,985 INFO org.apache.hadoop.mapred.JobInProgress: 
tip:task_201205100740_1745_m_00 has split on node:/192.168.0.1
/my.totally.madeup.host.com
{noformat}

I looked at one cluster and these messages were taking up more than 30% of the 
JT log. If jobs have large number of maps, it can be worse. I think it is 
reasonable to lower the log level of the statement from INFO to DEBUG.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4207) Remove System.out.println() in FileInputFormat

[Kihwal Lee (JIRA)]

Kihwal Lee created MAPREDUCE-4207:
-

 Summary: Remove System.out.println() in FileInputFormat
 Key: MAPREDUCE-4207
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4207
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv1
Affects Versions: 1.0.2
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Fix For: 1.1.0, 1.0.3


MAPREDUCE-3607 accidentally left the println statement. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3894) MR commit build for 0.23 and trunk fails intermittently

[Kihwal Lee (Created) (JIRA)]

MR commit build for 0.23 and trunk fails intermittently
---

 Key: MAPREDUCE-3894
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3894
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.24.0, 0.23.2
Reporter: Kihwal Lee


The commit builds occasionally report ABORTED or FAILURE, which is not caused 
by the new code change included in the builds. We are not sure since when it 
has been broken this way, but Bobby's guess is around Feb 10.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3851) Allow more aggressive action on detection of the jetty issue

[Kihwal Lee (Created) (JIRA)]

Allow more aggressive action on detection of the jetty issue


 Key: MAPREDUCE-3851
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3851
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker
Affects Versions: 1.0.0
Reporter: Kihwal Lee
 Fix For: 1.1.0, 1.0.1


MAPREDUCE-2529 added the useful failure detection mechanism. In this jira, I 
propose we add a periodic check inside TT and configurable action to 
self-destruct. Blacklisting helps but is not enough. Hung jetty still accepts 
connection and it takes very long time for clients to fail out. Short jobs are 
delayed for hours because of this. This feature will be a nice companion to 
MAPREDUCE-3184.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: some weird problem with hadoop, Task process exit with nonzero status of 134

[Kihwal Lee]

Hi James,

It means that the jvm's crashed.  134-128=6, which is SIGABRT. This is not the 
cause, but what the jvm is sending to itself to exit after handling the crash 
condition (e.g. seg fault). To find the real cause, you need to examine the 
crash dump. You can specify the dump location with -XX:ErrorFile=.

Kihwal


On 2/8/12 3:07 AM, James drja...@163.com wrote:

Hello everybody,
I've met a weird problem when running a hadoop job, the error msg is as follows:
发 送
2012-02-08 16:55:20,686 WARN org.apache.hadoop.mapred.TaskRunner: 
attempt_201202081654_0001_m_03_0 : Child Error
java.io.IOException: Task process exit with nonzero status of 134.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

I've been troubled by this for quite a long time, wish someone can help me, 
thanks!


Best Regards,
James

[jira] [Created] (MAPREDUCE-3741) Conflicting dependency in hadoop-mapreduce-examples

[Kihwal Lee (Created) (JIRA)]

Conflicting dependency in hadoop-mapreduce-examples
---

 Key: MAPREDUCE-3741
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3741
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 0.23.1, 0.24.0
Reporter: Kihwal Lee
 Fix For: 0.23.1, 0.24.0


{code:xml}
 dependency
   groupIdorg.apache.hadoop/groupId
   artifactIdhadoop-mapreduce-client-hs/artifactId
   scopeprovided/scope
 /dependency
 dependency
   groupIdorg.apache.hadoop/groupId
   artifactIdhadoop-mapreduce-client-hs/artifactId
   scopetest/scope
 /dependency
{code}

Are we missing type here?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3259) ContainerLocalizer should get the proper java.library.path from LinuxContainerExecutor

[Kihwal Lee (Created) (JIRA)]

ContainerLocalizer should get the proper java.library.path from 
LinuxContainerExecutor
--

 Key: MAPREDUCE-3259
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3259
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 0.23.0, 0.24.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical


As seen in MAPREDUCE-2915, java.library.path is not being passed when the LCE 
spawns a JVM for ContainerLocalizer. 

However, unlike branch-0.20-security, the task runtime in 0.23 is unaffected by 
this. This is because tasks' run-time environment is specified in the launch 
script by client. Setting LD_LIBRARY_PATH is the primary way of specifying the 
locations of required native library in this case. The config property, 
mapreduce.admin.user.env is always set in the job environment and the default 
value is to add the path to the hadoop native library to LD_LABRARY_PATH.

For JVM's being launched by the hadoop system scripts, java.library.path is set.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-2915) LinuxTaskController does not work when JniBasedUnixGroupsNetgroupMapping or JniBasedUnixGroupsMapping is enabled

[Kihwal Lee (JIRA)]

LinuxTaskController does not work when JniBasedUnixGroupsNetgroupMapping or 
JniBasedUnixGroupsMapping is enabled


 Key: MAPREDUCE-2915
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2915
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task-controller
Affects Versions: 0.20.205.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Fix For: 0.20.205.0


When a job is submitted, LinuxTaskController launches the native 
task-controller binary for job initialization. The native program does a series 
of prep work and call execv() to run JobLocalizer.  It was observed that 
JobLocalizer does fails to run when JniBasedUnixGroupsNetgroupMapping or 
JniBasedUnixGroupsMapping is enabled, resulting in 100% job failures.

JobLocalizer normally does not need the native library (libhadoop) for its 
functioning, but enabling a JNI user-to-group mapping function cause it to load 
the library. However, JobLocalizer cannot locate the library since 
java.library.path is not set.

The proposed solution is to pass the java.library.path property through 
task-controller. LinuxTaskController already does it when launching the task 
log truncater.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira