Re: Porting to Solaris

[Jun Ping Du]

Also, for development process, please follow 
http://wiki.apache.org/hadoop/HowToContribute to file JIRAs and attach doc, 
patches, etc.

Thanks,

Junping

- Original Message -
From: "Colin McCabe" 
To: "Hadoop Common" , malc...@kavalsky.net
Sent: Wednesday, February 19, 2014 8:35:07 AM
Subject: Re: Porting to Solaris

There are a few existing portability JIRAs for libhadoop.  Check out
HADOOP-7147, HADOOP-9934, HADOOP-6767, HADOOP-7824, and HDFS-5642.

best,
Colin

On Tue, Feb 18, 2014 at 2:14 AM, Malcolm  wrote:
> I have started porting the native libraries of Hadoop 2.2.0 to Solaris and
> would like to eventually get my changes into the main Hadoop source tree.
>
> Are there any public coding guidelines/requirements that I should follow ?
>
> Is there a specific procedure to follow once I have completed the work i.e.
> who to send the changes to, for example.
>
> Thanks,
> Malcolm

Re: Next releases

[Jun Ping Du]

Great news! Thanks Arpit!
I think we should update Roadmap wiki to include this work. :)

Thanks,

Junping

- Original Message -
From: "Arpit Agarwal" 
To: common-dev@hadoop.apache.org
Cc: yarn-...@hadoop.apache.org, hdfs-...@hadoop.apache.org, 
mapreduce-...@hadoop.apache.org
Sent: Tuesday, January 7, 2014 8:47:32 AM
Subject: Re: Next releases

This merge to branch-2 is complete. The changes have been merged to
branch-2 and target version set to 2.4.0 (r1556076).


On Fri, Jan 3, 2014 at 4:13 PM, Arpit Agarwal wrote:

> We plan to merge HDFS-2832 to branch-2 next week for inclusion in 2.4.
>
>
>
> On Fri, Dec 6, 2013 at 1:53 PM, Arun C Murthy  wrote:
>
>> Thanks Suresh & Colin.
>>
>> Please update the Roadmap wiki with your proposals.
>>
>> As always, we will try our best to get these in - but we can collectively
>> decide to slip some of these to subsequent releases based on timelines.
>>
>> Arun
>>
>> On Dec 6, 2013, at 10:43 AM, Suresh Srinivas 
>> wrote:
>>
>> > Arun,
>> >
>> > I propose the following changes for 2.3:
>> > - There have been a lot of improvements related to supporting http
>> policy.
>> > - There is a still discussion going on, but I would like to deprecate
>> > BackupNode in 2.3 as well.
>> > - We are currently working on rolling upgrades related change in HDFS.
>> We
>> > might add a couple of changes that enables rolling upgrades from 2.3
>> > onwards (hopefully we can this done by December)
>> >
>> > I propose the following for 2.4 release, if they are tested and stable:
>> > - Heterogeneous storage support - HDFS-2832
>> > - Datanode cache related change - HDFS-4949
>> > - HDFS ACLs - HDFS-4685
>> > - Rolling upgrade changes
>> >
>> > Let me know if you want me to update the wiki.
>> >
>> > Regards,
>> > Suresh
>> >
>>
>> On Dec 6, 2013, at 12:27 PM, Colin McCabe  wrote:
>>
>> > If 2.4 is released in January, I think it's very unlikely to include
>> > symlinks.  There is still a lot of work to be done before they're
>> > usable.  You can look at the progress on HADOOP-10019.  For some of
>> > the subtasks, it will require some community discussion before any
>> > code can be written.
>> >
>> > For better or worse, symlinks have not been requested by users as
>> > often as features like NFS export, HDFS caching, ACLs, etc, so effort
>> > has been focused on those instead.
>> >
>> > For now, I think we should put the symlinks-disabling patches
>> > (HADOOP-10020, etc) into branch-2, so that they will be part of the
>> > next releases without additional effort.
>> >
>> > I would like to see HDFS caching make it into 2.4.  The APIs and
>> > implementation are beginning to stabilize, and around January it
>> > should be ok to backport to a stable branch.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Thu, Nov 7, 2013 at 6:42 PM, Arun C Murthy 
>> wrote:
>> >
>> >> Gang,
>> >>
>> >> Thinking through the next couple of releases here, appreciate f/b.
>> >>
>> >> # hadoop-2.2.1
>> >>
>> >> I was looking through commit logs and there is a *lot* of content here
>> >> (81 commits as on 11/7). Some are features/improvements and some are
>> fixes
>> >> - it's really hard to distinguish what is important and what isn't.
>> >>
>> >> I propose we start with a blank slate (i.e. blow away branch-2.2 and
>> >> start fresh from a copy of branch-2.2.0)  and then be very careful and
>> >> meticulous about including only *blocker* fixes in branch-2.2. So,
>> most of
>> >> the content here comes via the next minor release (i.e. hadoop-2.3)
>> >>
>> >> In future, we continue to be *very* parsimonious about what gets into a
>> >> patch release (major.minor.patch) - in general, these should be only
>> >> *blocker* fixes or key operational issues.
>> >>
>> >> # hadoop-2.3
>> >>
>> >> I'd like to propose the following features for YARN/MR to make it into
>> >> hadoop-2.3 and punt the rest to hadoop-2.4 and beyond:
>> >> * Application History Server - This is happening in  a branch and is
>> >> close; with it we can provide a reasonable experience for new
>> frameworks
>> >> being built on top of YARN.
>> >> * Bug-fixes in RM Restart
>> >> * Minimal support for long-running applications (e.g. security) via
>> >> YARN-896
>> >> * RM Fail-over via ZKFC
>> >> * Anything else?
>> >>
>> >> HDFS???
>> >>
>> >> Overall, I feel like we have a decent chance of rolling hadoop-2.3 by
>> the
>> >> end of the year.
>> >>
>> >> Thoughts?
>> >>
>> >> thanks,
>> >> Arun
>> >>
>> >>
>> >> --
>> >> Arun C. Murthy
>> >> Hortonworks Inc.
>> >> https://urldefense.proofpoint.com/v1/url?u=http://hortonworks.com/&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=mPOfB21VcWnmM3DU81gzph9Quwrh%2BSSBvfQ2CNr5G0g%3D%0A&s=35573f6f0811fd2b7cc2274469110563eca6b20dce4ed5961ae05ad601d552a0
>> >>
>> >>
>> >>
>> >> --
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> entity to
>> >> which it is addressed and may contain information that is confidential,
>> >> privile

Re: Video: how to commit a patch to hadoop

[Jun Ping Du]

Thank you Steve! I think we can put the video link on Hadoop wiki and update it 
if anything change in future. Thoughts?

Thanks,

Junping

- Original Message -
From: "Steve Loughran" 
To: common-dev@hadoop.apache.org
Sent: Friday, January 3, 2014 8:07:25 PM
Subject: Video: how to commit a patch to hadoop

For new committers and for the curious, I've just stuck up a screen capture
with commentary on how to commit a patch to the Hadoop SVN repository

https://urldefense.proofpoint.com/v1/url?u=http://youtu.be/txW3m7qWdzw&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=dQmz4YgDBYrEohn2%2Bh4yof8U%2F0DcsSvkf7PO9Y0z4U4%3D%0A&s=b4775e4262718d49f385d7ad4b8205b5c646c89b25d991e3fb8fee8a57ca6b5b

It's 27 minutes long, not just because of the commentary but because you
have to be as rigorous committing a one-line patch as you do to a whole new
module.

I'd love to see how the git+gerrit projects work (like accumulo) to see if
we could speed up both the review and the checkin.

-steve

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You

Re: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk

[Jun Ping Du]

+1. Good to see HDFS can support different storage tiers. 
I have been involved with minor development & bug fixing effort and I agree it 
is ready to merge too. 

Thanks,

Junping

- Original Message -
From: "Suresh Srinivas" 
To: hdfs-...@hadoop.apache.org
Cc: common-dev@hadoop.apache.org
Sent: Tuesday, December 3, 2013 8:15:26 AM
Subject: Re: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk

Great work Arpit and Nicholas!

+1. I have been part of design. I have been following the changes closely.
This is ready to be merged into trunk.




On Mon, Dec 2, 2013 at 4:06 PM, Arpit Agarwal wrote:

> Hello all,
>
> I would like to call a vote to merge phase 1 of the Heterogeneous Storage
> feature into trunk.
>
> *Scope of the changes:*
> The changes allow exposing the DataNode as a collection of storages and set
> the foundation for subsequent work to present Heterogeneous Storages to
> applications. This allows DataNodes to send block and storage reports
> per-storage. In addition this change introduces the ability to add a
> 'storage type' tag to the storage directories. This enables supporting
> different types of storages in addition to disk storage.
>
> Development of the feature is tracked in the jira
> https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/browse/HDFS-2832&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=Z%2F4r%2B%2FNk9yYikPHTUsPHx9kGN2a1jV0DGMDT3uJYLqw%3D%0A&s=1d962b101464bff32d5f6339d6616e1dc05d7ece8890923c322549257b696ecf.
>
> *Details of development and testing:*
> Development has been done in a separate branch -
> https://urldefense.proofpoint.com/v1/url?u=https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=Z%2F4r%2B%2FNk9yYikPHTUsPHx9kGN2a1jV0DGMDT3uJYLqw%3D%0A&s=09ee6285bc4abc6bcac8ba4f05f9ef65d66cf1e94d8e705b356bd429659eb689.
>  The
> updated design is posted at -
>
> https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/secure/attachment/12615761/20131125-HeterogeneousStorage.pdf&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=Z%2F4r%2B%2FNk9yYikPHTUsPHx9kGN2a1jV0DGMDT3uJYLqw%3D%0A&s=9ca29507f2c8e258ab7d4ddcdae82926bf66e4052c31368b2abead87c7df110f
> .
> The changes involve ~6K changed lines of code, with a third of those
> changes being to tests.
>
> Please see the test plan
>
> https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/secure/attachment/12616642/20131202-HeterogeneousStorage-TestPlan.pdffor&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=Z%2F4r%2B%2FNk9yYikPHTUsPHx9kGN2a1jV0DGMDT3uJYLqw%3D%0A&s=f1d595e4b4cb6ee14cd92093cc1d7f173671f606606528931cd34cab4a08fabd
> the details. Once the feature is
> merged into trunk, we will continue to test and fix any bugs that may be
> found on trunk as well as add further tests as outlined in the test plan.
>
> The bulk of the design and implementation was done by Suresh Srinivas,
> Sanjay Radia, Nicholas Sze, Junping Du and me. Also, thanks to Eric
> Sirianni, Chris Nauroth, Steve Loughran, Bikas Saha, Andrew Wang and Todd
> Lipcon for providing feedback on the Jiras and in discussions.
>
> This vote runs for a week and closes on 12/9/2013 at 11:59 pm PT.
>
> Thanks,
> Arpit
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
https://urldefense.proofpoint.com/v1/url?u=http://hortonworks.com/download/&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=Z%2F4r%2B%2FNk9yYikPHTUsPHx9kGN2a1jV0DGMDT3uJYLqw%3D%0A&s=167e3a51d20384a08fa0a4a1636e40ce4ecf7a996bfcaccd97e60a3883b42b61

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You

Re: Next releases

[Jun Ping Du]

Hi Arun,
   Thanks for working out this list which looks great to me. In addition, I 
would like to add an item: YARN-291 to 2.3 release which enhance YARN's 
resource elasticity in cloud scenario and can benefit other scenarios i.e. 
graceful NM decommission (YARN-914), non job/app regression (or maintenance 
model) in NM rolling upgrade (YARN-671), etc. With great help from Luke, Bikas 
and Vinod, we already get the first and the most important work (YARN-311) in. 
Now, I am working on the left parts include: interfaces (RPC, CLI, REST, etc.) 
and a few enhancements (persistent, supporting different policies, etc.) and be 
optimistic on completing most of work by the end of 2013. Would you help to 
embrace it in if we can make it on time? :)

Thanks,

Junping

- Original Message -
From: "Arun C Murthy" 
To: common-dev@hadoop.apache.org, hdfs-...@hadoop.apache.org, 
yarn-...@hadoop.apache.org, mapreduce-...@hadoop.apache.org
Sent: Friday, November 8, 2013 10:42:36 AM
Subject: Next releases

Gang,

 Thinking through the next couple of releases here, appreciate f/b.

 # hadoop-2.2.1

 I was looking through commit logs and there is a *lot* of content here (81 
commits as on 11/7). Some are features/improvements and some are fixes - it's 
really hard to distinguish what is important and what isn't.

 I propose we start with a blank slate (i.e. blow away branch-2.2 and start 
fresh from a copy of branch-2.2.0)  and then be very careful and meticulous 
about including only *blocker* fixes in branch-2.2. So, most of the content 
here comes via the next minor release (i.e. hadoop-2.3)

 In future, we continue to be *very* parsimonious about what gets into a patch 
release (major.minor.patch) - in general, these should be only *blocker* fixes 
or key operational issues.

 # hadoop-2.3
 
 I'd like to propose the following features for YARN/MR to make it into 
hadoop-2.3 and punt the rest to hadoop-2.4 and beyond:
 * Application History Server - This is happening in  a branch and is close; 
with it we can provide a reasonable experience for new frameworks being built 
on top of YARN.
 * Bug-fixes in RM Restart
 * Minimal support for long-running applications (e.g. security) via YARN-896
 * RM Fail-over via ZKFC
 * Anything else?

 HDFS???

 Overall, I feel like we have a decent chance of rolling hadoop-2.3 by the end 
of the year.

 Thoughts?

thanks,
Arun
 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You

Re: [VOTE] Release Apache Hadoop 2.2.0

[Jun Ping Du]

+1 (non-binding). 
Test build and deploy it on a tiny cluster and run a few jobs.

Thanks,

Junping 

- Original Message -
From: "Arun C Murthy" 
To: common-dev@hadoop.apache.org, hdfs-...@hadoop.apache.org, 
yarn-...@hadoop.apache.org, mapreduce-...@hadoop.apache.org
Sent: Monday, October 7, 2013 3:00:52 PM
Subject: [VOTE] Release Apache Hadoop 2.2.0

Folks,

I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to 
get released - this release fixes a small number of bugs and some protocol/api 
issues which should ensure they are now stable and will not change in 
hadoop-2.x.

The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down 
the symlinks-related issues. I'll release note the fact that we have disabled 
it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the 
last couple of weeks.





--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You

Re: [ACTION NEEDED]: protoc 2.5.0 in trunk/branch-2/branch-2.1-beta/branch-2.1.0-beta

[Jun Ping Du]

Hi Tsuyoshi,
   I just checked Hadoop wiki on HowToContribute and it points ProtocolBuffer 
things to YARN Readme which is already updated to 2.5.0 now.

Thanks,

Junping

- Original Message -
From: "Tsuyoshi OZAWA" 
To: hdfs-...@hadoop.apache.org
Cc: common-dev@hadoop.apache.org, yarn-...@hadoop.apache.org, 
mapreduce-...@hadoop.apache.org
Sent: Friday, August 16, 2013 1:55:23 PM
Subject: Re: [ACTION NEEDED]: protoc 2.5.0 in 
trunk/branch-2/branch-2.1-beta/branch-2.1.0-beta

Thanks for sharing! We also need to update Wiki or some documents, don't we?
http://wiki.apache.org/hadoop/HowToContribute

On Thu, Aug 15, 2013 at 8:03 AM, Alejandro Abdelnur  wrote:
> Following up on this.
>
> HADOOP-9845 & HADOOP-9872 have been committed
> to trunk/branch-2/branch-2.1-beta/branch-2.1.0-beta.
>
> All Hadoop developers must install protoc 2.5.0 in their development
> machines for the build to run.
>
> All Hadoop jenkins boxes are using protoc 2.5.0
>
> The BUILDING.txt file has been updated to reflect that protoc 2.5.0 is the
> required one and includes instructions on how to use a different protoc
> from multiple local versions (using an ENV var). This may be handy for
> folks working with Hadoop versions using protoc 2.4.1.
>
> INTERIM SOLUTION IF YOU CANNOT UPGRADE TO PROTOC 2.5.0 IMMEDIATELY
>
> Use the following option with all your Maven commands
>  '-Dprotobuf.version=2.4.1'.
>
> Note that this option will make the build use protoc and protobuf 2.4.1.
>
> Though you should upgrade to 2.5.0 at the earliest.
>
> As soon as we start using the new goodies from protobuf 2.5.0 (like the
> non-copy bytearrays) 2.4.1 will not work anymore.
>
> Thanks and apologies again for the noise through out this change.
>
> --
> Alejandro



-- 
- Tsuyoshi

Re: [VOTE] Hadoop 1.2.1 release candidate 0 -- stable release

[Jun Ping Du]

I run some simple jobs (teragen, terasort, etc.) on a 7 nodes cluster deployed 
with 1.2.1-rc0 and tried configure with new topology layer - node group. 
Everything seems to be fine so far.
So +1 (non-binding).

Thanks,

Junping

- Original Message -
From: "Matt Foley" 
To: common-dev@hadoop.apache.org
Sent: Thursday, July 25, 2013 5:08:20 AM
Subject: [VOTE] Hadoop 1.2.1 release candidate 0 -- stable release

Colleagues,
This is a stabilization release of the Hadoop-1.2 codeline.  It has 18
patches over the 1.2.0 release, which may be seen in the Release
Notes.
 Please vote on two items:
a) accepting this as an Apache Hadoop release; and
b) accepting this as a Stable Release.

If you vote against accepting it as stable, please state the Jiras you
believe need to be fixed in the 1.2 line that would allow you to accept it
as stable.

If 1.2.1 is accepted as stable, I will move on and produce a 1.3.0 release.
 My build environment seems to be fixed up (thanks Giri!) so it shouldn't
take so long to put 1.3.0-rc0 out.

Release information and artifacts for 1.2.1-rc0:
Source code:
http://svn.apache.org/viewvc/hadoop/common/tags/release-1.2.1-rc0/
Release notes:
http://people.apache.org/~mattf/hadoop-1.2.1-rc0/releasenotes-1.2.1.html
Tarballs, RPMs, and DEBs: http://people.apache.org/~mattf/hadoop-1.2.1-rc0/
Nexus:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-core/1.2.1/

Voting will close in seven days, at 2:15pm PDT on 7/31/2013.
Thank you,
--Matt

Re: [ANNOUNCE] New Hadoop PMC members

[Jun Ping Du]

Congratulations!


- Original Message -
From: "Tom White" 
To: "common-dev" 
Sent: Monday, June 10, 2013 11:38:45 PM
Subject: [ANNOUNCE] New Hadoop PMC members

On behalf of the Apache Hadoop PMC, I'm pleased to announce the
addition of the following new Hadoop PMC members:

* Daryn Sharp
* Hitesh Shah
* Jonathan Eagles
* Kihwal Lee
* Luke Lu
* Steve Loughran
* Uma Maheswara Rao G

Thank you for all of your work on the project! Please join me in welcoming them.

Cheers,
Tom

Re: Cannot create a new Jira issue for MapReduce

[Jun Ping Du]

Thanks Ted. Those are very good suggestions as backup solutions when JIRA is 
down.
Besides alleviating the impact of JIRA downtime as you mentioned above, do we 
think of some way to keep JIRA system highly available? It is a little 
embarrassing that we deliver all kinds of HA systems to rest of world, but we 
suffering from this. :(

- Original Message -
From: "Ted Yu" 
To: mapreduce-...@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.org, common-dev@hadoop.apache.org
Sent: Sunday, August 12, 2012 12:17:36 PM
Subject: Re: Cannot create a new Jira issue for MapReduce

I made some suggestions to hbase dev mailing list a few weeks ago. The
following suggestion is about hbase development which can be extrapolated
to other Apache projects.


People can continue discussion through dev mailing list when JIRA is down.
When JIRA comes back up, transcript of such discussion can be posted back
on related issues.
Use of https://reviews.apache.org is encouraged. The review board wasn't
affected by JIRA downtime.
Running test suite by contributors and committers is encouraged which
alleviates the burden on Hadoop QA.

Goal for the above suggestions is for alleviating the impact of JIRA down
time.

BTW I have kept notifications from iss...@hbase.apache.org in my Inbox.
This shows benefit when JIRA is down.

Cheers

On Sat, Aug 11, 2012 at 7:14 PM, Jun Ping Du  wrote:

> Yes. I saw JIRA is in maintenance now and the schedule is as below:
>
> Host Name   Service Entry Time  Author  Comment Start Time
>  End TimeTypeDurationDowntime ID Trigger ID
>  Actions
> ull.zones.apache.orgIssues - JIRA - General 2012-08-11 19:06:08
> danielshMigrating to a different physical host  2012-08-11 19:06:08
> 2012-08-13 19:06:08 Fixed   2d 0h 0m 0s 1663N/A
> Delete/Cancel This Scheduled Downtime Entry
>
> Looks like it will take 2 days to migrate to a different host. As JIRA is
> a key component to dev process in community, do we think of some ways to
> lower the maintenance overhead?
>
>
> Thanks,
>
> Junping
>
> - Original Message -
> From: "Steve Loughran" 
> To: mapreduce-...@hadoop.apache.org
> Sent: Friday, August 10, 2012 7:33:04 AM
> Subject: Re: Cannot create a new Jira issue for MapReduce
>
> There's been disk problems w/ Jira recently. Githubs been playing up
> this morning to. Time to put away the dev tools and get powerpoint out
> instead
>
> On 9 August 2012 13:38, Robert Evans  wrote:
> > It is a bit worse then that though.  I found that it did create the JIRA,
> > but it is in a bad state where you cannot put it in patch available or
> > close it. So we may need to do some cleanup of these JIRAs later.
> >
> > --Bobby
> >
> > On 8/9/12 3:19 PM, "Ted Yu"  wrote:
> >
> >>This has been reported by HBase developers as well.
> >>
> >>See https://issues.apache.org/jira/browse/INFRA-5131
> >>
> >>On Thu, Aug 9, 2012 at 1:10 PM, Benoy Antony  wrote:
> >>
> >>> Hi,
> >>>
> >>> I am getting the following error when I try to create a Jira issue.
> >>>
> >>> Error creating issue: com.atlassian.jira.util.RuntimeIOException:
> >>> java.io.IOException: read past EOF
> >>>
> >>> Anyone else face the same problem ?
> >>>
> >>> Thanks ,
> >>> Benoy
> >>>
> >
>

Re: Cannot create a new Jira issue for MapReduce

[Jun Ping Du]

Yes. I saw JIRA is in maintenance now and the schedule is as below:

Host Name   Service Entry Time  Author  Comment Start Time  End 
TimeTypeDurationDowntime ID Trigger ID  Actions
ull.zones.apache.orgIssues - JIRA - General 2012-08-11 19:06:08 
danielshMigrating to a different physical host  2012-08-11 19:06:08 
2012-08-13 19:06:08 Fixed   2d 0h 0m 0s 1663N/A Delete/Cancel 
This Scheduled Downtime Entry

Looks like it will take 2 days to migrate to a different host. As JIRA is a key 
component to dev process in community, do we think of some ways to lower the 
maintenance overhead? 


Thanks,

Junping

- Original Message -
From: "Steve Loughran" 
To: mapreduce-...@hadoop.apache.org
Sent: Friday, August 10, 2012 7:33:04 AM
Subject: Re: Cannot create a new Jira issue for MapReduce

There's been disk problems w/ Jira recently. Githubs been playing up
this morning to. Time to put away the dev tools and get powerpoint out
instead

On 9 August 2012 13:38, Robert Evans  wrote:
> It is a bit worse then that though.  I found that it did create the JIRA,
> but it is in a bad state where you cannot put it in patch available or
> close it. So we may need to do some cleanup of these JIRAs later.
>
> --Bobby
>
> On 8/9/12 3:19 PM, "Ted Yu"  wrote:
>
>>This has been reported by HBase developers as well.
>>
>>See https://issues.apache.org/jira/browse/INFRA-5131
>>
>>On Thu, Aug 9, 2012 at 1:10 PM, Benoy Antony  wrote:
>>
>>> Hi,
>>>
>>> I am getting the following error when I try to create a Jira issue.
>>>
>>> Error creating issue: com.atlassian.jira.util.RuntimeIOException:
>>> java.io.IOException: read past EOF
>>>
>>> Anyone else face the same problem ?
>>>
>>> Thanks ,
>>> Benoy
>>>
>

Can someone review MAPREDUCE-4309 and MAPREDUCE-4310?

[Jun Ping Du]

These two patches are for Hadoop Network Topology extension (YARN part) for 
virtualization environment.

Thanks,

Junping

- Original Message -
From: "Jun Ping Du" 
To: common-dev@hadoop.apache.org, hdfs-...@hadoop.apache.org, 
mapreduce-...@hadoop.apache.org
Cc: "Mark Pollack" , "Jurgen Leschner" 
, "Richard McDougall" 
Sent: Monday, June 4, 2012 11:48:35 PM
Subject: Make Hadoop NetworkTopology and data locality more pluggable for other 
deploying topology like: virtualization.

Hello Folks,
  I just filed a Umbrella jira today to address current NetworkTopology 
issue that binding strictly to three tier network. The motivation here is to 
make hadoop more flexible for deploying topology (especially for 
cloud/virtualization case) and more configurable in data locality related 
policies like: replica placement, task scheduling, choosing block for DFSClient 
reading, balancing. 
  We submit a draft proposal in this Umbrella as well as the implementation 
code. As code base is large (~260K), the code is separated into 7 sub JIRA 
issues which seems to be more convenient for reviewing. However, we split the 
code based on functionality which cause some dependencies between patches which 
way we are not sure the best. Welcome to provide comments and suggestions on 
doc and code, and look forward to work with all of you to enhance hadoop in 
some new situations towards perfect.
  Hope this is a good start.

Cheers,

Junping

- Original Message -
From: "Junping Du (JIRA)" 
To: common-iss...@hadoop.apache.org
Sent: Monday, June 4, 2012 12:09:22 PM
Subject: [jira] [Created] (HADOOP-8468) Umbrella of enhancements to support 
different failure and locality topologies

Junping Du created HADOOP-8468:
--

 Summary: Umbrella of enhancements to support different failure and 
locality topologies
 Key: HADOOP-8468
 URL: https://issues.apache.org/jira/browse/HADOOP-8468
 Project: Hadoop Common
  Issue Type: Bug
  Components: ha, io
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du
Priority: Critical


The current hadoop network topology (described in some previous issues like: 
Hadoop-692) works well in classic three-tiers network when it comes out. 
However, it does not take into account other failure models or changes in the 
infrastructure that can affect network bandwidth efficiency like: 
virtualization. 
Virtualized platform has following genes that shouldn't been ignored by hadoop 
topology in scheduling tasks, placing replica, do balancing or fetching block 
for reading: 
1. VMs on the same physical host are affected by the same hardware failure. In 
order to match the reliability of a physical deployment, replication of data 
across two virtual machines on the same host should be avoided.
2. The network between VMs on the same physical host has higher throughput and 
lower latency and does not consume any physical switch bandwidth.
Thus, we propose to make hadoop network topology extend-able and introduce a 
new level in the hierarchical topology, a node group level, which maps well 
onto an infrastructure that is based on a virtualized environment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: PreCommit-Admin not running

[Jun Ping Du]

Move to dev alias, it seems to stop working since weekend. 

Thanks,

Junping

- Original Message -
From: "Kihwal Lee" 
To: gene...@hadoop.apache.org
Sent: Tuesday, July 3, 2012 3:59:28 AM
Subject: PreCommit-Admin not running

It looks like the PreCommit-Admin build job is not running.
Can anyone give it a gentle nudge?

Kihwal

Make Hadoop NetworkTopology and data locality more pluggable for other deploying topology like: virtualization.

[Jun Ping Du]

Hello Folks,
  I just filed a Umbrella jira today to address current NetworkTopology 
issue that binding strictly to three tier network. The motivation here is to 
make hadoop more flexible for deploying topology (especially for 
cloud/virtualization case) and more configurable in data locality related 
policies like: replica placement, task scheduling, choosing block for DFSClient 
reading, balancing. 
  We submit a draft proposal in this Umbrella as well as the implementation 
code. As code base is large (~260K), the code is separated into 7 sub JIRA 
issues which seems to be more convenient for reviewing. However, we split the 
code based on functionality which cause some dependencies between patches which 
way we are not sure the best. Welcome to provide comments and suggestions on 
doc and code, and look forward to work with all of you to enhance hadoop in 
some new situations towards perfect.
  Hope this is a good start.

Cheers,

Junping

- Original Message -
From: "Junping Du (JIRA)" 
To: common-iss...@hadoop.apache.org
Sent: Monday, June 4, 2012 12:09:22 PM
Subject: [jira] [Created] (HADOOP-8468) Umbrella of enhancements to support 
different failure and locality topologies

Junping Du created HADOOP-8468:
--

 Summary: Umbrella of enhancements to support different failure and 
locality topologies
 Key: HADOOP-8468
 URL: https://issues.apache.org/jira/browse/HADOOP-8468
 Project: Hadoop Common
  Issue Type: Bug
  Components: ha, io
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du
Priority: Critical


The current hadoop network topology (described in some previous issues like: 
Hadoop-692) works well in classic three-tiers network when it comes out. 
However, it does not take into account other failure models or changes in the 
infrastructure that can affect network bandwidth efficiency like: 
virtualization. 
Virtualized platform has following genes that shouldn't been ignored by hadoop 
topology in scheduling tasks, placing replica, do balancing or fetching block 
for reading: 
1. VMs on the same physical host are affected by the same hardware failure. In 
order to match the reliability of a physical deployment, replication of data 
across two virtual machines on the same host should be avoided.
2. The network between VMs on the same physical host has higher throughput and 
lower latency and does not consume any physical switch bandwidth.
Thus, we propose to make hadoop network topology extend-able and introduce a 
new level in the hierarchical topology, a node group level, which maps well 
onto an infrastructure that is based on a virtualized environment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira