Re: [DISCUSS] Removing the archiac master branch

[Zhe Zhang]

+1 Thanks Owen

I scratched my head for a while and couldn't think of a downside.

On Fri, Jun 19, 2020 at 10:20 AM Owen O'Malley 
wrote:

> We unfortunately have a lot of master/slave and whitelist/blacklist
> terminology usage in Hadoop. It will take a while to fix them all, but one
> is easy to fix. In particular, we have a "master" branch that hasn't been
> used since the project reunification and we use "trunk" as the main branch.
>
> I propose that we delete the "master" branch. Thoughts?
>
> .. Owen
>

Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)

[Zhe Zhang]

+1 on RC1

- Verified signatures
- Built from source (RHEL7)
- Ran single node cluster and tested basic HDFS commands

Thanks for putting this together Jonathan!

On Mon, Oct 28, 2019 at 7:54 PM zhankun tang  wrote:

> Thanks for the efforts! Jonathan!
>
> +1 (non-binding) on RC1.
> - Set up a single node cluster with the binary tarball
> - Run Spark Pi and PySpark job
>
> BR,
> Zhankun
>
> On Mon, 28 Oct 2019 at 23:16, Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp>
> wrote:
>
> > Thanks for putting this up, Jonathan Hung.
> >
> > +1(non-binding)
> >
> > * verified signature and checksum.
> > * built source tarball by OpenJDK 7 on CentOS 7 with native profile
> > enabled.
> > * launched pseudo distributed cluster with security configurations.
> > * create encryption zone and put files in it.
> > * accessed files in encryption zone via httpfs.
> > * checked that file names containing '%' or '+' or ';' can be handled by
> > webhdfs.
> >
> > Masatake Iwasaki
> >
> > On 10/23/19 06:55, Jonathan Hung wrote:
> > > Hi folks,
> > >
> > > This is the second release candidate for the first release of Apache
> > Hadoop
> > > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It
> includes
> > > features such as:
> > >
> > > - User-defined resource types
> > > - Native GPU support as a schedulable resource type
> > > - Consistent reads from standby node
> > > - Namenode port based selective encryption
> > > - Improvements related to rolling upgrade support from 2.x to 3.x
> > > - Cost based fair call queue
> > >
> > > The RC1 artifacts are at:
> > http://home.apache.org/~jhung/hadoop-2.10.0-RC1/
> > >
> > > RC tag is release-2.10.0-RC1.
> > >
> > > The maven artifacts are hosted here:
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1243/
> > >
> > > My public key is available here:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm
> > PDT.
> > >
> > > Thanks,
> > > Jonathan Hung
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0)
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Zhe Zhang]

My late +1. Me and my colleagues have been involved in Submarine
development (in collaboration with our TonY project). I think Submarine
will be valuable as a new Apache project.

On Fri, Sep 6, 2019 at 10:57 PM Dinesh Chitlangia
 wrote:

> +1
>
> -Dinesh
>
>
>
>
> On Fri, Sep 6, 2019 at 11:23 PM 俊平堵  wrote:
>
> > +1. Please include me also.
> >
> > Thanks,
> >
> > Junping
> >
> > Wangda Tan  于2019年9月1日周日 下午1:19写道：
> >
> > > Hi all,
> > >
> > > As we discussed in the previous thread [1],
> > >
> > > I just moved the spin-off proposal to CWIKI and completed all TODO
> parts.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
> > >
> > > If you have interests to learn more about this. Please review the
> > proposal
> > > let me know if you have any questions/suggestions for the proposal.
> This
> > > will be sent to board post voting passed. (And please note that the
> > > previous voting thread [2] to move Submarine to a separate Github repo
> > is a
> > > necessary effort to move Submarine to a separate Apache project but not
> > > sufficient so I sent two separate voting thread.)
> > >
> > > Please let me know if I missed anyone in the proposal, and reply if
> you'd
> > > like to be included in the project.
> > >
> > > This voting runs for 7 days and will be concluded at Sep 7th, 11 PM
> PDT.
> > >
> > > Thanks,
> > > Wangda Tan
> > >
> > > [1]
> > >
> > >
> >
> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
> > > [2]
> > >
> > >
> >
> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
> > >
> >
>

Re: [VOTE] Merge YARN-8200 to branch-2 and branch-3.0

[Zhe Zhang]

+1 (binding)

As Jonathan said this feature in branch-2 has been running stably for over
a year. Therefore I’m supportive to the merge

On Mon, Aug 26, 2019 at 2:00 PM Jim Brennan 
wrote:

> +1 (non-binding).
> I have built branch-2 with the latest YARN-8200 patch
> (YARN-8200-branch-2.003.patch).  I ran all of the NM/RM tests and ran a few
> test jobs on a one-node cluster with default settings.
>
>
> On Mon, Aug 26, 2019 at 3:51 PM Oliver Hu  wrote:
>
> > +1 (non-binding)
> >
> > We have used this patch internally for more than a year to acquire GPU
> > reliably at LinkedIn. I don't think it is necessary to merge this to
> > branch-3.0 tho, as we are EOLing that branch.
> >
> > On Thu, Aug 22, 2019 at 4:43 PM Jonathan Hung 
> > wrote:
> >
> > > Hi folks,
> > >
> > > As per [1], starting a vote to merge YARN-8200 (and YARN-8200.branch3)
> > > feature branch to branch-2 (and branch-3.0).
> > >
> > > Vote runs for 7 days, to Thursday, Aug 29 5:00PM PDT.
> > >
> > > Thanks.
> > >
> > > [1]
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAHzWLgcX7f5Tr3q=csrqgysvpdf7mh-iu17femgx89dhr+1...@mail.gmail.com%3e
> > >
> > > Jonathan Hung
> > >
> >
>

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

[Zhe Zhang]

+1

On Mon, Feb 4, 2019 at 6:14 PM Jonathan Hung  wrote:

> Hello,
>
> Starting a vote based on the discuss thread [1] for moving branch-2
> precommit/nightly test builds to openjdk8. After this change, the test
> phase for precommit builds [2] and branch-2 nightly build [3] will run on
> openjdk8. To maintain source compatibility, these builds will still run
> their compile phase for branch-2 on openjdk7 as they do now (in addition to
> compiling on openjdk8).
>
> Vote will run for three business days until Thursday Feb 7 6:00PM PDT.
>
> [1]
>
> https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
>
> [2]
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
>
> [3]
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
>
> Jonathan Hung
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Propose to start new Hadoop sub project "submarine"

[Zhe Zhang]

+1, binding

On Mon, Feb 4, 2019 at 1:20 PM Takanobu Asanuma 
wrote:

> +1 (non-binding). Thanks, Wangda!
>
> - Takanobu
>
> on 2019/02/02 7:24, "Wangda Tan" wrote:
>
> Hi all,
>
> According to positive feedbacks from the thread [1]
>
> This is vote thread to start a new subproject named "hadoop-submarine"
> which follows the release process already established for ozone.
>
> The vote runs for usual 7 days, which ends at Feb 8th 5 PM PDT.
>
> Thanks,
> Wangda Tan
>
> [1]
>
> https://lists.apache.org/thread.html/f864461eb188bd12859d51b0098ec38942c4429aae7e4d001a633d96@%3Cyarn-dev.hadoop.apache.org%3E
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [DISCUSS] Making submarine to different release model like Ozone

[Zhe Zhang]

+1 on the proposal and looking forward to the progress of the project!

On Thu, Jan 31, 2019 at 10:51 PM Weiwei Yang  wrote:

> Thanks for proposing this Wangda, my +1 as well.
> It is amazing to see the progress made in Submarine last year, the
> community grows fast and quiet collaborative. I can see the reasons to get
> it release faster in its own cycle. And at the same time, the Ozone way
> works very well.
>
> —
> Weiwei
> On Feb 1, 2019, 10:49 AM +0800, Xun Liu , wrote:
> > +1
> >
> > Hello everyone,
> >
> > I am Xun Liu, the head of the machine learning team at Netease Research
> Institute. I quite agree with Wangda.
> >
> > Our team is very grateful for getting Submarine machine learning engine
> from the community.
> > We are heavy users of Submarine.
> > Because Submarine fits into the direction of our big data team's hadoop
> technology stack,
> > It avoids the needs to increase the manpower investment in learning
> other container scheduling systems.
> > The important thing is that we can use a common YARN cluster to run
> machine learning,
> > which makes the utilization of server resources more efficient, and
> reserves a lot of human and material resources in our previous years.
> >
> > Our team have finished the test and deployment of the Submarine and will
> provide the service to our e-commerce department (http://www.kaola.com/)
> shortly.
> >
> > We also plan to provides the Submarine engine in our existing YARN
> cluster in the next six months.
> > Because we have a lot of product departments need to use machine
> learning services,
> > for example:
> > 1) Game department (http://game.163.com/) needs AI battle training,
> > 2) News department (http://www.163.com) needs news recommendation,
> > 3) Mailbox department (http://www.163.com) requires anti-spam and
> illegal detection,
> > 4) Music department (https://music.163.com/) requires music
> recommendation,
> > 5) Education department (http://www.youdao.com) requires voice
> recognition,
> > 6) Massive Open Online Courses (https://open.163.com/) requires
> multilingual translation and so on.
> >
> > If Submarine can be released independently like Ozone, it will help us
> quickly get the latest features and improvements, and it will be great
> helpful to our team and users.
> >
> > Thanks hadoop Community!
> >
> >
> > > 在 2019年2月1日，上午2:53，Wangda Tan  写道：
> > >
> > > Hi devs,
> > >
> > > Since we started submarine-related effort last year, we received a lot
> of
> > > feedbacks, several companies (such as Netease, China Mobile, etc.) are
> > > trying to deploy Submarine to their Hadoop cluster along with big data
> > > workloads. Linkedin also has big interests to contribute a Submarine
> TonY (
> > > https://github.com/linkedin/TonY) runtime to allow users to use the
> same
> > > interface.
> > >
> > > From what I can see, there're several issues of putting Submarine under
> > > yarn-applications directory and have same release cycle with Hadoop:
> > >
> > > 1) We started 3.2.0 release at Sep 2018, but the release is done at Jan
> > > 2019. Because of non-predictable blockers and security issues, it got
> > > delayed a lot. We need to iterate submarine fast at this point.
> > >
> > > 2) We also see a lot of requirements to use Submarine on older Hadoop
> > > releases such as 2.x. Many companies may not upgrade Hadoop to 3.x in a
> > > short time, but the requirement to run deep learning is urgent to
> them. We
> > > should decouple Submarine from Hadoop version.
> > >
> > > And why we wanna to keep it within Hadoop? First, Submarine included
> some
> > > innovation parts such as enhancements of user experiences for YARN
> > > services/containerization support which we can add it back to Hadoop
> later
> > > to address common requirements. In addition to that, we have a big
> overlap
> > > in the community developing and using it.
> > >
> > > There're several proposals we have went through during Ozone merge to
> trunk
> > > discussion:
> > >
> https://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201803.mbox/%3ccahfhakh6_m3yldf5a2kq8+w-5fbvx5ahfgs-x1vajw8gmnz...@mail.gmail.com%3E
> > >
> > > I propose to adopt Ozone model: which is the same master branch,
> different
> > > release cycle, and different release branch. It is a great example to
> show
> > > agile release we can do (2 Ozone releases after Oct 2018) with less
> > > overhead to setup CI, projects, etc.
> > >
> > > *Links:*
> > > - JIRA: https://issues.apache.org/jira/browse/YARN-8135
> > > - Design doc
> > > <
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit
> >
> > > - User doc
> > > <
> https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/Index.html
> >
> > > (3.2.0
> > > release)
> > > - Blogposts, {Submarine} : Running deep learning workloads on Apache
> Hadoop
> > > <
> https://hortonworks.com/blog/submarine-running-deep-learning-workloads-apache-hadoop/
> >,
> > > (Chinese 

Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Zhe Zhang]

+1

Thanks for addressing concerns from the previous vote.

On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko 
wrote:

> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Zhe Zhang]

+1 (binding)

Thanks Konstantin for leading the merge effort!

I worked very closely with Chen, Konstantin, and Erik in the testing stage
and I feel confident that the feature has now completed designed
functionalities and has proven to be stable.

Great team work with contributors from multiple companies!

On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko 
wrote:

> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> There are a few outstanding issues:
> 1. Need to provide proper documentation - a user guide for the new feature
> 2. Need to fix automatic failover with ZKFC. Currently it does not doesn't
> know about ObserverNodes trying to convert them to SBNs.
> 3. Scale testing and performance fine-tuning
> 4. As testing progresses, we continue fixing non-critical bugs like
> HDFS-14116.
>
> I attached a unified patch to the umbrella jira for the review and Jenkins
> build.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 12.
>
> Thanks,
> --Konstantin
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

[Zhe Zhang]

+1 (binding)

- Downloaded source
- Verified checksum
- Built and started local cluster
- Checked YARN UI
- Performed basic HDFS operations

On Fri, Apr 13, 2018 at 1:40 PM Chen Liang <vagaryc...@gmail.com> wrote:

> Thanks for working on this Konstantin!
>
> +1 (non-binding)
>
> - verified checksum
> - built from source
> - started a single node HDFS cluster
> - performed basic operations of ls/mkdir/put/get
> - checked web UI
> - ran the MR job pi
>
> Regards,
> Chen
>
>
> > On Apr 9, 2018, at 4:14 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
> >
> > Hi everybody,
> >
> > This is the next dot release of Apache Hadoop 2.7 line. The previous one
> 2.7.5
> > was released on December 14, 2017.
> > Release 2.7.6 includes critical bug fixes and optimizations. See more
> > details in Release Note:
> > http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html
> >
> > The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/
> >
> > Please give it a try and vote on this thread. The vote will run for 5
> days
> > ending 04/16/2018.
> >
> > My up to date public key is available from:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Thanks,
> > --Konstantin
>
>
> ---------
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Zhe Zhang]

Great work Konstantin.

+1 (binding)
- Downloaded both source and binary tar files and verified checksums
- Deployed pseudo-distributed cluster and verified basic HDFS operations
(mkdir, copyFromLocal)
- Verified both NameNode and RM UIs

On Wed, Dec 13, 2017 at 12:05 PM Jonathan Hung <jyhung2...@gmail.com> wrote:

> Thanks Konstantin for working on this.
>
> +1 (non-binding)
> - Downloaded binary and verified md5
> - Deployed RM HA and tested failover
>
>
>
>
> Jonathan Hung
>
> On Wed, Dec 13, 2017 at 11:02 AM, Eric Payne <erichadoo...@yahoo.com
> .invalid
> > wrote:
>
> > Thanks for the hard work on this release, Konstantin.
> > +1 (binding)
> > - Built from source
> > - Verified that refreshing of queues works as expected.
> >
> > - Verified can run multiple users in a single queue
> > - Ran terasort test
> > - Verified that cross-queue preemption works as expected
> > Thanks. Eric Payne
> >
> >   From: Konstantin Shvachko <shv.had...@gmail.com>
> >  To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>; "
> > hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; "
> > mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>; "
> > yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>
> >  Sent: Thursday, December 7, 2017 9:22 PM
> >  Subject: [VOTE] Release Apache Hadoop 2.7.5 (RC1)
> >
> > Hi everybody,
> >
> > I updated CHANGES.txt and fixed documentation links.
> > Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
> >
> > This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
> > previous one 2.7.4 was release August 4, 2017.
> > Release 2.7.5 includes critical bug fixes and optimizations. See more
> > details in Release Note:
> > http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
> >
> > The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
> >
> > Please give it a try and vote on this thread. The vote will run for 5
> days
> > ending 12/13/2017.
> >
> > My up to date public key is available from:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Thanks,
> > --Konstantin
> >
> >
> >
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler configuration) to trunk, branch-3.0, branch-2

[Zhe Zhang]

+1 (binding)

LinkedIn has been running an internal version of OrgQueue for almost 2
years. It is an essential feature for running YARN scheduler on clusters
shared by multiple internal organizations.

I have participated in the YARN-5734 design and have regularly synced with
Jonathan on development and implementation details. I believe the feature
is of decent quality now and is ready to merge into trunk.

On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jyhung2...@gmail.com> wrote:

> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=pkec9dpggyx9rzf3n...@mail.gmail.com%3E
>
> Jonathan Hung
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [RESULT] [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Zhe Zhang]

Thanks Konstantin, great work!

On Sat, Aug 5, 2017 at 1:36 PM Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> My formal vote
> +1 (binding)
>
> I am glad to summarize that
> with 7 binding and 13 non-binding +1s and no -1s the vote for Apache
> Release 2.7.4 passes.
> Thank you everybody for contributing to the release, testing it, and
> voting.
>
> Binding +1s (7)
> Zhe Zhang
> Jason Lowe
> Eric Payne
> Sunil G
> Akira Ajisaka
> Chris Douglas
> Konstantin Shvachko
>
> Non-binding +1s (13)
> John Zhuge
> Surendra Lilhore
> Masatake Iwasaki
> Hanisha Koneru
> Chen Liang
> Fnu Ajay Kumar
> Brahma Reddy Battula
> Edwina Lu
> Ye Zhou
> Eric Badger
> Mingliang Liu
> Kuhu Shukla
> Erik Krogen
>
> Thanks,
> --Konstantin
>
> On Sat, Jul 29, 2017 at 4:29 PM, Konstantin Shvachko <shv.had...@gmail.com
> >
> wrote:
>
> > Hi everybody,
> >
> > Here is the next release of Apache Hadoop 2.7 line. The previous stable
> > release 2.7.3 was available since 25 August, 2016.
> > Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
> > critical bug fixes and major optimizations. See more details in Release
> > Note:
> > http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
> >
> > The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
> >
> > Please give it a try and vote on this thread. The vote will run for 5
> days
> > ending 08/04/2017.
> >
> > Please note that my up to date public key are available from:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > Please don't forget to refresh the page if you've been there recently.
> > There are other place on Apache sites, which may contain my outdated key.
> >
> > Thanks,
> > --Konstantin
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Zhe Zhang]

Quick question for Chris: was your +1 for the RC, or to support
Konstantin's statement regarding packaging?

Thanks,

On Mon, Jul 31, 2017 at 3:56 PM Chris Douglas <cdoug...@apache.org> wrote:

> On Mon, Jul 31, 2017 at 3:02 PM, Konstantin Shvachko
> <shv.had...@gmail.com> wrote:
> > For the packaging, here is the exact phrasing from the sited
> release-policy
> > document relevant to binaries:
> > "As a convenience to users that might not have the appropriate tools to
> > build a compiled version of the source, binary/bytecode packages MAY be
> > distributed alongside official Apache releases. In all such cases, the
> > binary/bytecode package MUST have the same version number as the source
> > release and MUST only add binary/bytecode files that are the result of
> > compiling that version of the source code release and its dependencies."
> > I don't think my binary package violates any of these.
>
> +1 The PMC VOTE applies to source code, only. If someone wants to
> rebuild the binary tarball with native libs and replace this one,
> that's fine.
>
> My reading of the above is that source code must be distributed with
> binaries, not that we omit the source code from binary releases... -C
>
> > But I'll upload an additional tar.gz with native bits and no src, as you
> > guys requested.
> > Will keep it as RC0 as there is no source code change and it comes from
> the
> > same build.
> > Hope this is satisfactory.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang <andrew.w...@cloudera.com>
> > wrote:
> >
> >> I agree with Brahma on the two issues flagged (having src in the binary
> >> tarball, missing native libs). These are regressions from prior
> releases.
> >>
> >> As an aside, "we release binaries as a convenience" doesn't relax the
> >> quality bar. The binaries are linked on our website and distributed
> through
> >> official Apache channels. They have to adhere to Apache release
> >> requirements. And, most users consume our work via Maven dependencies,
> >> which are binary artifacts.
> >>
> >> http://www.apache.org/legal/release-policy.html goes into this in more
> >> detail. A release must minimally include source packages, and can also
> >> include binary artifacts.
> >>
> >> Best,
> >> Andrew
> >>
> >> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko <
> >> shv.had...@gmail.com> wrote:
> >>
> >>> To avoid any confusion in this regard. I built RC0 manually in
> compliance
> >>> with Apache release policy
> >>> http://www.apache.org/legal/release-policy.html
> >>> I edited the HowToReleasePreDSBCR page to make sure people don't use
> >>> Jenkins option for building.
> >>>
> >>> A side note. This particular build is broken anyways, so no worries
> there.
> >>> I think though it would be useful to have it working for testing and
> as a
> >>> packaging standard.
> >>>
> >>> Thanks,
> >>> --Konstantin
> >>>
> >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
> >>> a...@effectivemachines.com
> >>> > wrote:
> >>>
> >>> >
> >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
> >>> shv.had...@gmail.com>
> >>> > wrote:
> >>> > >
> >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
> >>> >
> >>> > FYI:
> >>> >
> >>> > If you are using ASF Jenkins to create an ASF release
> >>> > artifact, it's pretty much an automatic vote failure as any such
> >>> release is
> >>> > in violation of ASF policy.
> >>> >
> >>> >
> >>>
> >>
> >>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Zhe Zhang]

gt; > > >>> I edited the HowToReleasePreDSBCR page to make sure people don't
> use
> > > >>> Jenkins option for building.
> > > >>>
> > > >>> A side note. This particular build is broken anyways, so no worries
> > > there.
> > > >>> I think though it would be useful to have it working for testing
> and
> > > as a
> > > >>> packaging standard.
> > > >>>
> > > >>> Thanks,
> > > >>> --Konstantin
> > > >>>
> > > >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
> > > >>> a...@effectivemachines.com
> > > >>> > wrote:
> > > >>>
> > > >>> >
> > > >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
> > > >>> shv.had...@gmail.com>
> > > >>> > wrote:
> > > >>> > >
> > > >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
> > > >>> >
> > > >>> > FYI:
> > > >>> >
> > > >>> > If you are using ASF Jenkins to create an ASF
> > release
> > > >>> > artifact, it's pretty much an automatic vote failure as any such
> > > >>> release is
> > > >>> > in violation of ASF policy.
> > > >>> >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > >
> >
>
>
>
> --
>
> *Zhou, Ye  **周晔*
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: About 2.7.4 Release

[Zhe Zhang]

Thanks for volunteering as RM Konstantin! The plan LGTM.

I've created a nightly Jenkins job for branch-2.7 (unit tests):
https://builds.apache.org/job/Hadoop-branch2.7-nightly/

On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hey guys,
>
> I and a few of my colleagues would like to help here and move 2.7.4 release
> forward. A few points in this regard.
>
> 1. Reading through this thread since March 1 I see that Vinod hinted on
> managing the release. Vinod, if you still want the job / have bandwidth
> will be happy to work with you.
> Otherwise I am glad to volunteer as the release manager.
>
> 2. In addition to current blockers and criticals, I would like to propose a
> few issues to be included in the release, see the list below. Those are
> mostly bug fixes and optimizations, which we already have in our internal
> branch and run in production. Plus one minor feature "node labeling", which
> we found very handy, when you have heterogeneous environments and mixed
> workloads, like MR and Spark.
>
> 3. For marking issues for the release I propose to
>  - set the target version to 2.7.4, and
>  - add a new label "release-blocker"
> That way we will know issues targeted for the release without reopening
> them for backports.
>
> 4. I see quite a few people are interested in the release. With all the
> help I think we can target to release by the end of May.
>
> Other things include fixing CHANGES.txt and fixing Jenkins build for 2.7.4
> branch.
>
> Thanks,
> --Konstantin
>
> ==  List of issue for 2.7.4  ===
> -- Backports
> HADOOP-12975 <https://issues.apache.org/jira/browse/HADOOP-12975>. Add du
> jitters
> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR batching
> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>. NPE when
> applying AvailableSpaceBlockPlacementPolicy
> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>. fsck removal
> of dot printing
> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
> space-balanced
> policy for balancer
> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>. abort
> balancer
> if upgrade in progress
> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>. skip small
> blocks in getBlocks
>
> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>. SLS simulator
> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>. SLS
> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>. SLS
> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>. SLS
>
> - Node labeling
> MAPREDUCE-6304 <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: Local trunk build is flaky

[Zhe Zhang]

Thanks John! Updating maven version worked.

On Mon, Apr 17, 2017 at 11:31 AM John Zhuge <john.zh...@gmail.com> wrote:

> This command works for me (similar to Eric's):
> mvn clean && mvn install package -Pdist -Dtar -DskipTests -DskipShade
> -Dmaven.javadoc.skip
>
>
> $ mvn -v
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T08:41:47-08:00)
> Maven home: /Users/jzhuge/apache-maven-3.3.9
> Java version: 1.8.0_66, vendor: Oracle Corporation
> Java home:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.12.4", arch: "x86_64", family: "mac"
>
>
> On Mon, Apr 17, 2017 at 10:46 AM, Zhe Zhang <z...@apache.org> wrote:
>
> > Thanks Eric. Same build command failed for me.
> >
> > On Mon, Apr 17, 2017 at 10:38 AM Eric Badger <ebad...@yahoo-inc.com>
> > wrote:
> >
> > > For what it's worth, I successfully built trunk just now on macOS
> Sierra
> > > using mvn install -Pdist -Dtar -DskipTests -DskipShade -Dmaven
> > > .javadoc.skip
> > >
> > >
> > >
> > > On Monday, April 17, 2017 12:32 PM, Zhe Zhang <z...@apache.org> wrote:
> > >
> > >
> > > Starting from last week, building trunk on my local Mac has been
> flaky. I
> > > haven't tried Linux yet. The error is:
> > >
> > > [ERROR] Failed to execute goal
> > > org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (clean) on
> > > project hadoop-assemblies: Some Enforcer rules have failed. Look above
> > for
> > > specific messages explaining why the rule failed. -> [Help 1]
> > > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> > execute
> > > goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce
> > (clean)
> > > on project hadoop-assemblies: Some Enforcer rules have failed. Look
> above
> > > for specific messages explaining why the rule failed.
> > > at
> > >
> > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> > MojoExecutor.java:217)
> > > ...
> > >
> > > Anyone else seeing the same issue?
> > >
> > > Thanks,
> > >
> > > --
> > > Zhe Zhang
> > > Apache Hadoop Committer
> > > http://zhe-thoughts.github.io/about/ | @oldcap
> > >
> > >
> > > --
> > Zhe Zhang
> > Apache Hadoop Committer
> > http://zhe-thoughts.github.io/about/ | @oldcap
> >
>
>
>
> --
> John
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: Local trunk build is flaky

[Zhe Zhang]

Thanks Eric. Same build command failed for me.

On Mon, Apr 17, 2017 at 10:38 AM Eric Badger <ebad...@yahoo-inc.com> wrote:

> For what it's worth, I successfully built trunk just now on macOS Sierra
> using mvn install -Pdist -Dtar -DskipTests -DskipShade -Dmaven
> .javadoc.skip
>
>
>
> On Monday, April 17, 2017 12:32 PM, Zhe Zhang <z...@apache.org> wrote:
>
>
> Starting from last week, building trunk on my local Mac has been flaky. I
> haven't tried Linux yet. The error is:
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (clean) on
> project hadoop-assemblies: Some Enforcer rules have failed. Look above for
> specific messages explaining why the rule failed. -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (clean)
> on project hadoop-assemblies: Some Enforcer rules have failed. Look above
> for specific messages explaining why the rule failed.
> at
>
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
> ...
>
> Anyone else seeing the same issue?
>
> Thanks,
>
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Local trunk build is flaky

[Zhe Zhang]

Starting from last week, building trunk on my local Mac has been flaky. I
haven't tried Linux yet. The error is:

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (clean) on
project hadoop-assemblies: Some Enforcer rules have failed. Look above for
specific messages explaining why the rule failed. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (clean)
on project hadoop-assemblies: Some Enforcer rules have failed. Look above
for specific messages explaining why the rule failed.
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
...

Anyone else seeing the same issue?

Thanks,

-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

[jira] [Resolved] (HADOOP-14147) Offline Image Viewer bug

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang resolved HADOOP-14147.

Resolution: Duplicate

> Offline Image Viewer  bug
> -
>
> Key: HADOOP-14147
> URL: https://issues.apache.org/jira/browse/HADOOP-14147
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: gehaijiang
>
> $ hdfs oiv -p Delimited  -i fsimage_13752447421 -o fsimage.xml
> 17/03/04 08:40:22 INFO offlineImageViewer.FSImageHandler: Loading 757 strings
> 17/03/04 08:40:22 INFO offlineImageViewer.PBImageTextWriter: Loading 
> directories
> 17/03/04 08:40:22 INFO offlineImageViewer.PBImageTextWriter: Loading 
> directories in INode section.
> 17/03/04 08:41:59 INFO offlineImageViewer.PBImageTextWriter: Found 4374109 
> directories in INode section.
> 17/03/04 08:41:59 INFO offlineImageViewer.PBImageTextWriter: Finished loading 
> directories in 96798ms
> 17/03/04 08:41:59 INFO offlineImageViewer.PBImageTextWriter: Loading INode 
> directory section.
> Exception in thread "main" java.lang.IllegalStateException
>   at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.buildNamespace(PBImageTextWriter.java:570)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.loadINodeDirSection(PBImageTextWriter.java:522)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.visit(PBImageTextWriter.java:460)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageDelimitedTextWriter.visit(PBImageDelimitedTextWriter.java:46)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:182)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:124)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14147) Offline Image Viewer bug

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang reopened HADOOP-14147:


> Offline Image Viewer  bug
> -
>
> Key: HADOOP-14147
> URL: https://issues.apache.org/jira/browse/HADOOP-14147
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: gehaijiang
>
> $ hdfs oiv -p Delimited  -i fsimage_13752447421 -o fsimage.xml
> 17/03/04 08:40:22 INFO offlineImageViewer.FSImageHandler: Loading 757 strings
> 17/03/04 08:40:22 INFO offlineImageViewer.PBImageTextWriter: Loading 
> directories
> 17/03/04 08:40:22 INFO offlineImageViewer.PBImageTextWriter: Loading 
> directories in INode section.
> 17/03/04 08:41:59 INFO offlineImageViewer.PBImageTextWriter: Found 4374109 
> directories in INode section.
> 17/03/04 08:41:59 INFO offlineImageViewer.PBImageTextWriter: Finished loading 
> directories in 96798ms
> 17/03/04 08:41:59 INFO offlineImageViewer.PBImageTextWriter: Loading INode 
> directory section.
> Exception in thread "main" java.lang.IllegalStateException
>   at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.buildNamespace(PBImageTextWriter.java:570)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.loadINodeDirSection(PBImageTextWriter.java:522)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageTextWriter.visit(PBImageTextWriter.java:460)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageDelimitedTextWriter.visit(PBImageDelimitedTextWriter.java:46)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:182)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:124)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Jenkins patch-compile fails on trunk

[Zhe Zhang]

https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-compile-root.txt

Looks related to yarn-ui? The HDFS-10872 patch doesn't touch YARN, but the
patch-compile failed at YARN.

Thanks,
Zhe

Re: Multiple node Hadoop cluster on Docker for test and debugging

[Zhe Zhang]

Nice work Kai! Quick comment: have you considered Bigtop? It allows
creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
a custom built Hadoop (from the instructions I read, it builds from a
certain git branch; I'm not sure if that customizable).

On Wed, Nov 9, 2016 at 5:39 AM Sasaki Kai <lewua...@me.com> wrote:

> Hi Hadoop developers
>
> The other day I created a tool for launching multiple node hadoop cluster
> on docker container.
> You can easily launch multiple node hadoop cluster from your Hadoop source
> code.
> It is useful for testing and debugging. Actually I often use it before
> submitting a patch to Hadoop project.
> https://github.com/Lewuathe/docker-hadoop-cluster <
> https://github.com/Lewuathe/docker-hadoop-cluster>
>
> And I also updated to build the latest trunk image automatically and
> upload onto Docker Hub.
> So you can easily check and test the latest trunk branch in the
> environment which is more close to actual usage.
>
> If you already installed docker and docker-compose, what needed is
> docker-compose.yml like this.
>
> version: '2'
>
> services:
>   master:
> image: lewuathe/hadoop-master
> ports:
>   - "9870:9870"
>   - "8088:8088"
>   - "19888:19888"
>   - "8188:8188"
> container_name: "master"
>   slave1:
> image: lewuathe/hadoop-slave
> container_name: "slave1"
> depends_on:
>   - master
> ports:
>   - "9901:9864"
>   - "8041:8042"
>   slave2:
> image: lewuathe/hadoop-slave
> container_name: "slave2"
> depends_on:
>   - master
> ports:
>   - "9902:9864"
>   - "8042:8042"
>
> The usage in detail is described in the repository.
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>
>
> I would be glad if you use this tool for developing and debugging and make
> our development more efficient.
> Please give me any feedbacks to me. Thanks you!
>
> 
> Kai Sasaki
> mail: lewua...@me.com <mailto:lewua...@me.com>
> github: https://github.com/Lewuathe <https://github.com/Lewuathe>
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: Updated 2.8.0-SNAPSHOT artifact

[Zhe Zhang]

I think recutting branch-2.8 from branch-2 is a good idea. We are running
2.6.x in production, and as a result we spend quite some effort for each
patch we want to backport (branch-2 => branch-2.8 => branch-2.7).

On Mon, Oct 31, 2016 at 1:20 AM Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> Thanks Akira and Karthik for replies.
>
>
> If there is no significant benefit over current branch-2.8 ,re-cutting
> will be an good idea, As we can reduce the maintenance effort.
>
>
>
> --Brahma Reddy Battula
>
>
> -Original Message-
> From: Akira Ajisaka [mailto:ajisa...@oss.nttdata.co.jp]
> Sent: 31 October 2016 14:06
> To: Karthik Kambatla
> Cc: Brahma Reddy Battula; Vinod Kumar Vavilapalli;
> common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: Updated 2.8.0-SNAPSHOT artifact
>
> Thanks Karthik for the comment.
>
> Pros:
>
> * (Probably) we can release 2.8.0 earlier.
>
>* New feature in 2.8.0 will be available sooner.
>* We can deprecate APIs sooner. After 2.8.0 is released, we can remove
> them from trunk.
>
> Cons:
>
> * Maintenance cost becomes higher because # of branches becomes more.
> * New feature in 2.9.0 will be available later
>
> I'm okay with either releasing current branch-2.8 or re-cutting it.
>
> Thoughts?
>
> Regards,
> Akira
>
> On 10/26/16 07:30, Karthik Kambatla wrote:
> > Is there value in releasing current branch-2.8? Aren't we better off
> > re-cutting the branch off of branch-2?
> >
> > On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka
> > <ajisa...@oss.nttdata.co.jp>
> > wrote:
> >
> >> It's almost a year since branch-2.8 has cut.
> >> I'm thinking we need to release 2.8.0 ASAP.
> >>
> >> According to the following list, there are 5 blocker and 6 critical
> issues.
> >> https://issues.apache.org/jira/issues/?filter=12334985
> >>
> >> Regards,
> >> Akira
> >>
> >>
> >> On 10/18/16 10:47, Brahma Reddy Battula wrote:
> >>
> >>> Hi Vinod,
> >>>
> >>> Any plan on first RC for branch-2.8 ? I think, it has been long time.
> >>>
> >>>
> >>>
> >>>
> >>> --Brahma Reddy Battula
> >>>
> >>> -Original Message-
> >>> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org]
> >>> Sent: 20 August 2016 00:56
> >>> To: Jonathan Eagles
> >>> Cc: common-dev@hadoop.apache.org
> >>> Subject: Re: Updated 2.8.0-SNAPSHOT artifact
> >>>
> >>> Jon,
> >>>
> >>> That is around the time when I branched 2.8, so I guess you were
> >>> getting SNAPSHOT artifacts till then from the branch-2 nightly builds.
> >>>
> >>> If you need it, we can set up SNAPSHOT builds. Or just wait for the
> >>> first RC, which is around the corner.
> >>>
> >>> +Vinod
> >>>
> >>> On Jul 28, 2016, at 4:27 PM, Jonathan Eagles <jeag...@gmail.com>
> wrote:
> >>>>
> >>>> Latest snapshot is uploaded in Nov 2015, but checkins are still
> >>>> coming in quite frequently.
> >>>> https://repository.apache.org/content/repositories/snapshots/org/ap
> >>>> ach
> >>>> e/hadoop/hadoop-yarn-api/
> >>>>
> >>>> Are there any plans to start producing updated SNAPSHOT artifacts
> >>>> for current hadoop development lines?
> >>>>
> >>>
> >>>
> >>> 
> >>> - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>>
> >>>
> >>> 
> >>> - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>>
> >>>
> >>
> >> -
> >> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >>
> >>
> >
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

[jira] [Reopened] (HADOOP-10300) Allowed deferred sending of call responses

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang reopened HADOOP-10300:


I think this'd be a good addition to branch-2.7; all other subtasks under the 
umbrella JIRA are actually in 2.3. Attaching a branch-2.7 patch to trigger 
Jenkins.

[~daryn] [~kihwal] LMK if you have any concerns about the backport.

> Allowed deferred sending of call responses
> --
>
> Key: HADOOP-10300
> URL: https://issues.apache.org/jira/browse/HADOOP-10300
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: ipc
>Affects Versions: 2.0.0-alpha, 3.0.0-alpha1
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>  Labels: BB2015-05-TBR
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HADOOP-10300-branch-2.7.0.patch, HADOOP-10300.patch, 
> HADOOP-10300.patch, HADOOP-10300.patch
>
>
> RPC handlers currently do not return until the RPC call completes and 
> response is sent, or a partially sent response has been queued for the 
> responder.  It would be useful for a proxy method to notify the handler to 
> not yet the send the call's response.
> An potential use case is a namespace handler in the NN might want to return 
> before the edit log is synced so it can service more requests and allow 
> increased batching of edits per sync.  Background syncing could later trigger 
> the sending of the call response to the client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-10597) RPC Server signals backoff to clients when all request queues are full

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-10597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang reopened HADOOP-10597:


Thanks [~mingma] for the nice work!

Since the umbrella JIRA HADOOP-9640 (and the FairCallQueue feature) is 
available in 2.7 and 2.6, I think it makes sense to backport this to at least 
2.7. Reopening to test branch-2.7 patch. Please let me know if you have any 
concern about adding this to branch-2.7.

> RPC Server signals backoff to clients when all request queues are full
> --
>
> Key: HADOOP-10597
> URL: https://issues.apache.org/jira/browse/HADOOP-10597
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HADOOP-10597-2.patch, HADOOP-10597-3.patch, 
> HADOOP-10597-4.patch, HADOOP-10597-5.patch, HADOOP-10597-6.patch, 
> HADOOP-10597-branch-2.7.patch, HADOOP-10597.patch, 
> MoreRPCClientBackoffEvaluation.pdf, RPCClientBackoffDesignAndEvaluation.pdf
>
>
> Currently if an application hits NN too hard, RPC requests be in blocking 
> state, assuming OS connection doesn't run out. Alternatively RPC or NN can 
> throw some well defined exception back to the client based on certain 
> policies when it is under heavy load; client will understand such exception 
> and do exponential back off, as another implementation of 
> RetryInvocationHandler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang reopened HADOOP-12325:


Sorry to reopen the JIRA. I think it is a good addition to branch-2.7 and want 
to test branch-2.7 patch.

> RPC Metrics : Add the ability track and log slow RPCs
> -
>
> Key: HADOOP-12325
> URL: https://issues.apache.org/jira/browse/HADOOP-12325
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: ipc, metrics
>Affects Versions: 2.7.1
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: Callers of WritableRpcEngine.call.png, 
> HADOOP-12325-branch-2.7.00.patch, HADOOP-12325.001.patch, 
> HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, 
> HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch
>
>
> This JIRA proposes to add a counter called RpcSlowCalls and also a 
> configuration setting that allows users to log really slow RPCs.  Slow RPCs 
> are RPCs that fall at 99th percentile. This is useful to troubleshoot why 
> certain services like name node freezes under heavy load.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13747) Use LongAdder for more efficient metrics tracking

[Zhe Zhang (JIRA)]

Zhe Zhang created HADOOP-13747:
--

 Summary: Use LongAdder for more efficient metrics tracking
 Key: HADOOP-13747
 URL: https://issues.apache.org/jira/browse/HADOOP-13747
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Reporter: Zhe Zhang
Assignee: Erik Krogen


Currently most metrics, including {{RpcMetrics}} and {{RpcDetailedMetrics}}, 
use a synchronized counter to be updated by all handler threads (multiple 
hundreds in large production clusters). As [~andrew.wang] suggested, it'd be 
more efficient to use the [LongAdder | 
http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co]
 library which dynamically create intermediate-result variables.

Assigning to [~xkrogen] who has already done some investigation on this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13657) IPC Reader thread could silently die and leave NameNode unresponsive

[Zhe Zhang (JIRA)]

Zhe Zhang created HADOOP-13657:
--

 Summary: IPC Reader thread could silently die and leave NameNode 
unresponsive
 Key: HADOOP-13657
 URL: https://issues.apache.org/jira/browse/HADOOP-13657
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc
Reporter: Zhe Zhang
Priority: Critical


For each listening port, IPC {{Server#Listener#Reader}} is a single thread in 
charge of moving {{Connection}} items from {{pendingConnections}} (capacity 
100) to the {{callQueue}}.

We have experienced an incident where the {{Reader}} thread for HDFS NameNode 
died from run time exception. Then the {{pendingConnections}} queue became full 
and the NameNode port became inaccessible.

In our particular case, what killed {{Reader}} was a NPE caused by 
https://bugs.openjdk.java.net/browse/JDK-8024883. But in general, other types 
of runtime exceptions could cause this issue as well.

We should add logic to either make the {{Reader}} more robust in case of 
runtime exceptions, or at least treat it as a FATAL exception so that NameNode 
can fail over to standby, and admins get alerted of the real issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: how about syncing namenode metadata to glusterfs?

[Zhe Zhang]

Hi Levi,

This clearly should go to hdfs-dev instead of common-dev. Just using an HA
storage for fsimage won't allow HDFS work without the current NN-HA
framework. After all fsimage is only a checkpoint of NameNode state.

On Tue, Sep 13, 2016 at 2:19 AM Levi Nie <levinie...@gmail.com> wrote:

> Use glusterfs instead of disk to keep fsimage, will it work without HA?
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Zhe Zhang]

+1 (non-binding)

Did the following on 7 RHEL 6.6 servers
- Downloaded and built from source
- Downloaded and verified checksum of the binary tar.gz file
- Setup a cluster with 1 NN and 6 DNs
- Tried regular HDFS commands
- Tried EC commands (listPolicies, getPolicy, setPolicy), they work fine
- Verified that with a 3-2 policy, 1.67x capacity is used. Below is the
output after copying the binary tar.gz file into an EC folder. The file is
318MB.

Configured Capacity: 3221225472 (3 GB)
Present Capacity: 3215348743 (2.99 GB)
DFS Remaining: 2655666176 (2.47 GB)
DFS Used: 559682567 (533.75 MB)

Thanks Allen for clarifying on the markdown files. I also verified the site
html files (content of the index.html, randomly selected some links).


On Tue, Aug 30, 2016 at 2:20 PM Eric Badger 
wrote:

> Well that's embarrassing. I had accidentally slightly renamed my
> log4j.properties file in my conf directory, so it was there, just not being
> read. Apologies for the unnecessary spam. With this and the public key from
> Andrew, I give my non-binding +1.
>
> Eric
>
>
>
> On Tuesday, August 30, 2016 4:11 PM, Allen Wittenauer <
> a...@effectivemachines.com> wrote:
>
>
> > On Aug 30, 2016, at 2:06 PM, Eric Badger 
> wrote:
> >
> >
> > WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be
> incomplete.
>
> ^^
>
>
> >
> > After running the above command, the RM UI showed a successful job, but
> as you can see, I did not have anything printed onto the command line.
> Hopefully this is just a misconfiguration on my part, but I figured that I
> would point it out just in case.
>
>
> It gave you a very important message in the output ...
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Zhe Zhang]

Thanks Andrew for the great work! It's really exciting to finally see a
Hadoop 3 RC.

I noticed CHANGES and RELEASENOTES markdown files which were not in
previous RCs like 2.7.3. What are good tools to verify them? I tried
reading them on IntelliJ but format looks odd.

I'm still testing the RC:
- Downloaded and verified checksum
- Built from source
- Will start small cluster and test simple programs, focusing on EC
functionalities

-- Zhe

On Tue, Aug 30, 2016 at 8:51 AM Andrew Wang 
wrote:

> Hi all,
>
> Thanks to the combined work of many, many contributors, here's an RC0 for
> 3.0.0-alpha1:
>
> http://home.apache.org/~wang/3.0.0-alpha1-RC0/
>
> alpha1 is the first in a series of planned alpha releases leading up to GA.
> The objective is to get an artifact out to downstreams for testing and to
> iterate quickly based on their feedback. So, please keep that in mind when
> voting; hopefully most issues can be addressed by future alphas rather than
> future RCs.
>
> Sorry for getting this out on a Tuesday, but I'd still like this vote to
> run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend
> if we lack the votes.
>
> Please try it out and let me know what you think.
>
> Best,
> Andrew
>

Re: [DISCUSS] 2.6.x line releases

[Zhe Zhang]

Thanks for pointing it out Allen. I'll work on an addendum 2.6 patch for
HADOOP-12800.

On Fri, Jul 22, 2016 at 3:59 PM Allen Wittenauer <a...@effectivemachines.com>
wrote:

>
> > On Jul 22, 2016, at 9:07 AM, Zhe Zhang <zhe.zhang.resea...@gmail.com>
> wrote:
> >
> > Thanks Allen for the note. I thought the 2.6 Dockerfile issue was
> addressed in https://issues.apache.org/jira/browse/HADOOP-12800?
>
> 2.6 builds on JDK6 and JDK7.  2.x, where x>6, builds on JDK7 and JDK8.
> [1] This should have been a big red flag in that patch:
>
> ---
> +RUN apt-get install -y oracle-java8-installer
> ---
>
> (It's interesting that ~2 years later, we're still dealing with the
> fallout of the JRE compatibility break in 2.7.  I wonder how many of those
> PMCs who voted for it are still actively involved.)
>
> [1] For completeness, 3.x only builds on JDK8 but probably not JDK9, given
> log4j 1.x, etc, etc.

-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [DISCUSS] 2.6.x line releases

[Zhe Zhang]

Thanks Allen for the note. I thought the 2.6 Dockerfile issue was addressed
in https://issues.apache.org/jira/browse/HADOOP-12800?

On Fri, Jul 22, 2016 at 9:04 AM Allen Wittenauer <a...@effectivemachines.com>
wrote:

>
> > On Jul 21, 2016, at 5:06 PM, Zhe Zhang <zhe.zhang.resea...@gmail.com>
> wrote:
> >
> > We are using 2.6 releases and would like to see 2.6.5.
> >
> > One related issue, pre-commit is not working for 2.6 (see comment from
> > Allen
> >
> https://issues.apache.org/jira/browse/HDFS-10653?focusedCommentId=15388621=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15388621
> ).
> > Should we fix for the purpose of the release?
>
>
> Someone probably just needs to invest some time in modifying the
> Dockerfile in 2.6 to actually meet the build requirements.  Right now, it's
> way way wrong.

-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [DISCUSS] 2.6.x line releases

[Zhe Zhang]

We are using 2.6 releases and would like to see 2.6.5.

One related issue, pre-commit is not working for 2.6 (see comment from
Allen
https://issues.apache.org/jira/browse/HDFS-10653?focusedCommentId=15388621=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15388621).
Should we fix for the purpose of the release?

Thanks,

On Thu, Jul 21, 2016 at 5:03 PM Chris Trezzo <ctre...@gmail.com> wrote:

> Sangjin, I would be happy to help you with the work for cutting 2.6.5.
>
> Thanks!
> Chris Trezzo
>
> On Wed, Jul 20, 2016 at 2:44 PM, Akira AJISAKA <ajisa...@oss.nttdata.co.jp
> >
> wrote:
>
> > Hi Sangjin,
> >
> > I'm thinking it's time for 2.6.5 release. Now there are 51 fixed issues
> in
> > 2.6.5 and the number is more than 2.6.4 (47).
> >
> > > How long do we maintain this line? What would be a sensible EOL policy?
> >
> > Ideally 5 years, because most of our customers in Japan are using Hadoop
> > cluster without upgrading for about 5 years. However, it's probably
> > impossible for us to achieve the long term maintenance in the community
> > because the maintenance cost is too high. Therefore, my answer is "longer
> > is better".
> >
> > By the way, I'm thinking we can reduce the maintenance cost by selecting
> > which branch to maintenance for a long time. An example is Ubuntu LTS.
> > Ubuntu Server LTS is maintained for 5 years but the other releases are
> not.
> >
> > Regards,
> > Akira
> >
> > On 7/20/16 12:45, Sean Busbey wrote:
> >
> >> The HBase community would like more 2.6.z releases.
> >>
> >> On Wed, Jul 20, 2016 at 2:00 PM, Ravi Prakash <ravihad...@gmail.com>
> >> wrote:
> >>
> >>> We for one are not using 2.6.*
> >>>
> >>> On Tue, Jul 19, 2016 at 1:21 PM, Sangjin Lee <sj...@apache.org> wrote:
> >>>
> >>> It's been a while since we had a release on the 2.6.x line. Is it time
> to
> >>>> get ready for a 2.6.5 release? Are folks using and relying on releases
> >>>> on
> >>>> 2.6.x? If there is enough interest, I could take that on. Let me know.
> >>>>
> >>>> I also want to gauge the community's interest in maintaining the 2.6.x
> >>>> line. How long do we maintain this line? What would be a sensible EOL
> >>>> policy? Note that as the main code lines start diverging (java
> version,
> >>>> features, etc.), the cost of maintaining multiple release lines does
> go
> >>>> up.
> >>>> I'd love to hear your thoughts.
> >>>>
> >>>> Regards,
> >>>> Sangjin
> >>>>
> >>>>
> >>
> >>
> >>
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [DISCUSS] Increased use of feature branches

[Zhe Zhang]

Thanks for the notes Andrew, Junping, Karthik.

Here are some of my understandings:

- Trunk is the "latest and greatest" of Hadoop. If a user starts using
Hadoop today, without legacy workloads, trunk is what he/she should use.
- Therefore, each commit to trunk should be transactional -- atomic,
consistent, isolated (from other uncommitted patches); I'm not so sure
about durability, Hadoop might be gone in 50 years :). As a committer, I
should be able to look at a patch and determine whether it's a
self-contained improvement of trunk, without looking at other uncommitted
patches.
- Some comments inline:

On Fri, Jun 10, 2016 at 6:56 AM Junping Du  wrote:

> Comparing with advantages, I believe the disadvantages of shipping any
> releases directly from trunk are more obvious and significant:
> - A lot of commits (incompatible, risky, uncompleted feature, etc.) have
> to wait to commit to trunk or put into a separated branch that could delay
> feature development progress as additional vote process get involved even
> the feature is simple and harmless.
>
Thanks Junping, those are valid concerns. I think we should clearly
separate incompatible with  uncompleted / half-done work in this
discussion. Whether people should commit incompatible changes to trunk is a
much more tricky question (related to trunk-incompat etc.). But per my
comment above, IMHO, *not committing uncompleted work to trunk* should be a
much easier principle to agree upon.


> - For small feature with only 1 or 2 commits, that need three +1 from PMCs
> will increase the bar largely for contributors who just start to contribute
> on Hadoop features but no such sufficient support.
>
Development overhead is another valid concern. I think our rule-of-thumb
should be that, small-medium new features should be proposed as a single
JIRA/patch (as we recently did for HADOOP-12666). If the complexity goes
beyond a single JIRA/patch, use a feature branch.


>
> Given these concerns, I am open to other options, like: proposed by Vinod
> or Chris, but rather than to release anything directly from trunk.
>
> - This point doesn't necessarily need to be resolved now though, since
> again we're still doing alphas.
> No. I think we have to settle down this first. Without a common agreed and
> transparent release process and branches in community, any release (alpha,
> beta) bits is only called a private release but not a official apache
> hadoop release (even alpha).
>
>
> Thanks,
>
> Junping
> 
> From: Karthik Kambatla 
> Sent: Friday, June 10, 2016 7:49 AM
> To: Andrew Wang
> Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: [DISCUSS] Increased use of feature branches
>
> Thanks for restarting this thread Andrew. I really hope we can get this
> across to a VOTE so it is clear.
>
> I see a few advantages shipping from trunk:
>
>- The lack of need for one additional backport each time.
>- Feature rot in trunk
>
> Instead of creating branch-3, I recommend creating branch-3.x so we can
> continue doing 3.x releases off branch-3 even after we move trunk to 4.x (I
> said it :))
>
> On Thu, Jun 9, 2016 at 11:12 PM, Andrew Wang 
> wrote:
>
> > Hi all,
> >
> > On a separate thread, a question was raised about 3.x branching and use
> of
> > feature branches going forward.
> >
> > We discussed this previously on the "Looking to a Hadoop 3 release"
> thread
> > that has spanned the years, with Vinod making this proposal (building on
> > ideas from others who also commented in the email thread):
> >
> >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201604.mbox/browser
> >
> > Pasting here for ease:
> >
> > On an unrelated note, offline I was pitching to a bunch of
> > contributors another idea to deal
> > with rotting trunk post 3.x: *Make 3.x releases off of trunk directly*.
> >
> > What this gains us is that
> >  - Trunk is always nearly stable or nearly ready for releases
> >  - We no longer have some code lying around in some branch (today’s
> > trunk) that is not releasable
> > because it gets mixed with other undesirable and incompatible changes.
> >  - This needs to be coupled with more discipline on individual
> > features - medium to to large
> > features are always worked upon in branches and get merged into trunk
> > (and a nearing release!)
> > when they are ready
> >  - All incompatible changes go into some sort of a trunk-incompat
> > branch and stay there till
> > we accumulate enough of those to warrant another major release.
> >
> > Regarding "trunk-incompat", since we're still in the alpha stage for
> 3.0.0,
> > there's no need for this branch yet. This aspect of Vinod's proposal was
> > still under a bit of discussion; Chris Douglas though we should cut a
> > branch-3 for the first 3.0.0 beta, which aligns with my original
> 

Branch-2.6 Pre-Commit Jenkins broken?

[Zhe Zhang]

Has anyone gotten a successful run recently? Thanks.

[jira] [Created] (HADOOP-13206) Delegation token cannot be fetched and used by different versions of client

[Zhe Zhang (JIRA)]

Zhe Zhang created HADOOP-13206:
--

 Summary: Delegation token cannot be fetched and used by different 
versions of client
 Key: HADOOP-13206
 URL: https://issues.apache.org/jira/browse/HADOOP-13206
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.6.1, 2.3.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang


We have observed that an HDFS delegation token fetched by a 2.3.0 client cannot 
be used by a 2.6.1 client, and vice versa. Through some debugging I found that 
it's a mismatch between the token's {{service}} and the {{service}} of the 
filesystem (e.g. {{webhdfs://host.something.com:50070/}}). One would be in 
numerical IP address and one would be in non-numerical hostname format.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Release numbering for 3.x leading up to GA

[Zhe Zhang]

Thanks Andrew. I also had a talk with Kai offline. Agreed that we should
try our best to finalize coder config changes for alpha1.

On Tue, May 17, 2016 at 5:34 PM Andrew Wang <andrew.w...@cloudera.com>
wrote:

> The sooner the better for incompatible changes, but at this point we are
> explicitly not guaranteeing any compatibility between alpha releases.
>
> For EC, my understanding is that we're still working on the coder
> configuration. Given that we're still working on L changes, I think it's
> possible that the coder configuration will be finished in time.
>
> On Tue, May 17, 2016 at 4:42 PM, Zhe Zhang <zhe.zhang.resea...@gmail.com>
> wrote:
>
> > Naming convention looks good. Thanks Andrew for driving this!
> >
> > Could you explain a little more on the criteria of cutting alpha1 /
> > alpha2? What are the goals we want to achieve for alpha1? From EC's
> > perspective, maybe we should target on having all compatibility-related
> > changes in alpha1, like new config keys and fsimage format?
> >
> > Thanks,
> >
> > On Thu, May 12, 2016 at 11:35 AM Andrew Wang <andrew.w...@cloudera.com>
> > wrote:
> >
> >> Hi folks,
> >>
> >> I think it's working, though it takes some time for the rename to
> >> propagate
> >> in JIRA. JIRA is also currently being hammered by spammers, which might
> be
> >> related.
> >>
> >> Anyway, the new "3.0.0-alpha1" version should be live for all four
> >> subprojects, so have at it!
> >>
> >> Best,
> >> Andrew
> >>
> >> On Thu, May 12, 2016 at 11:01 AM, Gangumalla, Uma <
> >> uma.ganguma...@intel.com>
> >> wrote:
> >>
> >> > Thanks Andrew for driving. Sounds good. Go ahead please.
> >> >
> >> > Good luck :-)
> >> >
> >> > Regards,
> >> > Uma
> >> >
> >> > On 5/12/16, 10:52 AM, "Andrew Wang" <andrew.w...@cloudera.com> wrote:
> >> >
> >> > >Hi all,
> >> > >
> >> > >Sounds like we have general agreement on this release numbering
> scheme
> >> for
> >> > >3.x.
> >> > >
> >> > >I'm going to attempt some mvn and JIRA invocations to get the version
> >> > >numbers lined up for alpha1, wish me luck.
> >> > >
> >> > >Best,
> >> > >Andrew
> >> > >
> >> > >On Tue, May 3, 2016 at 9:52 AM, Roman Shaposhnik <
> ro...@shaposhnik.org
> >> >
> >> > >wrote:
> >> > >
> >> > >> On Tue, May 3, 2016 at 8:18 AM, Karthik Kambatla <
> ka...@cloudera.com
> >> >
> >> > >> wrote:
> >> > >> > The naming scheme sounds good. Since we want to start out sooner,
> >> I am
> >> > >> > assuming we are not limiting ourselves to two alphas as the email
> >> > >>might
> >> > >> > indicate.
> >> > >> >
> >> > >> > Also, as the release manager, can you elaborate on your
> >> definitions of
> >> > >> > alpha and beta? Specifically, when do we expect downstream
> >> projects to
> >> > >> try
> >> > >> > and integrate and when we expect Hadoop users to try out the
> bits?
> >> > >>
> >> > >> Not to speak of all the downstream PMC,s but Bigtop project will
> jump
> >> > >> on the first alpha the same way we jumped on the first alpha back
> >> > >> in the 1 -> 2 transition period.
> >> > >>
> >> > >> Given that Bigtop currently integrates quite a bit of Hadoop
> >> ecosystem
> >> > >> that work is going to produce valuable feedback that we plan to
> >> > >>communicate
> >> > >> to the individual PMCs. What PMCs do with that feedback, of course,
> >> will
> >> > >> be up to them (obviously Bigtop can't take the ownership of issues
> >> that
> >> > >> go outside of integration work between projects in the Hadoop
> >> ecoystem).
> >> > >>
> >> > >> Thanks,
> >> > >> Roman.
> >> > >>
> >> >
> >> >
> >> > -
> >> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >> >
> >> >
> >>
> > --
> > Zhe Zhang
> > Apache Hadoop Committer
> > http://zhe-thoughts.github.io/about/ | @oldcap
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: Release numbering for 3.x leading up to GA

[Zhe Zhang]

Naming convention looks good. Thanks Andrew for driving this!

Could you explain a little more on the criteria of cutting alpha1 / alpha2?
What are the goals we want to achieve for alpha1? From EC's perspective,
maybe we should target on having all compatibility-related changes in
alpha1, like new config keys and fsimage format?

Thanks,

On Thu, May 12, 2016 at 11:35 AM Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Hi folks,
>
> I think it's working, though it takes some time for the rename to propagate
> in JIRA. JIRA is also currently being hammered by spammers, which might be
> related.
>
> Anyway, the new "3.0.0-alpha1" version should be live for all four
> subprojects, so have at it!
>
> Best,
> Andrew
>
> On Thu, May 12, 2016 at 11:01 AM, Gangumalla, Uma <
> uma.ganguma...@intel.com>
> wrote:
>
> > Thanks Andrew for driving. Sounds good. Go ahead please.
> >
> > Good luck :-)
> >
> > Regards,
> > Uma
> >
> > On 5/12/16, 10:52 AM, "Andrew Wang" <andrew.w...@cloudera.com> wrote:
> >
> > >Hi all,
> > >
> > >Sounds like we have general agreement on this release numbering scheme
> for
> > >3.x.
> > >
> > >I'm going to attempt some mvn and JIRA invocations to get the version
> > >numbers lined up for alpha1, wish me luck.
> > >
> > >Best,
> > >Andrew
> > >
> > >On Tue, May 3, 2016 at 9:52 AM, Roman Shaposhnik <ro...@shaposhnik.org>
> > >wrote:
> > >
> > >> On Tue, May 3, 2016 at 8:18 AM, Karthik Kambatla <ka...@cloudera.com>
> > >> wrote:
> > >> > The naming scheme sounds good. Since we want to start out sooner, I
> am
> > >> > assuming we are not limiting ourselves to two alphas as the email
> > >>might
> > >> > indicate.
> > >> >
> > >> > Also, as the release manager, can you elaborate on your definitions
> of
> > >> > alpha and beta? Specifically, when do we expect downstream projects
> to
> > >> try
> > >> > and integrate and when we expect Hadoop users to try out the bits?
> > >>
> > >> Not to speak of all the downstream PMC,s but Bigtop project will jump
> > >> on the first alpha the same way we jumped on the first alpha back
> > >> in the 1 -> 2 transition period.
> > >>
> > >> Given that Bigtop currently integrates quite a bit of Hadoop ecosystem
> > >> that work is going to produce valuable feedback that we plan to
> > >>communicate
> > >> to the individual PMCs. What PMCs do with that feedback, of course,
> will
> > >> be up to them (obviously Bigtop can't take the ownership of issues
> that
> > >> go outside of integration work between projects in the Hadoop
> ecoystem).
> > >>
> > >> Thanks,
> > >> Roman.
> > >>
> >
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

[jira] [Created] (HADOOP-13055) Implement linkMergeSlash for ViewFs

[Zhe Zhang (JIRA)]

Zhe Zhang created HADOOP-13055:
--

 Summary: Implement linkMergeSlash for ViewFs
 Key: HADOOP-13055
 URL: https://issues.apache.org/jira/browse/HADOOP-13055
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs, viewfs
Reporter: Zhe Zhang
Assignee: Zhe Zhang


In a multi-cluster environment it is sometimes useful to operate on the root / 
slash directory of an HDFS cluster. E.g., list all top level directories. 
Quoting the comment in {{ViewFs}}:
{code}
 *   A special case of the merge mount is where mount table's root is merged
 *   with the root (slash) of another file system:
 *   
 *   fs.viewfs.mounttable.default.linkMergeSlash=hdfs://nn99/
 *   
 *   In this cases the root of the mount table is merged with the root of
 *hdfs://nn99/  
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: CHANGES.txt is gone from trunk, branch-2, branch-2.8

[Zhe Zhang]

Great change! Saving ~1min for each commit. Thanks Andrew and Allen.

On Thu, Mar 3, 2016 at 9:32 PM Yongjun Zhang <yzh...@cloudera.com> wrote:

> That's nice, thanks Andrew and Allen.
>
> --Yongjun
>
> On Thu, Mar 3, 2016 at 9:11 PM, Andrew Wang <andrew.w...@cloudera.com>
> wrote:
>
> > Hi all,
> >
> > With the inclusion of HADOOP-12651 going back to branch-2.8, CHANGES.txt
> > and release notes are now generated by Yetus. I've gone ahead and deleted
> > the manually updated CHANGES.txt from trunk, branch-2, and branch-2.8
> > (HADOOP-11792). Many thanks to Allen for the releasedocmaker.py rewrite,
> > and the Yetus integration.
> >
> > I'll go ahead and update the HowToCommit and HowToRelease wiki pages, but
> > at a high-level, this means we no longer need to edit CHANGES.txt on new
> > commit, streamlining our commit process. CHANGES.txt updates will still
> be
> > necessary for backports to older release lines like 2.6.x and 2.7.x.
> >
> > Happy committing!
> >
> > Best,
> > Andrew
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: Looking to a Hadoop 3 release

[Zhe Zhang]

Thanks Andrew for driving the effort!

+1 (non-binding) on starting the 3.0 release process now with 3.0 as an
alpha.

I wanted to echo Andrew's point that backporting EC to branch-2 is a lot of
work. Considering that no concrete backporting plan has been proposed, it
seems quite uncertain whether / when it can be released in 2.9. I think we
should rather concentrate our EC dev efforts to harden key features under
the follow-on umbrella HDFS-8031 and make it solid for a 3.0 release.

Sincerely,
Zhe

On Mon, Feb 22, 2016 at 9:25 AM Colin P. McCabe  wrote:

> +1 for a release of 3.0.  There are a lot of significant,
> compatibility-breaking, but necessary changes in this release... we've
> touched on some of them in this thread.
>
> +1 for a parallel release of 2.8 as well.  I think we are pretty close
> to this, barring a dozen or so blockers.
>
> best,
> Colin
>
> On Mon, Feb 22, 2016 at 2:56 AM, Steve Loughran 
> wrote:
> >
> >> On 20 Feb 2016, at 15:34, Junping Du  wrote:
> >>
> >> Shall we consolidate effort for 2.8.0 and 3.0.0? It doesn't sounds
> reasonable to have two alpha releases to go in parallel. Is EC feature the
> main motivation of releasing hadoop 3 here? If so, I don't understand why
> this feature cannot land on 2.8.x or 2.9.x as an alpha feature.
> >
> >
> >
> >> If we release 3.0 in a month like plan proposed below, it means we will
> have 4 active releases going in parallel - two alpha releases (2.8 and 3.0)
> and two stable releases (2.6.x and 2.7.x). It brings a lot of challenges in
> issues tracking and patch committing, not even mention the tremendous
> effort of release verification and voting.
> >> I would like to propose to wait 2.8 release become stable (may be 2nd
> release in 2.8 branch cause first release is alpha due to discussion in
> another email thread), then we can move to 3.0 as the only alpha release.
> In the meantime, we can bring more significant features (like ATS v2, etc.)
> to trunk and consolidate stable releases in 2.6.x and 2.7.x. I believe that
> make life easier. :)
> >> Thoughts?
> >>
> >
> > 2.8.0 is relatively close to shipping. I say relatively as I'm doing
> some work with ATS 1.5 downstream and I'd like to make sure all that works.
> There's also a large collection of S3 and swift patches needing attention
> from any reviewers with time and credentials.
> >
> > 3.x is going to take multiple iterations to stabilise, and with more
> changes, more significant a rollout. I'd also like to do a complete update
> of all the dependencies before a final release, so we can have less
> pressure to upgrade for a while, and get Sean's classloader patch in so
> it's slightly less visible.
> >
> > That means 3.0 is going to be an alpha release, not final.
> >
> > one thing that could be shared is any build.xml automation of the
> release process, to at least take away most of the manual steps in the
> process, to have something more repeatable.
> >
> > -steve
> >
> >
> >> Thanks,
> >>
> >> Junping
> >> 
> >> From: Yongjun Zhang 
> >> Sent: Friday, February 19, 2016 8:05 PM
> >> To: hdfs-...@hadoop.apache.org
> >> Cc: common-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> >> Subject: Re: Looking to a Hadoop 3 release
> >>
> >> Thanks Andrew for initiating the effort!
> >>
> >> +1 on pushing 3.x with extended alpha cycle, and continuing the more
> stable
> >> 2.x releases.
> >>
> >> --Yongjun
> >>
> >> On Thu, Feb 18, 2016 at 5:58 PM, Andrew Wang 
> >> wrote:
> >>
> >>> Hi Kai,
> >>>
> >>> Sure, I'm open to it. It's a new major release, so we're allowed to
> make
> >>> these kinds of big changes. The idea behind the extended alpha cycle is
> >>> that downstreams can give us feedback. This way if we do anything too
> >>> radical, we can address it in the next alpha and have downstreams
> re-test.
> >>>
> >>> Best,
> >>> Andrew
> >>>
> >>> On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai 
> wrote:
> >>>
>  Thanks Andrew for driving this. Wonder if it's a good chance for
>  HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note
> >>> it's
>  not an incompatible change, but feel better to be done in the major
> >>> release.
> 
>  Regards,
>  Kai
> 
>  -Original Message-
>  From: Andrew Wang [mailto:andrew.w...@cloudera.com]
>  Sent: Friday, February 19, 2016 7:04 AM
>  To: hdfs-...@hadoop.apache.org; Kihwal Lee 
>  Cc: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
>  yarn-...@hadoop.apache.org
>  Subject: Re: Looking to a Hadoop 3 release
> 
>  Hi Kihwal,
> 
>  I think there's still value in continuing the 2.x releases. 3.x comes
> >>> with
>  the incompatible bump to a JDK8 runtime, and also the fact that 3.x
> won't
>  be 

[jira] [Created] (HADOOP-12800) Copy docker directory from 2.8 to 2.7 repo to enable pre-commit on 2.7 patches

[Zhe Zhang (JIRA)]

Zhe Zhang created HADOOP-12800:
--

 Summary: Copy docker directory from 2.8 to 2.7 repo to enable 
pre-commit on 2.7 patches
 Key: HADOOP-12800
 URL: https://issues.apache.org/jira/browse/HADOOP-12800
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.7.2
Reporter: Zhe Zhang
Assignee: Zhe Zhang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HADOOP-12764) Increase default value of KMX maxHttpHeaderSize and make it configurable

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang reopened HADOOP-12764:


Created branch-2 patch.

> Increase default value of KMX maxHttpHeaderSize and make it configurable
> 
>
> Key: HADOOP-12764
> URL: https://issues.apache.org/jira/browse/HADOOP-12764
> Project: Hadoop Common
>  Issue Type: Improvement
>    Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-12764-branch-2.00.patch, HADOOP-12764.00.patch
>
>
> The Tomcat default value of {{maxHttpHeaderSize}} is 4096, which is too low 
> for certain Hadoop workloads. This JIRA proposes to change it to 65536 in 
> {{server.xml}} and make it configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12764) Increase default value of KMX maxHttpHeaderSize and make it configurable

[Zhe Zhang (JIRA)]

Zhe Zhang created HADOOP-12764:
--

 Summary: Increase default value of KMX maxHttpHeaderSize and make 
it configurable
 Key: HADOOP-12764
 URL: https://issues.apache.org/jira/browse/HADOOP-12764
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor


The Tomcat default value of {{maxHttpHeaderSize}} is 4096, which is too low for 
certain Hadoop workloads. This JIRA proposes to change it to 65536 in 
{{server.xml}} and make it configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Can you help us Hadoop Community?

[Zhe Zhang]

>
> it is difficult to find files to change in a
> specific issue.

I guess this can be a useful reminder "you might also want to update file
Y". Maybe richer insights can be found on method level.

---
Zhe Zhang

On Mon, Dec 14, 2015 at 7:07 PM, Igor Wiese <igor.wi...@gmail.com> wrote:

> Hi Zhe! Thanks for your answer.
>
> In fact, we are predicting the "co-change" based on contextual
> information collected from issues, commits and developers
> communication. Considering the files that i described in the example
> ("/ipc/Client.java" and
> "security/SecurityUtil.java") I collected metrics in each issue and
> commit from Client.java to predict when Client.java is prone to change
> with SecurityUtil.java.
>
> We are thinking to build a webservice to help newcomers during their
> first contributions. Our research group interviewed some newcomers and
> they told us that it is difficult to find files to change in a
> specific issue. We can recommend files to be checked.
>
> From the committer perspective, we could help in code review tasks.
>
> What do you think?
>
> Our idea
>
> 2015-12-14 22:16 GMT-02:00 Zhe Zhang <z...@apache.org>:
> > Hi Igor,
> >
> > It's an interesting direction to study tickets/commits in the Hadoop
> > community.
> >
> > A research group from Univ. Wisconsin did a similar study on Linux file
> > systems and I found it quite insightful:
> > http://research.cs.wisc.edu/wind/Publications/fsstudy-tos14.pdf
> >
> > For your results, could you elaborate why you picked "co-change" as the
> > metric, and how to improve software tools from the "co-change"
> predictions?
> >
> > Thanks,
> > Zhe
> >
> > On Mon, Dec 14, 2015 at 3:01 PM, Igor Wiese <igor.wi...@gmail.com>
> wrote:
> >
> >> Hi, Hadoop Community.
> >>
> >> My name is Igor Wiese, phd Student from Brazil. I sent an email a week
> >> ago about my research. We received some visit to inspect the results
> >> but any feedback was provided.
> >>
> >> I am investigating two important questions: What makes two files
> >> change together? Can we predict when they are going to co-change
> >> again?
> >>
> >> I've tried to investigate this question on the Hadoop project. I've
> >> collected data from issue reports, discussions and commits and using
> >> some machine learning techniques to build a prediction model.
> >>
> >>
> >> I collected a total of 950 commits in which a pair of files changed
> >> together and could correctly predict 47% commits. These were the most
> >> useful information for predicting co-changes of files:
> >>
> >> - sum of number of lines of code added, modified and removed,
> >>
> >> - number of words used to describe and discuss the issues,
> >>
> >> - median value of closeness, a social network measure obtained from
> >> issue comments,
> >>
> >> - median value of constraint, a social network measure obtained from
> >> issue comments, and
> >>
> >> - median value of hierarchy, a social network measure obtained from
> >> issue comments.
> >>
> >> To illustrate, consider the following example from our analysis. For
> >> release 0.22, the files "/ipc/Client.java" and
> >> "security/SecurityUtil.java" changed together in 3 commits. In another
> >> 1 commit, only the first file changed, but not the second. Collecting
> >> contextual information for each commit made to first file in the
> >> previous release, we were able to predict 2 commits in which both
> >> files changed together in release 0.22, and we only issued 1 wrong
> >> prediction. For this pair of files, the most important contextual
> >> information were the social network metrics (density, hierarchy,
> >> efficiency) obtained from issue comments.
> >>
> >>
> >> - Do these results surprise you? Can you think in any explanation for
> >> the results?
> >>
> >> - Do you think that our rate of prediction is good enough to be used
> >> for building tool support for the software community?
> >>
> >> - Do you have any suggestion on what can be done to improve the change
> >> recommendation?
> >>
> >> You can visit our webpage to inspect the results in details:
> >> http://flosscoach.com/index.php/17-cochanges/70-hadoop
> >>
> >> All the best,
> >> Igor Wiese
> >>
> >> Phd Candidate
> >>
> >> --
> >> =
> >> Igor Scaliante Wiese
> >> PhD Candidate - Computer Science @ IME/USP
> >> Faculty in Dept. of Computing at Universidade Tecnológica Federal do
> Paraná
> >>
>
>
>
> --
> =
> Igor Scaliante Wiese
> PhD Candidate - Computer Science @ IME/USP
> Faculty in Dept. of Computing at Universidade Tecnológica Federal do Paraná
>

Re: Can you help us Hadoop Community?

[Zhe Zhang]

Hi Igor,

It's an interesting direction to study tickets/commits in the Hadoop
community.

A research group from Univ. Wisconsin did a similar study on Linux file
systems and I found it quite insightful:
http://research.cs.wisc.edu/wind/Publications/fsstudy-tos14.pdf

For your results, could you elaborate why you picked "co-change" as the
metric, and how to improve software tools from the "co-change" predictions?

Thanks,
Zhe

On Mon, Dec 14, 2015 at 3:01 PM, Igor Wiese  wrote:

> Hi, Hadoop Community.
>
> My name is Igor Wiese, phd Student from Brazil. I sent an email a week
> ago about my research. We received some visit to inspect the results
> but any feedback was provided.
>
> I am investigating two important questions: What makes two files
> change together? Can we predict when they are going to co-change
> again?
>
> I've tried to investigate this question on the Hadoop project. I've
> collected data from issue reports, discussions and commits and using
> some machine learning techniques to build a prediction model.
>
>
> I collected a total of 950 commits in which a pair of files changed
> together and could correctly predict 47% commits. These were the most
> useful information for predicting co-changes of files:
>
> - sum of number of lines of code added, modified and removed,
>
> - number of words used to describe and discuss the issues,
>
> - median value of closeness, a social network measure obtained from
> issue comments,
>
> - median value of constraint, a social network measure obtained from
> issue comments, and
>
> - median value of hierarchy, a social network measure obtained from
> issue comments.
>
> To illustrate, consider the following example from our analysis. For
> release 0.22, the files "/ipc/Client.java" and
> "security/SecurityUtil.java" changed together in 3 commits. In another
> 1 commit, only the first file changed, but not the second. Collecting
> contextual information for each commit made to first file in the
> previous release, we were able to predict 2 commits in which both
> files changed together in release 0.22, and we only issued 1 wrong
> prediction. For this pair of files, the most important contextual
> information were the social network metrics (density, hierarchy,
> efficiency) obtained from issue comments.
>
>
> - Do these results surprise you? Can you think in any explanation for
> the results?
>
> - Do you think that our rate of prediction is good enough to be used
> for building tool support for the software community?
>
> - Do you have any suggestion on what can be done to improve the change
> recommendation?
>
> You can visit our webpage to inspect the results in details:
> http://flosscoach.com/index.php/17-cochanges/70-hadoop
>
> All the best,
> Igor Wiese
>
> Phd Candidate
>
> --
> =
> Igor Scaliante Wiese
> PhD Candidate - Computer Science @ IME/USP
> Faculty in Dept. of Computing at Universidade Tecnológica Federal do Paraná
>

Re: Disable some of the Hudson integration comments on JIRA

[Zhe Zhang]

I think that's a good idea. Thanks for the proposal Andrew.

---
Zhe Zhang

On Wed, Nov 25, 2015 at 5:41 PM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Hi all,
>
> Right now we get something like 7 comments from Hudson whenever a change is
> committed. Would anyone object if I turned off 6 of them? We have
> variations like:
>
> Hadoop-trunk-Commit
> Hadoop-Hdfs-trunk-Java8
> Hadoop-Yarn-trunk
> ...etc
>
> I propose leaving notifications on for just Hadoop-trunk-Commit.
>
> Side note is that we could probably stand to delete the disabled jobs on
> our view, I'll do that if I see a job has been disabled for a while with no
> recent builds:
>
> https://builds.apache.org/view/H-L/view/Hadoop/
>
> Best,
> Andrew
>

Update on potential Gerrit integration

[Zhe Zhang]

Hi,

Per our earlier discussion on the "Github integration for Hadoop" thread, I
surveyed HBase, Flume, and HTrace about their interest on Gerrit. The
feedbacks have been positive:

HBase:
http://mail-archives.apache.org/mod_mbox/hbase-dev/201511.mbox/%3CCAAjoTXkeAR5xjNfWtEP87Y%3DPToaL7pyPeE2o1C4N5fh5d5s3Pw%40mail.gmail.com%3E


Flume:
http://mail-archives.apache.org/mod_mbox/flume-dev/201511.mbox/%3CCAAjoTXmm0tMYyjbHs%3DgGsnCvt%3DWj1ZAb90JuiO2Ks5TORoJ3aA%40mail.gmail.com%3E

HTrace (earlier than my survey):
http://mail-archives.apache.org/mod_mbox/incubator-htrace-dev/201510.mbox/%3CCA%2BqbEUOirX7Ob29Ozr7Qr94LdWzQp%3DWQ2BPzVBEzv53J_EmCvQ%40mail.gmail.com%3E

Based on the above I created an INFRA JIRA (
https://issues.apache.org/jira/browse/INFRA-10743). Let me know if you have
questions or comments.

Thanks,
Zhe

[jira] [Created] (HADOOP-12559) KMS connection failures should trigger TGT renewal

[Zhe Zhang (JIRA)]

Zhe Zhang created HADOOP-12559:
--

 Summary: KMS connection failures should trigger TGT renewal
 Key: HADOOP-12559
 URL: https://issues.apache.org/jira/browse/HADOOP-12559
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.7.1
Reporter: Zhe Zhang
Assignee: Zhe Zhang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Github integration for Hadoop

[Zhe Zhang]

>
> So I think Gerrit/Crucible/whatever are on the table, with some work.
> Does anyone want to take the token for asking other projects and
> assembling a proposal for an alternative they're excited about?


Thanks for the update and the suggestion Chris, I think it's a great idea.
I'd like to volunteer for the above plan.

---
Zhe Zhang

On Sun, Nov 1, 2015 at 2:07 PM, Chris Douglas <cdoug...@apache.org> wrote:

> Wow, this happened quickly.
>
> Owen, could you please create a Wiki describing the proposal and
> cataloging infra references so others can understand the
> implementation in detail? Even after reading this thread, I'm still
> confused what changes this proposes and how the integration works. A
> document pairing open questions with answers/workarounds would help
> this converge.
>
> CC'd David Nalley, who reached out ~6 months ago to ask about
> Gerrit/Crucible/etc. He said that infra would be open to supporting
> any of these tools, but they'll want to study what's required.
> Further, they don't want to support *all* these tools, so if there are
> projects other than Hadoop that want to be part of an experimental
> program, we should collectively pick one and work with infra to ensure
> the foundation's criteria are satisfied (i.e., clear IP provenance)
> and the maintenance burden is manageable.
>
> So I think Gerrit/Crucible/whatever are on the table, with some work.
> Does anyone want to take the token for asking other projects and
> assembling a proposal for an alternative they're excited about? -C
>
> On Fri, Oct 30, 2015 at 4:55 PM, Owen O'Malley <omal...@apache.org> wrote:
> > On Fri, Oct 30, 2015 at 2:37 PM, Andrew Wang <andrew.w...@cloudera.com>
> > wrote:
> >
> >>
> >> Strongly agree that we need documentation for all this.
> >
> >
> > Agreed.
> >
> >
> >> Some of my open questions are also still pending.
> >>
> >
> > Which open questions?
> >
> > Owen, are you chasing all of this down? Do we need some JIRAs to track?
> >>
> >
> > Yeah, I'll take a first pass at it. I think jiras are overkill.
> >
> > .. Owen
>

[jira] [Resolved] (HADOOP-9311) hadoop-auth: Kerberos token expiration results in an error

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang resolved HADOOP-9311.
---
   Resolution: Duplicate
Fix Version/s: 2.4.0

HADOOP-10301 has added a similar logic to address the issue:
{code}
  try {
token = getToken(httpRequest);
  }
  catch (AuthenticationException ex) {
LOG.warn("AuthenticationToken ignored: " + ex.getMessage());
// will be sent back in a 401 unless filter authenticates
authenticationEx = ex;
token = null;
  }
{code}

> hadoop-auth: Kerberos token expiration results in an error
> --
>
> Key: HADOOP-9311
> URL: https://issues.apache.org/jira/browse/HADOOP-9311
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: net
>Affects Versions: 1.0.4, 1.1.0
>Reporter: Joel Firehammer
>Assignee: Joel Firehammer
> Fix For: 2.4.0
>
> Attachments: HADOOP-9311-release-1.0.4.patch
>
>
> When using Kerberos for auth, if a page is left open longer than the token 
> timeout, a subsequent request fails (typically to an error page.) Forcing a 
> refresh will re-establish the session. 
> A minor annoyance, but automatically re-authenticating is friendlier.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[VOTE] Merge HDFS-7285 (erasure coding) branch to trunk

[Zhe Zhang]

Hi,

I'd like to propose a vote to merge the HDFS-7285 feature branch back to
trunk. Since November 2014 we have been designing and developing this
feature under the umbrella JIRAs HDFS-7285 and HADOOP-11264, and have
committed approximately 210 patches.

The HDFS-7285 feature branch was created to support the first phase of HDFS
erasure coding (HDFS-EC). The objective of HDFS-EC is to significantly
reduce storage space usage in HDFS clusters. Instead of always creating 3
replicas of each block with 200% storage space overhead, HDFS-EC provides
data durability through parity data blocks. With most EC configurations,
the storage overhead is no more than 50%. Based on profiling results of
production clusters, we decided to support EC with the striped block layout
in the first phase, so that small files can be better handled. This means
dividing each logical HDFS file block into smaller units (striping cells)
and spreading them on a set of DataNodes in round-robin fashion. Parity
cells are generated for each stripe of original data cells. We have made
changes to NameNode, client, and DataNode to generalize the block concept
and handle the mapping between a logical file block and its internal
storage blocks. For further details please see the design doc on HDFS-7285.
HADOOP-11264 focuses on providing flexible and high-performance codec
calculation support.

The nightly Jenkins job of the branch has reported several successful runs,
and doesn't show new flaky tests compared with trunk. We have posted
several versions of the test plan including both unit testing and cluster
testing, and have executed most tests in the plan. The most basic
functionalities have been extensively tested and verified in several real
clusters with different hardware configurations; results have been very
stable. We have created follow-on tasks for more advanced error handling
and optimization under the umbrella HDFS-8031. We also plan to implement or
harden the integration of EC with existing features such as WebHDFS,
snapshot, append, truncate, hflush, hsync, and so forth.

Development of this feature has been a collaboration across many companies
and institutions. I'd like to thank J. Andreina, Takanobu Asanuma,
Vinayakumar B, Li Bo, Takuya Fukudome, Uma Maheswara Rao G, Rui Li, Yi Liu,
Colin McCabe, Xinwei Qin, Rakesh R, Gao Rui, Kai Sasaki, Walter Su, Tsz Wo
Nicholas Sze, Andrew Wang, Yong Zhang, Jing Zhao, Hui Zheng and Kai Zheng
for their code contributions and reviews. Andrew and Kai Zheng also made
fundamental contributions to the initial design. Rui Li, Gao Rui, Kai
Sasaki, Kai Zheng and many other contributors have made great efforts in
system testing. Many thanks go to Weihua Jiang for proposing the JIRA, and
ATM, Todd Lipcon, Silvius Rus, Suresh, as well as many others for providing
helpful feedbacks.

Following the community convention, this vote will last for 7 days (ending
September 29th). Votes from Hadoop committers are binding but non-binding
votes are very welcome as well. And here's my non-binding +1.

Thanks,
---
Zhe Zhang

Re: [DISCUSS] Proposal for allowing merge workflows on feature branches

[Zhe Zhang]

Thanks Andrew for the proposal! I think a more flexible git workflow will
be very helpful in feature branch development.

I just had an offline discussion with Walter Su, and it happens to be very
relevant to Ravi's question. At least for the EC branch, we find it
problematic to use only 1 approach (git rebase or git merge).
Retrospectively, the optimal approach would have been 1) weekly git
rebase to resolve trivial conflicts + 2) occasional git merge for trunk
changes that overlaps with many branch commits.

So I agree with Andrew to focus on allowing git merge in this [VOTE] and
leave further details (e.g. how to use the 2 options) to each branch's
contributors.

---
Zhe Zhang

On Thu, Aug 20, 2015 at 10:55 AM, Andrew Wang andrew.w...@cloudera.com
wrote:

 Hi Ravi,

 Thanks for reviewing. I think the choice between merge and rebase is very
 situational, so I don't want to be too prescriptive in the text we're
 voting on. On the wiki page or docs, I'll include more discussion about
 when one or the other might be preferred.

 Responses have been positive thus far, I'll start a real [VOTE] tomorrow
 unless there are more edits.

 Best,
 Andrew

 On Thu, Aug 20, 2015 at 10:12 AM, Ravi Prakash ravi...@ymail.com wrote:

  Thanks for the work Andrew! Should we specify a preference for one
  workflow over another? If not, this looks good.
 
 
 
   On Wednesday, August 19, 2015 12:04 PM, Colin McCabe 
  cmcc...@alumni.cmu.edu wrote:
 
 
   +1.  Rebasing can really make the history much clearer when used
  correctly.
 
  Colin
 
  On Tue, Aug 18, 2015 at 2:57 PM, Andrew Wang andrew.w...@cloudera.com
  wrote:
   Hi common-dev,
  
   Based on the prior [DISCUSS] thread, I've put together a new [VOTE]
   proposal which modifies the branch development practices edified by the
   [VOTE] when we switched from SVN to git [1]. This new proposal modifies
  the
   third and fourth points of the earlier [VOTE], copied here for your
   convenience:
  
   3. Force-push on feature-branches is allowed. Before pulling in a
  feature,
   the feature-branch should be rebased on latest trunk and the changes
   applied to trunk through git rebase --onto or git cherry-pick
   commit-range.
  
   4. Every time a feature branch is rebased on trunk, a tag that
 identifies
   the state before the rebase needs to be created (e.g.
tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted
 once
   the feature is pulled into trunk and the tags are no longer useful.
  
   Said new proposal expands and modifies as follows:
  
   
  
   Feature branch development can use either a merge or rebase workflow,
 as
   decided by contributors working on the branch.
  
   When using a rebase workflow, the feature branch is periodically
 rebased
  on
   trunk via git rebase trunk and force pushed.
  
   Before performing a force-push, a tag should be created of the current
   feature branch HEAD to preserve history. The tag should identify the
   feature and date of most recent commit, e.g.
   tag_feature_HDFS-7285_2015-08-11. It can also be convenient to use a
   temporary branch to review rebase conflict resolution before
  force-pushing
   the main feature branch, e.g. HDFS-7285-rebase. Temporary branches
 should
   be deleted after they are force-pushed over the feature branch.
  
   Developers are allowed to squash and reorder commits to make rebasing
   easier. Use this judiciously. When squashing, please maintain the
  original
   commit messages in the squashed commit message to preserve history.
  
   When using a merge workflow, changes are periodically integrated from
  trunk
   to the branch via git merge trunk.
  
   Merge conflict resolution can be reviewed by posting the diff of the
  merge
   commit.
  
   For both rebase and merge workflows, integration of the branch into
 trunk
   should happen via git merge --no-ff. --no-ff is important since it
   generates a merge commit even if the branch applies cleanly on top of
   trunk. This clearly denotes the set of commits that were made on the
   branch, and makes it easier to revert the branch if necessary.
  
   git merge --no-ff is also the preferred way of integrating a feature
   branch to other branches, e.g. branch-2.
  
   
  
   LMK what you think about the above, when we finalize I'll kick off a
   [VOTE]. Last time we did Adoption of New Codebase but this feels more
   like Modifying bylaws, if it needs a [VOTE] at all. Bylaws is
 easier,
   since it's just a lazy majority of active PMC members rather than
 2/3rds.
  
   If the eventual [VOTE] passes, I'll put it on the wiki somewhere for
  easier
   reference. I'll also expand said wiki page with discussion about merge
  vs.
   rebase based on the mailing list threads, since I think we've got some
  good
   information here.
  
   Best,
   Andrew
  
   [1]:
  
 
 http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201408.mbox/%3CCALwhT94Y64M9keY25Ry_QOLUSZQT29tJQ95twsoa8xXrcNTxpQ

[jira] [Created] (HADOOP-12017) Hadoop archives command should use configurable replication factor when closing

[Zhe Zhang (JIRA)]

Zhe Zhang created HADOOP-12017:
--

 Summary: Hadoop archives command should use configurable 
replication factor when closing
 Key: HADOOP-12017
 URL: https://issues.apache.org/jira/browse/HADOOP-12017
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang


{{HadoopArchives#close}} uses hard-coded replication factor. It should use 
{{repl}} instead, which is parsed from command line parameters.
{code}
  // try increasing the replication 
  fs.setReplication(index, (short) 5);
  fs.setReplication(masterIndex, (short) 5);
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11740) Combine erasure encoder and decoder interfaces

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang resolved HADOOP-11740.

   Resolution: Fixed
Fix Version/s: HDFS-7285

 Combine erasure encoder and decoder interfaces
 --

 Key: HADOOP-11740
 URL: https://issues.apache.org/jira/browse/HADOOP-11740
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Zhe Zhang
Assignee: Zhe Zhang
 Fix For: HDFS-7285

 Attachments: HADOOP-11740-000.patch, HADOOP-11740-001.patch, 
 HADOOP-11740-002.patch


 Rationale [discussed | 
 https://issues.apache.org/jira/browse/HDFS-7337?focusedCommentId=14376540page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14376540]
  under HDFS-7337.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11664) Loading predefined EC schemas from configuration

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang resolved HADOOP-11664.

Resolution: Fixed

 Loading predefined EC schemas from configuration
 

 Key: HADOOP-11664
 URL: https://issues.apache.org/jira/browse/HADOOP-11664
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HADOOP-11664-v2.patch, HADOOP-11664-v3.patch, 
 HDFS-7371_v1.patch


 System administrator can configure multiple EC codecs in hdfs-site.xml file, 
 and codec instances or schemas in a new configuration file named 
 ec-schema.xml in the conf folder. A codec can be referenced by its instance 
 or schema using the codec name, and a schema can be utilized and specified by 
 the schema name for a folder or EC ZONE to enforce EC. Once a schema is used 
 to define an EC ZONE, then its associated parameter values will be stored as 
 xattributes and respected thereafter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11740) Combine erasure encoder and decoder interfaces

[Zhe Zhang (JIRA)]

Zhe Zhang created HADOOP-11740:
--

 Summary: Combine erasure encoder and decoder interfaces
 Key: HADOOP-11740
 URL: https://issues.apache.org/jira/browse/HADOOP-11740
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Zhe Zhang


Rationale [discussed | 
https://issues.apache.org/jira/browse/HDFS-7337?focusedCommentId=14376540page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14376540]
 under HDFS-7337.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HADOOP-11676) Add API to NetworkTopology for getting all racks

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang reopened HADOOP-11676:


 Add API to NetworkTopology for getting all racks
 

 Key: HADOOP-11676
 URL: https://issues.apache.org/jira/browse/HADOOP-11676
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
 Fix For: HDFS-7285

 Attachments: HADOOP-11676.002.patch, HADOOP-11676.patch


 The existing two NetworkTopology.chooseRandom(..) API support choosing node 
 from scope and choosing outside scope. BlockPlacementPolicyDefault class use 
 these two API to choose node from one rack or choose outside one rack.
 We want to implement a new placement policy called 
 BlockPlacementPolicyFaultTolerant which tries its best to place replicas to 
 most racks. To achieve this, We need to know how many replicas each rack has. 
 And first, we need to get all racks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11676) Add API to NetworkTopology for getting all racks

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang resolved HADOOP-11676.

Resolution: Won't Fix

Reverted per discussion under HDFS-7891

 Add API to NetworkTopology for getting all racks
 

 Key: HADOOP-11676
 URL: https://issues.apache.org/jira/browse/HADOOP-11676
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
 Fix For: HDFS-7285

 Attachments: HADOOP-11676.002.patch, HADOOP-11676.patch


 The existing two NetworkTopology.chooseRandom(..) API support choosing node 
 from scope and choosing outside scope. BlockPlacementPolicyDefault class use 
 these two API to choose node from one rack or choose outside one rack.
 We want to implement a new placement policy called 
 BlockPlacementPolicyFaultTolerant which tries its best to place replicas to 
 most racks. To achieve this, We need to know how many replicas each rack has. 
 And first, we need to get all racks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)