Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Ravi Prakash]

Hi folks!

Thank you for sharing the design docs and the tremendous amount of work
that has gone into Ozone. I'm grateful that atleast someone is trying to
drastically improve HDFS.

*If* there is a meeting to discuss this merge, could I please also be
invited?

Have we ever thought about distributing the Namenode metadata across nodes
dynamically based on load and RPC times (unlike static federation that we
have now)?

Also, I think a major feature that HDFS still lacks (and a lot of our users
ask for) is BCP / Disaster Recovery. I only bring this up to see if the
choice of proposed design would have implications for that later on.

Thanks,
Ravi

On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia  wrote:

> Konstantine,
>  Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
>  that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> > On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko 
> wrote:
> >
> > Hey guys,
> >
> > It is an interesting question whether Ozone should be a part of Hadoop.
> > There are two main reasons why I think it should not.
> >
> > 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> > sizable community behind, it looks to me like a whole new project.
> > It is essentially a new storage system, with different (than HDFS)
> > architecture, separate S3-like APIs. This is really great - the World
> sure
> > needs more distributed file systems. But it is not clear why Ozone should
> > co-exist with HDFS under the same roof.
> >
> > 2. Ozone is probably just the first step in rebuilding HDFS under a new
> > architecture. With the next steps presumably being HDFS-10419 and
> > HDFS-8.
> > The design doc for the new architecture has never been published. I can
> > only assume based on some presentations and personal communications that
> > the idea is to use Ozone as a block storage, and re-implement NameNode,
> so
> > that it stores only a partial namesapce in memory, while the bulk of it
> > (cold data) is persisted to a local storage.
> > Such architecture makes me wonder if it solves Hadoop's main problems.
> > There are two main limitations in HDFS:
> >  a. The throughput of Namespace operations. Which is limited by the
> number
> > of RPCs the NameNode can handle
> >  b. The number of objects (files + blocks) the system can maintain. Which
> > is limited by the memory size of the NameNode.
> > The RPC performance (a) is more important for Hadoop scalability than the
> > object count (b). The read RPCs being the main priority.
> > The new architecture targets the object count problem, but in the expense
> > of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> > Also based on the use patterns on our large clusters we read up to 90% of
> > the data we write, so cold data is a small fraction and most of it must
> be
> > cached.
> >
> > To summarize:
> > - Ozone is a big enough system to deserve its own project.
> > - The architecture that Ozone leads to does not seem to solve the
> intrinsic
> > problems of current HDFS.
> >
> > I will post my opinion in the Ozone jira. Should be more convenient to
> > discuss it there for further reference.
> >
> > Thanks,
> > --Konstantin
> >
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei 
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >>  1.  All services, management scripts
> >>  2.  Object store APIs, exposed via both REST and RPC
> >>  3.  Master service UIs, command line interfaces
> >>  4.  Pluggable pipeline Integration
> >>  5.  Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >>  6.  Corona - a load generator for Ozone.
> >>  7.  Essential documentation added to Hadoop site.
> >>  8.  Version specific Ozone Documentation, accessible via service UI.
> >>  9.  Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confl
> >> uence/display/HADOOP/Dev+cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ contributors/committers from
> >> different countries and affiliations. We also want to t

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Ravi Prakash]

Thanks for all your hard work Junping!

* Checked signature.
* Ran a sleep job.
* Checked NN File browser UI works.

+1 (binding)

Cheers
Ravi

On Tue, Oct 24, 2017 at 12:26 PM, Rakesh Radhakrishnan 
wrote:

> Thanks Junping for getting this out.
>
> +1 (non-binding)
>
> * Built from source on CentOS 7.3.1611, jdk1.8.0_111
> * Deployed 3 node cluster
> * Ran some sample jobs
> * Ran balancer
> * Operate HDFS from command line: ls, put, dfsadmin etc
> * HDFS Namenode UI looks good
>
>
> Thanks,
> Rakesh
>
> On Fri, Oct 20, 2017 at 6:12 AM, Junping Du  wrote:
>
> > Hi folks,
> >  I've created our new release candidate (RC1) for Apache Hadoop
> 2.8.2.
> >
> >  Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line
> > and will be the latest stable/production release for Apache Hadoop - it
> > includes 315 new fixed issues since 2.8.1 and 69 fixes are marked as
> > blocker/critical issues.
> >
> >   More information about the 2.8.2 release plan can be found here:
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> >
> >   New RC is available at: http://home.apache.org/~
> > junping_du/hadoop-2.8.2-RC1 > du/hadoop-2.8.2-RC0>
> >
> >   The RC tag in git is: release-2.8.2-RC1, and the latest commit id
> > is: 66c47f2a01ad9637879e95f80c41f798373828fb
> >
> >   The maven artifacts are available via repository.apache.org
>  > repository.apache.org/> at: https://repository.apache.org/
> > content/repositories/orgapachehadoop-1064 > repository.apache.org/content/repositories/orgapachehadoop-1062>
> >
> >   Please try the release and vote; the vote will run for the usual 5
> > days, ending on 10/24/2017 6pm PST time.
> >
> > Thanks,
> >
> > Junping
> >
> >
>

Re: native folder not found in hadoop-common build on Ubuntu

[Ravi Prakash]

Please use -Pnative profile

On Thu, Aug 31, 2017 at 3:53 PM, Ping Liu  wrote:

> Hi John,
>
> Thank you for your quick response.
>
> I used
>
> mvn clean install -DskipTests
>
> I just did a comparison with my Windows build result. winutils is missing
> too.
>
> So both "native" and "winutils" folders are not generated in target folder,
> although it shows BUILD SUCCESS.
>
> Thanks.
>
> Ping
>
>
>
>
>
>
>
> On Thu, Aug 31, 2017 at 3:36 PM, John Zhuge  wrote:
>
> > Hi Ping,
> >
> > Thanks for using Hadoop. Linux is Unix-like. Hadoop supports native code
> > on Linux. Please read BUILDING.txt in the root of the Hadoop source tree.
> >
> > Could you provide the entire Maven command line when you built Hadoop?
> >
> > On Thu, Aug 31, 2017 at 3:06 PM, Ping Liu 
> wrote:
> >
> >> I built hadoop-common on Ubuntu in my VirtualBox.  But in target
> folder, I
> >> didn't find "native" folder that is supposed to contain the generated
> JNI
> >> header files for C.  On my Windows, native folder is found in target.
> >>
> >> As I check the POM file, I found "native build only supported on Mac or
> >> Unix".  Does this mean native is not supported on Linux?
> >>
> >> Thanks!
> >>
> >> Ping
> >>
> >
> >
> >
> > --
> > John
> >
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Ravi Prakash]

+1 to maintaining history.

On Wed, Aug 30, 2017 at 11:38 PM, varunsax...@apache.org <
varun.saxena.apa...@gmail.com> wrote:

> Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
> Maintaining history I believe can be useful as it can make reverts
> easier if at all required.
> And can be an easy reference point to look at who had contributed what
> without having to go back to the branch.
>
> Regards,
> Varun Saxena.
>
> On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C 
> wrote:
>
> > Thanks Sangjin for the link to the previous discussions on this! I think
> > that helps answer Steve's questions.
> >
> > As decided on that thread [1], YARN-5355 as a feature branch was merged
> to
> > trunk via "git merge --no-ff" .
> >
> > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > chose to develop on a feature branch YARN-5355 so that we could control
> > when changes went into trunk and didn't inadvertently disrupt trunk.
> >
> > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> >
> > thanks
> > Vrushali
> > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
> >
> >
> > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee  wrote:
> >
> >> I recall this discussion about a couple of years ago:
> >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> >> dev.hadoop.apache.org%3E
> >>
> >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran  >
> >> wrote:
> >>
> >>> I'd have assumed it would have gone in as one single patch, rather than
> >>> a full history. I don't see why the trunk needs all the evolutionary
> >>> history of a build.
> >>>
> >>> What should our policy/process be here?
> >>>
> >>> I do currently plan to merge the s3guard in as one single squashed
> >>> patch; just getting HADOOP-14809 sorted first.
> >>>
> >>>
> >>> > On 30 Aug 2017, at 07:09, Vrushali C 
> wrote:
> >>> >
> >>> > I'm adding my +1 (binding) to conclude the vote.
> >>> >
> >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
> >>> with
> >>> > the merge to trunk shortly. Thanks everyone!
> >>> >
> >>> > Regards
> >>> > Vrushali
> >>> >
> >>> >
> >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> >>> > varun.saxena.apa...@gmail.com> wrote:
> >>> >
> >>> >> +1 (binding).
> >>> >>
> >>> >> Kudos to all the team members for their great work!
> >>> >>
> >>> >> Being part of the ATSv2 team, I have been involved with either
> >>> development
> >>> >> or review of most of the JIRAs'.
> >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> >>> there
> >>> >> is no impact when ATSv2 is turned off.
> >>> >>
> >>> >> Regards,
> >>> >> Varun Saxena.
> >>> >>
> >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >>> >> vrushalic2...@gmail.com> wrote:
> >>> >>
> >>> >>> Hi folks,
> >>> >>>
> >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> merge
> >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> >>> vote
> >>> >>> will
> >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>> >>>
> >>> >>> We have previously completed one merge onto trunk [3] and Timeline
> >>> Service
> >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >>>
> >>> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> >>> Service v2 in a feature branch [2] for a while, and we are
> reasonably
> >>> >>> confident that the state of the feature meets the criteria to be
> >>> merged
> >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> >>> capacity
> >>> >>> and provide valuable feedback so that we can make it
> >>> production-ready.
> >>> >>>
> >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> scalability
> >>> and
> >>> >>> usability improvements based on a new architecture. What we would
> >>> like to
> >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> >>> complete end-to-end read/write flow with security and read level
> >>> >>> authorization via whitelists. You should be able to start setting
> it
> >>> up
> >>> >>> and
> >>> >>> testing it.
> >>> >>>
> >>> >>> At a high level, the following are the key features that have been
> >>> >>> implemented since alpha1:
> >>> >>> - Security via Kerberos Authentication and delegation tokens
> >>> >>> - Read side simple authorization via whitelist
> >>> >>> - Client configurable entity sort ordering
> >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> >>> metrics by
> >>> >>> timerange, pagination, sub-app entities
> >>> >>> - Support for storing sub-application entities (entities that exist
> >>> >>> outside
> >>> >>> the scope of an application)
> >>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> >>> prefixes,
> >>> >>> c

Re: Branch merges and 3.0.0-beta1 scope

[Ravi Prakash]

Also, when people +1 a merge, can they please describe if they did testing
/ use the feature in addition to what is already described in the thread?

On Wed, Aug 23, 2017 at 11:18 AM, Vrushali Channapattan <
vrushalic2...@gmail.com> wrote:

> For timeline service v2, we have completed all subtasks under YARN-5355
> [1].
>
> We initiated a merge-to-trunk vote [2] yesterday.
>
> thanks
> Vrushali
> [1] https://issues.apache.org/jira/browse/YARN-5355
> [2]
> http://mail-archives.apache.org/mod_mbox/hadoop-common-
> dev/201708.mbox/%3CCAE=b_fbLT2J+Ezb4wqdN_UwBiG1Sd5kpqGaw+9Br__zou5yNTQ@
> mail.gmail.com%3E
>
>
> On Wed, Aug 23, 2017 at 11:12 AM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
> > Agreed. I was very clearly not advocating for rushing in features. If you
> > have followed my past emails, I have only strongly advocated features be
> > worked in branches and get merged when they are in a reasonable state.
> >
> > Each branch contributor group should look at their readiness and merge
> > stuff in assuming that the branch reached a satisfactory state. That’s
> it.
> >
> > From release management perspective, blocking features just because we
> are
> > a month close to the deadline is not reasonable. Let the branch
> > contributors rationalize, make this decision and the rest of us can
> support
> > them in making the decision.
> >
> > +Vinod
> >
> > > At this point, there have been three planned alphas from September 2016
> > until July 2017 to "get in features".  While a couple of upcoming
> features
> > are "a few weeks" away, I think all of us are aware how predictable
> > software development schedules can be.  I think we can also all agree
> that
> > rushing just to meet a release deadline isn't the best practice when it
> > comes to software development either.
> > >
> > > Andrew has been very clear about his goals at each step and I think
> > Wangda's willingness to not rush in resource types was an appropriate
> > response.  I'm sympathetic to the goals of getting in a feature for 3.0,
> > but it might be a good idea for each project that is a "few weeks away"
> to
> > seriously look at the readiness compared to the features which have been
> > testing for 6+ months already.
> > >
> > > -Ray
> >
> >
>

Re: Are binary artifacts are part of a release?

[Ravi Prakash]

bq. My stance is that if we're going to publish something, it should be
good, or we shouldn't publish it at all.

I agree

On Tue, Aug 15, 2017 at 2:57 AM, Steve Loughran 
wrote:

>
> > On 15 Aug 2017, at 07:14, Andrew Wang  wrote:
> >
> > To close the thread on this, I'll try to summarize the LEGAL JIRA. I
> wasn't
> > able to convince anyone to make changes to the apache.org docs.
> >
> > Convenience binary artifacts are not official release artifacts and thus
> > are not voted on. However, since they are distributed by Apache, they are
> > still subject to the same distribution requirements as official release
> > artifacts. This means they need to have a LICENSE and NOTICE file, follow
> > ASF licensing rules, etc. The PMC needs to ensure that binary artifacts
> > meet these requirements.
> >
> > However, being a "convenience" artifact doesn't mean it isn't important.
> > The appropriate level of quality for binary artifacts is left up to the
> > project. An OpenOffice person mentioned the quality of their binary
> > artifacts is super important since very few of their users will compile
> > their own office suite.
> >
> > I don't know if we've discussed the topic of binary artifact quality in
> > Hadoop. My stance is that if we're going to publish something, it should
> be
> > good, or we shouldn't publish it at all. I think we do want to publish
> > binary tarballs (it's the easiest way for new users to get started with
> > Hadoop), so it's fair to consider them when evaluating a release.
> >
> > Best,
> > Andrew
> >
>
>
> Given we publish the artifacts to the m2 repo, which is very much a
> downstream distribution mechanism. For other redist mechanisms (yum,
> apt-get) its implicitly handled by whoever manages those repos.
>
> > On Mon, Jul 31, 2017 at 8:43 PM, Konstantin Shvachko <
> shv.had...@gmail.com>
> > wrote:
> >
> >> It does not. Just adding historical references, as Andrew raised the
> >> question.
> >>
> >> On Mon, Jul 31, 2017 at 7:38 PM, Allen Wittenauer <
> >> a...@effectivemachines.com> wrote:
> >>
> >>>
> >>> ... that doesn't contradict anything I said.
> >>>
>  On Jul 31, 2017, at 7:23 PM, Konstantin Shvachko <
> shv.had...@gmail.com>
> >>> wrote:
> 
>  The issue was discussed on several occasions in the past.
>  Took me a while to dig this out as an example:
>  http://mail-archives.apache.org/mod_mbox/hadoop-general/2011
> >>> 11.mbox/%3C4EB0827C.6040204%40apache.org%3E
> 
>  Doug Cutting:
>  "Folks should not primarily evaluate binaries when voting. The ASF
> >>> primarily produces and publishes source-code
>  so voting artifacts should be optimized for evaluation of that."
> 
>  Thanks,
>  --Konst
> 
>  On Mon, Jul 31, 2017 at 4:51 PM, Allen Wittenauer <
> >>> a...@effectivemachines.com> wrote:
> 
> > On Jul 31, 2017, at 4:18 PM, Andrew Wang 
> >>> wrote:
> >
> > Forking this off to not distract from release activities.
> >
> > I filed https://issues.apache.org/jira/browse/LEGAL-323 to get
> >>> clarity on the matter. I read the entire webpage, and it could be
> improved
> >>> one way or the other.
> 
> 
> IANAL, my read has always lead me to believe:
> 
> * An artifact is anything that is uploaded to dist.a.o
> >>> and repository.a.o
> * A release consists of one or more artifacts
> >>> ("Releases are, by definition, anything that is published beyond the
> group
> >>> that owns it. In our case, that means any publication outside the
> group of
> >>> people on the product dev list.")
> * One of those artifacts MUST be source
> * (insert voting rules here)
> * They must be built on a machine in control of the RM
> * There are no exceptions for alpha, nightly, etc
> * (various other requirements)
> 
> i.e., release != artifact  it's more like release =
> >>> artifact * n .
> 
> Do you have to have binaries?  No (e.g., Apache SpamAssassin
> >>> has no binaries to create).  But if you place binaries in dist.a.o or
> >>> repository.a.o, they are effectively part of your release and must
> follow
> >>> the same rules.  (Votes, etc.)
> 
> 
> >>>
> >>>
> >>
>
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Re: Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Ravi Prakash]

It would have been nice if MSDN had a windows image that I could spin up on
my favorite Linux distro :-) But I haven't been able to find it yet :(

On Thu, Aug 3, 2017 at 3:23 AM, Steve Loughran 
wrote:

> wow, a lot of failures.
>
> some appear to be test timeouts, some native library availability (
> https://builds.apache.org/job/hadoop-trunk-win/146/
> testReport/junit/org.apache.hadoop.util/TestNativeCodeLoader/
> testNativeCodeLoaded/), but I think many (most?) are due to them getting
> paths wrong once there's a drive prefix there, e.g.
> https://builds.apache.org/job/hadoop-trunk-win/146/
> testReport/junit/org.apache.hadoop.fs/TestFsShellList/testList/
>
> including those which think that file:// + file.getAbsolutePath is a valid
> URI
>
> https://builds.apache.org/job/hadoop-trunk-win/146/
> testReport/junit/org.apache.hadoop.conf/TestConfiguration/
>
> I think the way to deal with this is going to be some form of bug bash.
> Who out there wants to spend a day doing windows builds? It's not *too*
> bad...setting up the VM is the hard part.
>
>
> On 3 Aug 2017, at 05:45, Allen Wittenauer  mailto:a...@effectivemachines.com>> wrote:
>
> For more details, see https://builds.apache.org/job/hadoop-trunk-win/146/
>
> [Aug 2, 2017 4:25:19 PM] (yufei) YARN-6895. [FairScheduler] Preemption
> reservation may cause regular
>
>
>
>
> -1 overall
>
>
> The following subsystems voted -1:
>   unit
>
>
> The following subsystems are considered long running:
> (runtime bigger than 1h 00m 00s)
>   unit
>
>
> Specific tests:
>
>   Failed CTEST tests :
>
>  test_test_libhdfs_threaded_hdfs_static
>
>   Failed junit tests :
>
>  hadoop.conf.TestConfiguration
>  hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec
>  hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
>  hadoop.fs.TestFsShellCopy
>  hadoop.fs.TestFsShellList
>  hadoop.fs.TestLocalFileSystem
>  hadoop.io.nativeio.TestNativeIO
>  hadoop.io.TestSequenceFileAppend
>  hadoop.ipc.TestIPC
>  hadoop.ipc.TestSocketFactory
>  hadoop.metrics2.impl.TestStatsDMetrics
>  hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal
>  hadoop.security.TestShellBasedUnixGroupsMapping
>  hadoop.security.token.TestDtUtilShell
>  hadoop.util.TestNativeCodeLoader
>  hadoop.fs.TestWebHdfsFileContextMainOperations
>  hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy
>  hadoop.hdfs.qjournal.client.TestQuorumJournalManager
>  hadoop.hdfs.security.TestDelegationTokenForProxyUser
>  hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
>  hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
>  hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
>  hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
>  hadoop.hdfs.server.datanode.fsdataset.impl.
> TestLazyPersistLockedMemory
>  hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy
>  hadoop.hdfs.server.datanode.fsdataset.impl.
> TestLazyPersistReplicaPlacement
>  hadoop.hdfs.server.datanode.fsdataset.impl.
> TestLazyPersistReplicaRecovery
>  hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
>  hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
>  hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage
>  hadoop.hdfs.server.datanode.TestBlockScanner
>  hadoop.hdfs.server.datanode.TestDataNodeFaultInjector
>  hadoop.hdfs.server.datanode.TestDataNodeUUID
>  hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
>  hadoop.hdfs.server.datanode.TestDirectoryScanner
>  hadoop.hdfs.server.datanode.TestHSync
>  hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame
>  hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand
>  hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC
>  hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
>  hadoop.hdfs.server.namenode.ha.TestHAAppend
>  hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
>  hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
>  hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
>  hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands
>  hadoop.hdfs.server.namenode.TestEditLogRace
>  hadoop.hdfs.server.namenode.TestFileTruncate
>  hadoop.hdfs.server.namenode.TestFsck
>  hadoop.hdfs.server.namenode.TestNameNodeMXBean
>  hadoop.hdfs.server.namenode.TestNestedEncryptionZones
>  hadoop.hdfs.server.namenode.TestQuotaByStorageType
>  hadoop.hdfs.server.namenode.TestStartup
>  hadoop.hdfs.TestBlockStoragePolicy
>  hadoop.hdfs.TestDatanodeRegistration
>  hadoop.hdfs.TestDFSOutputStream
>  hadoop.hdfs.TestDFSShell
>  hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
>  hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
>  hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
>  hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
>  hadoop.hdfs.T

Re: [DISCUSS]: re-enable listing of secrets in S3x URIs

[Ravi Prakash]

On Thu, Aug 3, 2017 at 10:32 AM, Steve Loughran 
wrote:

> If there are places where this is still happening, I'd like to know -an
> email with the logs is fine. I know you can't stop it if you put the
> secrets inline, but if you do the secrets properly, they should never be
> visible. (which doesn't help diagnostics, not at all)
> If we agree to cutting support completely in 3.0, we could change the
> message in 2.8.2+ to "this will go away" and include a wiki link to how to
> prepare for this
>

Some of our users do use URLs with the credentials inline. Some places I
have seen it are in the distcp logs and the Oozie console . Should we stop
printing the URLs or perhaps stripping the secrets at all those end-points?
I think using credential caches are a lot better.

I agree with cutting support for it completely in 3.0 and your suggestion
for the wiki link. Unless someone has better ideas?

Cheers
Ravi

Re: [DISCUSS]: re-enable listing of secrets in S3x URIs

[Ravi Prakash]

Thanks for starting the discussion Steve.

This is a prickly issue and unfortunately we are hostages of past
decisions. Thanks a lot for attacking the problem in the first place and
sticking with it.

In my experience we have found a lot of places that AWS secrets were logged
for everyone to see. I'm not sure allowing people to do that is the right
thing to do in the long-term. We have to bite the bullet sometime. Perhaps
we should do that in trunk (3.0.0)? To unbreak clients of Hadoop-2.x we can
go with Vinayakumar's proposal but only in branch-2. Ofcourse technically
we have hadoop-2.8.0 already out with this, but I agree we can put the fix
in 2.8.2.

$0.02
Ravi

On Wed, Aug 2, 2017 at 5:52 AM, Steve Loughran 
wrote:

>
>
> HADOOP-3733 stripped
> out the user:password secret from the s3., s3a, s3n URLs for security
> grounds: everything logged Path entries without ever considering that they
> contained secret credentials.
>
> but that turns out to break things, as noted in HADOOP-14439  ...you can't
> any more go Path -> String -> Path without authentication details being
> lost, and of course, guess how paths are often marshalled around? As
> strings (after all, they weren't serializable until recently)
>
> Vinayakumar has proposed a patch reinstating retaining the secrets, at
> least enough for distcp
>
> https://issues.apache.org/jira/browse/HADOOP-3733?focusedCom
> mentId=16110297&page=com.atlassian.jira.plugin.system.
> issuetabpanels:comment-tabpanel#comment-16110297
>
> I think I'm going to go with this, once I get the tests & testing to go
> with, and if its enough to work with spark too .. targeting 2.8.2 if its
> not too late.
>
> If there's a risk, it's that if someone puts secrets into s3 URIs, the
> secrets are more likely to be logged. But even with the current code,
> there's no way to guarantee that the secrets will never be logged. The
> danger comes from having id:secret credentials in the URI —something people
> will be told off for doing.
>
>
>

[jira] [Created] (HADOOP-14682) cmake Makefiles in hadoop-common don't properly respect -Dopenssl.prefix

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-14682:
-

 Summary: cmake Makefiles in hadoop-common don't properly respect 
-Dopenssl.prefix
 Key: HADOOP-14682
 URL: https://issues.apache.org/jira/browse/HADOOP-14682
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ravi Prakash


Allen reported that while running tests, cmake didn't properly respect 
-Dopenssl.prefix that would allow us to build and run the tests with different 
versions of OpenSSL.
https://issues.apache.org/jira/browse/HADOOP-14597?focusedCommentId=16092114&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16092114

I too encountered some funny stuff while trying to build with a non-default 
OpenSSL library. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14597) Native compilation broken with OpenSSL-1.1.0 because EVP_CIPHER_CTX has been made opaque

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-14597:
-

 Summary: Native compilation broken with OpenSSL-1.1.0 because 
EVP_CIPHER_CTX has been made opaque
 Key: HADOOP-14597
 URL: https://issues.apache.org/jira/browse/HADOOP-14597
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.0-alpha4
 Environment: openssl-1.1.0
Reporter: Ravi Prakash


Trying to build Hadoop trunk on Fedora 26 which has openssl-devel-1.1.0 fails 
with this error
{code}[WARNING] 
/home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:
 In function ‘check_update_max_output_len’:
[WARNING] 
/home/raviprak/Code/hadoop/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14:
 error: dereferencing pointer to incomplete type ‘EVP_CIPHER_CTX {aka struct 
evp_cipher_ctx_st}’
[WARNING]if (context->flags & EVP_CIPH_NO_PADDING) {
[WARNING]   ^~
{code}

https://github.com/openssl/openssl/issues/962 mattcaswell says
{quote}
One of the primary differences between master (OpenSSL 1.1.0) and the 1.0.2 
version is that many types have been made opaque, i.e. applications are no 
longer allowed to look inside the internals of the structures
{quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14513) A little performance improvement of HarFileSystem

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-14513.
---
Resolution: Not A Problem

> A little performance improvement of HarFileSystem
> -
>
> Key: HADOOP-14513
> URL: https://issues.apache.org/jira/browse/HADOOP-14513
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.0.0-alpha3
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Trivial
> Attachments: HADOOP-14513.001.patch
>
>
> In the Java source of HarFileSystem.java:
> {code:title=HarFileSystem.java|borderStyle=solid}
> ...
> ...
> private Path archivePath(Path p) {
> Path retPath = null;
> Path tmp = p;
> 
> // I think p.depth() need not be loop many times, depth() is a complex 
> calculation
> for (int i=0; i< p.depth(); i++) {
>   if (tmp.toString().endsWith(".har")) {
> retPath = tmp;
> break;
>   }
>   tmp = tmp.getParent();
> }
> return retPath;
>   }
> ...
> ...
> {code}
>  
> I think the fellow is more suitable:
> {code:title=HarFileSystem.java|borderStyle=solid}
> ...
> ...
> private Path archivePath(Path p) {
> Path retPath = null;
> Path tmp = p;
> 
> // just loop once
> for (int i=0,depth=p.depth(); i< depth; i++) {
>   if (tmp.toString().endsWith(".har")) {
> retPath = tmp;
> break;
>   }
>   tmp = tmp.getParent();
> }
> return retPath;
>   }
> ...
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14319) Under replicated blocks are not getting re-replicated

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-14319.
---
Resolution: Invalid

Please send your queries to hdfs-user mailing list. 
https://hadoop.apache.org/mailing_lists.html
To answer your query please look at dfs.namenode.replication.max-streams , 
dfs.namenode.replication.max-streams-hard-limit,  
dfs.namenode.replication.work.multiplier.per.iteration etc.


> Under replicated blocks are not getting re-replicated
> -
>
> Key: HADOOP-14319
> URL: https://issues.apache.org/jira/browse/HADOOP-14319
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Anil
>
> Under replicated blocks are not getting re-replicated
> In production Hadoop cluster of 5 Manangement + 5 Data Nodes, under 
> replicated blocks are not re-replicated even after 2 days. 
> Here is quick view of required configurations;
>  Default replication factor:  3
>  Average block replication:   3.0
>  Corrupt blocks:  0
>  Missing replicas:0 (0.0 %)
>  Number of data-nodes:5
>  Number of racks: 1
> After bringing one of the DataNodes down, the replication factor for the 
> blocks allocated on the Data Node became 2. It is observed that, even after 2 
> days the replication factor remains as 2. Under replicated blocks are not 
> getting re-replicated to another DataNodes in the cluster. 
> If a Data Node goes down, HDFS will try to replicate the blocks from Dead DN 
> to other nodes and the priority. Are there any configuration changes to speed 
> up the re-replication process for the under replicated blocks? 
> When tested for blocks with replication factor 1, the re-replication happened 
> to 2 overnight in around 10 hours of time. But blocks with 2 replication 
> factor are not being re-replicated to default replication factor 3. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Ravi Prakash]

Thanks for all the effort Junping!

+1 (binding)
+ Verified signature and MD5, SHA1, SHA256 checksum of tarball
+ Verified SHA ID in git corresponds to RC3 tag
+ Verified wordcount for one small text file produces same output as
hadoop-2.7.3.
+ HDFS Namenode UI looks good.

I agree none of the issues reported so far are blockers. Looking forward to
another great release.

Thanks
Ravi

On Tue, Mar 21, 2017 at 8:10 PM, Junping Du  wrote:

> Thanks all for response with verification work and vote!
>
>
> Sounds like we are hitting several issues here, although none seems to be
> blockers so far. Given the large commit set - 2000+ commits first landed in
> branch-2 release, we may should follow 2.7.0 practice that to claim this
> release is not for production cluster, just like Vinod's suggestion in
> previous email. We should quickly come up with 2.8.1 release in next 1 or 2
> month for production deployment.
>
>
> We will close the vote in next 24 hours. For people who haven't vote,
> please keep on verification work and report any issues if founded - I will
> check if another round of RC is needed based on your findings. Thanks!
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Kuhu Shukla 
> Sent: Tuesday, March 21, 2017 3:17 PM
> Cc: Junping Du; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
>
>
> +1 (non-binding)
>
> - Verified signatures.
> - Downloaded and built from source tar.gz.
> - Deployed a pseudo-distributed cluster on Mac Sierra.
> - Ran example Sleep job successfully.
> - Deployed latest Apache Tez 0.9 and ran sample Tez orderedwordcount
> successfully.
>
> Thank you Junping and everyone else who worked on getting this release out.
>
> Warm Regards,
> Kuhu
> On Tuesday, March 21, 2017, 3:42:46 PM CDT, Eric Badger
>  wrote:
> +1 (non-binding)
>
> - Verified checksums and signatures of all files
> - Built from source on MacOS Sierra via JDK 1.8.0 u65
> - Deployed single-node cluster
> - Successfully ran a few sample jobs
>
> Thanks,
>
> Eric
>
> On Tuesday, March 21, 2017 2:56 PM, John Zhuge 
> wrote:
>
>
>
> +1. Thanks for the great effort, Junping!
>
>
>   - Verified checksums and signatures of the tarballs
>   - Built source code with Java 1.8.0_66-b17 on Mac OS X 10.12.3
>   - Built source and native code with Java 1.8.0_111 on Centos 7.2.1511
>   - Cloud connectors:
>   - s3a: integration tests, basic fs commands
>   - adl: live unit tests, basic fs commands. See notes below.
>   - Deployed a pseudo cluster, passed the following sanity tests in
>   both insecure and SSL mode:
>   - HDFS: basic dfs, distcp, ACL commands
>   - KMS and HttpFS: basic tests
>   - MapReduce wordcount
>   - balancer start/stop
>
>
> Needs the following JIRAs to pass all ADL tests:
>
>   - HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge.
>   - HDFS-11132. Allow AccessControlException in contract tests when
>   getFileStatus on subdirectory of existing files. Contributed by
> Vishwajeet
>   Dusane
>   - HADOOP-13928. TestAdlFileContextMainOperationsLive.testGetFileContext1
>   runtime error. (John Zhuge via lei)
>
>
> On Mon, Mar 20, 2017 at 10:31 AM, John Zhuge  wrote:
>
> > Yes, it only affects ADL. There is a workaround of adding these 2
> > properties to core-site.xml:
> >
> >  
> >fs.adl.impl
> >org.apache.hadoop.fs.adl.AdlFileSystem
> >  
> >
> >  
> >fs.AbstractFileSystem.adl.impl
> >org.apache.hadoop.fs.adl.Adl
> >  
> >
> > I have the initial patch ready but hitting these live unit test failures:
> >
> > Failed tests:
> >
> > TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> > testListStatus:257
> > expected:<1> but was:<10>
> >
> > Tests in error:
> >
> > TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.
> > testMkdirsFailsForSubdirectoryOfExistingFile:254
> > » AccessControl
> >
> > TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
> > testMkdirsFailsForSubdirectoryOfExistingFile:190
> > » AccessControl
> >
> >
> > Stay tuned...
> >
> > John Zhuge
> > Software Engineer, Cloudera
> >
> > On Mon, Mar 20, 2017 at 10:02 AM, Junping Du 
> wrote:
> >
> > > Thank you for reporting the issue, John! Does this issue only affect
> ADL
> > > (Azure Data Lake) which is a new feature for 2.8 rather than other
> > existing
> > > FS? If so, I think we can leave the fix to 2.8.1 to fix given this is
> > not a
> > > regression and just a new feature get broken.?
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Junping
> > > --
> > > *From:* John Zhuge 
> > > *Sent:* Monday, March 20, 2017 9:07 AM
> > > *To:* Junping Du
> > > *Cc:* common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > > yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> > > *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
> > >
> > > 

[jira] [Resolved] (HADOOP-11232) jersey-core-1.9 has a faulty glassfish-repo setting

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-11232.
---
Resolution: Duplicate

HADOOP-9613 seems to have upgraded jersey to 1.19 . Please reopen if I'm 
mistaken

> jersey-core-1.9 has a faulty glassfish-repo setting
> ---
>
> Key: HADOOP-11232
> URL: https://issues.apache.org/jira/browse/HADOOP-11232
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Sushanth Sowmyan
>
> The following was reported by [~sushanth].
> hadoop-common brings in jersey-core-1.9 as a dependency by default.
> This is problematic, since the pom file for jersey 1.9 hardcode-specifies 
> glassfish-repo as the place to get further transitive dependencies, which 
> leads to a site that serves a static "this has moved" page instead of a 404. 
> This results in faulty parent resolutions, which when asked for a pom file, 
> get erroneous results.
> The only way around this seems to be to add a series of exclusions for 
> jersey-core, jersey-json, jersey-server and a bunch of others to 
> hadoop-common, then to hadoop-hdfs, then to hadoop-mapreduce-client-core. I 
> don't know how many more excludes are necessary before I can get this to work.
> If you update your jersey.version to 1.14, this faulty pom goes away. Please 
> either update that, or work with build infra to update our nexus pom for 
> jersey-1.9 so that it does not include the faulty glassfish repo.
> Another interesting note about this is that something changed yesterday 
> evening to cause this break in behaviour. We have not had this particular 
> problem in about 9+ months.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Commit log Pattern Unification

[Ravi Prakash]

And sometimes there are multiple contributors, so it becomes Contributed by
XX1, XX2 and XX3.

I guess having the information in git logs makes for easy grepping, awking
and counting ;-)

On Mon, Nov 7, 2016 at 11:35 AM, Andrew Wang 
wrote:

> I've always done d), but isn't this information captured in JIRA anyway?
>
> On Mon, Nov 7, 2016 at 11:29 AM, Ravi Prakash 
> wrote:
>
>> I have a preference for d) Contributed by XXX.
>>
>> Wouldn't signed-off require the commit to come from the contributor? What
>> about people who submit patch files? I thought that was still the modus
>> operandi, no?
>>
>> On Sun, Nov 6, 2016 at 8:18 PM, Daniel Templeton 
>> wrote:
>>
>> > On 11/6/16 8:01 PM, Daniel Templeton wrote:
>> >
>> >> It's also how the committer is included in the log by git.
>> >>
>> >
>> > OK, git actually shows name and email in the log.  It shows the username
>> > in the annotations in NetBeans, which is what I was thinking of. :)
>> >
>> >
>> > Daniel
>> >
>> > -
>> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >
>> >
>>
>
>

Re: [DISCUSS] Commit log Pattern Unification

[Ravi Prakash]

I have a preference for d) Contributed by XXX.

Wouldn't signed-off require the commit to come from the contributor? What
about people who submit patch files? I thought that was still the modus
operandi, no?

On Sun, Nov 6, 2016 at 8:18 PM, Daniel Templeton 
wrote:

> On 11/6/16 8:01 PM, Daniel Templeton wrote:
>
>> It's also how the committer is included in the log by git.
>>
>
> OK, git actually shows name and email in the log.  It shows the username
> in the annotations in NetBeans, which is what I was thinking of. :)
>
>
> Daniel
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Pre-commit build in Windows platform

[Ravi Prakash]

Was anybody ever successful in getting Windows(tm) installed in a VM from
the developer accounts that Microsoft affords Apache?

On Fri, Oct 28, 2016 at 4:44 AM, Steve Loughran 
wrote:

>
> On 28 Oct 2016, at 04:20, Brahma Reddy Battula <
> brahmareddy.batt...@huawei.com>
> wrote:
>
> Hi All
>
> As we supporting the Hadoop in windows, I feel, we should have pre-commit
> build in windows( atleast in qbt).
>
> We've never had it work too well with Windows precommit & Jenkins; maybe
> someone could volunteer to try again. What's important is not to overwhelm
> the JIRAs with noise that gets ignored.
>
>
>
> Background:
>
> As of now pre-commit will not run on windows, we may end up with following
>
>
> 1)  Test cases  can fail in windows
>
> 2)  Compilation Error (now it's failing, after HADOOP-10075)
>
>
> I've not had branch-3 building for a while, though maybe it's my Windows
> Server VM that's a mess. Maybe I should rm it and grab the limited-lifespan
> dev box, then add the extra Hadoop bits https://developer.microsoft.
> com/en-us/windows/downloads/virtual-machines
>
> 3)  Some feature might not work
>
> It will be good, if we catch all these at time of commit. (Prevention is
> better than cure..:) )
>
> I think a first step would be to have scheduled builds on windows; ideally
> with people caring that they are playing up.
>
> Anyway,
>
> +1 for more windows testing
>

Re: adding contributor roles timing out again

[Ravi Prakash]

I can attest that "Contributor  1" works. IMHO we should also go back and
trim down the contributor list of people who are no longer expected to
contribute. They can always be added back if things change.

On Wed, Sep 7, 2016 at 7:53 PM, Akira Ajisaka 
wrote:

> Infra team created "Contributors 1" role. If you cannot add people to
> "Contributors" role, use "Contributors 1" instead.
>
> https://issues.apache.org/jira/browse/INFRA-12487
>
> > I've created the "Contributors 1" role which you should be able to add
> people to. Same permissions as "Contributors." This applies to all of the
> projects using Hadoop Permissions (includes common and HDFS.)
>
> -Akira
>
>
> On 8/24/16 14:16, Akira Ajisaka wrote:
>
>> How about we try to do some grouping, where we just have sub groups,
>>> hadoop-contributors-1, hadoop-contributors-2, ... and add them as
>>> contributors; we then edit group membership, adding a new group when the
>>> current one gets above some chosen size limit?
>>>
>>
>> Agreed. Filed INFRA-12487.
>>
>> -Akira
>>
>> On 8/24/16 02:00, Chris Trezzo wrote:
>>
>>> Thanks Chris!
>>>
>>> On Mon, Aug 22, 2016 at 11:05 PM, Chris Nauroth
>>> 
>>> wrote:
>>>
>>> Chris, I have taken care of adding you in the Contributors role on the
 HADOOP project.



 --Chris Nauroth



 *From: *Chris Trezzo 
 *Date: *Monday, August 22, 2016 at 3:20 PM
 *To: *Weiqing Yang 
 *Cc: *Chris Nauroth , Steve Loughran <
 ste...@hortonworks.com>, "common-dev@hadoop.apache.org" <
 common-dev@hadoop.apache.org>
 *Subject: *Re: adding contributor roles timing out again



 Would it be possible for someone to add myself (username: ctrezzo)? It
 looks like I am not on the list and can not edit jiras in the HADOOP
 project. Thank you!



 On Mon, Aug 22, 2016 at 1:20 AM, Weiqing Yang 
 wrote:

 Thanks a lot, Chris and Steve!





 On 8/21/16, 6:38 AM, "Chris Nauroth"  wrote:

 I just took care of adding WeiqinYang.
>
> --Chris Nauroth
>
> On 8/21/16, 2:56 AM, "Steve Loughran"  wrote:
>
>
>> On 18 Aug 2016, at 16:39, Chris Nauroth  >
>
 wrote:

>>
>> It’s odd that Firefox didn’t work for you.  My standard workaround
>
 is to use Firefox, and that’s what I just did successfully for
 shenyinjie.

>>
>> It’s quite mysterious to me that this problem would be
>
 browser-specific at all though.

>>
>
>Could you add WeiqingYang
>
>it's not working for me in Chrome, FF or Safari from an OSX box
>
>It's clear that there are too many people in that contributor group.
>
 We have hit a scale limit in the hadoop coding process.

>
>How about we try to do some grouping, where we just have sub groups,
>
 hadoop-contributors-1, hadoop-contributors-2, ... and add them as
 contributors; we then edit group membership, adding a new group when the
 current one gets above some chosen size limit?

>
>



>>>
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: Apache MSDN Offer is Back

[Ravi Prakash]

Thanks Chris!

I did avail of the offer a few months ago, and wasn't able to figure out if
a windows license was also available. I want to run windows inside a
virtual machine on my Linux laptop, for the rare cases that there are
patches that may affect that. Any clue if that is possible?

Thanks
Ravi

On Tue, Jul 19, 2016 at 4:09 PM, Chris Nauroth 
wrote:

> A few months ago, we learned that the offer for ASF committers to get an
> MSDN license had gone away.  I'm happy to report that as of a few weeks
> ago, that offer is back in place.  For more details, committers can check
> out https://svn.apache.org/repos/private/committers and read
> donated-licenses/msdn.txt.
>
> --Chris Nauroth
>

Re: [DISCUSS] 2.6.x line releases

[Ravi Prakash]

We for one are not using 2.6.*

On Tue, Jul 19, 2016 at 1:21 PM, Sangjin Lee  wrote:

> It's been a while since we had a release on the 2.6.x line. Is it time to
> get ready for a 2.6.5 release? Are folks using and relying on releases on
> 2.6.x? If there is enough interest, I could take that on. Let me know.
>
> I also want to gauge the community's interest in maintaining the 2.6.x
> line. How long do we maintain this line? What would be a sensible EOL
> policy? Note that as the main code lines start diverging (java version,
> features, etc.), the cost of maintaining multiple release lines does go up.
> I'd love to hear your thoughts.
>
> Regards,
> Sangjin
>

Re: Feedback on IRC channel

[Ravi Prakash]

I've never gone there either. +1 for retiring.

On Wed, Jul 13, 2016 at 11:34 PM, J. Rottinghuis 
wrote:

> Uhm, there is an IRC channel?!?
>
> Joep
>
> On Wed, Jul 13, 2016 at 3:13 PM, Sangjin Lee  wrote:
>
> > I seldom check out IRC (as my experience was the same). I'm OK with
> > retiring it if no committers are around.
> >
> > On a related note, I know Tsuyoshi set up a slack channel for the
> > committers. Even that one is pretty idle. :) Should we use it more often?
> > If that starts to gain traction, we could set up a more open room for
> users
> > as well.
> >
> > Sangjin
> >
> > On Wed, Jul 13, 2016 at 9:13 AM, Karthik Kambatla 
> > wrote:
> >
> > > Recently, Andrew Wang and I were at an academic conference where one of
> > the
> > > attendees (a grad student) was mentioning that his posts to the IRC
> > channel
> > > are never answered.
> > >
> > > Personally, I haven't been using the IRC channel. Neither do I know
> > anyone
> > > who is actively monitoring it.
> > >
> > > I am emailing to check:
> > >
> > >1. Are there folks actively monitoring the IRC channel and answering
> > >questions?
> > >2. If there is no one, should we consider retiring the channel?
> > >
> > > Thanks
> > > Karthik
> > >
> >
>

Re: [DICUSS] Upgrading Guice to 4.0(HADOOP-12064)

[Ravi Prakash]

Go Go Go! Thanks for all the upgrade work Tsuyoshi!

On Thu, Jun 30, 2016 at 12:03 PM, Tsuyoshi Ozawa  wrote:

> Thanks, Andrew.
>
> Based on discussion here, I would like to merge it into *trunk* if
> there are no objection tomorrow.
>
> Thanks,
> - Tsuyoshi
>
> On Wed, Jun 29, 2016 at 12:28 PM, Andrew Wang 
> wrote:
> > I think it's okay to merge. We've already bumped other deps in trunk.
> >
> > On Wed, Jun 29, 2016 at 12:27 PM, Tsuyoshi Ozawa 
> wrote:
> >>
> >> I forgot to mention about importance point: it's a blocker issue to
> >> compile Hadoop with JDK8. Hence, we need to merge it on both client
> >> side and server slide anyway.
> >>
> >> Thanks,
> >> - Tsuyoshi
> >>
> >> On Wed, Jun 29, 2016 at 12:24 PM, Tsuyoshi Ozawa 
> wrote:
> >> > Thanks Vinod, Sangjin, Sean for your comment.
> >> >
> >> > Okay, I will take a look at the class path isolation.
> >> > Should I postpone to merge Guice upgrade to trunk? IMHO, it works with
> >> > tests, so it's okay to merge to runk. Thoughts?
> >> >
> >> > - Tsuyoshi
> >> >
> >> > On Wed, Jun 29, 2016 at 12:10 PM, Sangjin Lee 
> wrote:
> >> >> Yeah it would be awesome if we can get feedback and/or suggestions on
> >> >> these
> >> >> JIRAs (HADOOP-11804 and HADOOP-13070).
> >> >>
> >> >> Thanks,
> >> >> Sangjin
> >> >>
> >> >> On Wed, Jun 29, 2016 at 10:55 AM, Sean Busbey 
> >> >> wrote:
> >> >>>
> >> >>> At the very least, I'm running through an updated shaded hadoop
> client
> >> >>> this week[1] (HBase is my test application and it wandered onto some
> >> >>> private things that broke in branch-2). And Sangjin has a good lead
> on
> >> >>> an lower-short-term-cost incremental improvement for runtime
> isolation
> >> >>> of apps built on yarn/mapreduce[2]. He's been patiently waiting for
> >> >>> more review feedback.
> >> >>>
> >> >>>
> >> >>> [1]: https://issues.apache.org/jira/browse/HADOOP-11804
> >> >>> [2]: https://issues.apache.org/jira/browse/HADOOP-13070
> >> >>>
> >> >>> On Wed, Jun 29, 2016 at 12:33 PM, Vinod Kumar Vavilapalli
> >> >>>  wrote:
> >> >>> > My strong expectation is that we’ll have a version of classpath
> >> >>> > isolation in our first release of 3.x. I’m planning to spending
> some
> >> >>> > cycles
> >> >>> > right away on this.
> >> >>> >
> >> >>> > Assuming classpath isolation gets in, it is reasonable to bump up
> >> >>> > our
> >> >>> > dependencies like Jetty / Guice to the latest stable versions.
> >> >>> >
> >> >>> > Thanks
> >> >>> > +Vinod
> >> >>> >
> >> >>> >> On Jun 27, 2016, at 6:01 AM, Tsuyoshi Ozawa 
> >> >>> >> wrote:
> >> >>> >>
> >> >>> >> Hi developers,
> >> >>> >>
> >> >>> >> I will plan to upgrade Google Guice dependency on trunk. The
> change
> >> >>> >> also includes asm and cglib upgrade.
> >> >>> >> I checked following points:
> >> >>> >>
> >> >>> >> * Both HDFS and YARN UIs work well.
> >> >>> >> * All webIU-related tests pass as described on HADOOP-12064.
> >> >>> >> * Ran mapreduce job, and it works well.
> >> >>> >>
> >> >>> >> https://issues.apache.org/jira/browse/HADOOP-12064
> >> >>> >>
> >> >>> >> Do you have any concern or opinion?  I would like to merge it to
> >> >>> >> trunk
> >> >>> >> on this Friday if you have no objections.
> >> >>> >>
> >> >>> >> Best,
> >> >>> >> - Tsuyoshi
> >> >>> >>
> >> >>> >>
> >> >>> >>
> -
> >> >>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> >>> >> For additional commands, e-mail:
> common-dev-h...@hadoop.apache.org
> >> >>> >>
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> -
> >> >>> > To unsubscribe, e-mail:
> mapreduce-dev-unsubscr...@hadoop.apache.org
> >> >>> > For additional commands, e-mail:
> >> >>> > mapreduce-dev-h...@hadoop.apache.org
> >> >>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> busbey
> >> >>>
> >> >>>
> -
> >> >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >> >>>
> >> >>
> >>
> >> -
> >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >>
> >
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[jira] [Resolved] (HADOOP-13295) Possible Vulnerability in DataNodes via SSH

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-13295.
---
Resolution: Invalid

> Possible Vulnerability in DataNodes via SSH
> ---
>
> Key: HADOOP-13295
> URL: https://issues.apache.org/jira/browse/HADOOP-13295
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Mobin Ranjbar
>
> I suspected something weird in my Hadoop cluster. When I run datanodes, after 
> a while my servers(except namenode) will be down for SSH Max Attempts. When I 
> checked the 'systemctl status ssh', I figured out there are some invalid 
> username/password attempts via SSH and the SSH daemon blocked all incoming 
> connections and I got connection refused.
> I have no problem when my datanodes are not running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Why there are so many revert operations on trunk?

[Ravi Prakash]

Lolz!

Thanks for your opinion Larry. I have often seen "-1 until this is done
according to my way rather than your way" (obviously not in those words),
even when both ways are perfectly reasonable. Anyway, I didn't expect the
voting rules to change. :-)

Cheers
Ravi

On Tue, Jun 7, 2016 at 11:02 AM, larry mccay  wrote:

> -1 needs not be a taken as a derogatory statement being a number should
> actually make it less emotional.
> It is dangerous to a community to become oversensitive to it.
>
> I generally see language such as "I am -1 on this until this particular
> thing is fixed" or that it violates some common pattern or precedence set
> in the project. This is perfectly reasonable language and there is no
> reason to make the reviewer provide an alternative.
>
> So, I am giving my -1 to any proposal for rule changes on -1 votes. :)
>
>
> On Tue, Jun 7, 2016 at 1:15 PM, Ravi Prakash  wrote:
>
>> +1 on being more respectful. We seem to be having a lot of distasteful
>> discussions recently. If we fight each other, we are only helping our
>> competitors win (and trust me, its out there).
>>
>> I would also respectfully request people not to throw -1s around. I have
>> faced this a few times and its really frustrating. Every one has opinions
>> and some times different people can't fathom why someone else thinks the
>> way they do. I am pretty sure none of us is acting with malicious intent,
>> so perhaps a little more tolerance, faith and trust will help all of us
>> improve Hadoop and the ecosystem much faster. That's not to say that
>> sometimes -1s are not warranted, but we should look to it as an extreme
>> measure. Unfortunately there is very little disincentive right now to vote
>> -1 . Maybe we should modify the rules. if you vote -1 , you have to
>> come up with an alternative implementation? (perhaps limit the amount of
>> time you have to the amount already spent in producing the patch that you
>> are against)?
>>
>> Just my 2 cents
>> Ravi
>>
>>
>> On Tue, Jun 7, 2016 at 7:54 AM, Junping Du  wrote:
>>
>> > - We need to at the least force a reset of expectations w.r.t how trunk
>> > and small / medium / incompatible changes there are treated. We should
>> hold
>> > off making a release off trunk before this gets fully discussed in the
>> > community and we all reach a consensus.
>> >
>> > +1. We should hold off any release work off trunk before we reach a
>> > consensus. Or more and more developing work/features could be affected
>> just
>> > like Larry mentioned.
>> >
>> >
>> > - Reverts (or revert and move to a feature-branch) shouldn’t have been
>> > unequivocally done without dropping a note / informing everyone /
>> building
>> > consensus.
>> >
>> > Agree. To revert commits from other committers, I think we need to: 1)
>> > provide technical evidence/reason that is solid as rack, like: break
>> > functionality, tests, API compatibility, or significantly offend code
>> > convention, etc. 2) Making consensus with related
>> contributors/committers
>> > based on these technical reasons/evidences. Unfortunately, I didn't see
>> we
>> > ever do either thing in this case.
>> >
>> >
>> > - Freaking out on -1’s and reverts - we as a community need to be less
>> > stigmatic about -1s / reverts.
>> >
>> > +1. As a community, I believe we all prefer to work in a more friendly
>> > environment. In many cases, -1 without solid reason will frustrate
>> people
>> > who are doing contributions. I think we should restraint our -1 unless
>> it
>> > is really necessary.
>> >
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Junping
>> >
>> >
>> > 
>> > From: Vinod Kumar Vavilapalli 
>> > Sent: Monday, June 06, 2016 9:36 PM
>> > To: Andrew Wang
>> > Cc: Junping Du; Aaron T. Myers; common-dev@hadoop.apache.org;
>> > hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
>> > yarn-...@hadoop.apache.org
>> > Subject: Re: Why there are so many revert operations on trunk?
>> >
>> > Folks,
>> >
>> > It is truly disappointing how we are escalating situations that can be
>> > resolved through basic communication.
>> >
>> > Things that shouldn’t have happened
>> > - After a few objections were raised, commits should have simply stopped
>> &g

Re: Why there are so many revert operations on trunk?

[Ravi Prakash]

+1 on being more respectful. We seem to be having a lot of distasteful
discussions recently. If we fight each other, we are only helping our
competitors win (and trust me, its out there).

I would also respectfully request people not to throw -1s around. I have
faced this a few times and its really frustrating. Every one has opinions
and some times different people can't fathom why someone else thinks the
way they do. I am pretty sure none of us is acting with malicious intent,
so perhaps a little more tolerance, faith and trust will help all of us
improve Hadoop and the ecosystem much faster. That's not to say that
sometimes -1s are not warranted, but we should look to it as an extreme
measure. Unfortunately there is very little disincentive right now to vote
-1 . Maybe we should modify the rules. if you vote -1 , you have to
come up with an alternative implementation? (perhaps limit the amount of
time you have to the amount already spent in producing the patch that you
are against)?

Just my 2 cents
Ravi


On Tue, Jun 7, 2016 at 7:54 AM, Junping Du  wrote:

> - We need to at the least force a reset of expectations w.r.t how trunk
> and small / medium / incompatible changes there are treated. We should hold
> off making a release off trunk before this gets fully discussed in the
> community and we all reach a consensus.
>
> +1. We should hold off any release work off trunk before we reach a
> consensus. Or more and more developing work/features could be affected just
> like Larry mentioned.
>
>
> - Reverts (or revert and move to a feature-branch) shouldn’t have been
> unequivocally done without dropping a note / informing everyone / building
> consensus.
>
> Agree. To revert commits from other committers, I think we need to: 1)
> provide technical evidence/reason that is solid as rack, like: break
> functionality, tests, API compatibility, or significantly offend code
> convention, etc. 2) Making consensus with related contributors/committers
> based on these technical reasons/evidences. Unfortunately, I didn't see we
> ever do either thing in this case.
>
>
> - Freaking out on -1’s and reverts - we as a community need to be less
> stigmatic about -1s / reverts.
>
> +1. As a community, I believe we all prefer to work in a more friendly
> environment. In many cases, -1 without solid reason will frustrate people
> who are doing contributions. I think we should restraint our -1 unless it
> is really necessary.
>
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Vinod Kumar Vavilapalli 
> Sent: Monday, June 06, 2016 9:36 PM
> To: Andrew Wang
> Cc: Junping Du; Aaron T. Myers; common-dev@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: Why there are so many revert operations on trunk?
>
> Folks,
>
> It is truly disappointing how we are escalating situations that can be
> resolved through basic communication.
>
> Things that shouldn’t have happened
> - After a few objections were raised, commits should have simply stopped
> before restarting again but only after consensus
> - Reverts (or revert and move to a feature-branch) shouldn’t have been
> unequivocally done without dropping a note / informing everyone / building
> consensus. And no, not even a release-manager gets this free pass. Not on
> branch-2, not on trunk, not anywhere.
> - Freaking out on -1’s and reverts - we as a community need to be less
> stigmatic about -1s / reverts.
>
> Trunk releases:
> This is the other important bit about huge difference of expectations
> between the two sides w.r.t trunk and branching. Till now, we’ve never made
> releases out of trunk. So in-progress features that people deemed to not
> need a feature branch could go into trunk without much trouble. Given that
> we are now making releases off trunk, I can see (a) the RM saying "no,
> don’t put in-progress stuff and (b) the contributors saying “no we don’t
> want the overhead of a branch”. I’ve raised related topics (but only
> focusing on incompatible changes) before -
> http://markmail.org/message/m6x73t6srlchywsn - but we never decided
> anything.
>
> We need to at the least force a reset of expectations w.r.t how trunk and
> small / medium / incompatible changes there are treated. We should hold off
> making a release off trunk before this gets fully discussed in the
> community and we all reach a consensus.
>
> * Without a user API, there's no way for people to use it, so not much
> advantage to having it in a release
>
> Since the code is separate and probably won't break any existing code, I
> won't -1 if you want to include this in a release without a user API, but
> again, I question the utility of including code that can't be used.
>
> Clearly, there are two sides to this argument. One side claims the absence
> of user-facing public / stable APIs, and that for all purposes this is
> dead-code for everyone other than the few early adopters who want to
> expe

Re: [DISCUSS] upgrade tomcat from 6.0.44 to 6.0.45 and use non apache.org mirror

[Ravi Prakash]

I'd filed https://issues.apache.org/jira/browse/HDFS-3135 an eon ago. Dunno
if it will be acted on... :(

On Mon, May 23, 2016 at 11:52 AM, Allen Wittenauer <
allenwittena...@yahoo.com.invalid> wrote:

> Some potential ways to fix this:
>
> a) Move away from tomcat to one of the other two? three? maybe
> even four? application servers that are already in the Apache Hadoop build
>
> b) Move to embedded tomcat so that the jars, etc, get downloaded
> as part of the maven build
>
> Either of these should allow for us to also replace the bootstrap
> code to use the standard Hadoop script bits and therefore give a much more
> familiar look-and-feel on the command line, log file location, log file
> naming, etc.
>
> c) Require tomcat be downloaded and pointed at separately from
> outside the build.  If it doesn’t exist, skip it.  (This has *HUGE*
> implications for jenkins testing, however…)
>
> d) Switch the maven build (SMOP?) to always grab “the latest” from
> a mirror and hope for the best
>
>
> > On May 23, 2016, at 11:35 AM, Colin McCabe  wrote:
> >
> > I think the Tomcat situation is concerning in a lot of ways.
> >
> > 1. We are downloading without authentication, using http rather than
> > https.
> > 2. We are downloading an obsolete release.
> > 3. Our build process is violating the apache.archive.org guidelines by
> > downloading from the site directly, rather than from a mirror.
> > 4. Clean builds take longer than necessary because this tar file needs
> > to be downloaded.
> >
> > I'm not too familiar with how Tomcat works... does anyone have any ideas
> > for fixing this?
> >
> > best,
> > Colin
> >
> >
> > On Tue, May 17, 2016, at 14:08, Wangda Tan wrote:
> >> Hi common-devs,
> >>
> >> When I tried to build Hadoop trunk today, Maven build failed because kms
> >> failed to download:
> >>
> http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.44/bin/apache-tomcat-6.0.44.tar.gz
> >> .
> >>
> >> When I open the link from browser, it shows bandwidth limit exceeded:
> >>
> >> ===
> >> Bandwidth limit exceeded
> >>
> >> The daily allowance of 5GB for this IP has been exceeded, and downloads
> >> disabled until midnight, UTC (circa 21 hours from now).
> >> If you have any questions about this, feel free to reach out to us at
> >> infrastruct...@apache.org.
> >> ===
> >>
> >> Then I investigated a little more, there're two issues from what I can
> >> see
> >> 1) If you look at http://archive.apache.org/dist/, it clearly says:
> >> *> Please do not download from apache.org !*
> >>
> >> 2) And also, http://archive.apache.org/dist/ says:
> >>> Older *non-recommended* releases can be found on our archive site
> >> .
> >>
> >> We're using tomcat with version=6.0.44, on other Apache mirror site,
> only
> >> 6.0.45 tomcat is downloadable. Which essentially means 6.0.44 is a
> >> non-recommended release.
> >>
> >> I would like to propose to make following changes:
> >> 1) Update tomcat download site from apache.org to another mirror.
> >> 2) Upgrade tomcat from 6.0.44 to 6.0.45.
> >>
> >> But would like to hear your thoughts before filing JIRA.
> >>
> >> Thanks,
> >> Wangda
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: ASF OS X Build Infrastructure

[Ravi Prakash]

FWIW, I was able to get a response from the form last month. I was issued a
new MSDN subscriber ID using which I could have downloaded Microsoft Visual
Studio (and some other products, I think). I was interested in downloading
an image of Windows to run in a VM, but the downloader is. wait for
it. an exe file :-) Haven't gotten around to begging someone with a
Windows OS to run that image downloader.

On Fri, May 20, 2016 at 10:39 AM, Sean Busbey  wrote:

> Some talk about the MSDN-for-committers program recently passed by on a
> private
> list. It's still active, it just changed homes within Microsoft. The
> info should still be in the committer repo. If something is amiss
> please let me know and I'll pipe up to the folks already plugged in to
> confirming it's active.
>
> On Fri, May 20, 2016 at 12:13 PM, Chris Nauroth
>  wrote:
> > It's very disappointing to see that vanish.  I'm following up to see if I
> > can learn more about what happened or if I can do anything to help
> > reinstate it.
> >
> > --Chris Nauroth
> >
> >
> >
> >
> > On 5/20/16, 6:11 AM, "Steve Loughran"  wrote:
> >
> >>
> >>> On 20 May 2016, at 10:40, Lars Francke  wrote:
> >>>
> 
>  Regarding lack of personal access to anything but Linux, I'll take
> this as
>  an opportunity to remind everyone that ASF committers (not just
> limited to
>  Hadoop committers) are entitled to a free MSDN license, which can get
> you
>  a Windows VM for validating Windows issues and any patches that touch
>  cross-platform concerns, like the native code.  Contributors who are
> not
>  committers still might struggle to get access to Windows, but all of
> us
>  reviewing and committing patches do have access.
> 
> >>>
> >>> Actually, from all I can tell this MSDN offer has been discontinued for
> >>> now. All the information has been removed from the committers repo. Do
> >>>you
> >>> have any more up to date information on this?
> >>>
> >>
> >>
> >>That's interesting.
> >>
> >>I did an SVN update and it went away..looks like something happened on
> >>April 26
> >>
> >>No idea, though the svn log has a bit of detail
> >>
> >>-
> >>To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> >>For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >>
> >>
> >
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
>
>
> --
> busbey
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Re: Compatibility guidelines for toString overrides

[Ravi Prakash]

Thanks sounds reasonable Colin. +1 to not using toString() as an API

On Thu, May 12, 2016 at 9:57 AM, Chris Nauroth 
wrote:

> I'm in favor of making a statement in the compatibility guidelines that
> there is no guarantee of stability or backwards-compatibility for
> toString() output.  If a proposed patch introduces new dependencies on a
> stable toString() output, such as for UI display or object serialization,
> then I'd consider -1'ing it and instead asking that the logic move to a
> different method that can provide that guarantee, i.e. toStableString().
> There are further comments justifying this on HADOOP-13028 and HDFS-9732.
>
> --Chris Nauroth
>
>
>
>
> On 5/12/16, 9:32 AM, "Colin McCabe"  wrote:
>
> >Hi all,
> >
> >Recently a discussion came up on HADOOP-13028 about the wisdom of
> >overloading S3AInputStream#toString to output statistics information.
> >It's a difficult judgement for me to make, since I'm not aware of any
> >compatibility guidelines for InputStream#toString.  Do we have
> >compatibility guidelines for toString functions?
> >
> >It seems like the output of toString functions is usually used as a
> >debugging aid, rather than as a stable format suitable for UI display or
> >object serialization.  Clearly, there are a few cases where we might
> >want to specifically declare that a toString method is a stable API.
> >However, I think if we attempt to treat the toString output of all
> >public classes as stable, we will have greatly increased the API
> >surface.  Should we formalize this and declare that toString functions
> >are @Unstable, Evolving unless declared otherwise?
> >
> >best,
> >Colin
> >
> >-
> >To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Merge Jersey upgrade(HADOOP-9613) to trunk

[Ravi Prakash]

+1. Awesome effort Tsuyoshi. This has been blocking compression on the wire
too.

On Tue, May 10, 2016 at 11:24 AM, Colin McCabe  wrote:

> +1 for updating this in trunk.  Thanks, Tsuyoshi Ozawa.
>
> cheers,
> Colin
>
> On Mon, May 9, 2016, at 12:12, Tsuyoshi Ozawa wrote:
> > Hi developers,
> >
> > We’ve worked on upgrading jersey(HADOOP-9613) for a years. It's
> > essential change to support complication with JDK8. It’s almost there.
> >
> > One concern to merge this to trunk is incompatibility. After the
> > release of Jersey 1.13, the root element whose content is empty
> > collection is changed from null to empty object({}).  Because of this
> > problem, I’ve marked HADOOP-9613 as incompatible change. Is it
> > acceptable for us? If it’s acceptable change in trunk, I’d like to
> > merge it into trunk.
> >
> > Thanks,
> > - Tsuyoshi
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Set minimum version of Hadoop 3 to JDK8 (HADOOP-11858)

[Ravi Prakash]

+1. Thanks for driving this Akira

On Tue, May 10, 2016 at 10:25 AM, Tsuyoshi Ozawa  wrote:

> > Before cutting 3.0.0-alpha RC, I'd like to drop JDK7 support in trunk.
>
> Sounds good. To do so, we need to check the blockers of 3.0.0-alpha
> RC, especially upgrading all dependencies which use refractions at
> first.
>
> Thanks,
> - Tsuyoshi
>
> On Tue, May 10, 2016 at 8:32 AM, Akira AJISAKA
>  wrote:
> > Hi developers,
> >
> > Before cutting 3.0.0-alpha RC, I'd like to drop JDK7 support in trunk.
> > Given this is a critical change, I'm thinking we should get the consensus
> > first.
> >
> > One concern I think is, when the minimum version is set to JDK8, we need
> to
> > configure Jenkins to disable multi JDK test only in trunk.
> >
> > Any thoughts?
> >
> > Thanks,
> > Akira
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

[jira] [Resolved] (HADOOP-12563) Updated utility to create/modify token files

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-12563.
---
Resolution: Fixed

> Updated utility to create/modify token files
> 
>
> Key: HADOOP-12563
> URL: https://issues.apache.org/jira/browse/HADOOP-12563
> Project: Hadoop Common
>  Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Matthew Paduano
> Fix For: 3.0.0
>
> Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch, 
> HADOOP-12563.03.patch, HADOOP-12563.04.patch, HADOOP-12563.05.patch, 
> HADOOP-12563.06.patch, HADOOP-12563.07.patch, HADOOP-12563.07.patch, 
> HADOOP-12563.08.patch, HADOOP-12563.09.patch, HADOOP-12563.10.patch, 
> HADOOP-12563.11.patch, HADOOP-12563.12.patch, HADOOP-12563.13.patch, 
> HADOOP-12563.14.patch, HADOOP-12563.15.patch, HADOOP-12563.16.patch, 
> dtutil-test-out, example_dtutil_commands_and_output.txt, 
> generalized_token_case.pdf
>
>
> hdfs fetchdt is missing some critical features and is geared almost 
> exclusively towards HDFS operations.  Additionally, the token files that are 
> created use Java serializations which are hard/impossible to deal with in 
> other languages. It should be replaced with a better utility in common that 
> can read/write protobuf-based token files, has enough flexibility to be used 
> with other services, and offers key functionality such as append and rename. 
> The old version file format should still be supported for backward 
> compatibility, but will be effectively deprecated.
> A follow-on JIRA will deprecrate fetchdt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-12563) Updated utility to create/modify token files

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash reopened HADOOP-12563:
---

> Updated utility to create/modify token files
> 
>
> Key: HADOOP-12563
> URL: https://issues.apache.org/jira/browse/HADOOP-12563
> Project: Hadoop Common
>  Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Matthew Paduano
> Fix For: 3.0.0
>
> Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch, 
> HADOOP-12563.03.patch, HADOOP-12563.04.patch, HADOOP-12563.05.patch, 
> HADOOP-12563.06.patch, HADOOP-12563.07.patch, HADOOP-12563.07.patch, 
> HADOOP-12563.08.patch, HADOOP-12563.09.patch, HADOOP-12563.10.patch, 
> HADOOP-12563.11.patch, HADOOP-12563.12.patch, HADOOP-12563.13.patch, 
> dtutil-test-out, example_dtutil_commands_and_output.txt, 
> generalized_token_case.pdf
>
>
> hdfs fetchdt is missing some critical features and is geared almost 
> exclusively towards HDFS operations.  Additionally, the token files that are 
> created use Java serializations which are hard/impossible to deal with in 
> other languages. It should be replaced with a better utility in common that 
> can read/write protobuf-based token files, has enough flexibility to be used 
> with other services, and offers key functionality such as append and rename. 
> The old version file format should still be supported for backward 
> compatibility, but will be effectively deprecated.
> A follow-on JIRA will deprecrate fetchdt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-13018) Make Kdiag fail fast if hadoop.token.files points to non-existent file

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-13018:
-

 Summary: Make Kdiag fail fast if hadoop.token.files points to 
non-existent file
 Key: HADOOP-13018
 URL: https://issues.apache.org/jira/browse/HADOOP-13018
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Ravi Prakash






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: CHANGES.txt is gone from trunk, branch-2, branch-2.8

[Ravi Prakash]

Yaayy!! +1

On Tue, Mar 8, 2016 at 10:59 AM, Colin P. McCabe  wrote:

> +1
>
> Thanks, Andrew.  This will avoid so many spurious conflicts when
> cherry-picking changes, and so much wasted time on commit.
>
> best,
> Colin
>
> On Thu, Mar 3, 2016 at 9:11 PM, Andrew Wang 
> wrote:
> > Hi all,
> >
> > With the inclusion of HADOOP-12651 going back to branch-2.8, CHANGES.txt
> > and release notes are now generated by Yetus. I've gone ahead and deleted
> > the manually updated CHANGES.txt from trunk, branch-2, and branch-2.8
> > (HADOOP-11792). Many thanks to Allen for the releasedocmaker.py rewrite,
> > and the Yetus integration.
> >
> > I'll go ahead and update the HowToCommit and HowToRelease wiki pages, but
> > at a high-level, this means we no longer need to edit CHANGES.txt on new
> > commit, streamlining our commit process. CHANGES.txt updates will still
> be
> > necessary for backports to older release lines like 2.6.x and 2.7.x.
> >
> > Happy committing!
> >
> > Best,
> > Andrew
>

Re: Looking to a Hadoop 3 release

[Ravi Prakash]

+1 for the plan to start cutting 3.x alpha releases. Thanks for the
initiative Andrew!

On Fri, Feb 19, 2016 at 6:19 AM, Steve Loughran 
wrote:

>
> > On 19 Feb 2016, at 11:27, Dmitry Sivachenko  wrote:
> >
> >
> >> On 19 Feb 2016, at 01:35, Andrew Wang  wrote:
> >>
> >> Hi all,
> >>
> >> Reviving this thread. I've seen renewed interest in a trunk release
> since
> >> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
> the
> >> shell script rewrite, and many other improvements, I think it's time to
> >> revisit Hadoop 3.0 release plans.
> >>
> >
>
> It's time to start ... I suspect it'll take a while to stabilise. I look
> forward to the new shell scripts already
>
> One thing I do want there is for all the alpha releases to make clear that
> there are no compatibility policies here; protocols may change and there is
> no requirement of the first 3.x release to be compatible with all the 3.0.x
> alphas. That's something we missed out on the 2.0.x-alpha process, or at
> least not repeated often enough.
>
> >
> > Hello,
> >
> > any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes
> out?
> >
> > Thanks!
> >
> >
>
> sounds like a good time for a status update on the FB work —and anything
> people can do to test it would be appreciated by all. That includes testing
> on ipv4 systems, and especially, IPv4/v6 systems with Kerberos turned on
> and both MIT and AD kerberos servers. At the same time, IPv6 support ought
> to be something that could be added in.
>
>
> I don't have any opinions on timescale, but
>
> +1 to anything related to classpath isolation
> +1 to a careful bump of versions of dependencies.
> +1 to fixing the outstanding Java 8 migration issues, especially the big
> Jersey patch that's just been updated.
> +1 to switching to JIRA-created release notes
>
> Having been doing the slider releases recently, it's clear to me that you
> can do a lot in automating the release process itself. All those steps in
> the release runbook can be turned into targets in a special ant release.xml
> build file, calling maven, gpg, etc.
>
> I think doing something like this for 3.0 will significantly benefit both
> the release phase here but the future releases
>
> This is the slider one:
> https://github.com/apache/incubator-slider/blob/develop/bin/release.xml
>
> It doesn't replace maven, instead it choreographs that along with all the
> other steps: signing and checksumming artifacts, publishing them, voting
>
> it includes
>  -refusing to release if the git repo is modified
>  -making the various git branch/tag/push operations
>  -issuing the various mvn versions:update commands
>  -signing
>  -publishing via asf SVN
>  -using GET calls too verify the artifacts made it
>  -generating the vote and vote result emails (it even counts the votes)
>
> I recommend this is included as part of the release process. It does make
> a difference; we can now cut new releases with no human intervention other
> than editing a properties file and running different targets as the process
> goes through its release and vote phases.
>
> -Steve

Re: [DISCUSS] fixing jenkins; policing the build

[Ravi Prakash]

With regard to the unit test failures, maybe it's time for another fix-it day?
+1
 


 On Monday, September 14, 2015 1:50 PM, Colin P. McCabe 
 wrote:
   

 I think building on Jenkins with jdk8 (even with source version = 1.7)
would help prevent the jdk8 javadoc failures from creeping in.

With regard to the unit test failures, maybe it's time for another fix-it day?

cheers,
Colin

On Sun, Sep 13, 2015 at 6:58 AM, Steve Loughran  wrote:
>
> Jenkins is pretty unhappy with the build ... its slowly been collecting 
> patched (including one i put in) triggering test failures, and we've all been 
> lax about fixing them. They're often pretty minor -as an example. trunk has 
> been breaking on java 8 with javadoc errors.
>
> I don't know what others think, but I personally think we need to be a lot 
> more focused on keeping the build happy. This is alongside the Yertus work: 
> that improves how we build; I'm more interested in the process of not 
> breaking the build —and addressing it when it does.
>
> If we were really ruthless, we'd be strict about reverting any patch that 
> triggers a failure. And maybe even halt all other commits until jenkins is 
> happy. That's pretty brutal, but it certainly gets people to care.
>
> A less ruthless but stricter-than-today policy could be
>
>
>  1.  Build failures are filed on JIRA @ critical or blocker. They need to 
>take priority.
>  2.  Patches that fix it get priority over everything else: don't be afraid 
>to ping people to keep that build up.
>  3.  We should recognise that trunk will be java8+ only, and even if don't 
>(yet) switch the source to being java8, the jenkins scheduled builds should go 
>to java8+. That way, we can't ignore java8+trunk failures, and don't have to 
>worry about java7 & trunk.
>  4.  Anyone who is a committer can get the login for 
>builds.apache.org to trigger rebuilds; It lets you 
>do a quick verify that the full builds are happy.
>
> Thoughts?

  

Re: [VOTE] Using rebase and merge for feature branch development

[Ravi Prakash]

+1. Thanks
 


 On Monday, August 24, 2015 3:02 PM, Jing Zhao  wrote:
   

 +1. Thanks.

On Mon, Aug 24, 2015 at 2:47 PM, Zhe Zhang  wrote:

> +1 non-binding. Thanks Andrew!
>
> ---
> Zhe Zhang
>
> On Mon, Aug 24, 2015 at 2:38 PM, Karthik Kambatla 
> wrote:
>
> > +1
> >
> > Thanks for driving this, Andrew.
> >
> > On Mon, Aug 24, 2015 at 11:00 AM, Vinayakumar B  >
> > wrote:
> >
> > > +1,
> > >
> > > -Vinay
> > > On Aug 24, 2015 11:29 PM, "Colin P. McCabe" 
> wrote:
> > >
> > > > +1
> > > >
> > > > cheers,
> > > > Colin
> > > >
> > > > On Mon, Aug 24, 2015 at 10:04 AM, Steve Loughran <
> > ste...@hortonworks.com
> > > >
> > > > wrote:
> > > > > +1 (binding)
> > > > >
> > > > >> On 21 Aug 2015, at 13:44, Andrew Wang 
> > > wrote:
> > > > >>
> > > > >> Hi common-dev,
> > > > >>
> > > > >> As promised, here is an official vote thread. Let's run it for the
> > > > standard
> > > > >> 7 days, closing on Aug 28th at noon. Only PMC members have binding
> > > > votes,
> > > > >> but of course everyone's input is welcomed.
> > > > >>
> > > > >> If the vote passes, I'll put the text on the website somewhere as
> > > > >> recommended by Steve.
> > > > >>
> > > > >> Previous discussion threads:
> > > > >>
> > > > >>
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201508.mbox/%3CCAGB5D2bPWeV2Hk%2B67%3DDamWpVfLTM6nkjb_wG3n4%3DWAN890zqfA%40mail.gmail.com%3E
> > > > >>
> > > > >>
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201508.mbox/%3CCAGB5D2aDXujQjwdmadVtg2-qrPAJeOgCS2_NHydv8jke8or1UA%40mail.gmail.com%3E
> > > > >>
> > > > >> Proposal:
> > > > >>
> > > > >> 
> > > > >>
> > > > >> Feature branch development can use either a merge or rebase
> > workflow,
> > > as
> > > > >> decided by contributors working on the branch.
> > > > >>
> > > > >> When using a rebase workflow, the feature branch is periodically
> > > > rebased on
> > > > >> trunk via "git rebase trunk" and force pushed.
> > > > >>
> > > > >> Before performing a force-push, a tag should be created of the
> > current
> > > > >> feature branch HEAD to preserve history. The tag should identify
> the
> > > > >> feature and date of most recent commit, e.g.
> > > > >> "tag_feature_HDFS-7285_2015-08-11". It can also be convenient to
> > use a
> > > > >> temporary branch to review rebase conflict resolution before
> > > > force-pushing
> > > > >> the main feature branch, e.g. HDFS-7285-rebase. Temporary branches
> > > > should
> > > > >> be deleted after they are force-pushed over the feature branch.
> > > > >>
> > > > >> Developers are allowed to squash and reorder commits to make
> > rebasing
> > > > >> easier. Use this judiciously. When squashing, please maintain the
> > > > original
> > > > >> commit messages in the squashed commit message to preserve
> history.
> > > > >>
> > > > >> When using a merge workflow, changes are periodically integrated
> > from
> > > > trunk
> > > > >> to the branch via "git merge trunk".
> > > > >>
> > > > >> Merge conflict resolution can be reviewed by posting the diff of
> the
> > > > merge
> > > > >> commit.
> > > > >>
> > > > >> For both rebase and merge workflows, integration of the branch
> into
> > > > trunk
> > > > >> should happen via "git merge --no-ff". "--no-ff" is important
> since
> > it
> > > > >> generates a merge commit even if the branch applies cleanly on top
> > of
> > > > >> trunk. This clearly denotes the set of commits that were made on
> the
> > > > >> branch, and makes it easier to revert the branch if necessary.
> > > > >>
> > > > >> "git merge --no-ff" is also the preferred way of integrating a
> > feature
> > > > >> branch to other branches, e.g. branch-2.
> > > > >>
> > > > >> 
> > > > >>
> > > > >> Thanks,
> > > > >> Andrew
> > > > >
> > > >
> > >
> >
>


   

Re: [DISCUSS] Proposal for allowing merge workflows on feature branches

[Ravi Prakash]

Thanks for the work Andrew! Should we specify a "preference" for one workflow 
over another? If not, this looks good.
 


 On Wednesday, August 19, 2015 12:04 PM, Colin McCabe 
 wrote:
   

 +1.  Rebasing can really make the history much clearer when used correctly.

Colin

On Tue, Aug 18, 2015 at 2:57 PM, Andrew Wang  wrote:
> Hi common-dev,
>
> Based on the prior [DISCUSS] thread, I've put together a new [VOTE]
> proposal which modifies the branch development practices edified by the
> [VOTE] when we switched from SVN to git [1]. This new proposal modifies the
> third and fourth points of the earlier [VOTE], copied here for your
> convenience:
>
> 3. Force-push on feature-branches is allowed. Before pulling in a feature,
> the feature-branch should be rebased on latest trunk and the changes
> applied to trunk through "git rebase --onto" or "git cherry-pick
> ".
>
> 4. Every time a feature branch is rebased on trunk, a tag that identifies
> the state before the rebase needs to be created (e.g.
>  tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once
> the feature is pulled into trunk and the tags are no longer useful.
>
> Said new proposal expands and modifies as follows:
>
> 
>
> Feature branch development can use either a merge or rebase workflow, as
> decided by contributors working on the branch.
>
> When using a rebase workflow, the feature branch is periodically rebased on
> trunk via "git rebase trunk" and force pushed.
>
> Before performing a force-push, a tag should be created of the current
> feature branch HEAD to preserve history. The tag should identify the
> feature and date of most recent commit, e.g.
> "tag_feature_HDFS-7285_2015-08-11". It can also be convenient to use a
> temporary branch to review rebase conflict resolution before force-pushing
> the main feature branch, e.g. HDFS-7285-rebase. Temporary branches should
> be deleted after they are force-pushed over the feature branch.
>
> Developers are allowed to squash and reorder commits to make rebasing
> easier. Use this judiciously. When squashing, please maintain the original
> commit messages in the squashed commit message to preserve history.
>
> When using a merge workflow, changes are periodically integrated from trunk
> to the branch via "git merge trunk".
>
> Merge conflict resolution can be reviewed by posting the diff of the merge
> commit.
>
> For both rebase and merge workflows, integration of the branch into trunk
> should happen via "git merge --no-ff". "--no-ff" is important since it
> generates a merge commit even if the branch applies cleanly on top of
> trunk. This clearly denotes the set of commits that were made on the
> branch, and makes it easier to revert the branch if necessary.
>
> "git merge --no-ff" is also the preferred way of integrating a feature
> branch to other branches, e.g. branch-2.
>
> 
>
> LMK what you think about the above, when we finalize I'll kick off a
> [VOTE]. Last time we did "Adoption of New Codebase" but this feels more
> like "Modifying bylaws," if it needs a [VOTE] at all. "Bylaws" is easier,
> since it's just a lazy majority of active PMC members rather than 2/3rds.
>
> If the eventual [VOTE] passes, I'll put it on the wiki somewhere for easier
> reference. I'll also expand said wiki page with discussion about merge vs.
> rebase based on the mailing list threads, since I think we've got some good
> information here.
>
> Best,
> Andrew
>
> [1]:
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201408.mbox/%3CCALwhT94Y64M9keY25Ry_QOLUSZQT29tJQ95twsoa8xXrcNTxpQ%40mail.gmail.com%3E


  

[jira] [Resolved] (HADOOP-12108) Erroneous behavior of use of wildcard character ( * ) in ls command of hdfs

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-12108.
---
Resolution: Invalid

Thanks Aman! Steve is right. You do need to use quotes when there is already a 
file on the local file system which would match the wildcard

> Erroneous behavior of use of wildcard character ( * ) in ls command of hdfs 
> 
>
> Key: HADOOP-12108
> URL: https://issues.apache.org/jira/browse/HADOOP-12108
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Aman Goyal
>Priority: Critical
>
> If you have following directories in your LOCAL file system 
> /data/hadoop/sample/00/contents1.txt
> /data/hadoop/sample/01/contents2.txt
> and following directories in hdfs : 
> /data/hadoop/sample/00/contents1.txt
> /data/hadoop/sample/01/contents2.txt
> /data/hadoop/sample/02/contents3.txt
> suppose you run the following hdfs ls command:
> hdfs dfs -ls -R /data/hadoop/sample/*
> the paths that are printed have a reference to local paths, and only 00 & 01 
> directories get listed. 
> this happens only when wildcard ( * ) character is used in input paths.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11972) hdfs dfs -copyFromLocal reports File Not Found instead of Permission Denied.

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-11972.
---
Resolution: Duplicate

This is a legitimate problem. Duping to HDFS-5033

> hdfs dfs -copyFromLocal reports File Not Found instead of Permission Denied.
> 
>
> Key: HADOOP-11972
> URL: https://issues.apache.org/jira/browse/HADOOP-11972
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 2.6.0
> Environment: Linux hadoop-8309-2.west.isilon.com 
> 2.6.32-504.16.2.el6.centos.plus.x86_64 #1 SMP Wed Apr 22 00:59:31 UTC 2015 
> x86_64 x86_64 x86_64 GNU/Linux
>Reporter: David Tucker
>
> userA creates a file in /home/userA with 700 permissions.
> userB tries to copy it to HDFS, and receives a "No such file or directory" 
> instead of "Permission denied".
> [hrt_qa@hadoop-8309-2 ~]$ touch ./foo
> [hrt_qa@hadoop-8309-2 ~]$ ls -l ./foo
> -rw-r--r--. 1 hrt_qa users 0 May 14 16:09 ./foo
> [hrt_qa@hadoop-8309-2 ~]$ sudo su hbase
> [hbase@hadoop-8309-2 hrt_qa]$ ls -l ./foo
> ls: cannot access ./foo: Permission denied
> [hbase@hadoop-8309-2 hrt_qa]$ hdfs dfs -copyFromLocal ./foo /tmp/foo
> copyFromLocal: `./foo': No such file or directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11992) ORA-00933: SQL command not properly ended

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-11992.
---
Resolution: Duplicate

Duplicate of https://issues.apache.org/jira/browse/MAPREDUCE-3695

> ORA-00933: SQL command not properly ended
> -
>
> Key: HADOOP-11992
> URL: https://issues.apache.org/jira/browse/HADOOP-11992
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: eiko
>Assignee: eiko
>  Labels: hadoop
>
> hello
> when I insert data into oracle database from hdfs with mapreduce.error occur 
> like this.
> ORA-00933: SQL command not properly ended
> -
> this is my solution
> hadoop version:hadoop-2.6.0
> file:DBOutputFormat.class
> method:constructQuery
> line:163 query.append(");");
> -
> modify like this
> query.append(")"+"\n");
> contact me
> email:minicoo...@gmail.com



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11972) hdfs dfs -copyFromLocal reports File Not Found instead of Permission Denied.

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-11972.
---
Resolution: Invalid

This is because '.' is translated to /user/hbase for user hbase, and 
/user/hrt_qa for hrt_qa.
If you think this is not the case, please reopen and tell us
1. If your environment is Kerberized?
2. Are you using NFS?
3. What happens when you do the same thing using HDFS CLI

> hdfs dfs -copyFromLocal reports File Not Found instead of Permission Denied.
> 
>
> Key: HADOOP-11972
> URL: https://issues.apache.org/jira/browse/HADOOP-11972
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 2.6.0
> Environment: Linux hadoop-8309-2.west.isilon.com 
> 2.6.32-504.16.2.el6.centos.plus.x86_64 #1 SMP Wed Apr 22 00:59:31 UTC 2015 
> x86_64 x86_64 x86_64 GNU/Linux
>Reporter: David Tucker
>
> userA creates a file in /home/userA with 700 permissions.
> userB tries to copy it to HDFS, and receives a "No such file or directory" 
> instead of "Permission denied".
> [hrt_qa@hadoop-8309-2 ~]$ touch ./foo
> [hrt_qa@hadoop-8309-2 ~]$ ls -l ./foo
> -rw-r--r--. 1 hrt_qa users 0 May 14 16:09 ./foo
> [hrt_qa@hadoop-8309-2 ~]$ sudo su hbase
> [hbase@hadoop-8309-2 hrt_qa]$ ls -l ./foo
> ls: cannot access ./foo: Permission denied
> [hbase@hadoop-8309-2 hrt_qa]$ hdfs dfs -copyFromLocal ./foo /tmp/foo
> copyFromLocal: `./foo': No such file or directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11976) submit job with oozie to between cluster

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-11976.
---
Resolution: Invalid

Sunmeng! JIRA is for tracking development changes to Hadoop. For user queries 
please send an email to u...@hadoop.apache.org . Being able to submit jobs to 
YARN using oozie is a pretty basic feature that I doubt would be broken in 
2.5.x . 

> submit job with oozie to between cluster
> 
>
> Key: HADOOP-11976
> URL: https://issues.apache.org/jira/browse/HADOOP-11976
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: sunmeng
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-7638) visibility of the security utils and things like getCanonicalService.

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-7638.
--
Resolution: Later

> visibility of the security utils and things like getCanonicalService.
> -
>
> Key: HADOOP-7638
> URL: https://issues.apache.org/jira/browse/HADOOP-7638
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 0.24.0
>Reporter: John George
>Priority: Minor
>
> It would be a good idea to file an additional jira to take another look at 
> the visibility of the security utils and things like getCanonicalService. 
> Doesn't seem like these should be fully public. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-7822) Hadoop startup script has a race condition : this causes failures in datanodes status and stop commands

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-7822.
--
Resolution: Fixed

This has probably been fixed as part of the shell script rewrite

> Hadoop startup script has a race condition : this causes failures in 
> datanodes status and stop commands
> ---
>
> Key: HADOOP-7822
> URL: https://issues.apache.org/jira/browse/HADOOP-7822
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 0.20.1, 0.20.2, 0.20.205.0
>Reporter: Rahul Jain
>
> The symptoms are the following:
> a) start-all.sh is able to start both hadoop dfs and map-reduce processes, 
> assuming same grid nodes are used for dfs and map-reduce
> b) stop-all.sh stops map-reduce but fails to stop dfs processes (datanode 
> tasks on grid nodes)  
> Instead, the warning message 'no datanode to stop' is seen for all data 
> nodes.
> c) The 'pid' files for datanode processes do not exist therefore the only way 
> to stop datanode processes is to manually execute kill commands.
> The root cause of the issue appears to be in hadoop startup scripts. 
> start-all.sh is really two parts:
> 1. start-dfs.sh : Start namenode and datanodes
> 2. start-mapred.sh: Jobtracker and task trackers.
> In this case, running start-dfs.sh did as expected and created the pid files 
> for different datanodes. However, start-mapred.sh script did end up forcing 
> another rsync from master to slaves, effectively wiping out the pid files 
> stored under "pid" directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-7692) hadoop single node setup script to create mapred dir

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-7692.
--
Resolution: Won't Fix

> hadoop single node setup script to create mapred dir
> 
>
> Key: HADOOP-7692
> URL: https://issues.apache.org/jira/browse/HADOOP-7692
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 0.20.205.0, 1.1.0
>Reporter: Giridharan Kesavan
>
> hadoop single node setup script should create /mapred directory and chown to 
> mapred:mapred ; jt requires this directory for startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-7643) Bump up the version of aspectj

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-7643.
--
Resolution: Won't Fix

> Bump up the version of aspectj
> --
>
> Key: HADOOP-7643
> URL: https://issues.apache.org/jira/browse/HADOOP-7643
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build, test
>Affects Versions: 0.20.205.0, 1.1.0
>Reporter: Kihwal Lee
>Priority: Minor
>
> When the fault injection target is enabled, aspectj fails with the following 
> message:
> "Can't parameterize a member of non-generic type:"
> This is fixed by upgrading aspectj. I tested with 1.6.11 and it worked.
> It will also apply to trunk, but I believe trunk has other problems.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: about CHANGES.txt

[Ravi Prakash]

+1 for automating the information contained in CHANGES.txt. There are some 
changes which go in without JIRAs sometimes (CVEs eg.) . I like git log because 
its the absolute source of truth (cryptographically secure, audited, 
distributed, yadadada). We could always use git hooks to force a commit message 
format.
a) cherry-picks have the same message (by default) as the original)b) I'm not 
sure why branch-mergers would be a problem?c) "Whoops I missed something in the 
previous commit" wouldn't happen if our hooks were smartishd) "no 
identification of what type of commit it was without hooking into JIRA anyway." 
This would be in the format of the commit message

Either way I think would be an improvement.
Thanks for your ideas folks



 On Monday, March 16, 2015 11:51 AM, Colin P. McCabe  
wrote:
   

 +1 for generating CHANGES.txt from JIRA and/or git as part of making a
release.  Or just dropping it altogether.  Keeping it under version
control creates lot of false conflicts whenever submitting a patch and
generally makes committing minor changes unpleasant.

Colin

On Sat, Mar 14, 2015 at 8:36 PM, Yongjun Zhang  wrote:
> Hi Allen,
>
> Thanks a lot for your input!
>
> Looks like problem a, c, d you listed is not too bad, assuming we can solve
> d by pulling this info from jira as Sean pointed out.
>
> Problem b (branch mergers) seems to be a real one, and your approach of
> using JIRA system to build changes.txt is a reasonably good way. This does
> count on that we update jira accurately. Since this update is a manual
> process, it's possible to have inconsistency, but may be not too bad. Since
> any mistake found here can be remedied by fixing the jira side and
> refreshing the result.
>
> I wonder if we as a community should switch to using your way, and save
> committer's effort of taking care of CHANGES.txt (quite some save IMO).
> Hope more people can share their thoughts.
>
> Thanks.
>
> --Yongjun
>
> On Fri, Mar 13, 2015 at 4:45 PM, Allen Wittenauer  wrote:
>
>>
>> I think the general consensus is don’t include the changes.txt file in
>> your commit. It won’t be correct for both branches if such a commit is
>> destined for both. (No, the two branches aren’t the same.)
>>
>> No, git log isn’t more accurate.  The problems are:
>>
>> a) cherry picks
>> b) branch mergers
>> c) “whoops i missed something in that previous commit”
>> d) no identification of what type of commit it was without hooking into
>> JIRA anyway.
>>
>> This is why I prefer building the change log from JIRA.  We already build
>> release notes from JIRA, BTW.  (Not that anyone appears to read them given
>> the low quality of our notes…)  Anyway, here’s what I’ve been
>> building/using as changes.txt and release notes:
>>
>> https://github.com/aw-altiscale/hadoop-release-metadata
>>
>> I try to update these every day. :)
>>
>> On Mar 13, 2015, at 4:07 PM, Yongjun Zhang  wrote:
>>
>> > Thanks Esteban, I assume this report gets info purely from the jira
>> > database, but not "git log" of a branch, right?
>> >
>> > I hope we get the info from "git log" of a release branch because that'd
>> be
>> > more accurate.
>> >
>> > --Yongjun
>> >
>> > On Fri, Mar 13, 2015 at 3:11 PM, Esteban Gutierrez > >
>> > wrote:
>> >
>> >> JIRA already provides a report:
>> >>
>> >>
>> >>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12327179&styleName=Html&projectId=12310240
>> >>
>> >>
>> >> cheers,
>> >> esteban.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Cloudera, Inc.
>> >>
>> >>
>> >> On Fri, Mar 13, 2015 at 3:01 PM, Sean Busbey 
>> wrote:
>> >>
>> >>> So long as you include the issue number, you can automate pulling the
>> >> type
>> >>> from jira directly instead of putting it in the message.
>> >>>
>> >>> On Fri, Mar 13, 2015 at 4:49 PM, Yongjun Zhang 
>> >>> wrote:
>> >>>
>>  Hi,
>> 
>>  I found that changing CHANGES.txt when committing a jira is error
>> prone
>>  because of the different sections in the file, and sometimes we forget
>>  about changing this file.
>> 
>>  After all, git log would indicate the history of a branch. I wonder if
>> >> we
>>  could switch to a new method:
>> 
>>  1. When committing, ensure the message include the type of the jira,
>> >> "New
>>  Feature", "Bug Fixes", "Improvement" etc.
>> 
>>  2. No longer need to make changes to CHANGES.txt for each commit
>> 
>>  3. Before releasing a branch, create the CHANGES.txt by using "git
>> log"
>>  command for the given branch..
>> 
>>  Thanks.
>> 
>>  --Yongjun
>> 
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Sean
>> >>>
>> >>
>>
>>

  

[jira] [Created] (HADOOP-11596) Allow smart-apply-patch.sh to add new files in binary git patches

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-11596:
-

 Summary: Allow smart-apply-patch.sh to add new files in binary git 
patches
 Key: HADOOP-11596
 URL: https://issues.apache.org/jira/browse/HADOOP-11596
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash


When a new file is added, the source is /dev/null (rather than the root of the 
tree (which would mean a a/b prefix)) Allow for this



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 2.7 status

[Ravi Prakash]

I would like the improvements to the Namenode UI be included in 2.7 too. 
HDFS-7588. All the code is ready and we can try to get as much of it in as 
possible piecemeal.
 

 On Saturday, February 14, 2015 3:52 AM, Steve Loughran 
 wrote:
   

 


On 14 February 2015 at 00:37:07, Karthik Kambatla 
(ka...@cloudera.com) wrote:

2 weeks from now (end of Feb) sounds reasonable. The one feature I would
like for to be included is shared-cache: we are pretty close - two more
main items to take care of.

In an offline conversation, Steve mentioned building Windows binaries for
our releases. Do we want to do that for 2.7? If so, can anyone with Windows
expertise setup a Jenkins job to build these artifacts, and may be hook it
up to https://builds.apache.org/job/HADOOP2_Release_Artifacts_Builder/





someone will have to first fix MiniYarnCluster to come up on the ASF jenkins 
machines; it currently fails with directory permission setup problems that may 
matter in production, but not in test runs.

[jira] [Created] (HADOOP-11516) Enable HTTP compression on all hadoop UIs

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-11516:
-

 Summary: Enable HTTP compression on all hadoop UIs
 Key: HADOOP-11516
 URL: https://issues.apache.org/jira/browse/HADOOP-11516
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Ravi Prakash


Some of our UI pages (e.g. # of jobs, # of task attempts, logs) are extremely 
big. It would drastically improve their load times if we used HTTP compression.
There exists a GzipFilter for jetty that can be added so that if the browser 
includes "Accept-Encoding: gzip, deflate" in the request header, the server 
will respond with the resource compressed and a response header 
"Content-Encoding: gzip" (or deflate). 
http://www.eclipse.org/jetty/documentation/current/gzip-filter.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Patch review process

[Ravi Prakash]

Just out of left field: Since we already migrated to git (Thanks everyone for 
that effort) should we try using github's UI? Isn't that how the majority of 
the rest of the world started doing code reviews?
 

 On Monday, January 26, 2015 3:57 PM, Arpit Agarwal 
 wrote:
   

 Thanks for that data point Chris.

It looks like reviews.apache.org no longer works. hadoop-hdfs-git may be
pointing to an outdated repository. I'll file a ticket with Infra.

On Mon, Jan 26, 2015 at 2:41 PM, Chris Nauroth 
wrote:

> reviews.apache.org is backed by Review Board, and I've had a very positive
> experience with that tool on other projects.  HADOOP-9629 is a Hadoop patch
> where we decided to go ahead and use it, and I think it helped.  AFAIK,
> there is no rule against using it in Hadoop, but it does have the side
> effect of splitting part of the conversation out of jira.  If Crucible can
> keep all the review notes sync'd with the jira and searchable, then that
> would be very interesting.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
> On Mon, Jan 26, 2015 at 1:54 PM, Arpit Agarwal 
> wrote:
>
> > IMO the number one improvement would be a web-based review tool. We could
> > evaluate Atlassian Crucible since it claims to integrate well with Jira.
> I
> > have not tried https://reviews.apache.org/r/new/.
> >
> > Some easy improvements that were also raised by others on the private
> list:
> > - Encourage contributors to batch related trivial fixes into a single
> > patch.
> > - Require more detailed descriptions with non-trivial patch
> contributions.
> > For patches that require knowledge of a specific subsystem a
> > background+design note would be a good start.
> > - Eliminate CHANGES.txt. This came up on the dev list not too long ago
> and
> > IIRC Allen did a PoC.
> >
> > I am not optimistic about Gerrit from past experience. It does help gate
> > checkins and enforce pre-commit checks (good). I did not find it
> > user-friendly and it will be an additional hurdle for contributors to
> > understand (bad).
> >
> > Andrew, can the community build on your distributed pre-commit work to
> make
> > it production ready?
> >
> > Regards,
> > Arpit
> >
> >
> > On Mon, Jan 26, 2015 at 11:55 AM, Andrew Wang 
> > wrote:
> >
> > > Let's move this over to common-dev@, general@ is normally used for
> > project
> > > announcements rather than discussion topics.
> > >
> > > I'd like to summarize a few things mentioned on the private@ thread,
> > > related to streamlining the code submission process.
> > >
> > > - Gerrit was brought up again, as it has in the past, as something that
> > > could make the actual process of reviewing and committing a lot easier.
> > > This would be especially helpful for small patches, where the mechanics
> > of
> > > committing can take longer than reviewing the patch.
> > > - There were also concerns about forking discussions between JIRA and
> > > Gerrit. This has been an issue in Spark, and we'd like to keep
> > discussions
> > > and issue tracking centralized.
> > >
> > > - Some talk about how to improve precommit. Right now it takes hours to
> > run
> > > the unit tests, which slows down patch iterations. One solution is
> > running
> > > tests in parallel (and even distributed). Previous distributed
> > experiments
> > > have done a full unit test run in a couple minutes, but it'd be a fair
> > > amount of work to actually make this production ready.
> > > - Also mention of putting in place more linting and static analysis.
> > > Automating this will save reviewer time.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Jan 26, 2015 at 9:16 AM, Ted Yu  wrote:
> > >
> > > > In some cases, contributor responded to review comments and attached
> > > > patches addressing the comments.
> > > >
> > > > Later on, there was simply no response to the latest patch - even
> with
> > > > follow-on ping.
> > > >
> > > > I wish this aspect can be improved.
> > > >
> > > > Cheers
> > > >
> > > > On Sun, Jan 25, 2015 at 6:03 PM, Tsz Wo (Nicholas), Sze <
> > > > s29752-hadoopgene...@yahoo.com.invalid> wrote:
> > > >
> > > > > Hi contributors,
> > > > > I would like to (re)start a discussion regrading to our patch
> review
> > > > > process.  A similar discussion has been happened in a the hadoop
> > > private
> > > > > mailing list, which is inappropriate.
> > > > > Here is the problem:The patch available queues become longer and
> > > longer.
> > > > > It seems that we never can catch up.  There are patches sitting in
> > the
> > > > > queues for years.  How could we speed up?
> > > > > Regrads,Tsz-Wo
> > > > >
> > > >
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing

Wiki permissions!

[Ravi Prakash]

Hi folks!
Could you please give me write priviledges to 
https://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support ?

ThanksRavi

Re: Contributing to Hadoop

[Ravi Prakash]

Please also go through BUILDING.txt in the code base to find out how to build 
the source code. A very good area to start off learning about Hadoop and 
helping the community is to fix a failing unit test case. 
 

 On Monday, December 15, 2014 10:02 AM, Jay Vyas 
 wrote:
   

 One easy place to contribute in small increments could be the reproducing of 
bugs in jiras that are filed and open.  

If every day you spent an hour reproducing a bug filed in a jira, you could 
come up to speed eventually on a lot of sharp corners of the source code, and 
probably contribute some value to the community as well.

> On Dec 15, 2014, at 12:30 PM, prem vishnoi  wrote:
> 
> I want to work on hadoop live project for 2 hr every day
> please help me
> 
> Warms Regards,
> Prema Vishnoi
> 
> “Try not to become a man of success but rather to become a man of value”
> 
> On Mon, Dec 15, 2014 at 8:05 PM, Raghavendra Vaidya <
> raghavendra.vai...@gmail.com> wrote:
>> 
>> [image: Boxbe]  This message is eligible
>> for Automatic Cleanup! (raghavendra.vai...@gmail.com) Add cleanup rule
>> 
>> | More info
>> 
>> 
>> Folks,
>> 
>> I want to contribute to Hadoop ... I have downloaded the hadoop source and
>> set up the same on Intellij on Mac ...
>> 
>> I would like to start by executing / writing unit test cases ... could some
>> one point me to some resources to how to do that ?
>> 
>> 
>> Regards
>> 
>> Raghavendra Vaidya
>> 
>> 

   

[jira] [Resolved] (HADOOP-11360) GraphiteSink reports data with wrong timestamp

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-11360.
---
Resolution: Duplicate

Thanks for the reporting this JIRA Kamil!
This looks like a duplicate of 
https://issues.apache.org/jira/browse/HADOOP-11182 which was fixed in Hadoop 
2.6.0. If not, please re-open this JIRA

> GraphiteSink reports data with wrong timestamp
> --
>
> Key: HADOOP-11360
> URL: https://issues.apache.org/jira/browse/HADOOP-11360
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: metrics
>Reporter: Kamil Gorlo
>
> I've tried to use GraphiteSink with metrics2 system, but it looks that 
> timestamp sent to Graphite is refreshed rarely (about every 2 minutes 
> approx.) no mather how small period is set.
> Here is my configuration:
> *.sink.graphite.server_host=graphite-relay.host
> *.sink.graphite.server_port=2013
> *.sink.graphite.metrics_prefix=graphite.warehouse-data-1
> *.period=10
> nodemanager.sink.graphite.class=org.apache.hadoop.metrics2.sink.GraphiteSink
> And here is dumped network traffic to graphite-relay.host (only selected 
> lines, every line appears in 10 seconds as period suggests):
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041472
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041472
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041472
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  3 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  4 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  2 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  3 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  2 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  2 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  1 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  1 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041600
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041728
> graphite.warehouse-data-1.yarn.NodeManagerMetrics.Context=yarn.Hostname=warehouse-data-1.AllocatedContainers
>  0 1418041728
> As you can see, AllocatedContainers value is refreshed every 10 seconds, but 
> timestamp is not.
> It looks that the problem is level above (in classes providing MetricsRecord 
> - because timestamp value is taken from MetricsRecord object provided in 
> argument to putMetrics method in Sink implementation) which implies that 
> every sink will have the same problem. Maybe I misconfigured something?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Test case failures with Hadoop trunk

[Ravi Prakash]

Hi Vijay!
Thanks a lot for your initiative. You are correct. There are often a lot of 
unit tests which fail in addition to flaky tests which fail intermittently. 
About two years ago I tried to bring the number of unit test failures down to 
zero, but since then a lot of features have been added and I'm guessing the 
number of test failures has crept back up. Most people just keep a list of 
tests which fail usually and ignore those. There also ought to be open JIRAs 
for failing tests usually.

As a community we should try to bring these down to 0, but everyone is busy 
putting in new features and improving our CI unfortunately falls in priority. 
Prime areas for contributions ;-)
HTHRavi
 

 On Monday, December 1, 2014 5:03 PM, "Bhat, Vijay (CONT)" 
 wrote:
   

 Hi all,

My name is Vijay Bhat and I am looking to contribute to the Hadoop YARN 
project. I have been using and benefiting from Hadoop ecosystem technologies 
for a few years now and I want to give back to the community that makes this 
happen.

I forked the apache/hadoop branch on github and synced to the last commit 
(https://github.com/apache/hadoop/commit/1556f86a31a54733d6550363aa0e027acca7823b)
 that successfully built on the Apache build server 
(https://builds.apache.org/view/All/job/Hadoop-Yarn-trunk/758/).

However, I get test case failures when I build the Hadoop source code on a VM 
running Ubuntu 12.04 LTS.

The maven command I am running from the hadoop base directory is:

mvn clean install -U

Console output

Tests run: 9, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 2.392 sec <<< 
FAILURE! - in org.apache.hadoop.ipc.TestDecayRpcScheduler
testAccumulate(org.apache.hadoop.ipc.TestDecayRpcScheduler)  Time elapsed: 
0.084 sec  <<< FAILURE!
java.lang.AssertionError: expected:<3> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.ipc.TestDecayRpcScheduler.testAccumulate(TestDecayRpcScheduler.java:136)

testPriority(org.apache.hadoop.ipc.TestDecayRpcScheduler)  Time elapsed: 0.052 
sec  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.ipc.TestDecayRpcScheduler.testPriority(TestDecayRpcScheduler.java:197)


Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 111.519 sec <<< 
FAILURE! - in org.apache.hadoop.ha.TestZKFailoverControllerStress
testExpireBackAndForth(org.apache.hadoop.ha.TestZKFailoverControllerStress)  
Time elapsed: 45.46 sec  <<< ERROR!
java.lang.Exception: test timed out after 4 milliseconds
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.ha.MiniZKFCCluster.waitForHAState(MiniZKFCCluster.java:164)
at 
org.apache.hadoop.ha.MiniZKFCCluster.expireAndVerifyFailover(MiniZKFCCluster.java:236)
at 
org.apache.hadoop.ha.TestZKFailoverControllerStress.testExpireBackAndForth(TestZKFailoverControllerStress.java:79)


Tests run: 19, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 62.514 sec <<< 
FAILURE! - in org.apache.hadoop.ha.TestZKFailoverController
testGracefulFailoverFailBecomingStandby(org.apache.hadoop.ha.TestZKFailoverController)
  Time elapsed: 15.062 sec  <<< ERROR!
java.lang.Exception: test timed out after 15000 milliseconds
at java.lang.Object.wait(Native Method)
at 
org.apache.hadoop.ha.ZKFailoverController.waitForActiveAttempt(ZKFailoverController.java:467)
at 
org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:657)
at 
org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:61)
at 
org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:602)
at 
org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:599)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at 
org.apache.hadoop.ha.ZKFailoverController.gracefulFailoverToYou(ZKFailoverController.java:599)
at org.apache.hadoop.ha.ZKFCRpcServer.gracefulFailover(ZKFCRpcServer.java:94)
at 
org.apache.hadoop.ha.TestZKFailoverController.testGracefulFailoverFailBecomingStandby(TestZKFailoverController.java:532)


When I skip the tests, the source code compiles successfully.

mvn clean install -U –DskipTests

Is there something I’m doing incorrectly that’s causing the test cases to fail? 
I’d really appreciate any insight from folks who have gone through this process 
before. I’ve looked at the JIRAs labeled newbie 
(http://wiki.apache.org/hadoop/HowToContribute) but didn’t find promising leads.

T

Re: [VOTE] Release Apache Hadoop 2.6.0

[Ravi Prakash]

Thanks for the respin Arun!
I've verified all checksums, and tested that the DockerContainerExecutor was 
able to launch jobs.

I'm a +1 on the release
 

 On Thursday, November 13, 2014 3:09 PM, Arun C Murthy 
 wrote:
   

 Folks,

I've created another release candidate (rc1) for hadoop-2.6.0 based on the 
feedback.

The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1
The RC tag in git is: release-2.6.0-rc1

The maven artifacts are available via repository.apache.org at 
https://repository.apache.org/content/repositories/orgapachehadoop-1013.

Please try the release and vote; the vote will run for the usual 5 days.

thanks,
Arun


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.6.0

[Ravi Prakash]

Hi Arun!
We are very close to completion on YARN-1964 (DockerContainerExecutor). I'd 
also like HDFS-4882 to be checked in. Do you think these issues merit another 
RC?
ThanksRavi
 

 On Tuesday, November 11, 2014 11:57 AM, Steve Loughran 
 wrote:
   

 +1 binding

-patched slider pom to build against 2.6.0

-verified build did download, which it did at up to ~8Mbps. Faster than a
local build.

-full clean test runs on OS/X & Linux


Windows 2012:

Same thing. I did have to first build my own set of the windows native
binaries, by checking out branch-2.6.0; doing a native build, copying the
binaries and then purging the local m2 repository of hadoop artifacts to be
confident I was building against. For anyone who wants those native libs
they will be up on
https://github.com/apache/incubator-slider/tree/develop/bin/windows/ once
it syncs with the ASF repos.

afterwords: the tests worked!


On 11 November 2014 02:52, Arun C Murthy  wrote:

> Folks,
>
> I've created a release candidate (rc0) for hadoop-2.6.0 that I would like
> to see released.
>
> The RC is available at:
> http://people.apache.org/~acmurthy/hadoop-2.6.0-rc0
> The RC tag in git is: release-2.6.0-rc0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1012.
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> thanks,
> Arun
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HADOOP-11192) Change old subversion links to git

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-11192:
-

 Summary: Change old subversion links to git
 Key: HADOOP-11192
 URL: https://issues.apache.org/jira/browse/HADOOP-11192
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ravi Prakash


e.g. hadoop-project/src/site/site.xml still references SVN. 
We should probably check our wiki's and other documentation. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11185) There should be a way to disable a kill -9 during stop

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-11185:
-

 Summary: There should be a way to disable a kill -9 during stop
 Key: HADOOP-11185
 URL: https://issues.apache.org/jira/browse/HADOOP-11185
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Ravi Prakash


eg. hadoop-common-project/hadoop-common/bin/src/main/bin/hadoop-functions.sh 
calls kill -9 after some time. This might not be the best thing to do for some 
processes (if HA is not enabled) . There should be ability to disable this kill 
-9.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.4.1

[Ravi Prakash]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

+1

Built and deployed clusters on Amazon. Ran a basic test suite.

Thanks Arun

On 06/25/14 17:11, Akira AJISAKA wrote:
> Thanks Arun for another RC!
>
> I'm +1 (non-binding) for RC2. HDFS-6527 should be reverted because the
issue is only in 2.5 and trunk. In addition, I hope HDFS-6591 to be merged.
>
> Other than that, RC1 is good to me. I tested RC1 with distributed
cluster on CentOS 6.3:
>
> - Successful build from src (including native library)
> - Successful RM automatic fail-over and running MapReduce job also
succeeded
> - Successful rolling upgrade HDFS from 2.4.0 to 2.4.1
> - Successful downgrade HDFS from 2.4.1 to 2.4.0
> - Documentation looks good
>
> Thanks,
> Akira
>
> (2014/06/23 9:59), Steve Loughran wrote:
>> someone's filed a JIRA on loops in Hedged Read, with tests ...
>>
>> https://issues.apache.org/jira/browse/HDFS-6591
>>
>>
>> On 23 June 2014 08:58, Mit Desai  wrote:
>>
>>> +1 (non-binding)
>>>
>>> Tested on: Fedora17
>>> -Successful build from src (including native)
>>> -Verified Signature
>>> -Deployed source to my single node cluster and ran couple of sample
MR jobs
>>>
>>>
>>> - M!T
>>>
>>>
>>>
>>>
>>>
>>> On 6/21/14, 1:51 AM, "Arun C Murthy"  wrote:
>>>
 Folks,

 I've created another release candidate (rc1) for hadoop-2.4.1 based on
 the feedback that I would like to push out.

 The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1
 The RC tag in svn is here:
 https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7
days.

 thanks,
 Arun



 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/hdp/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
entity
 to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the
reader
 of this message is not the intended recipient, you are hereby notified
 that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
 immediately
 and delete it from your system. Thank You.
>>>
>>>
>>
>

-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTq3XoAAoJEGunL/HJl4XexG0QAJnmxxfljAB2QWbkK5EfQNq3
DW8PkA2tZAyLrdCeMaBMOybnrrtHRUJpPloh34pm6aeFINIcdwjYlx/42Zfe5Wk8
24nLsDYad4Zai0N9hwOs1zk8z2Tj0cZbA3VoOqrexTnGQb6Z7O12yY/vRJ1iWanT
Qa2qCgrfleUoCxBwTBrtO68Z98EmJHOlWd8W6QyZpMZiKC/EsGpjkCTLZzXvirkF
5/u29HXo0yJT1xA8iCleOdp7MCTmnfCF7sLvCV2rLN2ERJPaPE19AXn8pFAqyqeH
9JVIO2SCXJbmTxIlKN/UFGgP/v0KaLleYupUltQ4CIM+omkbsBxLPN2vIHavoxup
/1w7JBfmq67RIX7AHkUJgS4Dzs+GOK81dpt2niEfu1dx7h7qq4eeAvLKImjIRlRi
EqKqxqWBoDAb6FGBPRHsJVXb2zxn1NAYVIYYD4AW27+S0OyrTvwQWwmurhjG+h45
XC5Z+jFG+FGc96On9DtNxSUTYB9a0GpBBnjU+u1enT99n3j0X5YGmi2B/ca4Cp9J
WQ4CDeQfp4+87LijF1ZH8ObQn7L0vWudehhcMjAC3qo9NK0oZ8eSMd48WaFCS0AI
/xdfJT069RN4U9633KoGT/HXIXf6pcULEc7kNCgqULjXZO7hGl2H6Q3hxCBh0Xs5
zA8LmbrvDThJYIwZRXRR
=rTDe
-END PGP SIGNATURE-

Re: hadoop-2.5 - June end?

[Ravi Prakash]

Does this also mean that there won't be a 2.4.1 Apache release?



On Tuesday, June 10, 2014 9:45 AM, Suresh Srinivas  
wrote:
 


We should also include extended attributes feature for HDFS from HDFS-2006
for release 2.5.


On Mon, Jun 9, 2014 at 9:39 AM, Arun C Murthy  wrote:

> Folks,
>
>  As you can see from the Roadmap wiki, it looks like several items are
> still a bit away from being ready.
>
>  I think rather than wait for them, it will be useful to create an
> intermediate release (2.5) this month - I think ATS security is pretty
> close, so we can ship that. I'm thinking of creating hadoop-2.5 by end of
> the month, with a branch a couple of weeks prior.
>
>  Thoughts?
>
> thanks,
> Arun
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
http://hortonworks.com/download/


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: #Contributors on JIRA

[Ravi Prakash]

You mean {common,mapreduce,yarn,hdfs}-{dev,commits,issues}@hadoop.apache.org 
lists mentioned on https://hadoop.apache.org/mailing_lists.html ?


On Friday, May 16, 2014 4:44 PM, Pete of Pete  wrote:
 


On behalf of those of us who keep an eye on posts for commercial reasons
(but do not contribute) Is there the option of setting up a list that
accesses the dev comms, but doesn't impact the contributor list?

P



On Wed, May 14, 2014 at 4:31 AM, Suresh Srinivas wrote:

> Last time we cleaned up names of people who had not contributed in a long
> time. That could be an option.
>
>
> On Mon, May 12, 2014 at 12:03 PM, Karthik Kambatla  >wrote:
>
> > Hi devs
> >
> > Looks like we ran over the max contributors allowed for a project,
> again. I
> > don't remember what we did last time and can't find it in my email
> either.
> >
> > Can we bump up the number of contributors allowed? Otherwise, we might
> have
> > to remove some of the currently inactive contributors from the list?
> >
> > Thanks
> > Karthik
> >
>
>
>
> --
> http://hortonworks.com/download/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: [VOTE] Release Apache Hadoop 2.3.0

[Ravi Prakash]

Thanks Arun for another release!

+1 non-binding

Verified signatures, deployed a single node cluster and ran sleep and 
wordcount. Everything looks fine.


Regards
Ravi




On Tuesday, February 11, 2014 5:36 PM, Travis Thompson  
wrote:
 
Everything looks good so far, running on 100 nodes with security enabled.

I've found two minor issues I've found with the new Namenode UI so far and will 
work on them over the next few days:

HDFS-5934
HDFS-5935

Thanks,

Travis


On Feb 11, 2014, at 4:53 PM, Mohammad Islam 
wrote:

> Thanks Arun for the initiative.
> 
> +1 non-binding.
> 
> 
> I tested the followings:
> 1. Build package from the source tar.
> 2. Verified with md5sum
> 3. Verified with gpg 
> 4. Basic testing
> 
> Overall, good to go.
> 
> Regards,
> Mohammad
> 
> 
> 
> 
> On Tuesday, February 11, 2014 2:07 PM, Chen He  wrote:
> 
> +1, non-binding
> successful compiled on MacOS 10.7
> deployed to Fedora 7 and run test job without any problem.
> 
> 
> 
> On Tue, Feb 11, 2014 at 8:49 AM, Arun C Murthy  wrote:
> 
>> Folks,
>> 
>> I've created a release candidate (rc0) for hadoop-2.3.0 that I would like
>> to get released.
>> 
>> The RC is available at:
>> http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0
>> The RC tag in svn is here:
>> https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0
>> 
>> The maven artifacts are available via repository.apache.org.
>> 
>> Please try the release and vote; the vote will run for the usual 7 days.
>> 
>> thanks,
>> Arun
>> 
>> PS: Thanks to Andrew, Vinod & Alejandro for all their help in various
>> release activities.
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.

[jira] [Resolved] (HADOOP-7985) maven build should be super fast when there are no changes

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-7985.
--

Resolution: Won't Fix

> maven build should be super fast when there are no changes
> --
>
> Key: HADOOP-7985
> URL: https://issues.apache.org/jira/browse/HADOOP-7985
> Project: Hadoop Common
>  Issue Type: Wish
>  Components: build
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
>  Labels: build, maven
> Attachments: HADOOP-7985.patch
>
>
> I use this command "mvn -Pdist -P-cbuild -Dmaven.javadoc.skip -DskipTests 
> install" to build. Without ANY changes in code, running this command takes 
> 1:32. It seems to me this is too long. Investigate if this time can be 
> reduced drastically.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9732) JOB_FAIL and JOB_KILL have different behaviors when they should ideally be same / similar.

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-9732:


 Summary: JOB_FAIL and JOB_KILL have different behaviors when they 
should ideally be same / similar.
 Key: HADOOP-9732
 URL: https://issues.apache.org/jira/browse/HADOOP-9732
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.23.9, 3.0.0, 2.2.0
Reporter: Ravi Prakash


After MAPREDUCE-5317, both JOB_KILL and JOB_FAIL wait for all the tasks to die 
/ be killed, and then move to their final states. We can make these two code 
paths the same. Ideally KILL_WAIT should also have a timeout like the one 
MAPREDUCE-5317 introduces.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9614) smart-test-patch.sh hangs for new version of patch (2.7.1)

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-9614:


 Summary: smart-test-patch.sh hangs for new version of patch (2.7.1)
 Key: HADOOP-9614
 URL: https://issues.apache.org/jira/browse/HADOOP-9614
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.4-alpha, 0.23.7, 3.0.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash


patch -p0 -E --dry-run prints "checking file " for the new version of 
patch(2.7.1) rather than "patching file" as it did for older versions. This 
causes TMP2 to become empty, which causes the script to hang on this command 
forever:
PREFIX_DIRS_AND_FILES=$(cut -d '/' -f 1 | sort | uniq)


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-5006) ganglia is now showing any graphs

[Ravi Prakash (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-5006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-5006.
--

  Resolution: Fixed
Release Note: 2409 seems to have fixed this. Closing

> ganglia is now showing any graphs
> -
>
> Key: HADOOP-5006
> URL: https://issues.apache.org/jira/browse/HADOOP-5006
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: contrib/cloud
>Affects Versions: 0.19.0
>Reporter: Stefan Groschupf
>Priority: Trivial
>
> Ganglia is not showing any graphs since the rdd tool require installed fonts, 
> though the fedora core image used as basis to build the hadoop image does not 
> come with fonts. 
> To fix this just install dejavu-fonts as another package.
> The line in create-hadoop-image-remote.sh should look like this:
> yum -y install rsync lynx screen ganglia-gmetad ganglia-gmond ganglia-web 
> dejavu-fonts httpd php

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Hadoop Source Code

[Ravi Prakash]

Running "ant eclipse" should produce the eclipse project files you need to 
import it into eclipse






 From: Samaneh Shokuhi 
To: common-dev@hadoop.apache.org; Ravi Prakash  
Sent: Monday, October 22, 2012 6:41 AM
Subject: Re: Hadoop Source Code
 
Hi Salma,
I have a question about your development enviroment.Did you use eclipse?
I am going to use branch-1  to do some experiments on that.The thing is i
want to import the source code into eclipse  and  modify and build it
.Seems the source code inside src pachage in branch-1 is not mavenized .
So how did you build your project ? Did you do that from eclipse or command
line?

Samaneh

On Mon, Oct 22, 2012 at 6:51 AM, Ravi Prakash  wrote:

> Hi Salma,
>
> 0.20.205 is a little old now. branch-1.x is the last stable release.
>
> Although 0.23 is "alpha", Yahoo has been deploying it to pretty big
> clusters and the community is stabilizing MRv2 on this branch.
>
> branch-2 is where a lot of innovative development is happening. It too is
> "alpha" but if I were you, I would probably choose this branch. Its builds
> are monitored actively and should be stable to run.
>
> trunk contains the latest and greatest new features / bug-fixes etc., but
> in my experience is very unstable and might have trouble building / running.
>
> HTH
> Ravi
>
>
>
>
>
> 
>  From: Salma Saber 
> To: common-dev@hadoop.apache.org
> Sent: Saturday, October 20, 2012 3:47 PM
> Subject: Hadoop Source Code
>
>
>
> Hi,
>
> I am a new Hadoop developer. I want to implement an algorithm which
> enhances the
> Hadoop performance in Heterogeneous environments. I tried to build and run
> many
> versions for the Hadoop source code but the only one that I have succeeded
> to
> build and run is hadoop-205.
>
> I want to know if this version is suitable to edit on it, or there is a
> better
> version I should use in my implementation.
>
> Thanks,
> Salma
>

Re: Hadoop Source Code

[Ravi Prakash]

Hi Salma,

0.20.205 is a little old now. branch-1.x is the last stable release.

Although 0.23 is "alpha", Yahoo has been deploying it to pretty big clusters 
and the community is stabilizing MRv2 on this branch.

branch-2 is where a lot of innovative development is happening. It too is 
"alpha" but if I were you, I would probably choose this branch. Its builds are 
monitored actively and should be stable to run.

trunk contains the latest and greatest new features / bug-fixes etc., but in my 
experience is very unstable and might have trouble building / running.

HTH
Ravi






 From: Salma Saber 
To: common-dev@hadoop.apache.org 
Sent: Saturday, October 20, 2012 3:47 PM
Subject: Hadoop Source Code
 


Hi,

I am a new Hadoop developer. I want to implement an algorithm which enhances the
Hadoop performance in Heterogeneous environments. I tried to build and run many
versions for the Hadoop source code but the only one that I have succeeded to
build and run is hadoop-205.

I want to know if this version is suitable to edit on it, or there is a better
version I should use in my implementation. 

Thanks,
Salma 

Re: Hadoop API performance testcases

[Ravi Prakash]

Hi Amir,

Did you have any specific APIs in mind?

There's obviously, sort, shuffle, gridmix, DFSIO etc. which measure
different aspects of Hadoop job execution.

Ravi

On Tue, Jul 10, 2012 at 12:58 PM, Amir Sanjar  wrote:

>
> Hi all, are there any performance testcases for hadoop APIs? We are looking
> for testcases to time performance of each API.
>
> Best Regards
> Amir Sanjar
>
>
>

[jira] [Created] (HADOOP-8504) OfflineImageViewer throws an NPE

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-8504:


 Summary: OfflineImageViewer throws an NPE
 Key: HADOOP-8504
 URL: https://issues.apache.org/jira/browse/HADOOP-8504
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ravi Prakash


Courtesy [~mithun]
Exception in thread "main" java.lang.NullPointerException
at 
org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:371)
at org.apache.hadoop.security.User.(User.java:48)
at org.apache.hadoop.security.User.(User.java:43)
at 
org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:857)
at 
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenIdentifier.getUser(AbstractDelegationTokenIdentifier.java:91)
at 
org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier.toString(DelegationTokenIdentifier.java:61)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDelegationTokens(ImageLoaderCurrent.java:222)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.loadImage(ImageLoaderCurrent.java:185)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.go(OfflineImageViewer.java:129)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.main(OfflineImageViewer.java:250)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8445) Token should not print the password in toString

[Ravi Prakash (JIRA)]

Ravi Prakash created HADOOP-8445:


 Summary: Token should not print the password in toString
 Key: HADOOP-8445
 URL: https://issues.apache.org/jira/browse/HADOOP-8445
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 1.0.3
Reporter: Ravi Prakash
Assignee: Ravi Prakash


This JIRA is for porting HADOOP-6622 to branch-1 since 6622 is already closed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Scan Benchmark

[Ravi Prakash]

https://issues.apache.org/jira/browse/MAPREDUCE-3524?focusedCommentId=13170564&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13170564

On Sat, May 5, 2012 at 6:35 AM, ASHISH ATTARDE wrote:

> Hi,
>
> Can anyone guide me for Benchmarking Hadoop Cluster?
> I am interested in Scan benchmark.
>
> Thanks
> -Ashish
>

[jira] [Created] (HADOOP-8288) Remove references of mapred.child.ulimit etc. since they are not being used any more

[Ravi Prakash (Created) (JIRA)]

Remove references of mapred.child.ulimit etc. since they are not being used any 
more


 Key: HADOOP-8288
 URL: https://issues.apache.org/jira/browse/HADOOP-8288
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.23.2
Reporter: Ravi Prakash
Assignee: Ravi Prakash


Courtesy Philip Su, we found that (mapred.child.ulimit, mapreduce.map.ulimit, 
mapreduce.reduce.ulimit) were not being used at all. The configuration exists 
but is never used. Its also mentioned in mapred-default.xml and 
templates/../mapred-site.xml . Also the method getUlimitMemoryCommand in 
Shell.java is now useless and can be removed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HADOOP-6963) Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic links

[Ravi Prakash (Reopened) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-6963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash reopened HADOOP-6963:
--


Reopening for committing to branch-1.

> Fix FileUtil.getDU. It should not include the size of the directory or follow 
> symbolic links
> 
>
> Key: HADOOP-6963
> URL: https://issues.apache.org/jira/browse/HADOOP-6963
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 0.20.205.0, 0.23.1
>Reporter: Owen O'Malley
>Assignee: Ravi Prakash
>Priority: Critical
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: HADOOP-6963.branch-1.0.2.patch, 
> HADOOP-6963.branch-1.0.2.patch, HADOOP-6963.branch-1.patch, 
> HADOOP-6963.branch-23.patch, HADOOP-6963.branch-23.patch, 
> HADOOP-6963.branch-23.patch
>
>
> The getDU method should not include the size of the directory. The Java 
> interface says that the value is undefined and in Linux/Sun it gets the 4096 
> for the inode. Clearly this isn't useful.
> It also recursively calls itself. In case the directory has a symbolic link 
> forming a cycle, getDU keeps spinning in the cycle. In our case, we saw this 
> in the org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects 
> call. This prevented other tasks on the same node from committing, causing 
> the TT to become effectively useless (because the JT thinks it already has 
> enough tasks running)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Add user to assignee list?

[Ravi Prakash]

Hi folks,

Can someone with the proper authorization please add Anty to the list of
users who can assign themselves JIRAs?

Cheers
Ravi.


-- Forwarded message --
From: Schubert Zhang (Commented) (JIRA) 
Date: Mon, Mar 19, 2012 at 10:00 AM
Subject: [jira] [Commented] (MAPREDUCE-3685) There are some bugs in
implementation of MergeManager
To: ravihad...@gmail.com



   [
https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232654#comment-13232654]

Schubert Zhang commented on MAPREDUCE-3685:
---

@Ravi
Thank you very much. Is someone can and Anty Rao into the assignee list?

@Anty
Seems we should add junit code, and make patch for a right version.

> There are some bugs in implementation of MergeManager
> -
>
> Key: MAPREDUCE-3685
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.1
>Reporter: anty.rao
>Assignee: Zhihong Yu
>Priority: Minor
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3685-branch-0.23.1.patch,
MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA
administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8180) Remove hsqldb since its not needed from pom.xml

[Ravi Prakash (Created) (JIRA)]

Remove hsqldb since its not needed from pom.xml
---

 Key: HADOOP-8180
 URL: https://issues.apache.org/jira/browse/HADOOP-8180
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.23.2
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Attachments: HADOOP-8180.patch

Related to MAPREDUCE-3621

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8129) ViewFileSystemTestSetup setupForViewFileSystem is erring when the user's home directory is somewhere other than /home (eg. /User) etc.

[Ravi Prakash (Created) (JIRA)]

ViewFileSystemTestSetup setupForViewFileSystem is erring when the user's home 
directory is somewhere other than /home (eg. /User) etc.
--

 Key: HADOOP-8129
 URL: https://issues.apache.org/jira/browse/HADOOP-8129
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash


All TestFSMainOperationsLocalFileSystem tests (99 in all) fail saying:
{noformat}
java.io.FileNotFoundException: /home
at org.apache.hadoop.fs.viewfs.InodeTree.resolve(InodeTree.java:403)
at 
org.apache.hadoop.fs.viewfs.ViewFileSystem.mkdirs(ViewFileSystem.java:373)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1684)
at 
org.apache.hadoop.fs.FSMainOperationsBaseTest.setUp(FSMainOperationsBaseTest.java:90)
at 
org.apache.hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem.setUp(TestFSMainOperationsLocalFileSystem.java:42)
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: datanode to blocks mapping

[Ravi Prakash]

Selwa,

I'm glad I could help. I suggest you import the source code in an IDE (we
use Eclipse) and navigate around it. The outline view in Eclipse is a
lifesaver :)

BlocksMap does not seem to have that API. I would try FSNamesystem on the
namenode. On the client side, the DFSClient class is what opens a file (and
takes care of all the block-mapping).

Hope this helps,
Ravi.



On Wed, Feb 15, 2012 at 6:56 PM, Selwa  wrote:

> Ravi,
>
> thank u very much for your reply.
> i am using version 0.21.0 . hope your comment will help me a lot.
> BTW, after i got the blocksMap, can i request this blocksMap to return
> blocks of a given file.
> i mean how to map a filename to its blocks?
>
> Best wishes,
> Selwa
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/datanode-to-blocks-mapping-tp3736421p3748957.html
> Sent from the Hadoop lucene-dev mailing list archive at Nabble.com.
>

Re: datanode to blocks mapping

[Ravi Prakash]

Selwa,

You are *creating* a new BlocksMap with the first line. It is NOT an API to
get the blockmap from the namenode. The Namenode stores its blocks in a
Blockmap in the BlockManager (for 0.23) or FSNameSystem (for 0.20)

Which version are you using?

Regards,
Ravi.

On Sat, Feb 11, 2012 at 7:41 PM, Selwa  wrote:

> Hello there,
> currently i am working on research about the block placement policy of
> HDFS.
> I want to find the blocks mapping of the pseudo cluster.
> then i request the name node for this data structure by the following code
>
>   BlocksMap blocksMap = new BlocksMap(16, 0.75f);
>  Collectionblocks = blocksMap.getBlocks();
>  but the collection does not hold any thing - always it is
> empty
> blocks.isempty() is becoming true even i saved some files on
> the pseudo -cluster.
> what is wrong with this code?
> is there any other way to find block mapping?
>
> thanks
>
> Selwa
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/datanode-to-blocks-mapping-tp3736421p3736421.html
> Sent from the Hadoop lucene-dev mailing list archive at Nabble.com.
>

[jira] [Created] (HADOOP-8039) mvn site:stage-deploy should not have broken links.

[Ravi Prakash (Created) (JIRA)]

mvn site:stage-deploy should not have broken links.
---

 Key: HADOOP-8039
 URL: https://issues.apache.org/jira/browse/HADOOP-8039
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, documentation
Affects Versions: 0.23.1
Reporter: Ravi Prakash


The stage-deployed site has a lot of broken links / missing pages. We should 
fix that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8032) mvn site:stage-deploy should be able to use the scp protocol to stage documents

[Ravi Prakash (Created) (JIRA)]

mvn site:stage-deploy should be able to use the scp protocol to stage documents
---

 Key: HADOOP-8032
 URL: https://issues.apache.org/jira/browse/HADOOP-8032
 Project: Hadoop Common
  Issue Type: Wish
  Components: build, documentation
Affects Versions: 0.23.0, 0.24.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash


mvn site:stage-deploy should be able to use the scp protocol to stage documents.

http://maven.apache.org/plugins/maven-site-plugin/examples/adding-deploy-protocol.html
 shows what we need to add to the pom.xml

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-7985) maven build should be super fast when there are no changes

[Ravi Prakash (Created) (JIRA)]

maven build should be super fast when there are no changes
--

 Key: HADOOP-7985
 URL: https://issues.apache.org/jira/browse/HADOOP-7985
 Project: Hadoop Common
  Issue Type: Wish
  Components: build
Affects Versions: 0.23.0
Reporter: Ravi Prakash


I use this command "mvn -Pdist -P-cbuild -Dmaven.javadoc.skip -DskipTests 
install" to build. Without ANY changes in code, running this command takes 
1:32. It seems to me this is too long. Investigate if this time can be reduced 
drastically.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [DISCUSS] Stop support for non-packaged i.e. developer build single-node installs

[Ravi Prakash]

I think also at issue is the maven build system. I don't know if its
intended, but if I change a single .java file, "mvn -Pdist -P-cbuild
-Dmaven.javadoc.skip -DskipTests install" runs a TON of stuff It takes
a 1m 29s to finish even if there are NO changes anywhere. I doubt this is
how it should be. Am I doing it right?

In my mind, the gmake model (where a recipe is run only if the target is
older than the prerequisites) is what maven should do for us. Its senseless
to rebuild stuff if it hasn't changed. Probably some configuration needs
fixing.

I'm afraid though that if we fix it, I might end up being too productive
and fix all of Hadoop.

On Wed, Jan 18, 2012 at 10:38 AM, Tom White  wrote:

> Arun,
>
> There are instructions on how to run 0.23/trunk from the dev tree here:
>
>
> http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment#Run_HDFS_in_pseudo-distributed_mode_from_the_dev_tree
>
> http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment#Run_MapReduce_in_pseudo-distributed_mode_from_the_dev_tree
>
> I think it's useful to be able to do this since the feedback cycle is
> shorter between making a change and seeing it reflected in running
> code. I don't think it's either one or the other: I use tarballs too
> in other circumstances - it's useful to have both.
>
> To prevent the packages being broken we should have automated tests
> that run on nightly builds:
> https://issues.apache.org/jira/browse/HADOOP-7650.
>
> Cheers,
> Tom
>
> On Wed, Jan 18, 2012 at 7:16 AM, Arun C Murthy 
> wrote:
> > Folks,
> >
> >  Somewhere between MR-279 and mavenization we have broken the support
> for allowing _developers_ to run single-node installs from the non-packaged
> 'build' i.e. ability to run single-node clusters without the need to use a
> tarball/rpm etc. (I fully suspect MR-279 is blame as much as anyone else!
> *smile*)
> >
> >  I propose we go ahead and stop support for this officially to prevent
> confusion I already see among several folks (this has come up several times
> on our lists in context of hadoop-0.23).
> >
> >  Some benefits I can think of:
> >  a) Focus on fewer 'features' in the core.
> >  b) Reduce maintenance/complexity in our scripts (bin/hadoop, bin/hdfs
> etc.).
> >  d) Force us devs to eat our own dogfood when it comes to packaging etc.
> (I can think of numerous cases where devs have broken the tarball/rpm
> generation since they don't use it all the time.)
> >
> >  Clearly, we *should* back this up by improving our docs for new devs
> (wiki and/or the site:
> http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/SingleCluster.html)
> and I'm happy to volunteer.
> >
> >  Thoughts?
> >
> > thanks,
> > Arun
>

[jira] [Reopened] (HADOOP-7664) o.a.h.conf.Configuration complains of overriding final parameter even if the value with which its attempting to override is the same.

[Ravi Prakash (Reopened) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash reopened HADOOP-7664:
--


Reopening to fix 0.20-security

> o.a.h.conf.Configuration complains of overriding final parameter even if the 
> value with which its attempting to override is the same. 
> --
>
> Key: HADOOP-7664
> URL: https://issues.apache.org/jira/browse/HADOOP-7664
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: conf
>Affects Versions: 0.20.205.0, 0.23.0
> Environment: commit a2f64ee8d9312fe24780ec53b15af439a315796d
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
>Priority: Minor
> Fix For: 0.20.206.0, 0.23.0
>
> Attachments: HADOOP-7664.branch-0.20-security.patch, 
> HADOOP-7664.patch, HADOOP-7664.patch, HADOOP-7664.patch
>
>   Original Estimate: 1m
>  Time Spent: 1m
>  Remaining Estimate: 0h
>
> o.a.h.conf.Configuration complains of overriding final parameter even if the 
> value with which its attempting to override is the same. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-7770) ViewFS getFileChecksum throws FileNotFoundException for files in /tmp and /user

[Ravi Prakash (Created) (JIRA)]

ViewFS getFileChecksum throws FileNotFoundException for files in /tmp and /user
---

 Key: HADOOP-7770
 URL: https://issues.apache.org/jira/browse/HADOOP-7770
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Priority: Blocker
 Fix For: 0.23.0


Thanks to Rohini Palaniswamy for discovering this bug. To quote
bq. When doing getFileChecksum for path /user/hadoopqa/somefile, it is trying 
to fetch checksum for /user/user/hadoopqa/somefile. If /tmp/file, it is trying 
/tmp/tmp/file. Works fine for other FS operations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-7509) Improve message when Authentication is required

[Ravi Prakash (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HADOOP-7509.
--

  Resolution: Fixed
Release Note: 
Thanks Aaron and Suresh!
Marking as resolved fixed since changes have gone in.

> Improve message when Authentication is required
> ---
>
> Key: HADOOP-7509
> URL: https://issues.apache.org/jira/browse/HADOOP-7509
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 0.20.204.0, 0.23.0
>    Reporter: Ravi Prakash
>Assignee: Ravi Prakash
>Priority: Trivial
> Fix For: 0.20.206.0, 0.24.0
>
> Attachments: HADOOP-7509.1.patch, HADOOP-7509.2.branch-0.23.patch, 
> HADOOP-7509.2.patch, correct7509.branch-0.23.patch, 
> patchToCorrect7509.branch-0.20.patch, patchToCorrect7509.patch
>
>
> The message when security is enabled and authentication is configured to be 
> simple is not explicit enough. It simply prints out "Authentication is 
> required" and prints out a stack trace. The message should be "Authorization 
> (hadoop.security.authorization) is enabled but authentication 
> (hadoop.security.authentication) is configured as simple. Please configure 
> another method."

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-7745) I switched variable names in HADOOP-7509

[Ravi Prakash (Created) (JIRA)]

I switched variable names in HADOOP-7509


 Key: HADOOP-7745
 URL: https://issues.apache.org/jira/browse/HADOOP-7745
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.24.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Fix For: 0.24.0


As Aaron pointed out on 
https://issues.apache.org/jira/browse/HADOOP-7509?focusedCommentId=13126725&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13126725
 I stupidly swapped CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION with 
CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HADOOP-7509) Improve message when Authentication is required

[Ravi Prakash (Reopened) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash reopened HADOOP-7509:
--


Switched between CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION and 
CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION

> Improve message when Authentication is required
> ---
>
> Key: HADOOP-7509
> URL: https://issues.apache.org/jira/browse/HADOOP-7509
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 0.20.204.0, 0.23.0
>    Reporter: Ravi Prakash
>Assignee: Ravi Prakash
>Priority: Trivial
> Fix For: 0.20.206.0, 0.24.0
>
> Attachments: HADOOP-7509.1.patch, HADOOP-7509.2.branch-0.23.patch, 
> HADOOP-7509.2.patch, patchToCorrect7509.patch
>
>
> The message when security is enabled and authentication is configured to be 
> simple is not explicit enough. It simply prints out "Authentication is 
> required" and prints out a stack trace. The message should be "Authorization 
> (hadoop.security.authorization) is enabled but authentication 
> (hadoop.security.authentication) is configured as simple. Please configure 
> another method."

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: problem of lost name-node

[Ravi Prakash]

Hi,

@Mirko: Please file a JIRA. This seems an appropriate time.

@Steve: If we store the absolute filenames (i.e. the whole path), would we
still have the problem you outlined in the 2nd para? I do agree updating
would have to be pushed out and that might be cumbersome, but hey, we are
processing heartbeats from the datanodes every 3 seconds anyway. Maybe we
can piggyback those updates? I'm sure there are better solutions as well and
I don't think these problems are show-stoppers. If this solutions helps to
decrease the FUD, then I think it might be worth it (apart from its merit)

Just my $.02
Ravi




On Wed, Sep 28, 2011 at 9:06 AM, Steve Loughran  wrote:

>
> One of the issues here is keeping that list up to date. You don't want
> filename operations on the NN to push out changes to datanodes (which may
> not be there, after all), and you don't necessarily want every block
> creation operation on a DN to force an update on what effectively becomes a
> mini-db of (filename, block) mappings. Yes, it could just be a text file,
> but you still need to push out atomic updates which don't lose the previous
> version on a power failure. That update would have to be thread safe, you
> would have to decide whether to make it save-immediately vs lazy-write.
>
> In the situation in which your NN loses the entire image -and all its
> backups- you are going to lose the directory tree. All the per-DN metadata
> would do is leave you with some useful filenames (2011_09_22_EMEA_paying_*
> *customers.csv.lzo) and lots that aren't (mapout0043.something). Someone
> is still going to have to try and recreate what appears to be a functional
> directory tree from it. Then once you add layers on top like HBase, life is
> even more complicated as the filenames will stop bearing any relationship to
> the content.
>
> I'd go for a process that makes checkpointing NN state more reliable. That
> could include making it easier for the secondary namenode to push out
> updates to worker nodes in the system that can store timestamped/version
> stamped copies of the state; it could be improving recovery of state, and it
> could be better code to make sure that the secondary Namenode is actually
> working. Because you will need a secondary namenode on any cluster of
> moderate size, and you will need to make sure it is working -and test it-
>
>
> On 28/09/11 14:27, Ravi Prakash wrote:
>
>> Hi Mirko,
>>
>> Its seems like a great idea to me!! The architects and senior developers
>> might have some more insight on this though.
>>
>> I think part of the reason why the community might be lazy about
>> implementing this is because the Namenode being a single point of failure
>> is
>> usually regarded as FUD. There are simple tricks (like writing the fsimage
>> and editslog to NFS) which can guard against some failure scenarios, and I
>> think most users of hadoop are satisfied with that.
>>
>> I wouldn't be too surprised if there is already a JIRA for this. But if
>> you
>> could come up with a patch, I'm hopeful the community would be interested
>> in
>> it.
>>
>> Cheers
>> Ravi
>>
>> 2011/9/27 Mirko 
>> Kämpf
>> >
>>
>>  Hi,
>>> during the Cloudera Developer Training at Berlin I came up with an idea,
>>> regarding a lost name-node.
>>> As in this case all data blocks are lost. The solution could be, to have
>>> a
>>> table which relates filenames and block_ids on that node, which can be
>>> scaned
>>> after a name-node is lost. Or on every block could be a kind of a
>>> backlink
>>> to the filename and the total nr of blocks and/or a total hashsum
>>> attached.
>>> This would it make easy to recover with minimal overhead.
>>>
>>> Now I would like to ask the developer community, if there is any good
>>> reason
>>> not to do this?
>>> Before I start to figure out where to start an implementation of such a
>>> feature.
>>>
>>> Thanks,
>>> Mirko
>>>
>>>
>>
>

Re: problem of lost name-node

[Ravi Prakash]

Hi Mirko,

Its seems like a great idea to me!! The architects and senior developers
might have some more insight on this though.

I think part of the reason why the community might be lazy about
implementing this is because the Namenode being a single point of failure is
usually regarded as FUD. There are simple tricks (like writing the fsimage
and editslog to NFS) which can guard against some failure scenarios, and I
think most users of hadoop are satisfied with that.

I wouldn't be too surprised if there is already a JIRA for this. But if you
could come up with a patch, I'm hopeful the community would be interested in
it.

Cheers
Ravi

2011/9/27 Mirko Kämpf 

> Hi,
> during the Cloudera Developer Training at Berlin I came up with an idea,
> regarding a lost name-node.
> As in this case all data blocks are lost. The solution could be, to have a
> table which relates filenames and block_ids on that node, which can be
> scaned
> after a name-node is lost. Or on every block could be a kind of a backlink
> to the filename and the total nr of blocks and/or a total hashsum attached.
> This would it make easy to recover with minimal overhead.
>
> Now I would like to ask the developer community, if there is any good
> reason
> not to do this?
> Before I start to figure out where to start an implementation of such a
> feature.
>
> Thanks,
> Mirko
>

[jira] [Created] (HADOOP-7664) o.a.h.conf.Configuration complains of overriding final parameter even if the value with which its attempting to override is the same.

[Ravi Prakash (JIRA)]

o.a.h.conf.Configuration complains of overriding final parameter even if the 
value with which its attempting to override is the same. 
--

 Key: HADOOP-7664
 URL: https://issues.apache.org/jira/browse/HADOOP-7664
 Project: Hadoop Common
  Issue Type: Improvement
  Components: conf
Affects Versions: 0.23.0
 Environment: commit a2f64ee8d9312fe24780ec53b15af439a315796d
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Priority: Minor
 Fix For: 0.23.0


o.a.h.conf.Configuration complains of overriding final parameter even if the 
value with which its attempting to override is the same. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: JIRA attachments order

[Ravi Prakash]

But what if I want to see an incremental diff between two patches? I don't
want to review the whole patch everytime. Maybe I just want to re-review the
changes made to a patch. I would then have sort the patches manually using
time. I think its better to have version numbers in that case


On Fri, Sep 9, 2011 at 10:48 AM, Doug Cutting  wrote:

> On 09/09/2011 07:27 AM, Ted Dunning wrote:
> > If you post the same patch with the same name, JIRA helps you out by
> greying
> > all the earlier versions out.
>
> Indeed.  That's the best practice, not to add version numbers to patch
> files, for this very reason.  We should perhaps note this on:
>
> http://wiki.apache.org/hadoop/HowToContribute
>
> I am a Jira administrator and would be happy to change the default
> ordering of attachments if it were possible, however I can see no option
> to do so.
>
> Doug
>

Re: Namenode not starting

[Ravi Prakash]

Hi Abhishek,

Try reading through the shell scripts before postiing. They are short and
simple enough and you should be able to debug them quite easily. I've seen
the same error many times.

Do you see JAVA_HOME set when you $ssh localhost?

Also which command are you using to start the daemons?

Fight on,
Ravi

On Thu, Sep 1, 2011 at 4:35 PM, abhishek sharma  wrote:

> Hi Hailong,
>
> I have installed JDK and set JAVA_HOME correctly (as far as I know).
>
> Output of java -version is:
> java version "1.6.0_04"
> Java(TM) SE Runtime Environment (build 1.6.0_04-b12)
> Java HotSpot(TM) Server VM (build 10.0-b19, mixed mode)
>
> I also have another version installed "1.6.0_27" but get same error with
> it.
>
> Abhishek
>
> On Thu, Sep 1, 2011 at 4:00 PM, hailong.yang1115
>  wrote:
> > Hi abhishek,
> >
> > Have you successfully installed java virtual machine like sun JDK before
> running Hadoop? Or maybe you forget to configure the environment variable
> JAVA_HOME? What is the output of the command 'java -version'?
> >
> > Regards
> >
> > Hailong
> >
> >
> >
> >
> > ***
> > * Hailong Yang, PhD. Candidate
> > * Sino-German Joint Software Institute,
> > * School of Computer Science&Engineering, Beihang University
> > * Phone: (86-010)82315908
> > * Email: hailong.yang1...@gmail.com
> > * Address: G413, New Main Building in Beihang University,
> > *  No.37 XueYuan Road,HaiDian District,
> > *  Beijing,P.R.China,100191
> > ***
> >
> > From: abhishek sharma
> > Date: 2011-09-02 03:51
> > To: common-user; common-dev
> > Subject: Namenode not starting
> > Hi all,
> >
> > I am trying to install Hadoop (release 0.20.203) on a machine with
> CentOS.
> >
> > When I try to start HDFS, I get the following error.
> >
> > : Unrecognized option: -jvm
> > : Could not create the Java virtual machine.
> >
> > Any idea what might be the problem?
> >
> > Thanks,
> > Abhishek
>

[jira] [Created] (HADOOP-7438) Using the hadoop-deamon.sh script to start nodes leads to a depricated warning

[Ravi Prakash (JIRA)]

Using the hadoop-deamon.sh script to start nodes leads to a depricated warning 
---

 Key: HADOOP-7438
 URL: https://issues.apache.org/jira/browse/HADOOP-7438
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 0.22.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash


hadoop-daemon.sh calls common/bin/hadoop for hdfs/bin/hdfs tasks and so 
common/bin/hadoop complains its deprecated for those uses.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-7430) Improve error message when moving to trash fails due to quota issue

[Ravi Prakash (JIRA)]

Improve error message when moving to trash fails due to quota issue
---

 Key: HADOOP-7430
 URL: https://issues.apache.org/jira/browse/HADOOP-7430
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Priority: Minor


-rm command doesn't suggest -skipTrash on failure.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira