Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

[Harsh J]

Thanks Enis and Andrew; I think I missed the key point of conformance
with other FSes' behavior.

On Wed, Apr 17, 2013 at 11:29 PM, Enis Söztutar  wrote:
> Harsh, the discussion above includes the reasoning behind forcing the
> change in hbase rather than hdfs. Although HBase has shipped with this,
> HDFS's snapshots are user visible, meaning that you can do:
>
> hadoop fs -ls /user/foo/.snapshot/
>
> Plus, it is a convention for file systems (netapp, etc) to expose snapshots
> this way, having a name ".snapshot". HBase's snapshot directories are not
> user visible, and not widely used yet.
>
> Enis
>
>
> On Wed, Apr 17, 2013 at 9:14 AM, Andrew Purtell  wrote:
>
>> Thanks for the consideration but we've just committed a change to address
>> this as HBASE-8352
>>
>>
>> On Wednesday, April 17, 2013, Harsh J wrote:
>>
>> > Pardon my late inquisition here but since HBase already shipped out
>> > with a name .snapshots/, why do we force them to change it, and not
>> > rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
>> > Snapshots has not been released for any users yet. The way I see it,
>> > that'd be much more easier to do than making a workaround for a done
>> > deal on HBase, which already has its snapshot's users.
>> >
>> > @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
>> > applicable to other FileSystem interface implementations as well, then
>> > .hdfs-snapshots should be fine for it - no?
>> >
>> > On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu  wrote:
>> > > Hi,
>> > > Please take a look at patch v5 attached to HBASE-8352.
>> > >
>> > > It would be nice to resolve this blocker today so that 0.94.7 RC can be
>> > cut.
>> > >
>> > > Thanks
>> > >
>> > > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl 
>> > wrote:
>> > >
>> > >> Please see my last comment on the jira. We can make this work without
>> > >> breaking users who are using HDFS snapshots.
>> > >>
>> > >>   --
>> > >>  *From:* Ted Yu 
>> > >> *To:* d...@hbase.apache.org
>> > >> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl 
>> > >> *Sent:* Tuesday, April 16, 2013 10:00 AM
>> > >> *Subject:* Re: collision in the naming of '.snapshot' directory
>> between
>> > >> hdfs snapshot and hbase snapshot
>> > >>
>> > >> Let's get proper release notes for HBASE-8352 .
>> > >>
>> > >> Either Lars or I can send out notification to user mailing list so
>> that
>> > >> there is enough preparation for this change.
>> > >>
>> > >> Cheers
>> > >>
>> > >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh 
>> > wrote:
>> > >>
>> > >> I was away from keyboard when I asserted that hdfs snapshot was a
>> hadoop
>> > >> 2.1 or 3.0 feature.  Apparently it is targeted as a hadoop 2.0.5
>> > feature.
>> > >>  (I'm a little surprised -- expected this to be a hadoop2 compat
>> > breaking
>> > >> feature) -- so I agree that this is a bit more urgent.
>> > >>
>> > >> Anyway, I agree that the fs .snapshot naming convention is long
>> standing
>> > >> and should win.
>> > >>
>> > >> My concern is with breaking compatibility in 0.94 again -- if we don't
>> > go
>> > >> down the conf variable route,  I consider having docs to properly
>> > document
>> > >> how to do the upgrade and caveats of doing the upgrade in the
>> > docs/release
>> > >> notes blocker to hbase 0.94.7.  (specifically mentioning from 0.94.6
>> to
>> > >> 0.94.7, and to possibly to 0.95).
>> > >>
>> > >> Jon.
>> > >>
>> > >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu  wrote:
>> > >>
>> > >> > bq. Alternatively, we can detect the underlying Hadoop version, and
>> > use
>> > >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>> > >> >
>> > >> > I think this would introduce more confusion, especially for
>> > operations.
>> > >> >
>> > >> > Cheers
>> > >> >
>> > >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar 
>> > >> wrote:
>> > >> >
>> > >> > > Because HDFS exposes the snapshots so that the normal file system
>> > >> > > operations are mapped inside snapshot dirs, I think HDFS reserving
>> > the
>> > >> > > .snapshot name makes sense. OTOH, nothing is specific about the
>> dir
>> > >> name
>> > >> > > that is chosen by HBase.
>> > >> > >
>> > >> > > I would prefer to change the dir name in 0.94 as well, since 0.94
>> is
>> > >> also
>> > >> > > being run on top of hadoop 2. Alternatively, we can detect the
>> > >> underlying
>> > >> > > Hadoop version, and use either .snapshot or .hbase_snapshot in
>> 0.94
>> > >> > > depending on h1 & h2.
>> > >> > >
>> > >> > > Enis
>> > >> > >
>> > >> > >
>> > >> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu 
>> > wrote:
>> > >> > >
>> > >> > > > bq. let's make the hbase snapshot for a conf variable.
>> > >> > > >
>> > >> > > > Once we decide on the new name of snapshot directory, we should
>> > still
>> > >> > use
>> > >> > > > hardcoded value. This aligns with current code base:
>> > >> > > > See this snippet from HConstants:
>> > >--
>> > Harsh J
>> >
>>
>>
>> --
>> Best regards,
>>
>>- Andy
>>
>> Pr

[jira] [Created] (HDFS-4709) TestDFSClientRetries#testGetFileChecksum fails using IBM java 6

[Tian Hong Wang (JIRA)]

Tian Hong Wang created HDFS-4709:


 Summary: TestDFSClientRetries#testGetFileChecksum fails using IBM 
java 6
 Key: HDFS-4709
 URL: https://issues.apache.org/jira/browse/HDFS-4709
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
 Fix For: 2.0.3-alpha


testGetFileChecksum(org.apache.hadoop.hdfs.TestDFSClientRetries)  Time elapsed: 
3993 sec  <<< ERROR!
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
/testGetFileChecksum could only be replicated to 0 nodes instead of 
minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are 
excluded in this operation.
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1339)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2186)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:491)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:351)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40744)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1735)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1731)
at 
java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1729)

at org.apache.hadoop.ipc.Client.call(Client.java:1235)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy10.addBlock(Unknown Source)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy10.addBlock(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:311)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1156)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1009)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Heads up - Snapshots feature merge into trunk

[Aaron T. Myers]

I'm very excited to see that this project is nearing completion. I've been
following the development pretty closely and am very much looking forward
to getting this merged to trunk.

One thing that I do think we should address before the merge is moving the
programmatic APIs for working with snapshots. I've brought this up before,
and was told that it would be done in a separate JIRA, but I don't think
that JIRA was ever filed.

As it stands right now, the API for using snapshots is the following:

1. The API to create/delete/rename snapshots are in FileSystem.
2. The API to mark directories as snapshottable or not only exists in
DistributedFileSystem and DFSAdmin, neither of which are intended to be
public APIs.

In my opinion (and I think this was shared by others at the last snapshots
design meetup?) we should move #1 out of the FileSystem class since these
are primarily administrative APIs, and it is unlikely that any other
FileSystem implementation besides HDFS will ever implement these commands.
Also, #2 should really be in some public (not necessarily stable, but
public) class for use by tools which are used to administer HDFS. In my
opinion the most natural place for both of these APIs is in the HdfsAdmin
class, which is a public/evolving interface explicitly for these sorts of
operations.

What are others thoughts on this subject?

Best,
Aaron

--
Aaron T. Myers
Software Engineer, Cloudera


On Sat, Apr 13, 2013 at 10:05 AM, Suresh Srinivas wrote:

> Support for snapshots feature is being worked on in the jira
> https://issues.apache.org/jira/browse/HDFS-2802. This is an important and
> a
> large feature in HDFS. Please see a brief presentation that describes the
> feature at a highlevel from the Snapshot discussion meetup we had a while
> back -
> https://issues.apache.org/jira/secure/attachment/12552861/Snapshots.pdf.
>
> I am exicted to announce that the feature development will soon be
> completed. Please see the jira for the design and the details of the
> subtasks. This is a heads up about the merge vote mail that will soon be
> sent.
>
> Details of development and testing:
> Development has been done in a separate branch -
> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2802. The
> design is posted at -
>
> https://issues.apache.org/jira/secure/attachment/12551474/Snapshots20121030.pdf
> .
> The feature development has involved close to 100 subtasks and close to 20K
> lines of code.
>
> A lot of unit tests have been added as a part of the feature. We also have
> been testing this in a cluster of 5 nodes with a long running test that
> mimics a real cluster usage with emphasis on use cases related to
> snapshots.  Please see the test plan
>
> https://issues.apache.org/jira/secure/attachment/12575442/snapshot-testplan.pdffor
> the details.
>
> Next steps, before calling for merge vote, we need to get the following
> done:
> - Add user documentation that describes the feature, and how to use it
> - Complete some of the pending tasks
> - Continue testing the feature and fix any bugs that might come up
> - Update the design document
>
> Thanks to everyone who has participated in design and development of this
> feature. Please review the work and help in testing the feature.
>
> Regards,
> Suresh
>

[jira] [Resolved] (HDFS-4706) disallowSnapshot does not work for root

[Tsz Wo (Nicholas), SZE (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE resolved HDFS-4706.
--

   Resolution: Fixed
Fix Version/s: Snapshot (HDFS-2802)
 Hadoop Flags: Reviewed

Thanks Jing for reviewing the patch.

I have committed this.

> disallowSnapshot does not work for root
> ---
>
> Key: HDFS-4706
> URL: https://issues.apache.org/jira/browse/HDFS-4706
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: Snapshot (HDFS-2802)
>
> Attachments: h4706_20130417.patch
>
>
> disallowSnapshot replaces a snapshottable directory to a normal directory.  
> However, we cannot replace root.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-4707) Fix FilterFileSystem and findbugs warning in Snapshot branch

[Tsz Wo (Nicholas), SZE (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE resolved HDFS-4707.
--

   Resolution: Fixed
Fix Version/s: Snapshot (HDFS-2802)
 Hadoop Flags: Reviewed

Thanks Jing for reviewing the patch.

I have committed this.

> Fix FilterFileSystem and findbugs warning in Snapshot branch
> 
>
> Key: HDFS-4707
> URL: https://issues.apache.org/jira/browse/HDFS-4707
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: Snapshot (HDFS-2802)
>
> Attachments: h4707_20130417.patch
>
>
> The snapshot methods are not added to FilterFileSystem.
> Findbugs warnings:
> - SIC Should snapshot.Snapshot$Root be a _static_ inner class?
> - WMI Method FSImageFormat$Saver.saveImage(ByteBuffer, 
> INodeDirectory, DataOutputStream, Snapshot, boolean) makes inefficient use of 
> keySet iterator instead of entrySet iterator
> - BC  Unchecked/unconfirmed cast from 
> FSImageSerialization.writeINodeFile(INodeFile, DataOutput, boolean)
> - BC  Unchecked/unconfirmed cast from 
> snapshot.SnapshotFSImageFormat.loadDirectoryDiffList(INodeDirectory, 
> DataInput, FSImageFormat$Loader)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-4708) Add snapshot user guide

[Tsz Wo (Nicholas), SZE (JIRA)]

Tsz Wo (Nicholas), SZE created HDFS-4708:


 Summary: Add snapshot user guide
 Key: HDFS-4708
 URL: https://issues.apache.org/jira/browse/HDFS-4708
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: documentation
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE


The guide should include the snapshot semantic, snapshot API and snapshot 
commands.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-4707) Fix FilterFileSystem and findbugs warning in Snapshot branch

[Tsz Wo (Nicholas), SZE (JIRA)]

Tsz Wo (Nicholas), SZE created HDFS-4707:


 Summary: Fix FilterFileSystem and findbugs warning in Snapshot 
branch
 Key: HDFS-4707
 URL: https://issues.apache.org/jira/browse/HDFS-4707
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE


The snapshot methods are not added to FilterFileSystem.

Findbugs warnings:
- SIC   Should snapshot.Snapshot$Root be a _static_ inner class?
- WMI   Method FSImageFormat$Saver.saveImage(ByteBuffer, INodeDirectory, 
DataOutputStream, Snapshot, boolean) makes inefficient use of keySet iterator 
instead of entrySet iterator
- BCUnchecked/unconfirmed cast from 
FSImageSerialization.writeINodeFile(INodeFile, DataOutput, boolean)
- BCUnchecked/unconfirmed cast from 
snapshot.SnapshotFSImageFormat.loadDirectoryDiffList(INodeDirectory, DataInput, 
FSImageFormat$Loader)



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

[Robert Evans]

+1 (binding)

Downloaded the tar ball and ran some simple jobs.

--Bobby Evans

On 4/17/13 2:01 PM, "Siddharth Seth"  wrote:

>+1 (binding)
>Verified checksums and signatures.
>Built from the source tar, deployed a single node cluster and tested a
>couple of simple MR jobs.
>
>- Sid
>
>
>On Fri, Apr 12, 2013 at 2:56 PM, Arun C Murthy 
>wrote:
>
>> Folks,
>>
>> I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I
>>would
>> like to release.
>>
>> The RC is available at:
>> http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/
>> The RC tag in svn is here:
>> 
>>http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc
>>2
>>
>> The maven artifacts are available via repository.apache.org.
>>
>> Please try the release and vote; the vote will run for the usual 7 days.
>>
>> thanks,
>> Arun
>>
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>

[jira] [Created] (HDFS-4706) disallowSnapshot does not work for root

[Tsz Wo (Nicholas), SZE (JIRA)]

Tsz Wo (Nicholas), SZE created HDFS-4706:


 Summary: disallowSnapshot does not work for root
 Key: HDFS-4706
 URL: https://issues.apache.org/jira/browse/HDFS-4706
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE


disallowSnapshot replaces a snapshottable directory to a normal directory.  
However, we cannot replace root.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Release Apache Hadoop 0.23.7

[Siddharth Seth]

+1 (binding).
Verified checksums and signature.
Built from source tar, deployed a single node cluster (CapacityScheduler)
and tried a couple of simple MR jobs.

- Sid


On Thu, Apr 11, 2013 at 12:55 PM, Thomas Graves wrote:

> I've created a release candidate (RC0) for hadoop-0.23.7 that I would like
> to release.
>
> This release is a sustaining release with several important bug fixes in
> it.
>
> The RC is available at:
> http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/
> The RC tag in svn is here:
> http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/
>
> The maven artifacts are available via repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Tom Graves
>
>

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

[Siddharth Seth]

+1 (binding)
Verified checksums and signatures.
Built from the source tar, deployed a single node cluster and tested a
couple of simple MR jobs.

- Sid


On Fri, Apr 12, 2013 at 2:56 PM, Arun C Murthy  wrote:

> Folks,
>
> I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I would
> like to release.
>
> The RC is available at:
> http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/
> The RC tag in svn is here:
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2
>
> The maven artifacts are available via repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

[Thomas Graves]

+1 (binding). 

I downloaded, verified signatures and checksums, built, and ran small jobs
on a single node unsecure cluster.

Tom

On 4/12/13 4:56 PM, "Arun C Murthy"  wrote:

>Folks,
>
>I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I
>would like to release.
>
>The RC is available at:
>http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/
>The RC tag in svn is here:
>http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2
>
>The maven artifacts are available via repository.apache.org.
>
>Please try the release and vote; the vote will run for the usual 7 days.
>
>thanks,
>Arun
>
>
>--
>Arun C. Murthy
>Hortonworks Inc.
>http://hortonworks.com/
>
>

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

[Chris Nauroth]

+1 (non-binding)

- downloaded binary tarball
- verified signatures and checksums
- deployed to 3 Ubuntu VMs: 1xNN, 1xRM, 2xDN, 2xNM, 1x2NN
- tested multiple HDFS operations
- ran wordcount MR job
- verified that 2NN can take a checkpoint


On Wed, Apr 17, 2013 at 1:24 AM, Chris Douglas  wrote:

> +1
>
> Verified checksum, signatures. Ran some tests, built the package. -C
>
> On Fri, Apr 12, 2013 at 2:56 PM, Arun C Murthy 
> wrote:
> > Folks,
> >
> > I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I
> would like to release.
> >
> > The RC is available at:
> http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/
> > The RC tag in svn is here:
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2
> >
> > The maven artifacts are available via repository.apache.org.
> >
> > Please try the release and vote; the vote will run for the usual 7 days.
> >
> > thanks,
> > Arun
> >
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>

[jira] [Reopened] (HDFS-3538) TestBlocksWithNotEnoughRacks fails

[Brandon Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Li reopened HDFS-3538:
--


> TestBlocksWithNotEnoughRacks fails
> --
>
> Key: HDFS-3538
> URL: https://issues.apache.org/jira/browse/HDFS-3538
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 0.24.0
>Reporter: Brandon Li
>
> It failed for a few days in jenkins test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Release Apache Hadoop 0.23.7

[Chris Nauroth]

+1 (non-binding)

- downloaded binary tarball
- verified signatures and checksums
- deployed to 3 Ubuntu VMs: 1xNN, 1xRM, 2xDN, 2xNM, 1x2NN
- tested multiple HDFS operations
- ran wordcount MR job
- verified that 2NN can take a checkpoint


On Wed, Apr 17, 2013 at 1:17 AM, Chris Douglas  wrote:

> +1
>
> Verified checksums and signatures, ran some tests, built the tarball. -C
>
> On Thu, Apr 11, 2013 at 12:55 PM, Thomas Graves 
> wrote:
> > I've created a release candidate (RC0) for hadoop-0.23.7 that I would
> like
> > to release.
> >
> > This release is a sustaining release with several important bug fixes in
> > it.
> >
> > The RC is available at:
> > http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/
> > The RC tag in svn is here:
> > http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/
> >
> > The maven artifacts are available via repository.apache.org.
> >
> > Please try the release and vote; the vote will run for the usual 7 days.
> >
> > thanks,
> > Tom Graves
> >
>

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

[Andrew Purtell]

Thanks for the consideration but we've just committed a change to address
this as HBASE-8352


On Wednesday, April 17, 2013, Harsh J wrote:

> Pardon my late inquisition here but since HBase already shipped out
> with a name .snapshots/, why do we force them to change it, and not
> rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
> Snapshots has not been released for any users yet. The way I see it,
> that'd be much more easier to do than making a workaround for a done
> deal on HBase, which already has its snapshot's users.
>
> @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
> applicable to other FileSystem interface implementations as well, then
> .hdfs-snapshots should be fine for it - no?
>
> On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu  wrote:
> > Hi,
> > Please take a look at patch v5 attached to HBASE-8352.
> >
> > It would be nice to resolve this blocker today so that 0.94.7 RC can be
> cut.
> >
> > Thanks
> >
> > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl 
> wrote:
> >
> >> Please see my last comment on the jira. We can make this work without
> >> breaking users who are using HDFS snapshots.
> >>
> >>   --
> >>  *From:* Ted Yu 
> >> *To:* d...@hbase.apache.org
> >> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl 
> >> *Sent:* Tuesday, April 16, 2013 10:00 AM
> >> *Subject:* Re: collision in the naming of '.snapshot' directory between
> >> hdfs snapshot and hbase snapshot
> >>
> >> Let's get proper release notes for HBASE-8352 .
> >>
> >> Either Lars or I can send out notification to user mailing list so that
> >> there is enough preparation for this change.
> >>
> >> Cheers
> >>
> >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh 
> wrote:
> >>
> >> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
> >> 2.1 or 3.0 feature.  Apparently it is targeted as a hadoop 2.0.5
> feature.
> >>  (I'm a little surprised -- expected this to be a hadoop2 compat
> breaking
> >> feature) -- so I agree that this is a bit more urgent.
> >>
> >> Anyway, I agree that the fs .snapshot naming convention is long standing
> >> and should win.
> >>
> >> My concern is with breaking compatibility in 0.94 again -- if we don't
> go
> >> down the conf variable route,  I consider having docs to properly
> document
> >> how to do the upgrade and caveats of doing the upgrade in the
> docs/release
> >> notes blocker to hbase 0.94.7.  (specifically mentioning from 0.94.6 to
> >> 0.94.7, and to possibly to 0.95).
> >>
> >> Jon.
> >>
> >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu  wrote:
> >>
> >> > bq. Alternatively, we can detect the underlying Hadoop version, and
> use
> >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> >> >
> >> > I think this would introduce more confusion, especially for
> operations.
> >> >
> >> > Cheers
> >> >
> >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar 
> >> wrote:
> >> >
> >> > > Because HDFS exposes the snapshots so that the normal file system
> >> > > operations are mapped inside snapshot dirs, I think HDFS reserving
> the
> >> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
> >> name
> >> > > that is chosen by HBase.
> >> > >
> >> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
> >> also
> >> > > being run on top of hadoop 2. Alternatively, we can detect the
> >> underlying
> >> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> >> > > depending on h1 & h2.
> >> > >
> >> > > Enis
> >> > >
> >> > >
> >> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu 
> wrote:
> >> > >
> >> > > > bq. let's make the hbase snapshot for a conf variable.
> >> > > >
> >> > > > Once we decide on the new name of snapshot directory, we should
> still
> >> > use
> >> > > > hardcoded value. This aligns with current code base:
> >> > > > See this snippet from HConstants:
> >--
> Harsh J
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

HDFS pread performance

[lei liu]

I test the HDFS pread performance, the avg time of pread is about 10ms,
but  pread max time reach 200ms, there is about one percent of pread
time is 200ms, that result to my application is timeout.

  I find  max time of the
RemoteBlockReader.readChunk
method also can reach 100ms.  The
RemoteBlockReader.readChunk
only read 512 bytes form socket, so  I think I can tuning linux socket
Kernel parameters to reduce the max time of pread. If anybody has the
experience,
please tell me.


Thansk,

LiuLei

Build failed in Jenkins: Hadoop-Hdfs-trunk #1375

[Apache Jenkins Server]

See 

Changes:

[vinodkv] YARN-500. Fixed YARN webapps to not roll-over ports when explicitly 
asked to use non-ephemeral ports. Contributed by Kenji Kikushima.

[llu] Fix HADOOP-9467 attribution.

[kihwal] MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes 
are different on source/target. Contributed by Mithun Radhakrishnan.

[tgraves] MAPREDUCE-5015. Coverage fix for 
org.apache.hadoop.mapreduce.tools.CLI (Aleksey Gorshkov via tgraves)

[suresh] HDFS-4693. Some test cases in TestCheckpoint do not clean up after 
themselves. Contributed by Arpit Agarwal and Suresh.

--
[...truncated 14112 lines...]
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Hadoop-Hdfs-trunk - Build # 1375 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1375/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 14305 lines...]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at 
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)

Running org.apache.hadoop.contrib.bkjournal.TestCurrentInprogress
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.677 sec
Running org.apache.hadoop.contrib.bkjournal.TestBookKeeperConfiguration
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.064 sec
Running org.apache.hadoop.contrib.bkjournal.TestBookKeeperJournalManager
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.477 sec

Results :

Failed tests:   
testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints):
 SBN should have still been checkpointing.

Tests run: 32, Failures: 1, Errors: 0, Skipped: 0

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  SUCCESS 
[1:28:52.775s]
[INFO] Apache Hadoop HttpFS .. SUCCESS [2:17.642s]
[INFO] Apache Hadoop HDFS BookKeeper Journal . FAILURE [1:00.140s]
[INFO] Apache Hadoop HDFS Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 1:32:11.364s
[INFO] Finished at: Wed Apr 17 13:05:56 UTC 2013
[INFO] Final Memory: 48M/798M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on 
project hadoop-hdfs-bkjournal: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-hdfs-bkjournal
Build step 'Execute shell' marked build as failure
Archiving artifacts
Updating YARN-500
Updating MAPREDUCE-5065
Updating MAPREDUCE-5015
Updating HADOOP-9467
Updating HDFS-4693
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
No tests ran.

Hadoop-Hdfs-0.23-Build - Build # 584 - Unstable

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/584/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 11796 lines...]
[INFO] No dependencies found.
[INFO] Wrote classpath file 
'/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/target/classes/mrapp-generated-classpath'.
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ 
hadoop-hdfs-project ---
[INFO] Installing 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-hdfs-project/0.23.8-SNAPHOST/hadoop-hdfs-project-0.23.8-SNAPHOST.pom
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-hdfs-project 
---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:build-classpath (build-classpath) @ 
hadoop-hdfs-project ---
[INFO] No dependencies found.
[INFO] Skipped writing classpath file 
'/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/target/classes/mrapp-generated-classpath'.
  No changes found.
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  SUCCESS [4:50.896s]
[INFO] Apache Hadoop HttpFS .. SUCCESS [54.577s]
[INFO] Apache Hadoop HDFS Project  SUCCESS [0.057s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 5:46.172s
[INFO] Finished at: Wed Apr 17 11:39:29 UTC 2013
[INFO] Final Memory: 51M/743M
[INFO] 
+ /home/jenkins/tools/maven/latest/bin/mvn test 
-Dmaven.test.failure.ignore=true -Pclover 
-DcloverLicenseLocation=/home/jenkins/tools/clover/latest/lib/clover.license
Archiving artifacts
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Publishing Javadoc
Recording fingerprints
Updating MAPREDUCE-5065
Updating MAPREDUCE-5015
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Unstable
Sending email for trigger: Unstable



###
## FAILED TESTS (if any) 
##
1 tests failed.
REGRESSION:  org.apache.hadoop.hdfs.server.namenode.TestStartup.testNNRestart

Error Message:
java.net.BindException: Port in use: 0.0.0.0:50075  at 
org.apache.hadoop.http.HttpServer.openListener(HttpServer.java:676)  at 
org.apache.hadoop.http.HttpServer.start(HttpServer.java:637)  at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:498)
  at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:703)
  at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:425)  
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1686)
  at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1619)
  at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1588)
  at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:943)  
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:828)  
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:595)
  at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:282)  at 
org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:88)  at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:268)  

Jenkins build became unstable: Hadoop-Hdfs-0.23-Build #584

[Apache Jenkins Server]

See 

How to understand Hadoop source code ?

[Mohammad Mustaqeem]

Hello everyone,
  I am new to this group. Since the source code of Hadoop is very
big, I am not able to understand it entirely.
Is there any document that describes the code?
Is there any way to understand the functionality of each classes and its
method?


-- 
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

[Chris Douglas]

+1

Verified checksum, signatures. Ran some tests, built the package. -C

On Fri, Apr 12, 2013 at 2:56 PM, Arun C Murthy  wrote:
> Folks,
>
> I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I would 
> like to release.
>
> The RC is available at: 
> http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/
> The RC tag in svn is here: 
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2
>
> The maven artifacts are available via repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

[Azuryy Yu]

good suggestion.


On Wed, Apr 17, 2013 at 4:10 PM, Harsh J  wrote:

> Pardon my late inquisition here but since HBase already shipped out
> with a name .snapshots/, why do we force them to change it, and not
> rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
> Snapshots has not been released for any users yet. The way I see it,
> that'd be much more easier to do than making a workaround for a done
> deal on HBase, which already has its snapshot's users.
>
> @Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
> applicable to other FileSystem interface implementations as well, then
> .hdfs-snapshots should be fine for it - no?
>
> On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu  wrote:
> > Hi,
> > Please take a look at patch v5 attached to HBASE-8352.
> >
> > It would be nice to resolve this blocker today so that 0.94.7 RC can be
> cut.
> >
> > Thanks
> >
> > On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl 
> wrote:
> >
> >> Please see my last comment on the jira. We can make this work without
> >> breaking users who are using HDFS snapshots.
> >>
> >>   --
> >>  *From:* Ted Yu 
> >> *To:* d...@hbase.apache.org
> >> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl 
> >> *Sent:* Tuesday, April 16, 2013 10:00 AM
> >> *Subject:* Re: collision in the naming of '.snapshot' directory between
> >> hdfs snapshot and hbase snapshot
> >>
> >> Let's get proper release notes for HBASE-8352 .
> >>
> >> Either Lars or I can send out notification to user mailing list so that
> >> there is enough preparation for this change.
> >>
> >> Cheers
> >>
> >> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh 
> wrote:
> >>
> >> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
> >> 2.1 or 3.0 feature.  Apparently it is targeted as a hadoop 2.0.5
> feature.
> >>  (I'm a little surprised -- expected this to be a hadoop2 compat
> breaking
> >> feature) -- so I agree that this is a bit more urgent.
> >>
> >> Anyway, I agree that the fs .snapshot naming convention is long standing
> >> and should win.
> >>
> >> My concern is with breaking compatibility in 0.94 again -- if we don't
> go
> >> down the conf variable route,  I consider having docs to properly
> document
> >> how to do the upgrade and caveats of doing the upgrade in the
> docs/release
> >> notes blocker to hbase 0.94.7.  (specifically mentioning from 0.94.6 to
> >> 0.94.7, and to possibly to 0.95).
> >>
> >> Jon.
> >>
> >> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu  wrote:
> >>
> >> > bq. Alternatively, we can detect the underlying Hadoop version, and
> use
> >> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
> >> >
> >> > I think this would introduce more confusion, especially for
> operations.
> >> >
> >> > Cheers
> >> >
> >> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar 
> >> wrote:
> >> >
> >> > > Because HDFS exposes the snapshots so that the normal file system
> >> > > operations are mapped inside snapshot dirs, I think HDFS reserving
> the
> >> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
> >> name
> >> > > that is chosen by HBase.
> >> > >
> >> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
> >> also
> >> > > being run on top of hadoop 2. Alternatively, we can detect the
> >> underlying
> >> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
> >> > > depending on h1 & h2.
> >> > >
> >> > > Enis
> >> > >
> >> > >
> >> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu 
> wrote:
> >> > >
> >> > > > bq. let's make the hbase snapshot for a conf variable.
> >> > > >
> >> > > > Once we decide on the new name of snapshot directory, we should
> still
> >> > use
> >> > > > hardcoded value. This aligns with current code base:
> >> > > > See this snippet from HConstants:
> >> > > >
> >> > > >   public static final List HBASE_NON_TABLE_DIRS =
> >> > > >
> >> > > > Collections.unmodifiableList(Arrays.asList(new String[] {
> >> > > > HREGION_LOGDIR_NAME,
> >> > > >
> >> > > >   HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
> >> > > >
> >> > > >   HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
> >> > SNAPSHOT_DIR_NAME,
> >> > > > HBASE_TEMP_DIRECTORY }));
> >> > > > Cheers
> >> > > >
> >> > > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh  >
> >> > > wrote:
> >> > > >
> >> > > > > constraints:
> >> > > > >
> >> > > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
> >> > > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that
> it
> >> > will
> >> > > > ever
> >> > > > > make it to 1.x.  This hdfs feature ideally this shouldn't affect
> >> > > current
> >> > > > A
> >> > > > > pache Hbase 0.94.x's.
> >> > > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these
> versions
> >> > > should
> >> > > > > pick a different table snapshot name to respect fs conventions.
> >> > > > >
> >> > > > > proposed actions:
> >> > > > >
> >> > > > > 1) let's make the hbase snapshot 

Re: [VOTE] Release Apache Hadoop 0.23.7

[Chris Douglas]

+1

Verified checksums and signatures, ran some tests, built the tarball. -C

On Thu, Apr 11, 2013 at 12:55 PM, Thomas Graves  wrote:
> I've created a release candidate (RC0) for hadoop-0.23.7 that I would like
> to release.
>
> This release is a sustaining release with several important bug fixes in
> it.
>
> The RC is available at:
> http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/
> The RC tag in svn is here:
> http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/
>
> The maven artifacts are available via repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Tom Graves
>

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

[Harsh J]

Pardon my late inquisition here but since HBase already shipped out
with a name .snapshots/, why do we force them to change it, and not
rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS
Snapshots has not been released for any users yet. The way I see it,
that'd be much more easier to do than making a workaround for a done
deal on HBase, which already has its snapshot's users.

@Tsz-Wo - If the snapshots in HDFS aren't a 'generic' feature
applicable to other FileSystem interface implementations as well, then
.hdfs-snapshots should be fine for it - no?

On Wed, Apr 17, 2013 at 2:32 AM, Ted Yu  wrote:
> Hi,
> Please take a look at patch v5 attached to HBASE-8352.
>
> It would be nice to resolve this blocker today so that 0.94.7 RC can be cut.
>
> Thanks
>
> On Tue, Apr 16, 2013 at 10:12 AM, lars hofhansl  wrote:
>
>> Please see my last comment on the jira. We can make this work without
>> breaking users who are using HDFS snapshots.
>>
>>   --
>>  *From:* Ted Yu 
>> *To:* d...@hbase.apache.org
>> *Cc:* hdfs-dev@hadoop.apache.org; lars hofhansl 
>> *Sent:* Tuesday, April 16, 2013 10:00 AM
>> *Subject:* Re: collision in the naming of '.snapshot' directory between
>> hdfs snapshot and hbase snapshot
>>
>> Let's get proper release notes for HBASE-8352 .
>>
>> Either Lars or I can send out notification to user mailing list so that
>> there is enough preparation for this change.
>>
>> Cheers
>>
>> On Tue, Apr 16, 2013 at 8:46 AM, Jonathan Hsieh  wrote:
>>
>> I was away from keyboard when I asserted that hdfs snapshot was a hadoop
>> 2.1 or 3.0 feature.  Apparently it is targeted as a hadoop 2.0.5 feature.
>>  (I'm a little surprised -- expected this to be a hadoop2 compat breaking
>> feature) -- so I agree that this is a bit more urgent.
>>
>> Anyway, I agree that the fs .snapshot naming convention is long standing
>> and should win.
>>
>> My concern is with breaking compatibility in 0.94 again -- if we don't go
>> down the conf variable route,  I consider having docs to properly document
>> how to do the upgrade and caveats of doing the upgrade in the docs/release
>> notes blocker to hbase 0.94.7.  (specifically mentioning from 0.94.6 to
>> 0.94.7, and to possibly to 0.95).
>>
>> Jon.
>>
>> On Mon, Apr 15, 2013 at 9:00 PM, Ted Yu  wrote:
>>
>> > bq. Alternatively, we can detect the underlying Hadoop version, and use
>> > either .snapshot or .hbase_snapshot in 0.94 depending on h1 & h2.
>> >
>> > I think this would introduce more confusion, especially for operations.
>> >
>> > Cheers
>> >
>> > On Mon, Apr 15, 2013 at 8:52 PM, Enis Söztutar 
>> wrote:
>> >
>> > > Because HDFS exposes the snapshots so that the normal file system
>> > > operations are mapped inside snapshot dirs, I think HDFS reserving the
>> > > .snapshot name makes sense. OTOH, nothing is specific about the dir
>> name
>> > > that is chosen by HBase.
>> > >
>> > > I would prefer to change the dir name in 0.94 as well, since 0.94 is
>> also
>> > > being run on top of hadoop 2. Alternatively, we can detect the
>> underlying
>> > > Hadoop version, and use either .snapshot or .hbase_snapshot in 0.94
>> > > depending on h1 & h2.
>> > >
>> > > Enis
>> > >
>> > >
>> > > On Mon, Apr 15, 2013 at 8:31 PM, Ted Yu  wrote:
>> > >
>> > > > bq. let's make the hbase snapshot for a conf variable.
>> > > >
>> > > > Once we decide on the new name of snapshot directory, we should still
>> > use
>> > > > hardcoded value. This aligns with current code base:
>> > > > See this snippet from HConstants:
>> > > >
>> > > >   public static final List HBASE_NON_TABLE_DIRS =
>> > > >
>> > > > Collections.unmodifiableList(Arrays.asList(new String[] {
>> > > > HREGION_LOGDIR_NAME,
>> > > >
>> > > >   HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, SPLIT_LOGDIR_NAME,
>> > > >
>> > > >   HBCK_SIDELINEDIR_NAME, HFILE_ARCHIVE_DIRECTORY,
>> > SNAPSHOT_DIR_NAME,
>> > > > HBASE_TEMP_DIRECTORY }));
>> > > > Cheers
>> > > >
>> > > > On Mon, Apr 15, 2013 at 8:24 PM, Jonathan Hsieh 
>> > > wrote:
>> > > >
>> > > > > constraints:
>> > > > >
>> > > > > 1) hbase 0.94.6 is released and .snapshot is hardcoded in there.
>> > > > > 2) hdfs snapshots is a Hadoop 2.1 or 3.0 feature. I doubt that it
>> > will
>> > > > ever
>> > > > > make it to 1.x.  This hdfs feature ideally this shouldn't affect
>> > > current
>> > > > A
>> > > > > pache Hbase 0.94.x's.
>> > > > > 3) hbase 95/96 may default to Hadoop1 or Hadoop 2. these versions
>> > > should
>> > > > > pick a different table snapshot name to respect fs conventions.
>> > > > >
>> > > > > proposed actions:
>> > > > >
>> > > > > 1) let's make the hbase snapshot for a conf variable. (hbase.
>> > > > > snapshots.dir)  let's change the default for hbase 95+. (maybe
>> > > > > .hbase-snapshots). we'll also port this patch to 0.94.x
>> > > > > 2) let's publish instructions on how to update the hbase snapshot
>> > dir:
>> > > > > shutdown hbase, config update, rename dir, restart hbase.
>> > > > > 3) I lean towa