Build failed in Jenkins: Hadoop-Hdfs-trunk #1484

[Apache Jenkins Server]

See 

Changes:

[cnauroth] HADOOP-9527. Add symlink support to LocalFileSystem on Windows. 
Contributed by Arpit Agarwal.

[jeagles] YARN-985. Nodemanager should log where a resource was localized (Ravi 
Prakash via jeagles)

[jing9] HADOOP-9821. ClientId should have getMsb/getLsb methods. Contributed by 
Tsuyoshi OZAWA.

--
[...truncated 11052 lines...]
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2031)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2027)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1493)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2025)

at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1352)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at $Proxy15.delete(Unknown Source)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at $Proxy15.delete(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:449)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1575)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$11.doCall(DistributedFileSystem.java:585)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$11.doCall(DistributedFileSystem.java:581)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:581)
at org.apache.hadoop.fs.TestGlobPaths.cleanupDFS(TestGlobPaths.java:788)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.ja

Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845

[Alejandro Abdelnur]

I' like to upgrade to protobuf 2.5.0 for the 2.1.0 release.

As mentioned in HADOOP-9845, Protobuf 2.5 has significant benefits to
justify the upgrade.

Doing the upgrade now, with the first beta, will make things easier for
downstream projects (like HBase) using protobuf and adopting Hadoop 2. If
we do the upgrade later, downstream projects will have to support 2
different versions and they my get in nasty waters due to classpath issues.

I've locally tested the patch in a pseudo deployment of 2.1.0-beta branch
and it works fine (something is broken in trunk in the RPC layer YARN-885).

Now, to do this it will require a few things:

* Make sure protobuf 2.5.0 is available in the jenkins box
* A follow up email to dev@ aliases indicating developers should install
locally protobuf 2.5.0

Thanks.

-- 
Alejandro

Feature request to provide DFSInputStream subclassing mechanism

[Jeff Dost]

Hello,

We work in a software development team at the UCSD CMS Tier2 Center.  We 
would like to propose a mechanism to allow one to subclass the 
DFSInputStream in a clean way from an external package.  First I'd like 
to give some motivation on why and then will proceed with the details.


We have a 3 Petabyte Hadoop cluster we maintain for the LHC experiment 
at CERN.  There are other T2 centers worldwide that contain mirrors of 
the same data we host.  We are working on an extension to Hadoop that, 
on reading a file, if it is found that there are no available replicas 
of a block, we use an external interface to retrieve this block of the 
file from another data center.  The external interface is necessary 
because not all T2 centers involved in CMS are running a Hadoop cluster 
as their storage backend.


In order to implement this functionality, we need to subclass the 
DFSInputStream and override the read method, so we can catch 
IOExceptions that occur on client reads at the block level.


The basic steps required:
1. Invent a new URI scheme for the customized "FileSystem" in core-site.xml:
  
fs.foofs.impl
my.package.FooFileSystem
My Extended FileSystem for foofs: uris.
  

2. Write new classes included in the external package that subclass the 
following:

FooFileSystem subclasses DistributedFileSystem
FooFSClient subclasses DFSClient
FooFSInputStream subclasses DFSInputStream

Now any client commands that explicitly use the foofs:// scheme in paths 
to access the hadoop cluster can open files with a customized 
InputStream that extends functionality of the default hadoop client 
DFSInputStream.  In order to make this happen for our use case, we had 
to change some access modifiers in the DistributedFileSystem, DFSClient, 
and DFSInputStream classes provided by Hadoop.  In addition, we had to 
comment out the check in the namenode code that only allows for URI 
schemes of the form "hdfs://".


Attached is a patch file we apply to hadoop.  Note that we derived this 
patch by modding the Cloudera release hadoop-2.0.0-cdh4.1.1 which can be 
found at:

http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.1.1.tar.gz

We would greatly appreciate any advise on whether or not this approach 
sounds reasonable, and if you would consider accepting these 
modifications into the official Hadoop code base.


Thank you,
Jeff, Alja & Matevz
UCSD Physics
diff -ru 
orig/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 
new/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
--- 
orig/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
2013-08-05 13:04:11.376533999 -0700
+++ 
new/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 2013-08-05 13:04:11.444567994 -0700
@@ -504,7 +504,7 @@
 return dfsClientConf.connectToDnViaHostname;
   }
 
-  void checkOpen() throws IOException {
+  public void checkOpen() throws IOException {
 if (!clientRunning) {
   IOException result = new IOException("Filesystem closed");
   throw result;
@@ -2077,4 +2077,12 @@
   void disableShortCircuit() {
 shortCircuitLocalReads = false;
   }
+  
+  public Configuration getConfig() {
+return conf;
+  }
+  
+  public FileSystem.Statistics getStats() {
+return stats;
+  }
 }
diff -ru 
orig/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
 
new/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
--- 
orig/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
   2013-08-05 13:04:11.400545996 -0700
+++ 
new/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
2013-08-05 13:04:11.455573494 -0700
@@ -62,8 +62,8 @@
 public class DFSInputStream extends FSInputStream implements 
ByteBufferReadable {
   private final SocketCache socketCache;
 
-  private final DFSClient dfsClient;
-  private boolean closed = false;
+  protected final DFSClient dfsClient;
+  protected boolean closed = false;
 
   private final String src;
   private final long prefetchSize;
@@ -87,7 +87,7 @@
* back to the namenode to get a new list of block locations, and is
* capped at maxBlockAcquireFailures
*/
-  private int failures = 0;
+  protected int failures = 0;
   private final int timeWindow;
 
   /* XXX Use of CocurrentHashMap is temp fix. Need to fix 
@@ -104,7 +104,7 @@
 deadNodes.put(dnInfo, dnInfo);
   }
   
-  DFSInputStream(DFSClient dfsClient, String src, int buffersize, boolean 
verifyChecksum
+  public DFSInputStream(DFSClient dfsClient, String src, int buffersize, 
boolean verifyChecksum
  ) throws IOException, UnresolvedLinkException {
 this.dfsClient = dfsClient;
 this.verifyChecksum = verifyChecksum;
@@ -346,7 +346,7 @@
* @return consequent segment of loc

Re: Feature request to provide DFSInputStream subclassing mechanism

[Joe Bounour]

Hello Jeff

Is it something that could go under HCFS project?
http://wiki.apache.org/hadoop/HCFS
(I might be wrong?)

Joe


On 8/7/13 10:59 AM, "Jeff Dost"  wrote:

>Hello,
>
>We work in a software development team at the UCSD CMS Tier2 Center.  We
>would like to propose a mechanism to allow one to subclass the
>DFSInputStream in a clean way from an external package.  First I'd like
>to give some motivation on why and then will proceed with the details.
>
>We have a 3 Petabyte Hadoop cluster we maintain for the LHC experiment
>at CERN.  There are other T2 centers worldwide that contain mirrors of
>the same data we host.  We are working on an extension to Hadoop that,
>on reading a file, if it is found that there are no available replicas
>of a block, we use an external interface to retrieve this block of the
>file from another data center.  The external interface is necessary
>because not all T2 centers involved in CMS are running a Hadoop cluster
>as their storage backend.
>
>In order to implement this functionality, we need to subclass the
>DFSInputStream and override the read method, so we can catch
>IOExceptions that occur on client reads at the block level.
>
>The basic steps required:
>1. Invent a new URI scheme for the customized "FileSystem" in
>core-site.xml:
>   
> fs.foofs.impl
> my.package.FooFileSystem
> My Extended FileSystem for foofs: uris.
>   
>
>2. Write new classes included in the external package that subclass the
>following:
>FooFileSystem subclasses DistributedFileSystem
>FooFSClient subclasses DFSClient
>FooFSInputStream subclasses DFSInputStream
>
>Now any client commands that explicitly use the foofs:// scheme in paths
>to access the hadoop cluster can open files with a customized
>InputStream that extends functionality of the default hadoop client
>DFSInputStream.  In order to make this happen for our use case, we had
>to change some access modifiers in the DistributedFileSystem, DFSClient,
>and DFSInputStream classes provided by Hadoop.  In addition, we had to
>comment out the check in the namenode code that only allows for URI
>schemes of the form "hdfs://".
>
>Attached is a patch file we apply to hadoop.  Note that we derived this
>patch by modding the Cloudera release hadoop-2.0.0-cdh4.1.1 which can be
>found at:
>http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.1.1.tar.gz
>
>We would greatly appreciate any advise on whether or not this approach
>sounds reasonable, and if you would consider accepting these
>modifications into the official Hadoop code base.
>
>Thank you,
>Jeff, Alja & Matevz
>UCSD Physics

Re: Feature request to provide DFSInputStream subclassing mechanism

[Todd Lipcon]

Hi Jeff,

Do you need to subclass or could you simply wrap? Generally composition as
opposed to inheritance is a lot safer way of integrating software written
by different parties, since inheritance exposes all the implementation
details which are subject to change.

-Todd

On Wed, Aug 7, 2013 at 10:59 AM, Jeff Dost  wrote:

> Hello,
>
> We work in a software development team at the UCSD CMS Tier2 Center.  We
> would like to propose a mechanism to allow one to subclass the
> DFSInputStream in a clean way from an external package.  First I'd like to
> give some motivation on why and then will proceed with the details.
>
> We have a 3 Petabyte Hadoop cluster we maintain for the LHC experiment at
> CERN.  There are other T2 centers worldwide that contain mirrors of the
> same data we host.  We are working on an extension to Hadoop that, on
> reading a file, if it is found that there are no available replicas of a
> block, we use an external interface to retrieve this block of the file from
> another data center.  The external interface is necessary because not all
> T2 centers involved in CMS are running a Hadoop cluster as their storage
> backend.
>
> In order to implement this functionality, we need to subclass the
> DFSInputStream and override the read method, so we can catch IOExceptions
> that occur on client reads at the block level.
>
> The basic steps required:
> 1. Invent a new URI scheme for the customized "FileSystem" in
> core-site.xml:
>   
> fs.foofs.impl
> my.package.**FooFileSystem
> My Extended FileSystem for foofs: uris.
>   
>
> 2. Write new classes included in the external package that subclass the
> following:
> FooFileSystem subclasses DistributedFileSystem
> FooFSClient subclasses DFSClient
> FooFSInputStream subclasses DFSInputStream
>
> Now any client commands that explicitly use the foofs:// scheme in paths
> to access the hadoop cluster can open files with a customized InputStream
> that extends functionality of the default hadoop client DFSInputStream.  In
> order to make this happen for our use case, we had to change some access
> modifiers in the DistributedFileSystem, DFSClient, and DFSInputStream
> classes provided by Hadoop.  In addition, we had to comment out the check
> in the namenode code that only allows for URI schemes of the form "hdfs://".
>
> Attached is a patch file we apply to hadoop.  Note that we derived this
> patch by modding the Cloudera release hadoop-2.0.0-cdh4.1.1 which can be
> found at:
> http://archive.cloudera.com/**cdh4/cdh/4/hadoop-2.0.0-cdh4.**1.1.tar.gz
>
> We would greatly appreciate any advise on whether or not this approach
> sounds reasonable, and if you would consider accepting these modifications
> into the official Hadoop code base.
>
> Thank you,
> Jeff, Alja & Matevz
> UCSD Physics
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Feature request to provide DFSInputStream subclassing mechanism

[Andrew Wang]

I don't think exposing DFSClient and DistributedFileSystem members is
necessary to achieve what you're trying to do. We've got wrapper
FileSystems like FilterFileSystem and ViewFileSystem which you might be
able to use for inspiration, and the HCFS wiki lists some third-party
FileSystems that might also be helpful too.


On Wed, Aug 7, 2013 at 11:11 AM, Joe Bounour  wrote:

> Hello Jeff
>
> Is it something that could go under HCFS project?
> http://wiki.apache.org/hadoop/HCFS
> (I might be wrong?)
>
> Joe
>
>
> On 8/7/13 10:59 AM, "Jeff Dost"  wrote:
>
> >Hello,
> >
> >We work in a software development team at the UCSD CMS Tier2 Center.  We
> >would like to propose a mechanism to allow one to subclass the
> >DFSInputStream in a clean way from an external package.  First I'd like
> >to give some motivation on why and then will proceed with the details.
> >
> >We have a 3 Petabyte Hadoop cluster we maintain for the LHC experiment
> >at CERN.  There are other T2 centers worldwide that contain mirrors of
> >the same data we host.  We are working on an extension to Hadoop that,
> >on reading a file, if it is found that there are no available replicas
> >of a block, we use an external interface to retrieve this block of the
> >file from another data center.  The external interface is necessary
> >because not all T2 centers involved in CMS are running a Hadoop cluster
> >as their storage backend.
> >
> >In order to implement this functionality, we need to subclass the
> >DFSInputStream and override the read method, so we can catch
> >IOExceptions that occur on client reads at the block level.
> >
> >The basic steps required:
> >1. Invent a new URI scheme for the customized "FileSystem" in
> >core-site.xml:
> >   
> > fs.foofs.impl
> > my.package.FooFileSystem
> > My Extended FileSystem for foofs: uris.
> >   
> >
> >2. Write new classes included in the external package that subclass the
> >following:
> >FooFileSystem subclasses DistributedFileSystem
> >FooFSClient subclasses DFSClient
> >FooFSInputStream subclasses DFSInputStream
> >
> >Now any client commands that explicitly use the foofs:// scheme in paths
> >to access the hadoop cluster can open files with a customized
> >InputStream that extends functionality of the default hadoop client
> >DFSInputStream.  In order to make this happen for our use case, we had
> >to change some access modifiers in the DistributedFileSystem, DFSClient,
> >and DFSInputStream classes provided by Hadoop.  In addition, we had to
> >comment out the check in the namenode code that only allows for URI
> >schemes of the form "hdfs://".
> >
> >Attached is a patch file we apply to hadoop.  Note that we derived this
> >patch by modding the Cloudera release hadoop-2.0.0-cdh4.1.1 which can be
> >found at:
> >http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.1.1.tar.gz
> >
> >We would greatly appreciate any advise on whether or not this approach
> >sounds reasonable, and if you would consider accepting these
> >modifications into the official Hadoop code base.
> >
> >Thank you,
> >Jeff, Alja & Matevz
> >UCSD Physics
>
>

[jira] [Created] (HDFS-5075) httpfs-config.sh calls out incorrect env script name

[Timothy St. Clair (JIRA)]

Timothy St. Clair created HDFS-5075:
---

 Summary: httpfs-config.sh calls out incorrect env script name
 Key: HDFS-5075
 URL: https://issues.apache.org/jira/browse/HDFS-5075
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Timothy St. Clair


looks just like a simple mistake

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-5076) Create http servlets to enable querying NN's last applied transaction ID and most recent checkpoint's transaction ID

[Jing Zhao (JIRA)]

Jing Zhao created HDFS-5076:
---

 Summary: Create http servlets to enable querying NN's last applied 
transaction ID and most recent checkpoint's transaction ID
 Key: HDFS-5076
 URL: https://issues.apache.org/jira/browse/HDFS-5076
 Project: Hadoop HDFS
  Issue Type: New Feature
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor


Currently NameNode already provides RPC calls to get its last applied 
transaction ID and most recent checkpoint's transaction ID. It can be helpful 
to provide servlets to enable querying these information through http, so that 
administrators and applications like Ambari can easily decide if a forced 
checkpoint by calling saveNamespace is necessary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-4926) namenode webserver's page has a tooltip that is inconsistent with the datanode HTML link

[Jing Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-4926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao resolved HDFS-4926.
-

   Resolution: Fixed
Fix Version/s: 2.1.1-beta
 Hadoop Flags: Reviewed

I've committed this to trunk, branch-2, and branch-2.1-beta.

> namenode webserver's page has a tooltip that is inconsistent with the 
> datanode HTML link
> 
>
> Key: HDFS-4926
> URL: https://issues.apache.org/jira/browse/HDFS-4926
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Joseph Lorenzini
>Assignee: Vivek Ganesan
>Priority: Trivial
> Fix For: 2.1.1-beta
>
> Attachments: DeadDN_Aug_01.png, dead_nodes.png, 
> Decommissioning_Aug_01.png, decommission.png, HDFS-4926.patch, 
> HDFS-4926.patch.1, live_nodes.png
>
>
> At http://localhost:50070/dfsnodelist.jsp?whatNodes=LIVE, there is a list of 
> Live datanodes in a table.  In the Node column, there is a link to each node. 
> If you hover your cursor over one of the links, it shows the IP address and 
> port number of the datanode. However, actual link is the hostname instead of 
> the IP. 
> Suggest that the tooltip reflect the actual url that the web browser would go 
> to if the link is selected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-5077) NPE in FSNamesystem.commitBlockSynchronization()

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HDFS-5077:
-

 Summary: NPE in FSNamesystem.commitBlockSynchronization()
 Key: HDFS-5077
 URL: https://issues.apache.org/jira/browse/HDFS-5077
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.0.5-alpha
Reporter: Konstantin Shvachko


NN starts a block recovery, which will synchronize block replicas on different 
DNs. In the end one of DNs will report the list of the nodes containing the 
consistent replicas to the NN via commitBlockSynchronization() call. The NPE 
happens if just before processing commitBlockSynchronization() NN removes from 
active one of DNs that are then reported in the call.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Feature request to provide DFSInputStream subclassing mechanism

[Jeff Dost]

Thank you for the suggestion, but we don't see how simply wrapping a 
FileSystem object would be sufficient in our use case.  The reason why 
is we need to catch and handle read exceptions at the block level.  
There aren't any public methods available in the high level FileSystem 
abstraction layer that would give us the fine grained control we need at 
block level read failures.


Perhaps if I outline the steps more clearly it will help explain what we 
are trying to do.  Without our enhancements, suppose a user opens a file 
stream and starts reading the file from Hadoop. After some time, at some 
position into the file, if there happen to be no replicas available for 
a particular block for whatever reason, datanodes have gone down due to 
disk issues, etc. the stream will throw an IOException 
(BlockMissingException or similar) and the read will fail.


What we are doing is rather than letting the stream fail, we have 
another stream queued up that knows how to fetch the blocks elsewhere 
outside of our Hadoop cluster that couldn't be retrieved.  So we need to 
be able to catch the exception at this point, and these externally 
fetched bytes then get read into the user supplied read buffer.  Now 
Hadoop can proceed to read in the stream the next blocks in the file.


So as you can see this method of fail over on demand allows an input 
stream to keep reading data, without having to start it all over again 
if a failure occurs (assuming the remote bytes were successfully fetched).


As a final note I would like to mention that we will be providing our 
failover module to the Open Science Grid.  Since we hope to provide this 
as a benefit to all OSG users running at participating T2 computing 
clusters, we will be committed to maintaining this software and any 
changes to Hadoop needed to make it work.  In other words we will be 
willing to maintain any implementation changes that may become necessary 
as Hadoop internals change in future releases.


Thanks,
Jeff

On 8/7/13 11:30 AM, Andrew Wang wrote:

I don't think exposing DFSClient and DistributedFileSystem members is
necessary to achieve what you're trying to do. We've got wrapper
FileSystems like FilterFileSystem and ViewFileSystem which you might be
able to use for inspiration, and the HCFS wiki lists some third-party
FileSystems that might also be helpful too.


On Wed, Aug 7, 2013 at 11:11 AM, Joe Bounour  wrote:


Hello Jeff

Is it something that could go under HCFS project?
http://wiki.apache.org/hadoop/HCFS
(I might be wrong?)

Joe


On 8/7/13 10:59 AM, "Jeff Dost"  wrote:


Hello,

We work in a software development team at the UCSD CMS Tier2 Center.  We
would like to propose a mechanism to allow one to subclass the
DFSInputStream in a clean way from an external package.  First I'd like
to give some motivation on why and then will proceed with the details.

We have a 3 Petabyte Hadoop cluster we maintain for the LHC experiment
at CERN.  There are other T2 centers worldwide that contain mirrors of
the same data we host.  We are working on an extension to Hadoop that,
on reading a file, if it is found that there are no available replicas
of a block, we use an external interface to retrieve this block of the
file from another data center.  The external interface is necessary
because not all T2 centers involved in CMS are running a Hadoop cluster
as their storage backend.

In order to implement this functionality, we need to subclass the
DFSInputStream and override the read method, so we can catch
IOExceptions that occur on client reads at the block level.

The basic steps required:
1. Invent a new URI scheme for the customized "FileSystem" in
core-site.xml:
   
 fs.foofs.impl
 my.package.FooFileSystem
 My Extended FileSystem for foofs: uris.
   

2. Write new classes included in the external package that subclass the
following:
FooFileSystem subclasses DistributedFileSystem
FooFSClient subclasses DFSClient
FooFSInputStream subclasses DFSInputStream

Now any client commands that explicitly use the foofs:// scheme in paths
to access the hadoop cluster can open files with a customized
InputStream that extends functionality of the default hadoop client
DFSInputStream.  In order to make this happen for our use case, we had
to change some access modifiers in the DistributedFileSystem, DFSClient,
and DFSInputStream classes provided by Hadoop.  In addition, we had to
comment out the check in the namenode code that only allows for URI
schemes of the form "hdfs://".

Attached is a patch file we apply to hadoop.  Note that we derived this
patch by modding the Cloudera release hadoop-2.0.0-cdh4.1.1 which can be
found at:
http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.1.1.tar.gz

We would greatly appreciate any advise on whether or not this approach
sounds reasonable, and if you would consider accepting these
modifications into the official Hadoop code base.

Thank you,
Jeff, Alja & Matevz
UCSD Physics

[jira] [Created] (HDFS-5078) Support file append in NFSv3 gateway to enable data streaming to HDFS

[Brandon Li (JIRA)]

Brandon Li created HDFS-5078:


 Summary: Support file append in NFSv3 gateway to enable data 
streaming to HDFS
 Key: HDFS-5078
 URL: https://issues.apache.org/jira/browse/HDFS-5078
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: nfs
Affects Versions: 3.0.0
Reporter: Brandon Li
Assignee: Brandon Li




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HDFS-5079:
-

 Summary: Cleaning up NNHAStatusHeartbeat.State 
DatanodeProtocolProtos.
 Key: HDFS-5079
 URL: https://issues.apache.org/jira/browse/HDFS-5079
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Konstantin Shvachko


NNHAStatusHeartbeat.State was removed from usage by HDFS-4268. The respective 
class should also be removed from DatanodeProtocolProtos.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Feature request to provide DFSInputStream subclassing mechanism

[Andrew Wang]

Blocks are supposed to be an internal abstraction within HDFS, and aren't
an inherent part of FileSystem (the user-visible class used to access all
Hadoop filesystems).

Is it possible to instead deal with files and offsets? On a read failure,
you could open a stream to the same file on the backup filesystem, seek to
the old file position, and retry the read. This feels like it's possible
via wrapping.


On Wed, Aug 7, 2013 at 3:29 PM, Jeff Dost  wrote:

> Thank you for the suggestion, but we don't see how simply wrapping a
> FileSystem object would be sufficient in our use case.  The reason why is
> we need to catch and handle read exceptions at the block level.  There
> aren't any public methods available in the high level FileSystem
> abstraction layer that would give us the fine grained control we need at
> block level read failures.
>
> Perhaps if I outline the steps more clearly it will help explain what we
> are trying to do.  Without our enhancements, suppose a user opens a file
> stream and starts reading the file from Hadoop. After some time, at some
> position into the file, if there happen to be no replicas available for a
> particular block for whatever reason, datanodes have gone down due to disk
> issues, etc. the stream will throw an IOException (BlockMissingException or
> similar) and the read will fail.
>
> What we are doing is rather than letting the stream fail, we have another
> stream queued up that knows how to fetch the blocks elsewhere outside of
> our Hadoop cluster that couldn't be retrieved.  So we need to be able to
> catch the exception at this point, and these externally fetched bytes then
> get read into the user supplied read buffer.  Now Hadoop can proceed to
> read in the stream the next blocks in the file.
>
> So as you can see this method of fail over on demand allows an input
> stream to keep reading data, without having to start it all over again if a
> failure occurs (assuming the remote bytes were successfully fetched).
>
> As a final note I would like to mention that we will be providing our
> failover module to the Open Science Grid.  Since we hope to provide this as
> a benefit to all OSG users running at participating T2 computing clusters,
> we will be committed to maintaining this software and any changes to Hadoop
> needed to make it work.  In other words we will be willing to maintain any
> implementation changes that may become necessary as Hadoop internals change
> in future releases.
>
> Thanks,
> Jeff
>
>
> On 8/7/13 11:30 AM, Andrew Wang wrote:
>
>> I don't think exposing DFSClient and DistributedFileSystem members is
>> necessary to achieve what you're trying to do. We've got wrapper
>> FileSystems like FilterFileSystem and ViewFileSystem which you might be
>> able to use for inspiration, and the HCFS wiki lists some third-party
>> FileSystems that might also be helpful too.
>>
>>
>> On Wed, Aug 7, 2013 at 11:11 AM, Joe Bounour  wrote:
>>
>>  Hello Jeff
>>>
>>> Is it something that could go under HCFS project?
>>> http://wiki.apache.org/hadoop/**HCFS
>>> (I might be wrong?)
>>>
>>> Joe
>>>
>>>
>>> On 8/7/13 10:59 AM, "Jeff Dost"  wrote:
>>>
>>>  Hello,

 We work in a software development team at the UCSD CMS Tier2 Center.  We
 would like to propose a mechanism to allow one to subclass the
 DFSInputStream in a clean way from an external package.  First I'd like
 to give some motivation on why and then will proceed with the details.

 We have a 3 Petabyte Hadoop cluster we maintain for the LHC experiment
 at CERN.  There are other T2 centers worldwide that contain mirrors of
 the same data we host.  We are working on an extension to Hadoop that,
 on reading a file, if it is found that there are no available replicas
 of a block, we use an external interface to retrieve this block of the
 file from another data center.  The external interface is necessary
 because not all T2 centers involved in CMS are running a Hadoop cluster
 as their storage backend.

 In order to implement this functionality, we need to subclass the
 DFSInputStream and override the read method, so we can catch
 IOExceptions that occur on client reads at the block level.

 The basic steps required:
 1. Invent a new URI scheme for the customized "FileSystem" in
 core-site.xml:

  fs.foofs.impl
  my.package.**FooFileSystem
  My Extended FileSystem for foofs: uris.


 2. Write new classes included in the external package that subclass the
 following:
 FooFileSystem subclasses DistributedFileSystem
 FooFSClient subclasses DFSClient
 FooFSInputStream subclasses DFSInputStream

 Now any client commands that explicitly use the foofs:// scheme in paths
 to access the hadoop cluster can open files with a customized
 InputStream that extends functionality of the default hadoop client
 DFSInp

Hadoop-Hdfs-trunk - Build # 1484 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1484/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 11245 lines...]
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-hdfs-project 
---
[INFO] Deleting 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/target
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-hdfs-project 
---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.0:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE 
[1:33:10.945s]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [1.892s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 1:33:13.693s
[INFO] Finished at: Wed Aug 07 13:06:46 UTC 2013
[INFO] Final Memory: 31M/255M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Build step 'Execute shell' marked build as failure
Archiving artifacts
Updating HADOOP-9821
Updating HADOOP-9527
Updating YARN-985
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
No tests ran.