Build failed in Jenkins: Hadoop-Common-trunk #1091
See https://builds.apache.org/job/Hadoop-Common-trunk/1091/
--
[...truncated 62608 lines...]
Adding reference: maven.local.repository
[DEBUG] Initialize Maven Ant Tasks
parsing buildfile
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml
with URI =
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml
from a zip file
parsing buildfile
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
with URI =
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
from a zip file
Class org.apache.maven.ant.tasks.AttachArtifactTask loaded from parent loader
(parentFirst)
+Datatype attachartifact org.apache.maven.ant.tasks.AttachArtifactTask
Class org.apache.maven.ant.tasks.DependencyFilesetsTask loaded from parent
loader (parentFirst)
+Datatype dependencyfilesets org.apache.maven.ant.tasks.DependencyFilesetsTask
Setting project property: test.build.dir -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir
Setting project property: test.exclude.pattern - _
Setting project property: hadoop.assemblies.version - 3.0.0-SNAPSHOT
Setting project property: test.exclude - _
Setting project property: distMgmtSnapshotsId - apache.snapshots.https
Setting project property: project.build.sourceEncoding - UTF-8
Setting project property: java.security.egd - file:///dev/urandom
Setting project property: distMgmtSnapshotsUrl -
https://repository.apache.org/content/repositories/snapshots
Setting project property: distMgmtStagingUrl -
https://repository.apache.org/service/local/staging/deploy/maven2
Setting project property: avro.version - 1.7.4
Setting project property: test.build.data -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir
Setting project property: commons-daemon.version - 1.0.13
Setting project property: hadoop.common.build.dir -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/../../hadoop-common-project/hadoop-common/target
Setting project property: testsThreadCount - 4
Setting project property: maven.test.redirectTestOutputToFile - true
Setting project property: jdiff.version - 1.0.9
Setting project property: build.platform - Linux-i386-32
Setting project property: project.reporting.outputEncoding - UTF-8
Setting project property: distMgmtStagingName - Apache Release Distribution
Repository
Setting project property: protobuf.version - 2.5.0
Setting project property: failIfNoTests - false
Setting project property: protoc.path - ${env.HADOOP_PROTOC_PATH}
Setting project property: jersey.version - 1.9
Setting project property: distMgmtStagingId - apache.staging.https
Setting project property: distMgmtSnapshotsName - Apache Development Snapshot
Repository
Setting project property: ant.file -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/pom.xml
[DEBUG] Setting properties with prefix:
Setting project property: project.groupId - org.apache.hadoop
Setting project property: project.artifactId - hadoop-common-project
Setting project property: project.name - Apache Hadoop Common Project
Setting project property: project.description - Apache Hadoop Common Project
Setting project property: project.version - 3.0.0-SNAPSHOT
Setting project property: project.packaging - pom
Setting project property: project.build.directory -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target
Setting project property: project.build.outputDirectory -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/classes
Setting project property: project.build.testOutputDirectory -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-classes
Setting project property: project.build.sourceDirectory -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/src/main/java
Setting project property: project.build.testSourceDirectory -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/src/test/java
Setting project property: localRepository -id: local
url: file:///home/jenkins/.m2/repository/
layout: none
Setting project property: settings.localRepository -
/home/jenkins/.m2/repository
Setting project property: maven.project.dependencies.versions -
[INFO] Executing tasks
Build sequence for target(s) `main' is [main]
Complete build sequence is [main, ]
main:
[mkdir] Created dir:
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir
[mkdir] Skipping
Re: Plans of moving towards JDK7 in trunk
On 5 April 2014 20:54, Raymie Stata rst...@altiscale.com wrote: To summarize the thread so far: a) Java7 is already a supported compile- and runtime environment for Hadoop branch2 and trunk b) Java6 must remain a supported compile- and runtime environment for Hadoop branch2 c) (b) implies that branch2 must stick to Java6 APIs I wonder if point (b) should be revised. We could immediately deprecate Java6 as a runtime (and thus compile-time) environment for Hadoop. We could end support for in some published time frame (perhaps 3Q2014). That is, we'd say that all future 2.x release past some date would not be guaranteed to run on Java6. This would set us up for using Java7 APIs into branch2. I'll let others deal with that question. An alternative might be to keep branch2 on Java6 APIs forever, and to start using Java7 APIs in trunk relatively soon. The concern here would be that trunk isn't getting the kind of production torture testing that branch2 is subjected to, and won't be for a while. If trunk and branch2 diverge too much too quickly, trunk could become a nest of bugs, endangering the timeline and quality of Hadoop 3. This would argue for keeping trunk and branch2 in closer sync (maybe until a branch3 is created and starts getting used by bleeding-edge users). However, as just suggested, keeping them in closer sync need _not_ imply that Java7 features be avoided indefinitely: again, with sufficient warning, Java6 support could be sunset within branch2. One thing we could do is have a policy towards new features where there's consensus that they won't go into branch-2, especially things in their own JARs. Here we could consider a policy of build set up to be Java 7 only, with Java7 APIs. That would be justified if there was some special java 7 feature -such as its infiniband support. Add a feature like that in its own module (under hadoop-tools, presumably), and use Java7 and Java 7 libraries. If someone really did want to use the feature in hadoop-2, they'd be able to, in a java7+ only backport. On a related note, Steve points out that we need to start thinking about Java8. YES!! Lambdas are a Really Big Deal! If we sunset Java6 in a few quarters, maybe we can add Java8 compile and runtime (but not API) support about the same time. This does NOT imply bringing Java8 APIs into branch2: Even if we do allow Java7 APIs into branch2 in the future, I doubt that bringing Java8 APIs into it will ever make sense. However, if Java8 is a supported runtime environment for Hadoop 2, that sets us up for using Java8 APIs for the eventual branch3 sometime in 2015. Testing Hadoop on Java 8 would let the rest of the stack move forward. In the meantime, can I point out that both Scala-on-Java7 and Groovy-on-Java 7 offer closures quite nicely, with performance by way of INVOKEDYNAMIC opcodes. -steve -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HADOOP-10464) Make TestTrash compatible with HADOOP-10461 .
jay vyas created HADOOP-10464:
-
Summary: Make TestTrash compatible with HADOOP-10461 .
Key: HADOOP-10464
URL: https://issues.apache.org/jira/browse/HADOOP-10464
Project: Hadoop Common
Issue Type: Test
Reporter: jay vyas
The current TestTrash implementation is like a static utility class
{noformat}
public static void trashShell(final Configuration conf, final Path base,
FileSystem trashRootFs, Path trashRoot)
{noformat}
It should be refactored to be a pure unit test that can be transparently
integrated into TestSuites which don't have any prior knowledge of the tests it
defines.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Hadoop v1.8 data transfer protocol
Hi, I'm trying to figure out how data is transferred between client and DataNode in Hadoop v1.8. This is my understanding so far: The client first fires an OP_READ_BLOCK request. The DataNode responds with a status code, checksum header, chunk offset, packet length, sequence number, the last packet boolean, the length and the data (in that order). However, I'm running into an issue. First of all, which of these lengths describes the length of the data? I tried both PacketLength and Length it seems that they leave data on the stream (I tried to cat a file with the numbers 1-1000 in it). Also, how does the DataNode signal the start of another packet? After Length number of bytes have been read, I assumed that the header would be repeated, but this is not the case (I'm not getting sane values for any of the fields of the header). I've looked through the DataXceiver, BlockSender, DFSClient (RemoteBlockReader) classes but I still can't quite grasp how this data transfer is conducted. Any help would be appreciated, Dhaivat Pandya
Re: Hadoop v1.8 data transfer protocol
There's been no Apache Hadoop release versioned v1.8 historically, nor is one upcoming. Do you mean 0.18? Either way, can you point to the specific code lines in BlockSender which have you confused? The sendBlock and sendPacket methods would interest you I assume, but they appear to be well constructed/named internally and commented in a few important spots. On Mon, Apr 7, 2014 at 6:39 AM, Dhaivat Pandya dhaivatpan...@gmail.com wrote: Hi, I'm trying to figure out how data is transferred between client and DataNode in Hadoop v1.8. This is my understanding so far: The client first fires an OP_READ_BLOCK request. The DataNode responds with a status code, checksum header, chunk offset, packet length, sequence number, the last packet boolean, the length and the data (in that order). However, I'm running into an issue. First of all, which of these lengths describes the length of the data? I tried both PacketLength and Length it seems that they leave data on the stream (I tried to cat a file with the numbers 1-1000 in it). Also, how does the DataNode signal the start of another packet? After Length number of bytes have been read, I assumed that the header would be repeated, but this is not the case (I'm not getting sane values for any of the fields of the header). I've looked through the DataXceiver, BlockSender, DFSClient (RemoteBlockReader) classes but I still can't quite grasp how this data transfer is conducted. Any help would be appreciated, Dhaivat Pandya -- Harsh J
