[jira] [Created] (HADOOP-10693) Implementation of AES-CTR CryptoCodec using JNI to OpenSSL
Yi Liu created HADOOP-10693: --- Summary: Implementation of AES-CTR CryptoCodec using JNI to OpenSSL Key: HADOOP-10693 URL: https://issues.apache.org/jira/browse/HADOOP-10693 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Yi Liu Assignee: Yi Liu Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) In HADOOP-10603, we have an implementation of AES-CTR CryptoCodec using Java JCE provider. To get high performance, the configured JCE provider should utilize native code and AES-NI, but in JDK6,7 the Java embedded provider doesn't support it. Considering not all hadoop user will use the provider like Diceros or able to get signed certificate from oracle to develop a custom provider, so this JIRA will have an implementation of AES-CTR CryptoCodec using JNI to OpenSSL directly. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Change proposal for FileInputFormat isSplitable
Hi, On Wed, Jun 11, 2014 at 8:25 PM, Chris Douglas cdoug...@apache.org wrote: On Wed, Jun 11, 2014 at 1:35 AM, Niels Basjes ni...@basjes.nl wrote: That's not what I meant. What I understood from what was described is that sometimes people use an existing file extension (like .gz) for a file that is not a gzipped file. Understood, but this change also applies to other loaded codecs, like .lzo, .bz, etc. Adding a new codec changes the default behavior for all InputFormats that don't override this method. Yes it would. I think that forcing the developer of the file based inputformat to implement this would be the best way to go. Making this method abstract is the first thing that spring to mind. This would break backwards compatibility so I think we can only do that with the 3.0.0 version I consider silently producing garbage one of the worst kinds of problem to tackle. Because many custom file based input formats have stumbled (getting silently produced garbage) over the current isSplitable implementation I really want to avoid any more of this in the future. That is why I want to change the implementations in this area of Hadoop in such a way that this silently producing garbage effect is taken out. Adding validity assumptions to a common base class will affect a lot of users, most of whom are not InputFormat authors. True, the thing is that if a user uses an InputFormat written by someone else and then it silently produces garbage they are also affected in a much worse way. So the question remains: What is the way this should be changed? I'm willing to build it and submit a patch. Would a logged warning suffice? This would aid debugging without an incompatible change in behavior. It could also be disabled easily. -C Hmmm, people only look at logs when they have a problem. So I don't think this would be enough. Perhaps this makes sense: - For 3.0: Shout at the developer who does it wrong (i.e. make it abstract and force them to think about this) i.e. Create new abstract method isSplittable (tt) in FileInputFormat, remove isSplitable (one t). To avoid needless code duplication (which we already have in the codebase) create a helper method something like 'fileNameIndicatesSplittableFile' ( returns enum: Splittable/NonSplittable/Unknown ). - For 2.x: Keep the enduser safe: Avoid silently producing garbage in all situations where the developer already did it wrong. (i.e. change isSplitable == return false) This costs performance only in those situations where the developer actually did it wrong (i.e. they didn't thing this through) How about that? P.S. I created an issue for the NLineInputFormat problem I found: https://issues.apache.org/jira/browse/MAPREDUCE-5925 -- Best regards / Met vriendelijke groeten, Niels Basjes
Build failed in Jenkins: Hadoop-Common-trunk #1138
See https://builds.apache.org/job/Hadoop-Common-trunk/1138/changes
Changes:
[wang] HADOOP-10691. Improve the readability of 'hadoop fs -help'. Contributed
by Lei Xu.
[arp] HADOOP-6350. Document Hadoop Metrics. (Contributed by Akira Ajisaka)
[vinodkv] YARN-1702. Added kill app functionality to RM web services.
Contributed by Varun Vasudev.
[kasha] YARN-2155. FairScheduler: Incorrect threshold check for preemption.
(Wei Yan via kasha)
[wang] Fixup CHANGES.txt message for HDFS-3493
[wang] HDFS-3493. Invalidate corrupted blocks as long as minimum replication is
satisfied. Contributed by Juan Yu and Vinayakumar B.
[wang] HDFS-6395. Skip checking xattr limits for non-user-visible namespaces.
Contributed by Yi Liu.
[cnauroth] MAPREDUCE-5912. Task.calculateOutputSize does not handle Windows
files after MAPREDUCE-5196. Contributed by Remus Rusanu.
[cnauroth] HADOOP-10561. Copy command with preserve option should handle
Xattrs. Contributed by Yi Liu.
--
[...truncated 68365 lines...]
parsing buildfile
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
with URI =
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
from a zip file
Class org.apache.maven.ant.tasks.AttachArtifactTask loaded from parent loader
(parentFirst)
+Datatype attachartifact org.apache.maven.ant.tasks.AttachArtifactTask
Class org.apache.maven.ant.tasks.DependencyFilesetsTask loaded from parent
loader (parentFirst)
+Datatype dependencyfilesets org.apache.maven.ant.tasks.DependencyFilesetsTask
Setting project property: test.build.dir -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir
Setting project property: test.exclude.pattern - _
Setting project property: zookeeper.version - 3.4.6
Setting project property: hadoop.assemblies.version - 3.0.0-SNAPSHOT
Setting project property: test.exclude - _
Setting project property: distMgmtSnapshotsId - apache.snapshots.https
Setting project property: project.build.sourceEncoding - UTF-8
Setting project property: java.security.egd - file:///dev/urandom
Setting project property: distMgmtSnapshotsUrl -
https://repository.apache.org/content/repositories/snapshots
Setting project property: distMgmtStagingUrl -
https://repository.apache.org/service/local/staging/deploy/maven2
Setting project property: avro.version - 1.7.4
Setting project property: test.build.data -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-dir
Setting project property: commons-daemon.version - 1.0.13
Setting project property: hadoop.common.build.dir -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/../../hadoop-common-project/hadoop-common/target
Setting project property: testsThreadCount - 4
Setting project property: maven.test.redirectTestOutputToFile - true
Setting project property: jdiff.version - 1.0.9
Setting project property: project.reporting.outputEncoding - UTF-8
Setting project property: distMgmtStagingName - Apache Release Distribution
Repository
Setting project property: build.platform - Linux-i386-32
Setting project property: protobuf.version - 2.5.0
Setting project property: failIfNoTests - false
Setting project property: protoc.path - ${env.HADOOP_PROTOC_PATH}
Setting project property: jersey.version - 1.9
Setting project property: distMgmtStagingId - apache.staging.https
Setting project property: distMgmtSnapshotsName - Apache Development Snapshot
Repository
Setting project property: ant.file -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/pom.xml
[DEBUG] Setting properties with prefix:
Setting project property: project.groupId - org.apache.hadoop
Setting project property: project.artifactId - hadoop-common-project
Setting project property: project.name - Apache Hadoop Common Project
Setting project property: project.description - Apache Hadoop Common Project
Setting project property: project.version - 3.0.0-SNAPSHOT
Setting project property: project.packaging - pom
Setting project property: project.build.directory -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target
Setting project property: project.build.outputDirectory -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/classes
Setting project property: project.build.testOutputDirectory -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/target/test-classes
Setting project property: project.build.sourceDirectory -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/src/main/java
Setting project property: project.build.testSourceDirectory -
https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/src/test/java
Setting project property: localRepository -id:
