[jira] [Created] (HADOOP-13887) Support for client-side encryption in S3A file system
Jeeyoung Kim created HADOOP-13887: - Summary: Support for client-side encryption in S3A file system Key: HADOOP-13887 URL: https://issues.apache.org/jira/browse/HADOOP-13887 Project: Hadoop Common Issue Type: New Feature Reporter: Jeeyoung Kim Priority: Minor Expose the client-side encryption option documented in Amazon S3 documentation - http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html Currently this is not exposed in Hadoop but it is exposed as an option in AWS Java SDK, which Hadoop currently includes. It should be trivial to propagate this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13886) s3guard: ITestS3AFileOperationCost.testFakeDirectoryDeletion failure
Aaron Fabbri created HADOOP-13886: - Summary: s3guard: ITestS3AFileOperationCost.testFakeDirectoryDeletion failure Key: HADOOP-13886 URL: https://issues.apache.org/jira/browse/HADOOP-13886 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri testFakeDirectoryDeletion(org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost) Time elapsed: 10.011 sec <<< FAILURE! java.lang.AssertionError: after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:431) at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:254) More details to follow in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13885) Implement getLinkTarget for ViewFileSystem
Manoj Govindassamy created HADOOP-13885: --- Summary: Implement getLinkTarget for ViewFileSystem Key: HADOOP-13885 URL: https://issues.apache.org/jira/browse/HADOOP-13885 Project: Hadoop Common Issue Type: Task Components: viewfs Affects Versions: 3.0.0-alpha1 Reporter: Manoj Govindassamy Assignee: Manoj Govindassamy ViewFileSystem doesn't override FileSystem#getLinkTarget(). So, when view filesystem is used to resolve the symbolic links, the default FileSystem implementation throws UnsupportedOperationException. The proposal is to define getLinkTarget() for ViewFileSystem and invoke the target FileSystem for resolving the symbolic links. Path thus returned is preferred to be a viewfs qualified path, so that it can be used again on the ViewFileSystem handle. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/181/ [Dec 8, 2016 5:57:15 PM] (stevel) HADOOP-13852 hadoop build to allow hadoop version property to be [Dec 8, 2016 8:31:12 PM] (sjlee) YARN-5922. Remove direct references of HBaseTimelineWriter/Reader in [Dec 8, 2016 8:40:20 PM] (xiao) HDFS-11197. Listing encryption zones fails when deleting a EZ that is on [Dec 8, 2016 9:36:08 PM] (kihwal) Revert "HDFS-11094. Send back HAState along with NamespaceInfo during a [Dec 8, 2016 9:41:49 PM] (kihwal) Revert "HDFS-5079. Cleaning up NNHAStatusHeartbeat.State from [Dec 9, 2016 3:49:27 AM] (aajisaka) Revert "HADOOP-13852 hadoop build to allow hadoop version property to be [Dec 9, 2016 9:23:50 AM] (vinayakumarb) HDFS-11224. Lifeline message should be ignored for dead nodes [Dec 9, 2016 12:51:42 PM] (brahma) HADOOP-13883. Add description of -fs option in generic command usage. -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.ha.TestZKFailoverController hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/181/artifact/out/patch-compile-root.txt [176K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/181/artifact/out/patch-compile-root.txt [176K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/181/artifact/out/patch-compile-root.txt [176K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/181/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [124K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/181/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [416K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/181/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/181/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [56K]
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/ [Dec 8, 2016 10:23:56 AM] (sunilg) YARN-5970. Validate application update timeout request parameters. [Dec 8, 2016 11:52:24 AM] (aajisaka) HDFS-11223. Fix typos in HttpFs documentations. Contributed by Yiqun [Dec 8, 2016 1:27:43 PM] (brahma) HADOOP-13867. FilterFileSystem should override rename(.., options) to [Dec 8, 2016 5:57:15 PM] (stevel) HADOOP-13852 hadoop build to allow hadoop version property to be [Dec 8, 2016 8:31:12 PM] (sjlee) YARN-5922. Remove direct references of HBaseTimelineWriter/Reader in [Dec 8, 2016 8:40:20 PM] (xiao) HDFS-11197. Listing encryption zones fails when deleting a EZ that is on [Dec 8, 2016 9:36:08 PM] (kihwal) Revert "HDFS-11094. Send back HAState along with NamespaceInfo during a [Dec 8, 2016 9:41:49 PM] (kihwal) Revert "HDFS-5079. Cleaning up NNHAStatusHeartbeat.State from [Dec 9, 2016 3:49:27 AM] (aajisaka) Revert "HADOOP-13852 hadoop build to allow hadoop version property to be -1 overall The following subsystems voted -1: asflicense findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestDiskFailures Timed out junit tests : org.apache.hadoop.hdfs.TestLeaseRecovery2 cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/diff-compile-javac-root.txt [172K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/diff-patch-shellcheck.txt [28K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/whitespace-tabs.txt [1.3M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [4.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [152K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [316K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/250/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13884) s3a create(overwrite=true) to only look for dir/ and list entries, not file
Steve Loughran created HADOOP-13884:
---
Summary: s3a create(overwrite=true) to only look for dir/ and list
entries, not file
Key: HADOOP-13884
URL: https://issues.apache.org/jira/browse/HADOOP-13884
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 2.9.0
Reporter: Steve Loughran
Priority: Minor
before doing a create(), s3a does a getFileStatus() to make sure there isn't a
directory there, and, if overwrite=false, that there isn't a file.
Because S3 caches negative HEAD/GET requests, if there isn't a file, then even
after the PUT, a later GET/HEAD may return 404; we are generating create
consistency where none need exist.
when overwrite=true we don't care whether the file exists or not, only that the
path isn't a directory. So we can just to the HEAD path +"/' and the LIST
calls, skipping the {{HEAD path}}. This will save an HTTP round trip of a few
hundred millis, and ensure that there's no 404 cached in the S3 front end for
later callers
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13883) Add description of -fs option in generic command usage
Yiqun Lin created HADOOP-13883:
--
Summary: Add description of -fs option in generic command usage
Key: HADOOP-13883
URL: https://issues.apache.org/jira/browse/HADOOP-13883
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 3.0.0-alpha2
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor
Currently the description of option '-fs' is missing in generic command usage
in documentation {{CommandManual.md}}. And the users won't know to use this
option, while this option already makes sense to {{hdfs dfsadmin}}, {{hdfs
fsck}}, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13882) TestLambdaTestUtils.testAwaitAlwaysFalse failing
Steve Loughran created HADOOP-13882:
---
Summary: TestLambdaTestUtils.testAwaitAlwaysFalse failing
Key: HADOOP-13882
URL: https://issues.apache.org/jira/browse/HADOOP-13882
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 3.0.0-alpha2
Reporter: Steve Loughran
{{org.apache.hadoop.test.TestLambdaTestUtils.testAwaitAlwaysFalse}} failing on
a Jenkins test run. No obvious info in the test as to why.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13881) Remove deprecated APIs added in HADOOP-6709
Akira Ajisaka created HADOOP-13881: -- Summary: Remove deprecated APIs added in HADOOP-6709 Key: HADOOP-13881 URL: https://issues.apache.org/jira/browse/HADOOP-13881 Project: Hadoop Common Issue Type: Improvement Reporter: Akira Ajisaka FileSystem#getName, getNamed, getReplication(Path), delete(Path), getFileStatus(), getBlockSize(Path) were re-instated about 6 years ago. They can be removed in Hadoop 3. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13880) Fix dead links in relevant APIs of Job setting
Yiqun Lin created HADOOP-13880: -- Summary: Fix dead links in relevant APIs of Job setting Key: HADOOP-13880 URL: https://issues.apache.org/jira/browse/HADOOP-13880 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 3.0.0-alpha2 Reporter: Yiqun Lin Assignee: Yiqun Lin Priority: Minor There are some dead links in Job relevant classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
