[jira] [Created] (HDFS-11963) Ozone: Documentation: Add getting started page
Anu Engineer created HDFS-11963: --- Summary: Ozone: Documentation: Add getting started page Key: HDFS-11963 URL: https://issues.apache.org/jira/browse/HDFS-11963 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anu Engineer We need to add the Ozone section to hadoop documentation and also a section on how to get started, that is what are configuration settings needed to start running ozone. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11962) Ozone: Add stop-ozone.sh script
Anu Engineer created HDFS-11962: --- Summary: Ozone: Add stop-ozone.sh script Key: HDFS-11962 URL: https://issues.apache.org/jira/browse/HDFS-11962 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anu Engineer This script should stop the cluster along with all ozone services. -- #run 'stop-dfs.sh' # run 'hdfs --daemon stop ksm' # run 'hdfs --daemon stop scm`' -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11961) Ozone: Add start-ozone.sh to quickly start ozone.
Anu Engineer created HDFS-11961: --- Summary: Ozone: Add start-ozone.sh to quickly start ozone. Key: HDFS-11961 URL: https://issues.apache.org/jira/browse/HDFS-11961 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anu Engineer Add start ozone script. Internally this script should call into # start-dfs.sh # run `hdfs --daemon start scm' # run `hdfs --daemon start ksm` This just makes it easy to start Ozone with a single command. This command assumes that Namenode format has been run before, since it will bring up HDFS also. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/ [Jun 8, 2017 5:08:08 PM] (templedf) YARN-6679. Reduce Resource instance overhead via non-PBImpl (Contributed [Jun 8, 2017 9:15:09 PM] (jzhuge) HDFS-11851. getGlobalJNIEnv() may deadlock if exception is thrown. [Jun 8, 2017 9:36:47 PM] (liuml07) HDFS-11945. Internal lease recovery may not be retried for a long time. [Jun 8, 2017 10:06:30 PM] (liuml07) HADOOP-14283. Upgrade AWS SDK to 1.11.134. Contributed by Aaron Fabbri [Jun 9, 2017 4:34:15 AM] (xiao) HADOOP-13174. Add more debug logs for delegation tokens and [Jun 9, 2017 5:35:31 AM] (xiao) HADOOP-13854. KMS should log error details in KMSExceptionsProvider. [Jun 9, 2017 5:40:13 AM] (naganarasimha_gr) YARN-6682. Improve performance of AssignmentInformation datastructures. -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.sftp.TestSFTPFileSystem hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.mover.TestMover hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.TestRollingUpgrade hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.namenode.ha.TestHASafeMode hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.api.impl.TestNMClient hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.yarn.sls.nodemanager.TestNMSimulator Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-mvninstall-root.txt [492K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-compile-root.txt [20K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-compile-root.txt [20K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-compile-root.txt [20K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-assemblies.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [416K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [72K]
[jira] [Created] (HDFS-11959) Ozone: Audit Logs
Weiwei Yang created HDFS-11959: -- Summary: Ozone: Audit Logs Key: HDFS-11959 URL: https://issues.apache.org/jira/browse/HDFS-11959 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Weiwei Yang Assignee: Weiwei Yang Add audit logs for ozone components. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11958) Ozone: Ensure KSM is initiated using ProtobufRpcEngine
Weiwei Yang created HDFS-11958:
--
Summary: Ozone: Ensure KSM is initiated using ProtobufRpcEngine
Key: HDFS-11958
URL: https://issues.apache.org/jira/browse/HDFS-11958
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Priority: Critical
Reproduce Steps
# Launch an ozone cluster
# Create a volume via commandline
{code}
hdfs oz -createVolume http://ozone1.fyre.ibm.com:9864/volume-wwei-0 -user root
{code}
it failed with following error
{noformat}
SEVERE: The RuntimeException could not be mapped to a response, re-throwing to
the HTTP container
java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:182)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invocation.(WritableRpcEngine.java:114)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:247)
at com.sun.proxy.$Proxy18.createVolume(Unknown Source)
...
Caused by: java.lang.NoSuchFieldException: versionID
at java.lang.Class.getField(Class.java:1703)
at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:178)
... 25 more
{noformat}
This was because {{keySpaceManagerClient}} in {{ObjectStoreHandler}} currently
is not properly initiated, it should be using {{ProtobufRpcEngine}} instead of
{{WritableRpcEngine}} which is deprecated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11957) Enable POSIX ACL inheritance by default
John Zhuge created HDFS-11957: - Summary: Enable POSIX ACL inheritance by default Key: HDFS-11957 URL: https://issues.apache.org/jira/browse/HDFS-11957 Project: Hadoop HDFS Issue Type: Improvement Components: security Affects Versions: 3.0.0-alpha2 Reporter: John Zhuge Assignee: John Zhuge It is time to enable POSIX ACL inheritance by default. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/429/ [Jun 8, 2017 5:08:08 PM] (templedf) YARN-6679. Reduce Resource instance overhead via non-PBImpl (Contributed [Jun 8, 2017 9:15:09 PM] (jzhuge) HDFS-11851. getGlobalJNIEnv() may deadlock if exception is thrown. [Jun 8, 2017 9:36:47 PM] (liuml07) HDFS-11945. Internal lease recovery may not be retried for a long time. [Jun 8, 2017 10:06:30 PM] (liuml07) HADOOP-14283. Upgrade AWS SDK to 1.11.134. Contributed by Aaron Fabbri [Jun 9, 2017 4:34:15 AM] (xiao) HADOOP-13174. Add more debug logs for delegation tokens and [Jun 9, 2017 5:35:31 AM] (xiao) HADOOP-13854. KMS should log error details in KMSExceptionsProvider. [Jun 9, 2017 5:40:13 AM] (naganarasimha_gr) YARN-6682. Improve performance of AssignmentInformation datastructures. -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-common-project/hadoop-minikdc Possible null pointer dereference in org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value of called method Dereferenced at MiniKdc.java:org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value of called method Dereferenced at MiniKdc.java:[line 368] FindBugs : module:hadoop-common-project/hadoop-auth org.apache.hadoop.security.authentication.server.MultiSchemeAuthenticationHandler.authenticate(HttpServletRequest, HttpServletResponse) makes inefficient use of keySet iterator instead of entrySet iterator At MultiSchemeAuthenticationHandler.java:of keySet iterator instead of entrySet iterator At MultiSchemeAuthenticationHandler.java:[line 192] FindBugs : module:hadoop-common-project/hadoop-common org.apache.hadoop.crypto.CipherSuite.setUnknownValue(int) unconditionally sets the field unknownValue At CipherSuite.java:unknownValue At CipherSuite.java:[line 44] org.apache.hadoop.crypto.CryptoProtocolVersion.setUnknownValue(int) unconditionally sets the field unknownValue At CryptoProtocolVersion.java:unknownValue At CryptoProtocolVersion.java:[line 67] Possible null pointer dereference in org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to return value of called method Dereferenced at FileUtil.java:org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to return value of called method Dereferenced at FileUtil.java:[line 118] Possible null pointer dereference in org.apache.hadoop.fs.RawLocalFileSystem.handleEmptyDstDirectoryOnWindows(Path, File, Path, File) due to return value of called method Dereferenced at RawLocalFileSystem.java:org.apache.hadoop.fs.RawLocalFileSystem.handleEmptyDstDirectoryOnWindows(Path, File, Path, File) due to return value of called method Dereferenced at RawLocalFileSystem.java:[line 387] Return value of org.apache.hadoop.fs.permission.FsAction.or(FsAction) ignored, but method has no side effect At FTPFileSystem.java:but method has no side effect At FTPFileSystem.java:[line 421] Useless condition:lazyPersist == true at this point At CommandWithDestination.java:[line 502] org.apache.hadoop.io.DoubleWritable.compareTo(DoubleWritable) incorrectly handles double value At DoubleWritable.java: At DoubleWritable.java:[line 78] org.apache.hadoop.io.DoubleWritable$Comparator.compare(byte[], int, int, byte[], int, int) incorrectly handles double value At DoubleWritable.java:int) incorrectly handles double value At DoubleWritable.java:[line 97] org.apache.hadoop.io.FloatWritable.compareTo(FloatWritable) incorrectly handles float value At FloatWritable.java: At FloatWritable.java:[line 71] org.apache.hadoop.io.FloatWritable$Comparator.compare(byte[], int, int, byte[], int, int) incorrectly handles float value At FloatWritable.java:int) incorrectly handles float value At FloatWritable.java:[line 89] Possible null pointer dereference in org.apache.hadoop.io.IOUtils.listDirectory(File, FilenameFilter) due to return value of called method Dereferenced at IOUtils.java:org.apache.hadoop.io.IOUtils.listDirectory(File, FilenameFilter) due to return value of called method Dereferenced at IOUtils.java:[line 351] org.apache.hadoop.io.erasurecode.ECSchema.toString() makes inefficient use of keySet iterator instead of entrySet iterator At ECSchema.java:keySet iterator instead of entrySet iterator At ECSchema.java:[line 193] Possible bad parsing of shift operation in org.apache.hadoop.io.file.tfile.Utils$Version.hashCode() At Utils.java:operation in org.apache.hadoop.io.file.tfile.Utils$Version.hashCode() At Utils.java:[line 398]
[jira] [Created] (HDFS-11956) Fix BlockToken compatibility with Hadoop 2.x clients
Andrew Wang created HDFS-11956:
--
Summary: Fix BlockToken compatibility with Hadoop 2.x clients
Key: HDFS-11956
URL: https://issues.apache.org/jira/browse/HDFS-11956
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0-alpha4
Reporter: Andrew Wang
Priority: Blocker
Seems like HDFS-9807 broke backwards compatibility with Hadoop 2.x clients.
When talking to a 3.0.0-alpha4 DN with security on:
{noformat}
2017-06-06 23:27:22,568 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Block token verification failed: op=WRITE_BLOCK,
remoteAddress=/172.28.208.200:53900, message=Block token with StorageIDs
[DS-c0f24154-a39b-4941-93cd-5b8323067ba2] not valid for access with StorageIDs
[]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
