[jira] [Created] (HDFS-12393) Fix incorrect package length for doRead in PacketReceiver
legend created HDFS-12393: - Summary: Fix incorrect package length for doRead in PacketReceiver Key: HDFS-12393 URL: https://issues.apache.org/jira/browse/HDFS-12393 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0-alpha4 Reporter: legend -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12392) Randomly read striped file failure
SammiChen created HDFS-12392: Summary: Randomly read striped file failure Key: HDFS-12392 URL: https://issues.apache.org/jira/browse/HDFS-12392 Project: Hadoop HDFS Issue Type: Bug Components: erasure-coding Affects Versions: 3.0.0-alpha3 Reporter: SammiChen Root cause: The buffer size returned by ElasticByteBufferPool.getBuffer() is more than caller expected. Exception stack: org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer, not of length 4096 at org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferEncodingState.checkBuffers(ByteBufferEncodingState.java:99) at org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferEncodingState.(ByteBufferEncodingState.java:46) at org.apache.hadoop.io.erasurecode.rawcoder.RawErasureEncoder.encode(RawErasureEncoder.java:67) at org.apache.hadoop.hdfs.DFSStripedOutputStream.encode(DFSStripedOutputStream.java:368) at org.apache.hadoop.hdfs.DFSStripedOutputStream.writeParityCells(DFSStripedOutputStream.java:942) at org.apache.hadoop.hdfs.DFSStripedOutputStream.writeChunk(DFSStripedOutputStream.java:547) at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:217) at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:125) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:111) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57) at java.io.DataOutputStream.write(DataOutputStream.java:107) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:94) at org.apache.hadoop.hdfs.DFSTestUtil.writeFile(DFSTestUtil.java:834) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/ [Sep 4, 2017 2:50:41 AM] (xiao) HDFS-12383. Re-encryption updater should handle canceled tasks better. -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Hard coded reference to an absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:absolute pathname in org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext) At DockerLinuxContainerRuntime.java:[line 490] Failed junit tests : hadoop.ha.TestZKFailoverController hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 hadoop.hdfs.TestLeaseRecoveryStriped hadoop.hdfs.TestClientProtocolForPipelineRecovery hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 hadoop.hdfs.TestFileAppendRestart hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 hadoop.hdfs.TestReadStripedFileWithMissingBlocks hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 hadoop.hdfs.TestWriteConfigurationToDFS hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 hadoop.tracing.TestTracing hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 hadoop.hdfs.TestDFSStripedOutputStreamWithFailure hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.cli.TestLogsCLI hadoop.mapreduce.v2.hs.webapp.TestHSWebApp hadoop.yarn.sls.TestReservationSystemInvariants hadoop.yarn.sls.TestSLSRunner Timed out junit tests : org.apache.hadoop.hdfs.TestWriteReadStripedFile org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/artifact/out/diff-compile-javac-root.txt [292K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/artifact/out/whitespace-tabs.txt [1.2M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/513/artifact/out/patch-javadoc-root.txt [2.0M] unit:
[jira] [Created] (HDFS-12391) Ozone: TestKSMSQLCli is not working as expected
Weiwei Yang created HDFS-12391:
--
Summary: Ozone: TestKSMSQLCli is not working as expected
Key: HDFS-12391
URL: https://issues.apache.org/jira/browse/HDFS-12391
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone, test
Affects Versions: HDFS-7240
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Priority: Minor
I found this issue while investigating the {{TestKSMSQLCli}} failure in [this
jenkins
report|https://builds.apache.org/job/PreCommit-HDFS-Build/20984/testReport/],
the test is supposed to use parameterized class to test both {{LevelDB}} and
{{RocksDB}} implementation of metadata stores, however it only tests default
{{RocksDB}} case twice.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12367) Ozone: Too many open files error while running corona
[
https://issues.apache.org/jira/browse/HDFS-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang resolved HDFS-12367.
Resolution: Duplicate
I think this issue no longer happens to me, closing it as a dup to HDFS-12382
as this looks like to be fixed there, thanks [~nandakumar131]. [~msingh] feel
free to create another lower severity JIRA to track resource leaks you found in
code level. I will close this one as it is no longer a blocker for tests.
> Ozone: Too many open files error while running corona
> -
>
> Key: HDFS-12367
> URL: https://issues.apache.org/jira/browse/HDFS-12367
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, tools
>Reporter: Weiwei Yang
>Assignee: Mukul Kumar Singh
>
> Too many open files error keeps happening to me while using corona, I have
> simply setup a single node cluster and run corona to generate 1000 keys, but
> I keep getting following error
> {noformat}
> ./bin/hdfs corona -numOfThreads 1 -numOfVolumes 1 -numOfBuckets 1 -numOfKeys
> 1000
> 17/08/28 00:47:42 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 17/08/28 00:47:42 INFO tools.Corona: Number of Threads: 1
> 17/08/28 00:47:42 INFO tools.Corona: Mode: offline
> 17/08/28 00:47:42 INFO tools.Corona: Number of Volumes: 1.
> 17/08/28 00:47:42 INFO tools.Corona: Number of Buckets per Volume: 1.
> 17/08/28 00:47:42 INFO tools.Corona: Number of Keys per Bucket: 1000.
> 17/08/28 00:47:42 INFO rpc.OzoneRpcClient: Creating Volume: vol-0-05000, with
> wwei as owner and quota set to 1152921504606846976 bytes.
> 17/08/28 00:47:42 INFO tools.Corona: Starting progress bar Thread.
> ...
> ERROR tools.Corona: Exception while adding key: key-251-19293 in bucket:
> bucket-0-34960 of volume: vol-0-05000.
> java.io.IOException: Exception getting XceiverClient.
> at
> org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:156)
> at
> org.apache.hadoop.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:122)
> at
> org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.getFromKsmKeyInfo(ChunkGroupOutputStream.java:289)
> at
> org.apache.hadoop.ozone.client.rpc.OzoneRpcClient.createKey(OzoneRpcClient.java:487)
> at
> org.apache.hadoop.ozone.tools.Corona$OfflineProcessor.run(Corona.java:352)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalStateException: failed to create a child event loop
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
> at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
> at
> org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:144)
> ... 9 more
> Caused by: java.lang.IllegalStateException: failed to create a child event
> loop
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68)
> at
> io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36)
> at org.apache.hadoop.scm.XceiverClient.connect(XceiverClient.java:76)
> at
> org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:151)
> at
> org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:145)
> at
> com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
> ... 12 more
> Caused by: io.netty.channel.ChannelException: failed to open a new selector
> at
[jira] [Created] (HDFS-12390) Supporting DNS to switch mapping
Jiandan Yang created HDFS-12390: Summary: Supporting DNS to switch mapping Key: HDFS-12390 URL: https://issues.apache.org/jira/browse/HDFS-12390 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs, hdfs-client Reporter: Jiandan Yang Assignee: Jiandan Yang As described in [HDFS-12200|https://issues.apache.org/jira/browse/HDFS-12200], ScriptBasedMapping may lead to NN cpu 100%. ScriptBasedMapping run sub_processor to get rack info of DN/Client, so we think it's a little heavy. We prepare to use TableMappingļ¼but TableMapping does not support refresh and can not reload rack info of newly added DataNodes. So we implement it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
