[jira] [Resolved] (HDFS-11819) HDFS client with hedged read, handle exceptions from callable when the hedged read thread pool is exhausted
[
https://issues.apache.org/jira/browse/HDFS-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huaxiang sun resolved HDFS-11819.
-
Resolution: Invalid
The exception is handled, close it as invalid.
> HDFS client with hedged read, handle exceptions from callable when the
> hedged read thread pool is exhausted
>
>
> Key: HDFS-11819
> URL: https://issues.apache.org/jira/browse/HDFS-11819
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>
> When the hedged read thread pool is exhausted, the current behavior is that
> callable will be executed in the current thread context. The callable can
> throw out IOExceptions which is not handled and it will not start a 'hedged'
> read.
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1131
> Please see the following exception:
> {code}
> 2017-05-11 22:42:35,883 WARN org.apache.hadoop.hdfs.BlockReaderFactory: I/O
> error constructing remote block reader.
> org.apache.hadoop.net.ConnectTimeoutException: 3000 millis timeout while
> waiting for channel to be ready for connect. ch :
> java.nio.channels.SocketChannel[connection-pending remote=/*.*.*.*:50010]
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
> at
> org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3527)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:840)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:755)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
> at
> org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1179)
> at
> org.apache.hadoop.hdfs.DFSInputStream.access$300(DFSInputStream.java:91)
> at
> org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1141)
> at
> org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1133)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2022)
> at
> org.apache.hadoop.hdfs.DFSClient$2.rejectedExecution(DFSClient.java:3571)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> at
> java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
> at
> org.apache.hadoop.hdfs.DFSInputStream.hedgedFetchBlockByteRange(DFSInputStream.java:1280)
> at
> org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1477)
> at
> org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1439)
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.read(FileLink.java:167)
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:757)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1457)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1682)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1542)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:445)
> at
> org.apache.hadoop.hbase.util.CompoundBloomFilter.contains(CompoundBloomFilter.java:100)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.passesGeneralBloomFilter(StoreFile.java:1383)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.passesBloomFilter(StoreFile.java:1247)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:469)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:393)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.getScannersNoCompaction(StoreScanner.java:312)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:192)
> at
>
[jira] [Created] (HDFS-11836) Ozone : add sql debug CLI to hdfs script
Chen Liang created HDFS-11836:
-
Summary: Ozone : add sql debug CLI to hdfs script
Key: HDFS-11836
URL: https://issues.apache.org/jira/browse/HDFS-11836
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Chen Liang
Assignee: Chen Liang
HDFS-11698 was missing one change, which is that {{SQLCLI}} should be exposed
to commandline via hdfs script. This JIRA addresses this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/ [May 15, 2017 3:32:01 PM] (jeagles) YARN-6598. History server getApplicationReport NPE when fetching report -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.ha.TestZKFailoverController hadoop.ha.TestZKFailoverControllerStress hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage hadoop.hdfs.qjournal.server.TestJournalNode hadoop.hdfs.server.namenode.ha.TestBootstrapStandby hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations hadoop.hdfs.server.namenode.TestReconstructStripedBlocks hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.datanode.TestDataNodeUUID hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.mapred.TestShuffleHandler hadoop.yarn.sls.TestSLSRunner hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestDiskFailures hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore Timed out junit tests : org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/artifact/out/patch-mvninstall-root.txt [496K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/artifact/out/patch-compile-root.txt [20K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/artifact/out/patch-compile-root.txt [20K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/artifact/out/patch-compile-root.txt [20K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/artifact/out/patch-unit-hadoop-assemblies.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [644K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/316/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [44K]
[jira] [Created] (HDFS-11835) Block Storage: Overwrite of blocks fails
Mukul Kumar Singh created HDFS-11835:
Summary: Block Storage: Overwrite of blocks fails
Key: HDFS-11835
URL: https://issues.apache.org/jira/browse/HDFS-11835
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
Overwrite of blocks fails, because "OverWriteRequested" flag is not set during
chunk creation.
{code}
2017-05-16 22:33:23,909 [nioEventLoopGroup-5-2] ERROR - Rejecting write
chunk request. Chunk overwrite without explicit request.
ChunkInfo{chunkName='1_chunk, offset=0, len=
11933}
{code}
This flag needs to be set here
{code}
public static void writeSmallFile(XceiverClientSpi client, String containerName,
String key, byte[] data, String traceID) throws IOException {
.
ChunkInfo chunk = ChunkInfo
.newBuilder()
.setChunkName(key + "_chunk")
.setOffset(0)
.setLen(data.length)
.build();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/405/ [May 15, 2017 3:32:01 PM] (jeagles) YARN-6598. History server getApplicationReport NPE when fetching report -1 overall The following subsystems voted -1: asflicense findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-common-project/hadoop-minikdc Possible null pointer dereference in org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value of called method Dereferenced at MiniKdc.java:org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value of called method Dereferenced at MiniKdc.java:[line 368] FindBugs : module:hadoop-common-project/hadoop-auth org.apache.hadoop.security.authentication.server.MultiSchemeAuthenticationHandler.authenticate(HttpServletRequest, HttpServletResponse) makes inefficient use of keySet iterator instead of entrySet iterator At MultiSchemeAuthenticationHandler.java:of keySet iterator instead of entrySet iterator At MultiSchemeAuthenticationHandler.java:[line 192] FindBugs : module:hadoop-common-project/hadoop-common org.apache.hadoop.crypto.CipherSuite.setUnknownValue(int) unconditionally sets the field unknownValue At CipherSuite.java:unknownValue At CipherSuite.java:[line 44] org.apache.hadoop.crypto.CryptoProtocolVersion.setUnknownValue(int) unconditionally sets the field unknownValue At CryptoProtocolVersion.java:unknownValue At CryptoProtocolVersion.java:[line 67] Possible null pointer dereference in org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to return value of called method Dereferenced at FileUtil.java:org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to return value of called method Dereferenced at FileUtil.java:[line 118] Possible null pointer dereference in org.apache.hadoop.fs.RawLocalFileSystem.handleEmptyDstDirectoryOnWindows(Path, File, Path, File) due to return value of called method Dereferenced at RawLocalFileSystem.java:org.apache.hadoop.fs.RawLocalFileSystem.handleEmptyDstDirectoryOnWindows(Path, File, Path, File) due to return value of called method Dereferenced at RawLocalFileSystem.java:[line 387] Return value of org.apache.hadoop.fs.permission.FsAction.or(FsAction) ignored, but method has no side effect At FTPFileSystem.java:but method has no side effect At FTPFileSystem.java:[line 421] Useless condition:lazyPersist == true at this point At CommandWithDestination.java:[line 502] org.apache.hadoop.io.DoubleWritable.compareTo(DoubleWritable) incorrectly handles double value At DoubleWritable.java: At DoubleWritable.java:[line 78] org.apache.hadoop.io.DoubleWritable$Comparator.compare(byte[], int, int, byte[], int, int) incorrectly handles double value At DoubleWritable.java:int) incorrectly handles double value At DoubleWritable.java:[line 97] org.apache.hadoop.io.FloatWritable.compareTo(FloatWritable) incorrectly handles float value At FloatWritable.java: At FloatWritable.java:[line 71] org.apache.hadoop.io.FloatWritable$Comparator.compare(byte[], int, int, byte[], int, int) incorrectly handles float value At FloatWritable.java:int) incorrectly handles float value At FloatWritable.java:[line 89] Possible null pointer dereference in org.apache.hadoop.io.IOUtils.listDirectory(File, FilenameFilter) due to return value of called method Dereferenced at IOUtils.java:org.apache.hadoop.io.IOUtils.listDirectory(File, FilenameFilter) due to return value of called method Dereferenced at IOUtils.java:[line 350] org.apache.hadoop.io.erasurecode.ECSchema.toString() makes inefficient use of keySet iterator instead of entrySet iterator At ECSchema.java:keySet iterator instead of entrySet iterator At ECSchema.java:[line 193] Possible bad parsing of shift operation in org.apache.hadoop.io.file.tfile.Utils$Version.hashCode() At Utils.java:operation in org.apache.hadoop.io.file.tfile.Utils$Version.hashCode() At Utils.java:[line 398] org.apache.hadoop.metrics2.lib.DefaultMetricsFactory.setInstance(MutableMetricsFactory) unconditionally sets the field mmfImpl At DefaultMetricsFactory.java:mmfImpl At DefaultMetricsFactory.java:[line 49] org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.setMiniClusterMode(boolean) unconditionally sets the field miniClusterMode At DefaultMetricsSystem.java:miniClusterMode At DefaultMetricsSystem.java:[line 100] Useless object stored in variable seqOs of method org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.addOrUpdateToken(AbstractDelegationTokenIdentifier,
[jira] [Created] (HDFS-11834) Fix TestArchive#testArchive
Xiaoyu Yao created HDFS-11834:
-
Summary: Fix TestArchive#testArchive
Key: HDFS-11834
URL: https://issues.apache.org/jira/browse/HDFS-11834
Project: Hadoop HDFS
Issue Type: Sub-task
Components: test
Affects Versions: HDFS-7240
Reporter: Xiaoyu Yao
This Alder32 based CRC check does not mismatch on MAC but does on some Jenkins
machines based on some recent Jenkins run:
{code}
org.apache.hadoop.scm.TestArchive.testArchive
Failing for the past 1 build (Since Failed#19352 )
Took 21 sec.
Error Message
expected:<3488429799> but was:<2161587943>
Stacktrace
java.lang.AssertionError: expected:<3488429799> but was:<2161587943>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at org.apache.hadoop.scm.TestArchive.testArchive(TestArchive.java:104)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11832) Switch leftover logs to slf4j format in BlockManager.java
Hui Xu created HDFS-11832:
-
Summary: Switch leftover logs to slf4j format in BlockManager.java
Key: HDFS-11832
URL: https://issues.apache.org/jira/browse/HDFS-11832
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.0.0-alpha1, 2.8.0, 2.7.0
Reporter: Hui Xu
Priority: Minor
HDFS-7706 Switch BlockManager logging to use slf4j. But the logging formats
were not changed appropriately. For example:
if (LOG.isDebugEnabled()) {
LOG.debug("blocks = " + java.util.Arrays.asList(blocks));
}
These codes should be changed to:
LOG.debug("blocks = {}", java.util.Arrays.asList(blocks));
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11831) Ozone: Improve the way of getting test file path in unit tests
Yiqun Lin created HDFS-11831: Summary: Ozone: Improve the way of getting test file path in unit tests Key: HDFS-11831 URL: https://issues.apache.org/jira/browse/HDFS-11831 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Yiqun Lin Assignee: Yiqun Lin -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11830) Ozone: Datanode needs to re-register to SCM if SCM is restarted
Weiwei Yang created HDFS-11830: -- Summary: Ozone: Datanode needs to re-register to SCM if SCM is restarted Key: HDFS-11830 URL: https://issues.apache.org/jira/browse/HDFS-11830 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Weiwei Yang Assignee: Weiwei Yang Problem description: # Start NN, DN, SCM # Restart SCM and will see following warnings in SCM log 17/05/02 00:47:08 WARN node.SCMNodeManager: SCM receive heartbeat from unregistered datanode Datanode could not re-establish communication with SCM afterwards. Propose to fix this by adding a new command in HB handling telling datanode to re-register with SCM. Datanode once received this command transits to REGISTER state again to proceed. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11828) Refactor FsDatasetImpl as the BlockAlias is in the wire protocol for PROVIDED blocks.
Ewan Higgs created HDFS-11828:
-
Summary: Refactor FsDatasetImpl as the BlockAlias is in the wire
protocol for PROVIDED blocks.
Key: HDFS-11828
URL: https://issues.apache.org/jira/browse/HDFS-11828
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Ewan Higgs
Assignee: Ewan Higgs
>From HDFS-11639:
{quote}[~virajith]
Looking over this patch, one thing that occurred to me is if it makes sense to
unify FileRegionProvider with BlockProvider? They both have very close
functionality.
I like the use of BlockProvider#resolve(). If we unify FileRegionProvider with
BlockProvider, then resolve can return null if the block map is accessible from
the Datanodes also. If it is accessible only from the Namenode, then a non-null
value can be propagated to the Datanode.
One of the motivations for adding the BlockAlias to the client protocol was to
have the blocks map only on the Namenode. In this scenario, the ReplicaMap in
FsDatasetImpl of will not have any replicas apriori. Thus, one way to ensure
that the FsDatasetImpl interface continues to function as today is to create a
FinalizedProvidedReplica in FsDatasetImpl#getBlockInputStream when BlockAlias
is not null.
{quote}
{quote}[~ehiggs]
With the pending refactoring of the FsDatasetImpl which won't have replicas a
priori, I wonder if it makes sense for the Datanode to have a
FileRegionProvider or BlockProvider at all. They are given the appropriate
block ID and block alias in the readBlock or writeBlock message. Maybe I'm
overlooking what's still being provided.{quote}
{quote}[~virajith]
I was trying to reconcile the existing design (FsDatasetImpl knows about
provided blocks apriori) with the new design where FsDatasetImpl will not know
about these before but just constructs them on-the-fly using the BlockAlias
from readBlock or writeBlock. Using BlockProvider#resolve() allows us to have
both designs exist in parallel. I was wondering if we should still retain the
earlier given the latter design.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11827) NPE is thrown when log level changed in BlockPlacementPolicyDefault#chooseRandom() method
xupeng created HDFS-11827: - Summary: NPE is thrown when log level changed in BlockPlacementPolicyDefault#chooseRandom() method Key: HDFS-11827 URL: https://issues.apache.org/jira/browse/HDFS-11827 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Environment: hadoop-version: 2.6.0-cdh5.4.4 Reporter: xupeng This issue was found by my colleague when changing log-level of BlockPlacementPolicy using "hadoop daemonlog" command. The exception stack trace is below: 2017-05-16 13:36:51,053 ERROR org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: ReplicationMonitor thread received Runtime exception. java.lang.NullPointerException at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:666) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:607) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:375) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:446) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:446) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:240) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:119) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithMultiDC.chooseTargetInternal(BlockPlacementPolicyWithMultiDC.java:263) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithMultiDC.chooseTarget(BlockPlacementPolicyWithMultiDC.java:214) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork.chooseTargets(BlockManager.java:3836) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork.access$200(BlockManager.java:3801) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1394) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1300) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3764) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3711) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
