[
https://issues.apache.org/jira/browse/HDFS-12618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204794#comment-16204794
]
Wellington Chevreuil commented on HDFS-12618:
---------------------------------------------
I was going to add some tests to TestFsck, but for some reason when running the
tests on trunk (without my changes), am getting below errors for some already
existing tests:
{noformat}
Running org.apache.hadoop.hdfs.server.namenode.TestFsck
Tests run: 32, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 136.329 sec
<<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestFsck
testFsckCorruptECFile(org.apache.hadoop.hdfs.server.namenode.TestFsck) Time
elapsed: 3.022 sec <<< ERROR!
java.lang.IllegalStateException: failed to create a child event loop
at sun.nio.ch.KQueueArrayWrapper.init(Native Method)
at sun.nio.ch.KQueueArrayWrapper.<init>(KQueueArrayWrapper.java:98)
at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:88)
at
sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:174)
at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:150)
at
io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:103)
at
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
at
io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:50)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:70)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:65)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:56)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:48)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:40)
at
org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.<init>(DatanodeHttpServer.java:132)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:954)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1402)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:497)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2778)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2681)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1635)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:882)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:494)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:453)
at
org.apache.hadoop.hdfs.server.namenode.TestFsck.testFsckCorruptECFile(TestFsck.java:2304)
testBlockIdCKDecommission(org.apache.hadoop.hdfs.server.namenode.TestFsck)
Time elapsed: 2.702 sec <<< ERROR!
java.lang.IllegalStateException: failed to create a child event loop
at sun.nio.ch.IOUtil.makePipe(Native Method)
at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
at
sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:174)
at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:150)
at
io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:103)
at
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
at
io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:50)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:70)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:65)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:56)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:48)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:40)
at
org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.<init>(DatanodeHttpServer.java:132)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:954)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1402)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:497)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2778)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2681)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1635)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:882)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:494)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:453)
at
org.apache.hadoop.hdfs.server.namenode.TestFsck.testBlockIdCKDecommission(TestFsck.java:1504)
testFsckOpenECFiles(org.apache.hadoop.hdfs.server.namenode.TestFsck) Time
elapsed: 0.428 sec <<< ERROR!
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes
- current valid volumes: 0, volumes configured: 2, volumes failed: 2, volume
failures tolerated: 0
at
org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:220)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2771)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2681)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1635)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:882)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:494)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:453)
at
org.apache.hadoop.hdfs.server.namenode.TestFsck.testFsckOpenECFiles(TestFsck.java:689)
testBlockIdCK(org.apache.hadoop.hdfs.server.namenode.TestFsck) Time elapsed:
2.376 sec <<< ERROR!
java.lang.IllegalStateException: failed to create a child event loop
at sun.nio.ch.KQueueArrayWrapper.init(Native Method)
at sun.nio.ch.KQueueArrayWrapper.<init>(KQueueArrayWrapper.java:98)
at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:88)
at
sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:174)
at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:150)
at
io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:103)
at
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
at
io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:50)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:70)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:65)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:56)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:48)
at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:40)
at
org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.<init>(DatanodeHttpServer.java:132)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:954)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1402)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:497)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2778)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2681)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1635)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:882)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:494)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:453)
at
org.apache.hadoop.hdfs.server.namenode.TestFsck.testBlockIdCK(TestFsck.java:1449)
Results :
Tests in error:
TestFsck.testFsckCorruptECFile:2304 » IllegalState failed to create a child
ev...
TestFsck.testBlockIdCKDecommission:1504 » IllegalState failed to create a
chil...
TestFsck.testFsckOpenECFiles:689 » DiskError Too many failed volumes -
current...
TestFsck.testBlockIdCK:1449 » IllegalState failed to create a child event loop
{noformat}
Is this something environmental?
> fsck -includeSnapshots reports wrong amount of total blocks
> -----------------------------------------------------------
>
> Key: HDFS-12618
> URL: https://issues.apache.org/jira/browse/HDFS-12618
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
> Affects Versions: 3.0.0-alpha3
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Minor
> Attachments: HDFS-121618.initial
>
>
> When snapshot is enabled, if a file is deleted but is contained by a
> snapshot, *fsck* will not reported blocks for such file, showing different
> number of *total blocks* than what is exposed in the Web UI.
> This should be fine, as *fsck* provides *-includeSnapshots* option. The
> problem is that *-includeSnapshots* option causes *fsck* to count blocks for
> every occurrence of a file on snapshots, which is wrong because these blocks
> should be counted only once (for instance, if a 100MB file is present on 3
> snapshots, it would still map to one block only in hdfs). This causes fsck to
> report much more blocks than what actually exist in hdfs and is reported in
> the Web UI.
> Here's an example:
> 1) HDFS has two files of 2 blocks each:
> {noformat}
> $ hdfs dfs -ls -R /
> drwxr-xr-x - root supergroup 0 2017-10-07 21:21 /snap-test
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16 /snap-test/file1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17 /snap-test/file2
> drwxr-xr-x - root supergroup 0 2017-05-13 13:03 /test
> {noformat}
> 2) There are two snapshots, with the two files present on each of the
> snapshots:
> {noformat}
> $ hdfs dfs -ls -R /snap-test/.snapshot
> drwxr-xr-x - root supergroup 0 2017-10-07 21:21
> /snap-test/.snapshot/snap1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16
> /snap-test/.snapshot/snap1/file1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17
> /snap-test/.snapshot/snap1/file2
> drwxr-xr-x - root supergroup 0 2017-10-07 21:21
> /snap-test/.snapshot/snap2
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16
> /snap-test/.snapshot/snap2/file1
> -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17
> /snap-test/.snapshot/snap2/file2
> {noformat}
> 3) *fsck -includeSnapshots* reports 12 blocks in total (4 blocks for the
> normal file path, plus 4 blocks for each snapshot path):
> {noformat}
> $ hdfs fsck / -includeSnapshots
> FSCK started by root (auth:SIMPLE) from /127.0.0.1 for path / at Mon Oct 09
> 15:15:36 BST 2017
> Status: HEALTHY
> Number of data-nodes: 1
> Number of racks: 1
> Total dirs: 6
> Total symlinks: 0
> Replicated Blocks:
> Total size: 1258291200 B
> Total files: 6
> Total blocks (validated): 12 (avg. block size 104857600 B)
> Minimally replicated blocks: 12 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 0 (0.0 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 1
> Average block replication: 1.0
> Missing blocks: 0
> Corrupt blocks: 0
> Missing replicas: 0 (0.0 %)
> {noformat}
> 4) Web UI shows the correct number (4 blocks only):
> {noformat}
> Security is off.
> Safemode is off.
> 5 files and directories, 4 blocks = 9 total filesystem object(s).
> {noformat}
> I would like to work on this solution, will propose an initial solution
> shortly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
