[ https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Zhuge updated HDFS-12528: ------------------------------ Attachment: HDFS-12528.000.patch Patch 000 * Unit test only Here is the test output. {{Unbuffer}} placed the replace into evictable: {noformat} 2017-10-23 17:19:37,189 [Thread-0] TRACE shortcircuit.ShortCircuitCache (ShortCircuitCache.java:unref(470)) - ShortCircuitCache(0x1d7b9bbe): unref replica ShortCircuitReplica{key=1073741825_BP-2140752653-10.16.1.161-1508804374205, metaHeader.version=1, metaHeader.checksum=DataChecksum(type=CRC32C, chunkSize=512), ident=0x54889b35, creationTimeMs=171553729}: added to evictable, refCount 2 -> 1 {noformat} {{trimEvictionMaps}} purged the replica immediately because the unit test sets the ShortCircuitCache size to 0: {noformat} 2017-10-23 17:19:37,189 [Thread-0] TRACE shortcircuit.ShortCircuitCache (ShortCircuitCache.java:trimEvictionMaps(554)) - ShortCircuitCache(0x1d7b9bbe): trimEvictionMaps is purging ShortCircuitReplica{key=1073741825_BP-2140752653-10.16.1.161-1508804374205, metaHeader.version=1, metaHeader.checksum=DataChecksum(type=CRC32C, chunkSize=512), ident=0x54889b35, creationTimeMs=171553729} {noformat} The next read after append hit this error: {noformat} 2017-10-23 17:19:37,250 [DataXceiver for client unix:/tmp/socks.1508804372834.-1113921213/testShortCircuitCacheUnbuffer.59920 [Passing file descriptors for block BP-2140752653-10.16.1.161-1508804374205:blk_1073741825_1001]] INFO datanode.DataNode (DataXceiver.java:requestShortCircuitFds(419)) - Unregistering SlotId(a74d30c4ed858b5846649090e3d8929d:0) because the requestShortCircuitFdsForRead operation failed. 2017-10-23 17:19:37,250 [Thread-0] WARN impl.BlockReaderFactory (BlockReaderFactory.java:requestFileDescriptors(647)) - BlockReaderFactory(fileName=/test_file, block=BP-2140752653-10.16.1.161-1508804374205:blk_1073741825_1001): unknown response code ERROR while attempting to set up short-circuit access. No data exists for block BP-2140752653-10.16.1 {noformat} Forced to read from TCP: {noformat} 2017-10-23 17:19:37,250 [Thread-0] WARN shortcircuit.ShortCircuitCache (ShortCircuitCache.java:create(792)) - ShortCircuitCache(0x1d7b9bbe): failed to load 1073741825_BP-2140752653-10.16.1.161-1508804374205 2017-10-23 17:19:37,251 [Thread-0] TRACE impl.BlockReaderFactory (BlockReaderFactory.java:getRemoteBlockReaderFromTcp(733)) - BlockReaderFactory(fileName=/test_file, block=BP-2140752653-10.16.1.161-1508804374205:blk_1073741825_1001): trying to create a remote block reader from a TCP socket {noformat} > Short-circuit reads getting disabled frequently in certain scenarios > -------------------------------------------------------------------- > > Key: HDFS-12528 > URL: https://issues.apache.org/jira/browse/HDFS-12528 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, performance > Affects Versions: 2.6.0 > Reporter: Andre Araujo > Assignee: John Zhuge > Attachments: HDFS-12528.000.patch > > > We have scenarios where data ingestion makes use of the -appendToFile > operation to add new data to existing HDFS files. In these situations, we're > frequently running into the problem described below. > We're using Impala to query the HDFS data with short-circuit reads (SCR) > enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce > the memory footprint. In some cases, though, Impala still keeps the HDFS file > handle open for reuse. > The "unbuffer" call, however, causes the file's current block reader to be > closed, which makes the associated ShortCircuitReplica evictable from the > ShortCircuitCache. When the cluster is under load, this means that the > ShortCircuitReplica can be purged off the cache pretty fast, which closes the > file descriptor to the underlying storage file. > That means that when Impala re-reads the file it has to re-open the storage > files associated with the ShortCircuitReplica's that were evicted from the > cache. If there were no appends to those blocks, the re-open will succeed > without problems. If one block was appended since the ShortCircuitReplica was > created, the re-open will fail with the following error: > {code} > Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087 > not found > {code} > This error is handled as an "unknown response" by the BlockReaderFactory [1], > which disables short-circuit reads for 10 minutes [2] for the client. > These 10 minutes without SCR can have a big performance impact for the client > operations. In this particular case ("Meta file not found") it would suffice > to return null without disabling SCR. This particular block read would fall > back to the normal, non-short-circuited, path and other SCR requests would > continue to work as expected. > It might also be interesting to be able to control how long SCR is disabled > for in the "unknown response" case. 10 minutes seems a bit to long and not > being able to change that is a problem. > [1] > https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646 > [2] > https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97 -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org