[
https://issues.apache.org/jira/browse/HIVE-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634774#comment-14634774
]
Harsh J commented on HIVE-11325:
--------------------------------
I missed the srcDir declare, which'd explain the loop (we're walking). I'm
checking why it doesn't abort at the family directory.
> Infinite loop in HiveHFileOutputFormat
> --------------------------------------
>
> Key: HIVE-11325
> URL: https://issues.apache.org/jira/browse/HIVE-11325
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
> Affects Versions: 1.0.0
> Reporter: Harsh J
>
> No idea why {{hbase_handler_bulk.q}} does not catch this if its being run
> regularly in Hive builds, but here's the gist of the issue:
> The condition at
> https://github.com/apache/hive/blob/master/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java#L152-L164
> indicates that we will infinitely loop until we find a file whose last path
> component (the name) is equal to the column family name.
> In execution, however, the iteration enters an actual infinite loop cause the
> file we end up considering as the srcDir name, is actually the region file,
> whose name will never match the family name.
> This is an example of the IPC the listing loop of a 100% progress task gets
> stuck in:
> {code}
> 2015-07-21 10:32:20,662 TRACE [main] org.apache.hadoop.ipc.ProtobufRpcEngine:
> 1: Call -> cdh54.vm/172.16.29.132:8020: getListing {src:
> "/user/hive/warehouse/hbase_test/_temporary/1/_temporary/attempt_1436935612068_0011_m_000000_0/family/97112ac1c09548ae87bd85af072d2e8c"
> startAfter: "" needLocation: false}
> 2015-07-21 10:32:20,662 DEBUG [IPC Parameter Sending Thread #1]
> org.apache.hadoop.ipc.Client: IPC Client (1551465414) connection to
> cdh54.vm/172.16.29.132:8020 from hive sending #510346
> 2015-07-21 10:32:20,662 DEBUG [IPC Client (1551465414) connection to
> cdh54.vm/172.16.29.132:8020 from hive] org.apache.hadoop.ipc.Client: IPC
> Client (1551465414) connection to cdh54.vm/172.16.29.132:8020 from hive got
> value #510346
> 2015-07-21 10:32:20,662 DEBUG [main] org.apache.hadoop.ipc.ProtobufRpcEngine:
> Call: getListing took 0ms
> 2015-07-21 10:32:20,662 TRACE [main] org.apache.hadoop.ipc.ProtobufRpcEngine:
> 1: Response <- cdh54.vm/172.16.29.132:8020: getListing {dirList {
> partialListing { fileType: IS_FILE path: "" length: 863 permission { perm:
> 4600 } owner: "hive" group: "hive" modification_time: 1437454718130
> access_time: 1437454717973 block_replication: 1 blocksize: 134217728 fileId:
> 33960 childrenNum: 0 storagePolicy: 0 } remainingEntries: 0 }}
> {code}
> The path we are getting out of the listing results is
> {{/user/hive/warehouse/hbase_test/_temporary/1/_temporary/attempt_1436935612068_0011_m_000000_0/family/97112ac1c09548ae87bd85af072d2e8c}},
> but instead of checking the path's parent {{family}} we're instead looping
> infinitely over its hashed filename {{97112ac1c09548ae87bd85af072d2e8c}}
> cause it does not match {{family}}.
> It stays in the infinite loop therefore, until the MR framework kills it away
> due to an idle task timeout (and then since the subsequent task attempts fail
> outright, the job fails).
> While doing a {{getPath().getParent()}} will resolve that, is that infinite
> loop even necessary? Especially given the fact that we throw exceptions if
> there are no entries or there is more than one entry.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
