Rajesh Balamohan created HIVE-24862:
---------------------------------------
Summary: Fix race condition causing NPE during dynamic partition
loading
Key: HIVE-24862
URL: https://issues.apache.org/jira/browse/HIVE-24862
Project: Hive
Issue Type: Improvement
Reporter: Rajesh Balamohan
Following properties default to 15 threads.
{noformat}
hive.load.dynamic.partitions.thread
hive.mv.files.thread
{noformat}
During loadDynamicPartitions, it ends ups initializing {{newFiles}} without
synchronization (HIVE-20661, HIVE-24738).
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2871]
This causes race condition when dynamic partition thread internally makes use
of {{hive.mv.files.threads}} in copyFiles/replaceFiles.
This causes "NPE" during retrieval in {{addInsertFileInformation()}}.
e.g stacktrace
{noformat}
Caused by: java.lang.NullPointerException
at org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2734)
at
org.apache.hadoop.hdfs.DistributedFileSystem.fixRelativePart(DistributedFileSystem.java:3396)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1740)
at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1740)
at
org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3566)
at org.apache.hadoop.hive.ql.metadata.Hive.fireInsertEvent(Hive.java:3540)
at
org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2414)
at
org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$4(Hive.java:2909)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
