Mehakmeet Singh created HADOOP-17195:
----------------------------------------
Summary: Intermittent OutOfMemory error while performing hdfs
CopyFromLocal to abfs
Key: HADOOP-17195
URL: https://issues.apache.org/jira/browse/HADOOP-17195
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 3.3.0
Reporter: Mehakmeet Singh
Assignee: Mehakmeet Singh
OutOfMemory error due to new ThreadPools being made each time AbfsOutputStream
is created. Since threadPool aren't limited a lot of data is loaded in buffer
and thus it causes OutOfMemory error.
Possible fixes:
- Limit the number of ThreadCounts while performing hdfs copyFromLocal (Using
-t property).
- Reducing OUTPUT_BUFFER_SIZE significantly which would limit the amount of
buffer to be loaded in threads.
- Don't create new ThreadPools each time AbfsOutputStream is created and limit
the number of ThreadPools each AbfsOutputStream could create.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
