Zihan Li created GOBBLIN-2024:
---------------------------------
Summary: Improve the performance of GobblinMCEWriter
Key: GOBBLIN-2024
URL: https://issues.apache.org/jira/browse/GOBBLIN-2024
Project: Apache Gobblin
Issue Type: Improvement
Reporter: Zihan Li
We are seeing slowness in the GobblinMCEWriter, which will cause us huge
backlog when handling large topic and affect the other topics in the same
container for GMIP. After benchmark, we identify that we can improve the
computeSpec part to reduce the duplicate work, also we can add the file size
information in GMCE to avoid calling HDFS for each file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
