nsivabalan commented on a change in pull request #2494:
URL: https://github.com/apache/hudi/pull/2494#discussion_r584328039
##########
File path:
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -112,13 +113,59 @@ private void initIfNeeded() {
@Override
protected Option<HoodieRecord<HoodieMetadataPayload>>
getRecordByKeyFromMetadata(String key) {
+ // This function can be called in parallel through multiple threads. For
each thread, we determine the thread-local
+ // versions of the baseFile and logRecord readers to use.
+ // - If reuse is enabled, we use the same readers and dont close them
+ // - if reuse is disabled, we open new readers in each thread and close
them
+ HoodieFileReader localFileReader = null;
+ HoodieMetadataMergedLogRecordScanner localLogRecordScanner = null;
+ synchronized (this) {
+ if (!metadataConfig.enableReuse()) {
+ // reuse is disabled so always open new readers
+ try {
+ Pair<HoodieFileReader, HoodieMetadataMergedLogRecordScanner> readers
= openReaders();
+ localFileReader = readers.getKey();
+ localLogRecordScanner = readers.getValue();
+ } catch (IOException e) {
+ throw new HoodieIOException("Error opening readers", e);
+ }
+ } else if (baseFileReader == null && logRecordScanner == null) {
+ // reuse is enabled but we haven't opened the readers yet
+ try {
+ Pair<HoodieFileReader, HoodieMetadataMergedLogRecordScanner> readers
= openReaders();
+ localFileReader = readers.getKey();
+ localLogRecordScanner = readers.getValue();
+ // cache the readers
+ baseFileReader = localFileReader;
+ logRecordScanner = localLogRecordScanner;
+ } catch (IOException e) {
+ throw new HoodieIOException("Error opening readers", e);
+ }
+ } else {
+ // reuse the already open readers
+ ValidationUtils.checkState((baseFileReader != null || logRecordScanner
!= null), "Readers should already be open");
+ localFileReader = baseFileReader;
+ localLogRecordScanner = logRecordScanner;
+ }
+ }
Review comment:
@vinothchandar : Can we sync up on this sometime and get a closure.
Would like to have this in before our next release. May be at the end of next
week's sync meeting, we can discuss on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
