Davis-Zhang-Onehouse commented on PR #20686:
URL: https://github.com/apache/hudi/pull/20686#issuecomment-7054748657

   > the doc already says it is utilitied per-thread/executor, not sure why 
there are concurrent access, for global cache, use either `HoodieSchemaCache` 
or `HoodieAvroSchemaCache`.
   
   @danny0405 You were right: the cache is meant to be per thread, and it 
should stay that way.
   
   The concurrent access came from two callers that broke that rule. 
`HoodieMergedLogRecordScanner#processNextRecord` and 
`RealtimeCompactedRecordReader#mergeRecord` merged through 
`AvroRecordContext.getFieldAccessorInstance()`, which is one static record 
context per JVM. Every task thread of an executor that merged log records wrote 
the same `LocalHoodieSchemaCache`. `HoodieSchemaCache` and 
`HoodieAvroSchemaCache` only intern schemas and do not issue the integer ids 
the record context needs, so they could not replace it.
   
   I reworked this PR in 43f9b777b8cc6f9c09670c077701192b3706756e. 
`LocalHoodieSchemaCache` is back to the upstream class, unchanged. The scanner 
and the realtime reader each create their own `AvroRecordContext` and merge 
through it, so the shared context is used only for field access. The new test 
checks that a merging scan no longer writes its schema into the shared context, 
and runs merging scans from eight threads at once. I updated the title and 
description to match.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to