bvaradar commented on a change in pull request #600: Timeline Service with
Incremental View Syncing support
URL: https://github.com/apache/incubator-hudi/pull/600#discussion_r276859404
##########
File path:
hoodie-common/src/main/java/com/uber/hoodie/common/util/collection/DiskBasedMap.java
##########
@@ -74,6 +73,22 @@ protected DiskBasedMap(String baseFilePath) throws
IOException {
this.filePosition = new AtomicLong(0L);
}
+ /**
+ * RandomAcessFile is not thread-safe. This API opens a new file handle and
returns.
+ * @return
+ */
+ private RandomAccessFile getNewRandomAccessFile() {
Review comment:
Discussed f2f. We agreed on following .
RandomAccessFile is not threadsafe. Hence, we would have to create separate
RandomAccessFileObjects for concurrent access. Agree, the current approach of
opening a new file per get() call is slow. I have made changes to use
ThreadLocal objects. This should reduce the open() calls greatly but for a long
running timeline service, there could be open handles getting accumulated per
thread. A better approach would be to use Resource Pool but needs to be config
tuned and tested. There is another place (ThreadLocal<BinaryEncoder> where we
might require similar treatment. Will open a ticket to address it. If current
perf testing points to the direction of Resource pool, will address it as part
of this PR. Otherwise, we can take this in subsequent PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
