lokeshj1703 opened a new issue, #18833:
URL: https://github.com/apache/hudi/issues/18833
### Bug Description
**What happened:**
The exception happens when rocks db handle is accessed by a reader.
Apparently there is another thread which can close the rocksdb handles and
remove them from
`org.apache.hudi.common.util.collection.RocksDBDAO#managedHandlesMap` since it
is not acquiring a write lock while closing the handles. The fix is to use
write lock during such operations.
```
org.apache.hudi.exception.HoodieException: java.lang.NullPointerException
at
org.apache.hudi.common.util.collection.RocksDBDAO.put(RocksDBDAO.java:259)
at
org.apache.hudi.common.util.collection.RocksDbDiskMap.put(RocksDbDiskMap.java:92)
at
org.apache.hudi.common.util.collection.RocksDbDiskMap.put(RocksDbDiskMap.java:43)
at
org.apache.hudi.common.util.collection.ExternalSpillableMap.put(ExternalSpillableMap.java:225)
at
org.apache.hudi.common.util.collection.ExternalSpillableMap.put(ExternalSpillableMap.java:58)
at
org.apache.hudi.common.table.view.HoodieTableFileSystemView.storePartitionView(HoodieTableFileSystemView.java:404)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$addFilesToView$5(AbstractTableFileSystemView.java:178)
at java.util.HashMap.forEach(HashMap.java:1290)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.addFilesToView(AbstractTableFileSystemView.java:166)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$17(AbstractTableFileSystemView.java:428)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:418)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestBaseFilesBeforeOrOn(AbstractTableFileSystemView.java:640)
at
org.apache.hudi.timeline.service.handlers.BaseFileHandler.getLatestDataFilesBeforeOrOn(BaseFileHandler.java:58)
at
org.apache.hudi.timeline.service.RequestHandler.lambda$registerDataFilesAPI$12(RequestHandler.java:278)
at
org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:614)
at
io.javalin.core.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:20)
at
io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96)
at
io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
at
io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
at
io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:99)
at
io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85)
at
java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
at
java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
at
io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at
io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at
org.apache.hudi.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at
org.apache.hudi.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at
org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.apache.hudi.org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at
org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52)
at
org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.apache.hudi.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at
org.apache.hudi.org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at
org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.apache.hudi.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at
org.apache.hudi.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.apache.hudi.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.apache.hudi.org.eclipse.jetty.server.Server.handle(Server.java:516)
at
org.apache.hudi.org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at
org.apache.hudi.org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at
org.apache.hudi.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at
org.apache.hudi.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at
org.apache.hudi.org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at
org.apache.hudi.org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at
org.apache.hudi.org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.apache.hudi.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at
org.apache.hudi.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException
at org.rocksdb.RocksDB.put(RocksDB.java:854)
at
org.apache.hudi.common.util.collection.RocksDBDAO.put(RocksDBDAO.java:257)
... 52 more
```
**What you expected:**
**Steps to reproduce:**
1. This is a concurrent access issue, difficult to reproduce
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
