Adam Gilmore created DRILL-3522:
-----------------------------------
Summary: IllegalStateException from Mongo storage plugin
Key: DRILL-3522
URL: https://issues.apache.org/jira/browse/DRILL-3522
Project: Apache Drill
Issue Type: Bug
Components: Storage - MongoDB
Affects Versions: 1.1.0
Reporter: Adam Gilmore
Assignee: B Anil Kumar
Priority: Critical
With a Mongo storage plugin enabled, we are sporadically getting the following
exception when running queries (even not against the Mongo storage plugin):
{code}
SYSTEM ERROR: IllegalStateException: state should be: open
(org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
during fragment initialization:
org.apache.drill.common.exceptions.DrillRuntimeException: state should be: open
org.apache.drill.exec.work.foreman.Foreman.run():253
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (com.google.common.util.concurrent.UncheckedExecutionException)
org.apache.drill.common.exceptions.DrillRuntimeException: state should be: open
com.google.common.cache.LocalCache$Segment.get():2263
com.google.common.cache.LocalCache.get():4000
com.google.common.cache.LocalCache.getOrLoad():4004
com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
org.apache.drill.exec.ops.QueryContext.getRootSchema():165
org.apache.drill.exec.ops.QueryContext.getRootSchema():154
org.apache.drill.exec.ops.QueryContext.getRootSchema():142
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
org.apache.drill.exec.work.foreman.Foreman.runSQL():901
org.apache.drill.exec.work.foreman.Foreman.run():242
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (org.apache.drill.common.exceptions.DrillRuntimeException) state
should be: open
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():98
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():82
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3599
com.google.common.cache.LocalCache$Segment.loadSync():2379
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2342
com.google.common.cache.LocalCache$Segment.get():2257
com.google.common.cache.LocalCache.get():4000
com.google.common.cache.LocalCache.getOrLoad():4004
com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
org.apache.drill.exec.ops.QueryContext.getRootSchema():165
org.apache.drill.exec.ops.QueryContext.getRootSchema():154
org.apache.drill.exec.ops.QueryContext.getRootSchema():142
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
org.apache.drill.exec.work.foreman.Foreman.runSQL():901
org.apache.drill.exec.work.foreman.Foreman.run():242
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
Caused By (java.lang.IllegalStateException) state should be: open
com.mongodb.assertions.Assertions.isTrue():70
com.mongodb.connection.BaseCluster.selectServer():79
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.():75
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.():71
com.mongodb.binding.ClusterBinding.getReadConnectionSource():63
com.mongodb.operation.OperationHelper.withConnection():166
com.mongodb.operation.ListDatabasesOperation.execute():100
com.mongodb.operation.ListDatabasesOperation.execute():52
com.mongodb.Mongo.execute():738
com.mongodb.Mongo$2.execute():725
com.mongodb.OperationIterable.iterator():47
com.mongodb.OperationIterable.forEach():66
com.mongodb.ListDatabasesIterableImpl.forEach():72
com.mongodb.MappingIterable.forEach():50
com.mongodb.MappingIterable.into():60
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():91
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():82
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3599
com.google.common.cache.LocalCache$Segment.loadSync():2379
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2342
com.google.common.cache.LocalCache$Segment.get():2257
com.google.common.cache.LocalCache.get():4000
com.google.common.cache.LocalCache.getOrLoad():4004
com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
org.apache.drill.exec.ops.QueryContext.getRootSchema():165
org.apache.drill.exec.ops.QueryContext.getRootSchema():154
org.apache.drill.exec.ops.QueryContext.getRootSchema():142
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
org.apache.drill.exec.work.foreman.Foreman.runSQL():901
org.apache.drill.exec.work.foreman.Foreman.run():242
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
{code}
Upon investigation, the issue appears to be registering the Mongo schemas on
connection (because the storage plugin is enabled).
Basically, it appears that recent changes when upgrading the Mongo driver meant
that MongoSchemaFactory indefinitely holds onto a MongoClient, even though they
are part of a cache that expire after 24 hours. This means that
MongoSchemaFactory ends up trying to use a MongoClient that is closed, and thus
the exception occurs.
Considering we already have a cache for MongoClient, it is safe for
MongoSchemaFactory to call plugin.getClient() every time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
