[ https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15010029#comment-15010029 ]
Rahul Challapalli commented on DRILL-3171: ------------------------------------------ [~hgunes] Below is the exception from the logs {code} 2015-11-17 18:01:20,517 [29b4240e-fb58-6b1b-0ce9-3b1341a6bac7:foreman] ERROR o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: ExecutionSetupException: Two processes tried to change a plugin at the same time. [Error Id: 1af07811-d7cd-4bea-a2b1-7732883b5b31 on atsqa6c63.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: ExecutionSetupException: Two processes tried to change a plugin at the same time. [Error Id: 1af07811-d7cd-4bea-a2b1-7732883b5b31 on atsqa6c63.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534) ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception during fragment initialization: Failure while updating storage plugins ... 4 common frames omitted Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Failure while updating storage plugins at org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas(StoragePluginRegistry.java:408) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:166) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:155) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:143) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema(QueryContext.java:129) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.planner.sql.DrillSqlWorker.<init>(DrillSqlWorker.java:93) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:907) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244) [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] ... 3 common frames omitted Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: Two processes tried to change a plugin at the same time. at org.apache.drill.exec.store.StoragePluginRegistry.createOrUpdate(StoragePluginRegistry.java:261) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.store.StoragePluginRegistry.getPlugin(StoragePluginRegistry.java:293) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] at org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas(StoragePluginRegistry.java:391) ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT] ... 10 common frames omitted {code} > Storage Plugins : Two processes tried to update the storage plugin at the > same time > ----------------------------------------------------------------------------------- > > Key: DRILL-3171 > URL: https://issues.apache.org/jira/browse/DRILL-3171 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Information Schema > Affects Versions: 1.0.0 > Reporter: Rahul Challapalli > Assignee: Hanifi Gunes > Labels: test > Fix For: Future > > > Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474 > We have seen this issue happen with the below steps > 1. Clear out the zookeeper > 2. Update the storage plugin using the rest API on one of the node > 3. Submit 10 queries concurrently > With randomized foreman node selection, the node executing the query might > not have the updated storage plugins info. This could be causing the issue. > - Rahul -- This message was sent by Atlassian JIRA (v6.3.4#6332)