[ https://issues.apache.org/jira/browse/SOLR-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504386#comment-16504386 ]
Shawn Heisey commented on SOLR-12283: ------------------------------------- Indeed, it looks like DIH doesn't work if its jar is loaded via the .system collection. If I do this the way you did, setting it up on Windows 10 with the cloud example in solr 7.3.0, running curl remotely on a Linux machine, I get a similar, but not absolutely identical, problem. Different line number for the NPE, but much the same issue. {noformat} ERROR - 2018-06-07 07:49:04.817; [c:foo s:shard1 r:core_node2 x:foo_shard1_replica_n1] org.apache.solr.common.SolrException; Full Import failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to PropertyWriter implementation:ZKPropertiesWriter at org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:339) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:420) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483) at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:941) at org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:335) ... 4 more {noformat} Then I got rid of that collection and the .system collection, created a lib directory in each of the solr homes for the example nodes, copied the two jars to those lib directories, restarted everything, made a NEW collection with the handler, and tried again. This time DIH itself worked and it was the MySQL connection that failed. {noformat} ERROR - 2018-06-07 08:00:42.981; [c:bar2 s:shard1 r:core_node2 x:bar2_shard1_replica_n1] org.apache.solr.common.SolrException; Exception while processing: events_Gesloc document : SolrInputDocument(fields: []):org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT id from view_indexation_advertisements Processing Document # 1 at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:327) at org.apache.solr.handler.dataimport.JdbcDataSource.createResultSetIterator(JdbcDataSource.java:288) at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:283) at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:52) at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59) at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73) at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483) at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466) at java.lang.Thread.run(Thread.java:748) Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2188) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330) at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:192) at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172) at org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:528) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:317) ... 14 more Caused by: java.net.UnknownHostException: srv-gesloc-sql at java.net.InetAddress.getAllByName0(InetAddress.java:1280) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:188) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:301) ... 30 more {noformat} This is quite odd. It looks like a classloader issue, but I would have expected the same problem either way, because in both cases, the jar is loaded by runtime classloaders. > Unable To Load ZKPropertiesWriter when dih.jar is added as runtimelib BLOB in > .system collection > ------------------------------------------------------------------------------------------------ > > Key: SOLR-12283 > URL: https://issues.apache.org/jira/browse/SOLR-12283 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: contrib - DataImportHandler > Affects Versions: 6.6.1, 7.3 > Environment: Debian > SolrCloud > Reporter: Maxence SAUNIER > Priority: Blocker > Attachments: modified-DIH.zip, modified-DIH.zip, > mysql-connector-java-5.1.46-bin.jar, mysql-connector-java-5.1.46.jar, > request_handler_config.json, solr-core-7.3.0.jar, > solr-dataimporthandler-7.3.0.jar, solr-dataimporthandler-extras-7.3.0.jar, > solr-solrj-7.3.0.jar, solr.log, solr.log, solr.log, solr.log > > > Hello, > It's been 2 weeks that I try to correct this problem with the community > user-solr but no success. I seriously wonder if this is not a problem in the > code. I do not have the impression that many people use DIH with Solr's cloud > version. > On Internet, no similar problem. > For information, the following configuration of DIH comes from DIHs that work > in production on a single Solr server. The connections to the databases are > therefore correct. > *Errors messages:* > {panel:title=DataImporter} > {code:java} > Full Import > failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable > to PropertyWriter implementation:ZKPropertiesWriter > at > org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:339) > at > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:420) > at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483) > at > org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at org.eclipse.jetty.server.Server.handle(Server.java:530) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.NullPointerException > at > org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:935) > at > org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:335) > ... 42 more > {code} > {panel} > {panel:title=RequestHandlerBase} > {code:java} > java.lang.NullPointerException > at > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:429) > at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483) > at > org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at org.eclipse.jetty.server.Server.handle(Server.java:530) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) > at java.lang.Thread.run(Thread.java:748) > {code} > {panel} > {panel:title=HttpSolrCall} > {code:java} > null:java.lang.NullPointerException > at > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:429) > at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483) > at > org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at org.eclipse.jetty.server.Server.handle(Server.java:530) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) > at java.lang.Thread.run(Thread.java:748) > {code} > {panel} > *Process to reproduce the problem:* > Create .system collection for the BLOB > {code} > curl http://localhost:8983/solr/admin/collections?action=CREATE&name=.system > {code} > Add Data Import Handler JAR file. > {code} > curl -X POST -H 'Content-Type: application/octet-stream' --data-binary > @solr-dataimporthandler-7.3.0.jar > http://localhost:8983/solr/.system/blob/data-import-handler > {code} > Add the runtime lib on the configoverlay.json with the API > {code} > curl http://localhost:8983/solr/unicorn/config -H > 'Content-type:application/json' -d '{"add-runtimelib": { > "name":"data-import-handler", "version":1 }}' > {code} > Send mysql connector on the blob. I have test with the connector-bin.jar but > I have the same error. > {code} > curl -X POST -H 'Content-Type: application/octet-stream' --data-binary > @mysql-connector-java-5.1.46.jar > http://localhost:8983/solr/.system/blob/mysql-connector-java > {code} > Add runtime > {code} > curl http://localhost:8983/solr/unicorn/config -H > 'Content-type:application/json' -d '{"add-runtimelib": { > "name":"mysql-connector-java", "version":1 }}' > {code} > Personally, I create a DIH folder on the collection config path on the cloud. > It's not necessary. > And I put the DIH config file on the cloud with the zkcli (it's a personnal > script to overload the zkcli.sh on Solr but it's the same result. My file is > on the cloud.) > {code} > ./zkcli.py putfile /configs/unicorn/DIH/ DIH/indexation_events.xml > {code} > Add request handler config: > {code} > curl -X POST http://{solr_server}:8983/solr/{collection_name}/config -H > 'Content-type:application/json' -d ' > { > "add-requesthandler": > { > "name": "/test-dih", > "class": "org.apache.solr.handler.dataimport.DataImportHandler", > "runtimeLib": true, > "version": 1, > "defaults": > { > "config": "DIH/indexation_events.xml" > } > } > } ' > {code} > Reload the config: > {code} > curl > http://localhost:8983/solr/unicorn/test-dih?command=reload-config&commit=true&debug=true > {code} > Import and have error: > {code} > curl > http://localhost:8983/solr/unicorn/test-dih?command=full-import&commit=true&debug=true > {code} > I have test to add on request handler and BLOB this jar files but not change: > # data-import-handler-extras > # solr-core > # solr-solrj > I have join all files used in this process. > Thansk you. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org