Re: Running multiple full-import commands via curl in a script
Literally, queue can be done by submitting as is (async) and polling command status. However, giving https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L200 you can try to add synchronous=true... that should hang request until it's completed. The other question is how run requests in parallel which is explicitly violated by https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L173 The only workaround I can suggest is to duplicate DIH definitions in solr config requestHandler name=/dataimport class=solr.DataImportHandler ... requestHandler name=/dataimport2 class=solr.DataImportHandler ... requestHandler name=/dataimport3 class=solr.DataImportHandler ... ... then those guys should be able to handle own request in parallel. Nasty stuff.. have a good hack On Wed, Jan 28, 2015 at 3:47 AM, Carl Roberts carl.roberts.zap...@gmail.com wrote: Hi, I am attempting to run all these curl commands from a script so that I can put them in a crontab job, however, it seems that only the first one executes and the other ones return with an error (below): curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2002 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2003 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2004 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2005 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2006 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2007 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2008 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2009 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2010 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2011 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2012 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2013 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2014 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2015 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= delta-importclean=falseentity=cve-last error: *A command is still running...* Question: Is there a way to queue the other requests in Solr so that they run as soon as the previous one is done? If not, how would you recommend I do this? Many thanks in advance, Joe -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics http://www.griddynamics.com mkhlud...@griddynamics.com
Re: Running multiple full-import commands via curl in a script
Thanks Mikhail - synchronous=true works like a charm...:) On 1/28/15, 5:16 AM, Mikhail Khludnev wrote: Literally, queue can be done by submitting as is (async) and polling command status. However, giving https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L200 you can try to add synchronous=true... that should hang request until it's completed. The other question is how run requests in parallel which is explicitly violated by https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L173 The only workaround I can suggest is to duplicate DIH definitions in solr config requestHandler name=/dataimport class=solr.DataImportHandler ... requestHandler name=/dataimport2 class=solr.DataImportHandler ... requestHandler name=/dataimport3 class=solr.DataImportHandler ... ... then those guys should be able to handle own request in parallel. Nasty stuff.. have a good hack On Wed, Jan 28, 2015 at 3:47 AM, Carl Roberts carl.roberts.zap...@gmail.com wrote: Hi, I am attempting to run all these curl commands from a script so that I can put them in a crontab job, however, it seems that only the first one executes and the other ones return with an error (below): curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2002 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2003 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2004 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2005 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2006 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2007 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2008 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2009 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2010 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2011 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2012 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2013 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2014 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= full-importclean=falseentity=cve-2015 curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command= delta-importclean=falseentity=cve-last error: *A command is still running...* Question: Is there a way to queue the other requests in Solr so that they run as soon as the previous one is done? If not, how would you recommend I do this? Many thanks in advance, Joe
Running multiple full-import commands via curl in a script
Hi, I am attempting to run all these curl commands from a script so that I can put them in a crontab job, however, it seems that only the first one executes and the other ones return with an error (below): curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2002; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2003; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2004; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2005; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2006; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2007; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2008; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2009; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2010; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2011; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2012; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2013; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2014; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=full-importclean=falseentity=cve-2015; curl http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=delta-importclean=falseentity=cve-last; error: *A command is still running...* Question: Is there a way to queue the other requests in Solr so that they run as soon as the previous one is done? If not, how would you recommend I do this? Many thanks in advance, Joe
