SV: data-import-handler for solr-7.5.0
I made it work with the simplest of xml-files with some inspiration from
https://opensolr.com/blog/2011/09/how-to-import-data-from-xml-files-into-your-solr-collection
.
Data-config is now:
And the document is simply:
2165432
5
28548113
89
Now I guess I just have to add to this solution.
Thanks for your help Alex, and also thanks to Jan answering the first mail.
Best regards
Martin Frank Hansen
-Oprindelig meddelelse-
Fra: Alexandre Rafalovitch
Sendt: 2. oktober 2018 19:52
Til: solr-user
Emne: Re: data-import-handler for solr-7.5.0
Ok, so then you can switch to debug mode and keep trying to figure it out. Also
try BinFileDataSource or URLDataSource, maybe it will have an easier way.
Or using relative path (example:
https://github.com/arafalov/solr-apachecon2018-presentation/blob/master/configsets/pets-final/pets-data-config.xml).
Regards,
Alex.
On Tue, 2 Oct 2018 at 12:46, Martin Frank Hansen (MHQ) wrote:
>
> Thanks for the info, the UI looks interesting... It does read the data-config
> correctly, so the problem is probably in this file.
>
> Martin Frank Hansen, Senior Data Analytiker
>
> Data, IM & Analytics
>
>
>
> Lautrupparken 40-42, DK-2750 Ballerup
> E-mail m...@kmd.dk Web www.kmd.dk
> Mobil +4525571418
>
> -Oprindelig meddelelse-
> Fra: Alexandre Rafalovitch
> Sendt: 2. oktober 2018 18:18
> Til: solr-user
> Emne: Re: data-import-handler for solr-7.5.0
>
> Admin UI for DIH will show you the config file read. So, if nothing is
> there, the path is most likely the issue
>
> You can also provide or update the configuration right in UI if you enable
> debug.
>
> Finally, the config file is reread on every invocation, so you don't need to
> restart the core after changing it.
>
> Hope this helps,
>Alex.
> On Tue, 2 Oct 2018 at 11:45, Jan Høydahl wrote:
> >
> > > url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
> >
> > Have you tried url="C:\\Users\\z6mhq/Desktop\\data_import\\nh_test.xml" ?
> >
> > --
> > Jan Høydahl, search solution architect Cominvent AS -
> > www.cominvent.com
> >
> > > 2. okt. 2018 kl. 17:15 skrev Martin Frank Hansen (MHQ) :
> > >
> > > Hi,
> > >
> > > I am having some problems getting the data-import-handler in Solr to
> > > work. I have tried a lot of things but I simply get no response from
> > > Solr, not even an error.
> > >
> > > When calling the API:
> > > http://localhost:8983/solr/nh/dataimport?command=full-import
> > > {
> > > "responseHeader":{
> > >"status":0,
> > >"QTime":38},
> > > "initArgs":[
> > >"defaults",[
> > >
> > > "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
> > > "command":"full-import",
> > > "status":"idle",
> > > "importResponse":"",
> > > "statusMessages":{}}
> > >
> > > The data looks like this:
> > >
> > >
> > >
> > > 2165432
> > > 5
> > >
> > >
> > > 28548113
> > > 89
> > >
> > >
> > > The data-config file looks like this:
> > >
> > >
> > >
> > >
> > > > >name="xml"
> > >pk="id"
> > >processor="XPathEntityProcessor"
> > >stream="true"
> > >forEach="/journal/doc"
> > >url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
> > >transformer="RegexTransformer,TemplateTransformer"
> > >>
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > And I referenced the jar files in the solr-config.xml as well as adding
> > > the request-handler by adding the following lines:
> > >
> > > > > regex="solr-dataimporthandler-\d.*\.jar" /> > > dir="${solr.install.dir:../../../..}/dist/"
> > > regex="solr-dataimporthandler-extras-\d.*\.jar" />
> > >
> > >
> > > > > class="org.apache.solr.handler.dataimport.DataImportHandler">
> > >
> > > > > name="config">C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml
> > >
> > >
> > >
> > > I am running a core residing in the folder
> > > “C:/Users/z6mhq/Desktop/nh/nh/conf” while the Solr installation is in
> > > “C:/Users/z6mhq/Documents/solr-7.5.0”.
> > >
> > > I really hope that someone can spot my mistake…
> > >
> > > Thanks in advance.
> > >
> > > Martin Frank Hansen
> > >
> > >
> > > Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder
> > > du KMD’s Privatlivspolitik, der
> > > fortæller, hvordan vi behandler oplysninger om dig.
> > >
> > > Protection of your personal data is important to us. Here you can read
> > > KMD’s Privacy Policy outlining how we
> > > process your personal data.
> > >
> > > Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information.
> > > Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst
> > > informere afsender om fejlen ved at bruge svarfunktionen. Samtidig beder
> > > vi dig slette e-mailen i dit system uden at videresende eller kopiere
> > > den. Selvom e-
SV: data-import-handler for solr-7.5.0
Thanks for the info, the UI looks interesting... It does read the data-config
correctly, so the problem is probably in this file.
Martin Frank Hansen, Senior Data Analytiker
Data, IM & Analytics
Lautrupparken 40-42, DK-2750 Ballerup
E-mail m...@kmd.dk Web www.kmd.dk
Mobil +4525571418
-Oprindelig meddelelse-
Fra: Alexandre Rafalovitch
Sendt: 2. oktober 2018 18:18
Til: solr-user
Emne: Re: data-import-handler for solr-7.5.0
Admin UI for DIH will show you the config file read. So, if nothing is there,
the path is most likely the issue
You can also provide or update the configuration right in UI if you enable
debug.
Finally, the config file is reread on every invocation, so you don't need to
restart the core after changing it.
Hope this helps,
Alex.
On Tue, 2 Oct 2018 at 11:45, Jan Høydahl wrote:
>
> > url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
>
> Have you tried url="C:\\Users\\z6mhq/Desktop\\data_import\\nh_test.xml" ?
>
> --
> Jan Høydahl, search solution architect Cominvent AS -
> www.cominvent.com
>
> > 2. okt. 2018 kl. 17:15 skrev Martin Frank Hansen (MHQ) :
> >
> > Hi,
> >
> > I am having some problems getting the data-import-handler in Solr to work.
> > I have tried a lot of things but I simply get no response from Solr, not
> > even an error.
> >
> > When calling the API:
> > http://localhost:8983/solr/nh/dataimport?command=full-import
> > {
> > "responseHeader":{
> >"status":0,
> >"QTime":38},
> > "initArgs":[
> >"defaults",[
> > "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
> > "command":"full-import",
> > "status":"idle",
> > "importResponse":"",
> > "statusMessages":{}}
> >
> > The data looks like this:
> >
> >
> >
> > 2165432
> > 5
> >
> >
> > 28548113
> > 89
> >
> >
> > The data-config file looks like this:
> >
> >
> >
> >
> > >name="xml"
> >pk="id"
> >processor="XPathEntityProcessor"
> >stream="true"
> >forEach="/journal/doc"
> >url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
> >transformer="RegexTransformer,TemplateTransformer"
> >>
> >
> >
> >
> >
> >
> >
> >
> > And I referenced the jar files in the solr-config.xml as well as adding the
> > request-handler by adding the following lines:
> >
> > > regex="solr-dataimporthandler-\d.*\.jar" /> > dir="${solr.install.dir:../../../..}/dist/"
> > regex="solr-dataimporthandler-extras-\d.*\.jar" />
> >
> >
> > > class="org.apache.solr.handler.dataimport.DataImportHandler">
> >
> > > name="config">C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml
> >
> >
> >
> > I am running a core residing in the folder
> > “C:/Users/z6mhq/Desktop/nh/nh/conf” while the Solr installation is in
> > “C:/Users/z6mhq/Documents/solr-7.5.0”.
> >
> > I really hope that someone can spot my mistake…
> >
> > Thanks in advance.
> >
> > Martin Frank Hansen
> >
> >
> > Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder du
> > KMD’s Privatlivspolitik, der
> > fortæller, hvordan vi behandler oplysninger om dig.
> >
> > Protection of your personal data is important to us. Here you can read
> > KMD’s Privacy Policy outlining how we
> > process your personal data.
> >
> > Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information.
> > Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst
> > informere afsender om fejlen ved at bruge svarfunktionen. Samtidig beder vi
> > dig slette e-mailen i dit system uden at videresende eller kopiere den.
> > Selvom e-mailen og ethvert vedhæftet bilag efter vores overbevisning er fri
> > for virus og andre fejl, som kan påvirke computeren eller it-systemet,
> > hvori den modtages og læses, åbnes den på modtagerens eget ansvar. Vi
> > påtager os ikke noget ansvar for tab og skade, som er opstået i forbindelse
> > med at modtage og bruge e-mailen.
> >
> > Please note that this message may contain confidential information. If you
> > have received this message by mistake, please inform the sender of the
> > mistake by sending a reply, then delete the message from your system
> > without making, distributing or retaining any copies of it. Although we
> > believe that the message and any attachments are free from viruses and
> > other errors that might affect the computer or it-system where it is
> > received and read, the recipient opens the message at his or her own risk.
> > We assume no responsibility for any loss or damage arising from the receipt
> > or use of this message.
>
SV: data-import-handler for solr-7.5.0
Unfortunately, still no luck.
{
"responseHeader":{
"status":0,
"QTime":8},
"initArgs":[
"defaults",[
"config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
"command":"full-import",
"status":"idle",
"importResponse":"",
"statusMessages":{
"Total Requests made to DataSource":"0",
"Total Rows Fetched":"0",
"Total Documents Processed":"0",
"Total Documents Skipped":"0",
"Full Dump Started":"2018-10-02 16:15:21",
"":"Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.",
"Committed":"2018-10-02 16:15:22",
"Time taken":"0:0:0.136"}}
Seems like it is not even trying to read the data.
Martin Frank Hansen
-Oprindelig meddelelse-
Fra: Jan Høydahl
Sendt: 2. oktober 2018 17:46
Til: solr-user@lucene.apache.org
Emne: Re: data-import-handler for solr-7.5.0
> url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
Have you tried url="C:\\Users\\z6mhq/Desktop\\data_import\\nh_test.xml" ?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 2. okt. 2018 kl. 17:15 skrev Martin Frank Hansen (MHQ) :
>
> Hi,
>
> I am having some problems getting the data-import-handler in Solr to work. I
> have tried a lot of things but I simply get no response from Solr, not even
> an error.
>
> When calling the API:
> http://localhost:8983/solr/nh/dataimport?command=full-import
> {
> "responseHeader":{
>"status":0,
>"QTime":38},
> "initArgs":[
>"defaults",[
> "config","C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml"]],
> "command":"full-import",
> "status":"idle",
> "importResponse":"",
> "statusMessages":{}}
>
> The data looks like this:
>
>
>
> 2165432
> 5
>
>
> 28548113
> 89
>
>
> The data-config file looks like this:
>
>
>
>
> name="xml"
>pk="id"
>processor="XPathEntityProcessor"
>stream="true"
>forEach="/journal/doc"
>url="C:/Users/z6mhq/Desktop/data_import/nh_test.xml"
>transformer="RegexTransformer,TemplateTransformer"
>>
>
>
>
>
>
>
>
> And I referenced the jar files in the solr-config.xml as well as adding the
> request-handler by adding the following lines:
>
> regex="solr-dataimporthandler-\d.*\.jar" /> dir="${solr.install.dir:../../../..}/dist/"
> regex="solr-dataimporthandler-extras-\d.*\.jar" />
>
>
> class="org.apache.solr.handler.dataimport.DataImportHandler">
>
> name="config">C:/Users/z6mhq/Desktop/nh/nh/conf/data-config.xml
>
>
>
> I am running a core residing in the folder
> “C:/Users/z6mhq/Desktop/nh/nh/conf” while the Solr installation is in
> “C:/Users/z6mhq/Documents/solr-7.5.0”.
>
> I really hope that someone can spot my mistake…
>
> Thanks in advance.
>
> Martin Frank Hansen
>
>
> Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder du
> KMD’s Privatlivspolitik, der fortæller,
> hvordan vi behandler oplysninger om dig.
>
> Protection of your personal data is important to us. Here you can read KMD’s
> Privacy Policy outlining how we process
> your personal data.
>
> Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information.
> Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst informere
> afsender om fejlen ved at bruge svarfunktionen. Samtidig beder vi dig slette
> e-mailen i dit system uden at videresende eller kopiere den. Selvom e-mailen
> og ethvert vedhæftet bilag efter vores overbevisning er fri for virus og
> andre fejl, som kan påvirke computeren eller it-systemet, hvori den modtages
> og læses, åbnes den på modtagerens eget ansvar. Vi påtager os ikke noget
> ansvar for tab og skade, som er opstået i forbindelse med at modtage og bruge
> e-mailen.
>
> Please note that this message may contain confidential information. If you
> have received this message by mistake, please inform the sender of the
> mistake by sending a reply, then delete the message from your system without
> making, distributing or retaining any copies of it. Although we believe that
> the message and any attachments are free from viruses and other errors that
> might affect the computer or it-system where it is received and read, the
> recipient opens the message at his or her own risk. We assume no
> responsibility for any loss or damage arising from the receipt or use of this
> message.
