Re: DIH problem with multiple (types of) resources

[Peter Blokland]

hi,

On Tue, Nov 15, 2016 at 02:54:49AM +1100, Alexandre Rafalovitch wrote:

>> 
>> 
 
> Attribute names are case sensitive as far as I remember. Try
> 'dataSource' for the second definition.

oh wow... that's sneaky. in the old version the case didn't seem to matter,
but now it certainly does. thx :)

-- 
CUL8R, Peter.

www.desk.nl

Your excuse is: It is a layer 8 problem

Re: DIH problem with multiple (types of) resources

[Alexandre Rafalovitch]

On 15 November 2016 at 02:19, Peter Blokland  wrote:
> 

> 

Attribute names are case sensitive as far as I remember. Try
'dataSource' for the second definition.

Regards,
   Alex.


Solr Example reading group is starting November 2016, join us at
http://j.mp/SolrERG
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/

Re: DIH Problem: create multiple docs from a single entity

[Raymond Wiker]

On Aug 7, 2013, at 18:10 , Lee Carroll lee.a.carr...@googlemail.com wrote:
 Hi
 
 I've 2 tables with the following data
 
 table 1
 id treatment_list
 1 a,b
 2 b,c
 
 table 2
 treatment id, name
 a  name1
 b  name 2
 c  name 3
 
 Using DIH can you create an index of the form
 
 id-treatment-id name
 1a  name1
 1b  name2
 2b  name2
 2c  name3
 
 In short can I splt the comma separated field and process each as an
 entity. From the docs and the wiki I can't see anything obvious.
 
 I feel I'm missing something easier here. (Note its not my data so can't do
 anything with the dodgy csv field )

I think this is an SQL problem, rather than a DIH one. A quick google shows 
several hits for splitting a string in SQL; I expect that it should be possible 
to come up with something that fits your purpose.

Re: DIH Problem: create multiple docs from a single entity

[SolrLover]

I suppose you can use Substring and Charindex to perform your task at SQL
level then use the value in another entity in DIH..



--
View this message in context: 
http://lucene.472066.n3.nabble.com/DIH-Problem-create-multiple-docs-from-a-single-entity-tp4083050p4083106.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: DIH Problem: create multiple docs from a single entity

[Mikhail Khludnev]

Hello Lee,

Unfortunately no. It's possible to read csv field by
http://wiki.apache.org/solr/DataImportHandler#FieldReaderDataSource but
there is no csv like EntityProcessor, which can broke line on entities.
Transformers can not emit new entities.


On Wed, Aug 7, 2013 at 8:10 PM, Lee Carroll lee.a.carr...@googlemail.comwrote:

 Hi

 I've 2 tables with the following data

 table 1
 id treatment_list
 1 a,b
 2 b,c

 table 2
 treatment id, name
 a  name1
 b  name 2
 c  name 3

 Using DIH can you create an index of the form

 id-treatment-id name
 1a  name1
 1b  name2
 2b  name2
 2c  name3

 In short can I splt the comma separated field and process each as an
 entity. From the docs and the wiki I can't see anything obvious.

 I feel I'm missing something easier here. (Note its not my data so can't do
 anything with the dodgy csv field )




-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

http://www.griddynamics.com
 mkhlud...@griddynamics.com

Re: DIH problem

[Gian Marco Tagliani]

Here the issue:

https://issues.apache.org/jira/browse/SOLR-3896




On Tue, Sep 25, 2012 at 1:41 PM, Gian Marco Tagliani
gm.tagli...@gmail.comwrote:

 Ok,
 I'll try to verify if there is the same issue in the 4.0 and I'll open the
 issue in Jira.

 thanks

 --
 Gian Marco




 On Sat, Sep 22, 2012 at 9:34 PM, Dyer, James james.d...@ingramcontent.com
  wrote:

 Gian,

 Even if you can't write a failing unit test (if you did it would be
 awesome), please open a JIRA issue on this and attach your patch.  Also,
 you may want to try 4.0 as opposed to 3.6 as some of the 3.6 issues with
 DIH are resolved in 4.0.

 https://issues.apache.org/jira/secure/Dashboard.jspa

 James Dyer
 E-Commerce Systems
 Ingram Content Group
 (615) 213-4311


 -Original Message-
 From: Mikhail Khludnev [mailto:mkhlud...@griddynamics.com]
 Sent: Friday, September 21, 2012 12:03 PM
 To: solr-user@lucene.apache.org
 Subject: Re: DIH problem

 Gian,

 The only way to handle it is to provide a test case and attach to jira.

 Thanks

 On Fri, Sep 21, 2012 at 6:03 PM, Gian Marco Tagliani
 gm.tagli...@gmail.comwrote:

  Hi,
  I'm updating my Solr from version 3.4 to version 3.6.1 and I'm facing a
  little problem with the DIH.
 
  In the delta-import I'm using the /parentDeltaQuery/ feature of the DIH
 to
  update the parent entity.
  I don't think this is working properly.
 
  I realized that it's just executing the /parentDeltaQuery/ with the
 first
  record of the /deltaQuery /result.
  Comparing the code with the previous versions I noticed that the
  rowIterator was never set to null.
 
  To solve this I wrote a simple patch:
 
  -
  Index: solr/contrib/**dataimporthandler/src/java/**
  org/apache/solr/handler/**dataimport/**EntityProcessorBase.java
  ==**==**===
  ---
 solr/contrib/**dataimporthandler/src/java/**org/apache/solr/handler/**
  dataimport/**EntityProcessorBase.java (revision 31454)
  +++
 solr/contrib/**dataimporthandler/src/java/**org/apache/solr/handler/**
  dataimport/**EntityProcessorBase.java (working copy)
  @@ -121,6 +121,7 @@
   if (rowIterator.hasNext())
 return rowIterator.next();
   query = null;
  +rowIterator = null;
   return null;
 } catch (Exception e) {
   SolrException.log(log, getNext() failed for query ' + query +
  ', e);
  -
 
 
  Do you think this is correct?
 
  Thanks for your help
 
  --
  Gian Marco Tagliani
 
 
 
 


 --
 Sincerely yours
 Mikhail Khludnev
 Tech Lead
 Grid Dynamics

 http://www.griddynamics.com
  mkhlud...@griddynamics.com

Re: DIH problem

[Gian Marco Tagliani]

Ok,
I'll try to verify if there is the same issue in the 4.0 and I'll open the
issue in Jira.

thanks

--
Gian Marco



On Sat, Sep 22, 2012 at 9:34 PM, Dyer, James
james.d...@ingramcontent.comwrote:

 Gian,

 Even if you can't write a failing unit test (if you did it would be
 awesome), please open a JIRA issue on this and attach your patch.  Also,
 you may want to try 4.0 as opposed to 3.6 as some of the 3.6 issues with
 DIH are resolved in 4.0.

 https://issues.apache.org/jira/secure/Dashboard.jspa

 James Dyer
 E-Commerce Systems
 Ingram Content Group
 (615) 213-4311


 -Original Message-
 From: Mikhail Khludnev [mailto:mkhlud...@griddynamics.com]
 Sent: Friday, September 21, 2012 12:03 PM
 To: solr-user@lucene.apache.org
 Subject: Re: DIH problem

 Gian,

 The only way to handle it is to provide a test case and attach to jira.

 Thanks

 On Fri, Sep 21, 2012 at 6:03 PM, Gian Marco Tagliani
 gm.tagli...@gmail.comwrote:

  Hi,
  I'm updating my Solr from version 3.4 to version 3.6.1 and I'm facing a
  little problem with the DIH.
 
  In the delta-import I'm using the /parentDeltaQuery/ feature of the DIH
 to
  update the parent entity.
  I don't think this is working properly.
 
  I realized that it's just executing the /parentDeltaQuery/ with the first
  record of the /deltaQuery /result.
  Comparing the code with the previous versions I noticed that the
  rowIterator was never set to null.
 
  To solve this I wrote a simple patch:
 
  -
  Index: solr/contrib/**dataimporthandler/src/java/**
  org/apache/solr/handler/**dataimport/**EntityProcessorBase.java
  ==**==**===
  ---
 solr/contrib/**dataimporthandler/src/java/**org/apache/solr/handler/**
  dataimport/**EntityProcessorBase.java (revision 31454)
  +++
 solr/contrib/**dataimporthandler/src/java/**org/apache/solr/handler/**
  dataimport/**EntityProcessorBase.java (working copy)
  @@ -121,6 +121,7 @@
   if (rowIterator.hasNext())
 return rowIterator.next();
   query = null;
  +rowIterator = null;
   return null;
 } catch (Exception e) {
   SolrException.log(log, getNext() failed for query ' + query +
  ', e);
  -
 
 
  Do you think this is correct?
 
  Thanks for your help
 
  --
  Gian Marco Tagliani
 
 
 
 


 --
 Sincerely yours
 Mikhail Khludnev
 Tech Lead
 Grid Dynamics

 http://www.griddynamics.com
  mkhlud...@griddynamics.com

RE: DIH problem

[Dyer, James]

Gian,

Even if you can't write a failing unit test (if you did it would be awesome), 
please open a JIRA issue on this and attach your patch.  Also, you may want to 
try 4.0 as opposed to 3.6 as some of the 3.6 issues with DIH are resolved in 
4.0.

https://issues.apache.org/jira/secure/Dashboard.jspa

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311


-Original Message-
From: Mikhail Khludnev [mailto:mkhlud...@griddynamics.com] 
Sent: Friday, September 21, 2012 12:03 PM
To: solr-user@lucene.apache.org
Subject: Re: DIH problem

Gian,

The only way to handle it is to provide a test case and attach to jira.

Thanks

On Fri, Sep 21, 2012 at 6:03 PM, Gian Marco Tagliani
gm.tagli...@gmail.comwrote:

 Hi,
 I'm updating my Solr from version 3.4 to version 3.6.1 and I'm facing a
 little problem with the DIH.

 In the delta-import I'm using the /parentDeltaQuery/ feature of the DIH to
 update the parent entity.
 I don't think this is working properly.

 I realized that it's just executing the /parentDeltaQuery/ with the first
 record of the /deltaQuery /result.
 Comparing the code with the previous versions I noticed that the
 rowIterator was never set to null.

 To solve this I wrote a simple patch:

 -
 Index: solr/contrib/**dataimporthandler/src/java/**
 org/apache/solr/handler/**dataimport/**EntityProcessorBase.java
 ==**==**===
 --- solr/contrib/**dataimporthandler/src/java/**org/apache/solr/handler/**
 dataimport/**EntityProcessorBase.java (revision 31454)
 +++ solr/contrib/**dataimporthandler/src/java/**org/apache/solr/handler/**
 dataimport/**EntityProcessorBase.java (working copy)
 @@ -121,6 +121,7 @@
  if (rowIterator.hasNext())
return rowIterator.next();
  query = null;
 +rowIterator = null;
  return null;
} catch (Exception e) {
  SolrException.log(log, getNext() failed for query ' + query +
 ', e);
 -


 Do you think this is correct?

 Thanks for your help

 --
 Gian Marco Tagliani






-- 
Sincerely yours
Mikhail Khludnev
Tech Lead
Grid Dynamics

http://www.griddynamics.com
 mkhlud...@griddynamics.com

Re: DIH problem

[Mikhail Khludnev]

Gian,

The only way to handle it is to provide a test case and attach to jira.

Thanks

On Fri, Sep 21, 2012 at 6:03 PM, Gian Marco Tagliani
gm.tagli...@gmail.comwrote:

 Hi,
 I'm updating my Solr from version 3.4 to version 3.6.1 and I'm facing a
 little problem with the DIH.

 In the delta-import I'm using the /parentDeltaQuery/ feature of the DIH to
 update the parent entity.
 I don't think this is working properly.

 I realized that it's just executing the /parentDeltaQuery/ with the first
 record of the /deltaQuery /result.
 Comparing the code with the previous versions I noticed that the
 rowIterator was never set to null.

 To solve this I wrote a simple patch:

 -
 Index: solr/contrib/**dataimporthandler/src/java/**
 org/apache/solr/handler/**dataimport/**EntityProcessorBase.java
 ==**==**===
 --- solr/contrib/**dataimporthandler/src/java/**org/apache/solr/handler/**
 dataimport/**EntityProcessorBase.java (revision 31454)
 +++ solr/contrib/**dataimporthandler/src/java/**org/apache/solr/handler/**
 dataimport/**EntityProcessorBase.java (working copy)
 @@ -121,6 +121,7 @@
  if (rowIterator.hasNext())
return rowIterator.next();
  query = null;
 +rowIterator = null;
  return null;
} catch (Exception e) {
  SolrException.log(log, getNext() failed for query ' + query +
 ', e);
 -


 Do you think this is correct?

 Thanks for your help

 --
 Gian Marco Tagliani






-- 
Sincerely yours
Mikhail Khludnev
Tech Lead
Grid Dynamics

http://www.griddynamics.com
 mkhlud...@griddynamics.com

Re: DIH problem passing HTTP parameters into data-config

[Noble Paul നോബിള്‍ नोब्ळ्]

On Thu, Aug 13, 2009 at 4:08 AM, Erik Hatchererik.hatc...@gmail.com wrote:
 My hunch, though I'll try to make some time to test this out thoroughly, is
 that the entity is parsed initially with variables resolved, but not per
 request.  Variables/expressions do get expanded for fields of course, but
 perhaps not for other high-level attributes?

All the variables are resolved just in time

        Erik

 On Aug 12, 2009, at 5:38 PM, John Lowe wrote:

 Hmmm...perhaps my original note was a bit TLTR.  Trying again:

 The v1.3 docs say that one can pass one's own parameters in to DIH via the
 HTTP request:


 http://wiki.apache.org/solr/DataImportHandler#head-520f8e527d9da55e8ed1e274e29709c8805c8eae

 SO if I have a URL like the following to dataimport:


 http://localhost:8983/solr/rss/dataimport?command=full-importfoo=http%3A%2F%2Fbar.com

 AND, in my data-config file I have something like:

 dataConfig
      dataSource type=HttpDataSource name=rss-feed /
      document
              entity name=article dataSource=rss-feed
                              pk=link
                              url=${dataimporter.request.foo}
                              processor=XPathEntityProcessor

 [blah blah blah...looks pretty much like the slashdot RSS
 rss-data-config.xml]

 the value of url should be filled in with 'http://bar.com', right?

 Well...it's not!

 I get an exception:

 SEVERE: Exception thrown while getting data
 java.net.MalformedURLException: no protocol: null

 Which seems to indicate that  the value is not getting passed. or I'm not
 referring to it correctly. Or something.

 Any hints would be greatly appreciated...apologies for re-spamming the
 list with my desperate request,

 John





-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: DIH problem passing HTTP parameters into data-config

[Shalin Shekhar Mangar]

On Thu, Aug 13, 2009 at 3:08 AM, John Lowe jbl...@johnblowe.com wrote:

 Hmmm...perhaps my original note was a bit TLTR.  Trying again:

 The v1.3 docs say that one can pass one's own parameters in to DIH via the
 HTTP request:


DIH in Solr 1.3 had a bug due to which request parameters in variables were
not resolved. The fix is in 1.4 trunk

https://issues.apache.org/jira/browse/SOLR-823

-- 
Regards,
Shalin Shekhar Mangar.

Re: DIH problem passing HTTP parameters into data-config

[Erik Hatcher]

My hunch, though I'll try to make some time to test this out  
thoroughly, is that the entity is parsed initially with variables  
resolved, but not per request.  Variables/expressions do get expanded  
for fields of course, but perhaps not for other high-level attributes?


Erik

On Aug 12, 2009, at 5:38 PM, John Lowe wrote:


Hmmm...perhaps my original note was a bit TLTR.  Trying again:

The v1.3 docs say that one can pass one's own parameters in to DIH  
via the HTTP request:


http://wiki.apache.org/solr/DataImportHandler#head-520f8e527d9da55e8ed1e274e29709c8805c8eae

SO if I have a URL like the following to dataimport:

http://localhost:8983/solr/rss/dataimport?command=full-importfoo=http%3A%2F%2Fbar.com

AND, in my data-config file I have something like:

dataConfig
  dataSource type=HttpDataSource name=rss-feed /
  document
  entity name=article dataSource=rss-feed
  pk=link
  url=${dataimporter.request.foo}
  processor=XPathEntityProcessor

[blah blah blah...looks pretty much like the slashdot RSS rss-data- 
config.xml]


the value of url should be filled in with 'http://bar.com', right?

Well...it's not!

I get an exception:

SEVERE: Exception thrown while getting data
java.net.MalformedURLException: no protocol: null

Which seems to indicate that  the value is not getting passed. or  
I'm not referring to it correctly. Or something.


Any hints would be greatly appreciated...apologies for re-spamming  
the list with my desperate request,


John

Re: DIH problem passing HTTP parameters into data-config

[John Lowe]

Oops, the url attribute of the entity element in the dataConfig  
snippet should read:


  url=${dataimporter.request.feed}

to match the http parameter...

John