Thanks all. Erick, is there documentation on doing things with SolrJ and a JDBC connection?
On Mon, Dec 12, 2011 at 1:34 PM, Erick Erickson <erickerick...@gmail.com>wrote: > You might want to consider just doing the whole > thing in SolrJ with a JDBC connection. When things > get complex, it's sometimes more straightforward. > > Best > Erick... > > P.S. Yes, it's pretty standard to have a single > field be the destination for several copyField > directives. > > On Mon, Dec 12, 2011 at 12:48 PM, Gora Mohanty <g...@mimirtech.com> wrote: > > On Mon, Dec 12, 2011 at 2:24 AM, Brian Lamb > > <brian.l...@journalexperts.com> wrote: > >> Hi all, > >> > >> I have a few questions about how the MySQL data import works. It seems > it > >> creates a separate connection for each entity I create. Is there any > way to > >> avoid this? > > > > Not sure, but I do not think that it is possible. However, from your > description > > below, I think that you are unnecessarily multiplying entities. > > > >> By nature of my schema, I have several multivalued fields. Each one I > >> populate with a separate entity. Is there a better way to do it? For > >> example, could I pull in all the singular data in one sitting and then > come > >> back in later and populate with the multivalued items. > > > > Not quite sure as to what you mean. Would it be possible for you > > to post your schema.xml, and the DIH configuration file? Preferably, > > put these on pastebin.com, and send us links. Also, you should > > obfuscate details like access passwords. > > > >> An alternate approach in some cases would be to do a GROUP_CONCAT and > then > >> populate the multivalued column with some transformation. Is that > possible? > > [...] > > > > This is how we have been handling it. A complete description would > > be long, but here is the gist of it: > > * A transformer will be needed. In this case, we found it easiest > > to use a Java-based transformer. Thus, your entity should include > > something like > > <entity name="myname" dataSource="mysource" > > transformer="com.mycompany.search.solr.handler.JobsNumericTransformer...> > > ... > > </entity> > > Here, the class name to be used for the transformer attribute follows > > the usual Java rules, and the .jar needs to be made available to Solr. > > * The SELECT statement for the entity looks something like > > select group_concat( myfield SEPARATOR '@||@')... > > The separator should be something that does not occur in your > > normal data stream. > > * Within the entity, define > > <field column="myfield"/> > > * There are complications involved if NULL values are allowed > > for the field, in which case you would need to use COALESCE, > > maybe along with CAST > > * The transformer would look up "myfield", split along the separator, > > and populate the multi-valued field. > > > > This *is* a little complicated, so I would also like to hear about > > possible alternatives. > > > > Regards, > > Gora >
