[
https://issues.apache.org/jira/browse/SOLR-4799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14020260#comment-14020260
]
Shawn Heisey commented on SOLR-4799:
------------------------------------
bq. I think a lot of DIH users are using it to import data from an RDBMS into
Solr.
This is exactly what I use it for. Based on mailing list and IRC traffic, I
think that most people who use DIH are using it for database import. DIH
works, and it's a lot more efficient than any single-threaded program that I
could write. I don't believe that it is a "playground tool."
Although DIH used to handle *all* our indexing, we currently only use it for
full index rebuilds. A SolrJ app handles the once-a-minute maintenance. I
have plans to build an internal multi-threaded SolrJ tool to handle full
rebuilds, but that effort still has not made it through the design phase.
Because DIH works so well, we don't have a strong need to replace it.
> SQLEntityProcessor for zipper join
> ----------------------------------
>
> Key: SOLR-4799
> URL: https://issues.apache.org/jira/browse/SOLR-4799
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Reporter: Mikhail Khludnev
> Priority: Minor
> Labels: dih
> Attachments: SOLR-4799.patch
>
>
> DIH is mostly considered as a playground tool, and real usages end up with
> SolrJ. I want to contribute few improvements target DIH performance.
> This one provides performant approach for joining SQL Entities with miserable
> memory at contrast to
> http://wiki.apache.org/solr/DataImportHandler#CachedSqlEntityProcessor
> The idea is:
> * parent table is explicitly ordered by it’s PK in SQL
> * children table is explicitly ordered by parent_id FK in SQL
> * children entity processor joins ordered resultsets by ‘zipper’ algorithm.
> Do you think it’s worth to contribute it into DIH?
> cc: [~goksron] [~jdyer]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
