[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Attachment: xpath-stream.patch xpath entity processor can stream rows one by one (for huge xml files) my making stream=true Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Shalin Shekhar Mangar Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, xpath-stream.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-469: --- Attachment: SOLR-469.patch A new patch containing the following changes: # On further thinking about Interface vs. Abstract classes, we have decided to replace all interfaces with abstract classes. Transformer, Context, EntityProcessor, Evaluator, DataSource and VariableResolver are now abstract classes. # The bug reported by Jonathan has been fixed and the TestCachedEntityProcessor has been updated to catch it. This exception used to be thrown only if the first request to CachedEntityProcessor needs a row which is not in cache. Subsequent requests were not affected. # Javadoc improvements. In particular, all the API related classes are marked as experimental and subject to change. # Propset Id in all classes. Users who have written their own custom transformers using the API will need to change their code. Sorry for the inconvenience. Grant - Is there anything else we need to do to get it committed? Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-469: --- Comment: was deleted Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-469: --- Attachment: (was: SOLR-469.patch) Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-469: --- Attachment: SOLR-469-contrib.patch Sorry for the spam due to my (multiple) mistakes. I think this one is the one :) A new patch containing the following changes: # On further thinking about Interface vs. Abstract classes, we have decided to replace all interfaces with abstract classes. Transformer, Context, EntityProcessor, Evaluator, DataSource and VariableResolver are now abstract classes. # The bug reported by Jonathan has been fixed and the TestCachedEntityProcessor has been updated to catch it. This exception used to be thrown only if the first request to CachedEntityProcessor needs a row which is not in cache. Subsequent requests were not affected. # Javadoc improvements. In particular, all the API related classes are marked as experimental and subject to change. # Propset Id in all classes. Users who have written their own custom transformers using the API will need to change their code. Sorry for the inconvenience. Grant - Is there anything else we need to do to get it committed? Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Attachment: SOLR-469-contrib.patch bug fix in _CachedSqlEntityProcessor_ Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Attachment: SOLR-469-contrib.patch ignore the previous patch Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Attachment: SOLR-469-contrib.patch The previous patch did not take care of multirow-transformers for CachedSqlEntityProcessor. Added a testcase and fixed that Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Attachment: SOLR-469-contrib.patch * Added a method _destroy()_ to _EntityProcessor_ . This coupled with init() can be used for pre/post actions * _JdbcDataSource_ uses _Statement#execute()_ instead of _Statement#executeQuery()_ . So users can execute DDL/DML using _JdbcDataSource_ Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Attachment: SOLR-469-contrib.patch it was a bad patch Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Attachment: (was: SOLR-469-contrib.patch) Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[
https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul updated SOLR-469:
Attachment: SOLR-469-contrib.patch
* All interfaces are marked as experimental
* The bug optimize=true fixed
* added a new variable to dataimporter namespace
_${dataimporter.index_start_time}_
Data Import RequestHandler
--
Key: SOLR-469
URL: https://issues.apache.org/jira/browse/SOLR-469
Project: Solr
Issue Type: New Feature
Components: update
Affects Versions: 1.3
Reporter: Noble Paul
Assignee: Grant Ingersoll
Fix For: 1.3
Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch,
SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch,
SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch,
SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch,
SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch
We need a RequestHandler Which can import data from a DB or other dataSources
into the Solr index .Think of it as an advanced form of SqlUpload Plugin
(SOLR-103).
The way it works is as follows.
* Provide a configuration file (xml) to the Handler which takes in the
necessary SQL queries and mappings to a solr schema
- It also takes in a properties file for the data source
configuraution
* Given the configuration it can also generate the solr schema.xml
* It is registered as a RequestHandler which can take two commands
do-full-import, do-delta-import
- do-full-import - dumps all the data from the Database into the
index (based on the SQL query in configuration)
- do-delta-import - dumps all the data that has changed since last
import. (We assume a modified-timestamp column in tables)
* It provides a admin page
- where we can schedule it to be run automatically at regular
intervals
- It shows the status of the Handler (idle, full-import,
delta-import)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-469: --- Attachment: SOLR-469.patch The last patch wasn't generated correctly. This one fixes it. No changes in the code since the last patch. Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-469: --- Attachment: (was: SOLR-469.patch) Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-469: --- Attachment: SOLR-469-contrib.patch This time with the correct name SOLR-469-contrib.patch Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Attachment: SOLR-469-contrib.patch Changes * classloading is done using SolrresourceLoader . adding jars to solrhome/lib must work * The request parameter can add optimize=false to disable optimize Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-469: --- Attachment: SOLR-469-contrib.patch *Changes* * Updated the build.xml to compile Solr before building DataImportHandler and place DataImportHandler's javadoc jar to solr/dist folder so that the javadocs are available in Solr nightly builds * Removed @author Javadoc tags from all source files in accordance with Solr coding conventions * Improved Javadocs for a lot of classes especially the public interfaces * Formatted code using the Eclipse codestyle xml given at HowToContribute wiki page * Added @since solr 1.3 to all source files * I've verified that the Apache license text is present in all the source files No changes have been made to the code (in terms of functionality) Note -- The SOLR-563 patch must be applied before this patch to build Solr with DataImportHandler as a contrib project. A lot of people are using this patch and it would be easier for them if DataImportHandler is available in the nightly builds. Also, this patch has become huge and enhancements and bug fixes would also be easier if it were committed. Grant -- We feel that this is ready to be committed now whenever you can take a look. Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Attachment: SOLR-469-contrib.patch This patch contains * integration with SOLR-505 ( disable cache headers) * Tests inTestSCriptTransformer Ignored. (it requires java 6) * New feature _CachedSqlEntityProcessor_. It can dramatically speed up indexing if there are sub-entities. It can cache the rows and avoid subsequent database calls. Consumes a lot of RAM. More documentation will be added to wiki Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-469: --- Attachment: SOLR-469.patch A new patch file (SOLR-469.patch) consisting of some important bug fixes and minor enhancements. The changes and the corresponding classes are given below *Changes* * Set fetch size to Integer.MIN_VALUE if batchSize in configuration is -1 as per Patrick's suggestion -- JdbcDataSource * Transformers can add a boost to a document by adding a key/value pair row.put($docBoost, 2.0f) from any entity -- DocBuilder,SolrWriter and DataImportHandler * Fixes for infinite loop in SqlEntityProcessor when delta query fails for some reason and NullPointerException is thrown in EntityProcessorBase -- EntityProcessorBase * Fix for NullPointerException in TemplateTransformer and corresponding test -- TemplateTransformer, TestTemplateTransformer * Enhancement for specifying table.column syntax for pk attribute in entity as per issue reported by Chris Moser and Olivier Poitrey -- SqlEntityProcessor,TestSqlEntityProcessor2 * Fix for NullPointerException in XPathRecordReader when attribute specified through xpath is null -- XPathRecordReader, TestXPathRecordReader * Enhancement to DataSource interface to provide a close method -- DataSource, FileDataSource, HttpDataSource, MockDataSource * Context interface has a new method getDataSource(String entityName) for getting a new DataSource instance for the given entity -- Context, ContextImpl, DataImporter, DocBuilder * FileListEntityProcessor implements olderThan and newerThan filtering parameters -- FileListEntityProcessor, TestFileListEntityProcessor * Debug Mode can be disabled from solrconfig.xml by enableDebug=false -- DataImporter, DataImportHandler * Running statistics are exposed on the Solr Statistics page in addition to cumulative statictics -- DataImportHandler, DocBuilder Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-469: --- Attachment: SOLR-469-contrib.patch This patch adds DataImportHandler as a contrib project into Solr. It uses standard Maven directory structure and a build.xml file. No changes have been made to the codebase. Note - I've opened SOLR-563 to track contrib area creation in Solr. Using this patch with the SOLR-563 patch lets you compile, test and package DataImportHandler with Solr war file. Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[
https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar updated SOLR-469:
---
Attachment: SOLR-469.patch
A new patch consisting of a few bug fixes and some major new features. The
changes include:
* No need to write fields in data-config if the field name from DB/XML and
field-name in schema.xml are the same. This removes a lot of useless verbosity
from data-config.xml
* A cool new interactive development page, in which you write/change
data-config.xml and see results immeadiately making interations extremely fast!
Use http://host:port/solr/admin/dataimport.jsp or if using multi-core
http://host:port/solr/core-name/admin/dataimport.jsp
* You can start using the interactive mode without specifying data-config file
in solrconfig.xml, however, specifying the data sources is necessary in
solrconfig.xml
* Interactive development uses a new debug mode in DataImportHandler, add
debug=on to the full-import command to see the actual documents which are
created by DataImportHandler. This shows the first 10 documents created by
DataImportHandler using the existing config without committing them to solr. It
supports the start and rows parameter (just like query params) which you can
use to see any document. This comes in very useful when suppose the 1000th
document failed during indexing and you want to see the reason. If there are
exceptions, the stacktrace is shown with the response.
* Verbose mode with verbose=on as a request parameter (used in conjunction
with debug=on) which shows exactly how DataImportHandler created each document.
** What query was executed?
** How much time it took?
** What rows it gave back?
** What transformers were applied and what was the result?
** Another advantage is that you can see the fields which are indexed but not
stored
* A show-config command has been added which gives the data-config.xml as a
raw response (uses RawResponseWriter)
* A new interface called Evaluator has been added which makes it possible to
plugin new expression evaluators (for resolving variable names)
* Using the same Evaluator interface, a few new evaluators have been added
** formatDate - use as ${dataimporter.functions.formatDate('NOW',-MM-dd
HH:mm)}, this will format NOW as per the given format and return a string which
can be used in queries or urls. It supports the full DateMathParser syntax. You
can also format fields e.g.
${dataimporter.functions.formatDate(A.purchase_date,dd-MM-)}
** encodeUrl - useful for URL-encoding parameters when making a HTTP call. Use
as ${dataimport.functions.encodeUrl(emp.name)}
** escapeSql - useful for escaping parameters supplied in sql statements. This
can replace quotes with two quotes to avoid sql syntax errors. Use as
${dataimporter.functions.escapeSql(emp.name)}
* Custom Evaluators can be specified in data-config.xml (more details and
example will be added to the wiki)
* HttpDataSource now reads the content encoding from the response by default.
Previously it assumed the default encoding to be UTF-8. This behavior can be
overriden by explicitly specifying an encoding in solrconfig.xml
* A FileDataSource has been added which can read content from local files
(e.g. XML feed files on local disk).
* Transformers can signal skipping a document by adding a key $skipDoc with
value true in the returned map.
* NumberFormatTransformer is a new transformer which can be used to
extract/convert numbers from strings. It uses the java.text.NumberFormat class
in Java to provide its features.
* The Context interface has been enhanced to add new methods for
getting/setting session variables which can be used by Transformers to share
data. Also a new method called getParentContext can enable a
Transformer/EntityProcessor to get the parent entity's context in full imports.
Please let us know your comments and feedback. More details and examples will
soon be added to the wiki page at http://wiki.apache.org/solr/DataImportHandler
Data Import RequestHandler
--
Key: SOLR-469
URL: https://issues.apache.org/jira/browse/SOLR-469
Project: Solr
Issue Type: New Feature
Components: update
Affects Versions: 1.3
Reporter: Noble Paul
Assignee: Grant Ingersoll
Fix For: 1.3
Attachments: SOLR-469.patch, SOLR-469.patch, SOLR-469.patch,
SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch
We need a RequestHandler Which can import data from a DB or other dataSources
into the Solr index .Think of it as an advanced form of SqlUpload Plugin
(SOLR-103).
The way it works is as follows.
* Provide a configuration file (xml) to the Handler which takes in the
necessary SQL queries and mappings to a solr schema
- It also takes in a
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Summary: Data Import RequestHandler (was: DB Import RequestHandler) The scope has been changed from consuming just DB data. It is designed to consume any type of structured data Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Priority: Minor Fix For: 1.3 Attachments: SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-469: Priority: Major (was: Minor) The priority is changed to major Data Import RequestHandler -- Key: SOLR-469 URL: https://issues.apache.org/jira/browse/SOLR-469 Project: Solr Issue Type: New Feature Components: update Affects Versions: 1.3 Reporter: Noble Paul Assignee: Grant Ingersoll Fix For: 1.3 Attachments: SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103). The way it works is as follows. * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema - It also takes in a properties file for the data source configuraution * Given the configuration it can also generate the solr schema.xml * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration) - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables) * It provides a admin page - where we can schedule it to be run automatically at regular intervals - It shows the status of the Handler (idle, full-import, delta-import) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
