dataimport

[Brian Lamb]

Hi all,

First of all, I'm quite new to solr.

I have the server set up and everything appears to work. I set it up so that
the indexed data comes through a mysql connection:

<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
  <lst name="defaults">
    <str name="config">db-data-config.xml</str>
   </lst>
</requestHandler>

And here is the contents of db-data-config.xml:

<dataConfig>
   <dataSource type="JdbcDataSource"
      name="mystuff"
      batchSize="-1"
      driver="com.mysql.jdbc.Driver"

url="jdbc:mysql://localhost/database?characterEncoding=UTF8&amp;zeroDateTimeBehavior=convertToNull"
      user="user"
      password="password"/>
    <document>
      <entity name="id"
        dataSource="mystuff"
        query="SELECT p.id, p.fielda, p.fieldb, p.fieldc, p.fieldd FROM
mytable p">
     </entity>
   </document>
</dataConfig>

When I point my browser at localhost:8983/solr/dataimport, the server
produces the following message:

Feb 24, 2011 8:58:24 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/dataimport params={command=full-import} status=0
QTime=10
Feb 24, 2011 8:58:24 PM org.apache.solr.handler.dataimport.DataImporter
doFullImport
INFO: Starting Full Import
Feb 24, 2011 8:58:24 PM org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
INFO: Read dataimport.properties
Feb 24, 2011 8:58:24 PM org.apache.solr.update.DirectUpdateHandler2
deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
Feb 24, 2011 8:58:24 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=/wwwroot/apps/apache-solr-1.4.1/example/solr/data/index,segFN=segments_p,version=1297781919778,generation=25,filenames=[_n.nrm,
_n.tis, _n.prx, segments_p, _n.fdt, _n.frq, _n.tii, _n.fdx, _n.fnm]
Feb 24, 2011 8:58:24 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1297781919778
Feb 24, 2011 8:58:24 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity id with URL:
jdbc:mysql://localhost/researchsquare_beta_library?characterEncoding=UTF8&zeroDateTimeBehavior=convertToNull
Feb 24, 2011 8:58:25 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Time taken for getConnection(): 137
Killed

So it looks like for whatever reason, the server crashes trying to do a full
import. When I add a LIMIT clause on the query, it works fine when the LIMIT
is only 250 records but if I try to do 500 records, I get the same message.

The fields types are:

SHOW CREATE TABLE mytable;
CREATE TABLE mytable (
   `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
   `fielda` varchar(650) COLLATE utf8_unicode_ci DEFAULT NULL,
   `fieldb` varchar(500) COLLATE utf8_unicode_ci DEFAULT NULL,
   `fieldc` text COLLATE utf8_unicode_ci,
   `fieldd` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
   PRIMARY KEY (`id`)
);

How can I get Solr to do a full import without crashing? Doing it 250
records at a time is not going to be feasible because there are about 500000
records.