Hi,
we're looking for some comparison-benchmarks for importing large tables from a
mysql database (full import).
Currently, a full-import of ~ 8 Million rows from a MySQL database takes around
3 hours, on a QuadCore Machine with 16 GB of
ram and a Raid 10 storage setup. Solr is running on a apache tomcat instance,
where it is the only app. The tomcat instance
has the following memory-related java_opts:
-Xms4096M -Xmx5120M
The data-config.xml looks like this (only 1 entity):
<entity name="track" query="select t.id as id, t.title as title, l.title
as label from track t left join label l on (l.id = t.label_id) where t.deleted
= 0" transformer="TemplateTransformer">
<field column="title" name="title_t" />
<field column="label" name="label_t" />
<field column="id" name="sf_meta_id" />
<field column="metaclass" template="Track" name="sf_meta_class"/>
<field column="metaid" template="${track.id}" name="sf_meta_id"/>
<field column="uniqueid" template="Track_${track.id}"
name="sf_unique_id"/>
<entity name="artists" query="select a.name as artist from artist a
left join track_artist ta on (ta.artist_id = a.id) where
ta.track_id=${track.id}">
<field column="artist" name="artists_t" />
</entity>
</entity>
We have the feeling that 3 hours for this import is quite long - regarding the
performance of the server running solr/mysql.
Are we wrong with that assumption, or do people experience similar import times
with this amount of data to be imported?
thanks!
-robert
