Thank you everyone for your patience and suggestions. It turns out I was doing something really unreasonable in my schema. I mistakenly edited the max EdgeNgram size to 512, when I meant to set the lengthFilter max to 512. I brought this to a more reasonable number, and my estimated time to import is now down to 4 hours. Based on the size of my record set, this time is more consistent with Walter's observations in his own project.
Thanks again for your help, Devon Baumgarten -----Original Message----- From: Devon Baumgarten [mailto:dbaumgar...@nationalcorp.com] Sent: Wednesday, February 22, 2012 12:42 PM To: 'solr-user@lucene.apache.org' Subject: RE: Unusually long data import time? Ahmet, I do not. I commented autoCommit out. Devon Baumgarten -----Original Message----- From: Ahmet Arslan [mailto:iori...@yahoo.com] Sent: Wednesday, February 22, 2012 12:25 PM To: solr-user@lucene.apache.org Subject: Re: Unusually long data import time? > Would it be unusual for an import of 160 million documents > to take 18 hours? Each document is less than 1kb and I > have the DataImportHandler using the jdbc driver to connect > to SQL Server 2008. The full-import query calls a stored > procedure that contains only a select from my target table. > > Is there any way I can speed this up? I saw recently someone > on this list suggested a new user could get all their Solr > data imported in under an hour. I sure hope that's true! Do have autoCommit or autoSoftCommit configured in solrconfig.xml?
