I can't do that, we don't have another server to manage that.
Noble Paul നോബിള് नोब्ळ् wrote: > > From the data-config.xml it is obvious that the your indexing will > take a lot of time. MySql has very poor join performance. It is not a > very good idea to run this on a production database. > > I would suggest you to configure another mysql server and do mysql > replication to that and run the import from there. > > > > On Mon, Nov 3, 2008 at 10:06 PM, sunnyfr <[EMAIL PROTECTED]> wrote: >> >> Hi, >> >> I've put a batchsize parameter at -1, it works fine, the point is I will >> monopolize the MySql's database for 10hours. >> And other request on it like update, or other process will be stack. And >> if >> I don't use batchsize -1 I will have an OOM error like below. I tried to >> put >> batchsize 1000 or 1 but it doesn't work either. >> >> What would you reckon ? >> I can't have another server just for that, I have to use MySql >> production's >> database. >> >>> Should I try to generate CSV files from MySql and then make a full >>> import >>> from it, and delta import on the database ?? >> >>> Should I modify the connection code in solr, to try to manage this >>> memory. >> >>> Do you have another idea ? >> >> If anyone has any suggestions/needs more info, i'd be very greatful. >> >> >> >> FYI : Linux server / 8G Mem >> >> TomCat55 : >> JAVA_OPTS="-Xms2000m -Xmx4000m -XX:+HeapDumpOnOutOfMemoryError >> -XX:+UseParallelGC -XX:+AggressiveOpts -XX:NewRatio=5 -Xloggc:gc.log >> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" >> >> data-config.xml: >> http://www.nabble.com/file/p20305986/data-config.xml data-config.xml >> >> >> My Error: >> Nov 3 16:45:43 solr-test jsvc.exec[29099]: [PSYoungGen: >> 227583K->0K(455104K)] [PSOldGen: 3413375K->142574K(1706688K)] >> 3640959K->142574K(2161792K) >> Nov 3 16:45:43 solr-test jsvc.exec[29099]: [PSPermGen: >> 20751K->20751K(21504K)], 1.0853010 secs] [Times: user=0.99 sys=0.10, >> real=1.09 secs] >> Nov 3 16:45:43 solr-test jsvc.exec[29099]: Nov 3, 2008 4:45:43 PM >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor >> processChildren SEVERE: Exception invoking periodic operation: >> java.lang.OutOfMemoryError: Java heap space ^Iat >> sun.nio.cs.US_ASCII.newDecoder(US_ASCII.java:39) ^Iat >> java.nio.charset.CharsetEncoder.isLegalReplacement(CharsetEncoder.java:311) >> ^Iat java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:267) >> ^Iat java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:186) ^Iat >> java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:209) ^Iat >> sun.nio.cs.US_ASCII$Encoder.<init>(US_ASCII.java:121) ^Iat >> sun.nio.cs.US_ASCII$Encoder.<init>(US_ASCII.java:118) ^Iat >> sun.nio.cs.US_ASCII.newEncoder(US_ASCII.java:43) ^Iat >> java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:215) ^Iat >> java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:207) ^Iat >> java.lang.StringCoding.encode(StringCoding.java:266) ^Iat >> java.lang.String.getBytes(String.java:947) ^Iat java.io.UnixFi >> Nov 3 16:45:43 solr-test jsvc.exec[29099]: >> java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228) ^Iat >> java.io.File.isDirectory(File.java:754) ^Iat >> org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:873) >> ^Iat >> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492) >> ^Iat org.apache.catalina.startup.HostConfig.check(HostConfig.java:1206) >> ^Iat >> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:293) >> ^Iat >> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120) >> ^Iat >> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1306) >> ^Iat >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1570) >> ^Iat >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1579) >> ^Iat >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1559) >> ^Iat java.lang.Thread.run(Thread.java:619) >> >> -- >> View this message in context: >> http://www.nabble.com/MySql---Solr-1.3---Tomcat55-----Full-Import-for-8%2C5M-of-data-%3E%3E-Exception-in-thread-%22Thread-33%22-tp20305986p20305986.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/MySql---Solr-1.3---Tomcat55-----Full-Import-for-8%2C5M-of-data-%3E%3E-Exception-in-thread-%22Thread-33%22-tp20305986p20319273.html Sent from the Solr - User mailing list archive at Nabble.com.
