Re: DataImporter : Java heap space
On Thu, Apr 16, 2009 at 10:31 AM, Mani Kumar manikumarchau...@gmail.comwrote: Aah, Bryan you got it ... Thanks! Noble: so i can hope that it'll be fixed soon :) thank you for fixing it ... please lemme know when its done.. This is fixed in trunk. The next nightly build should have this fix. -- Regards, Shalin Shekhar Mangar.
Re: DataImporter : Java heap space
I think there is a bug in the 1.4 daily builds of data import handler
which is causing the batchSize parameter to be ignored. This was
probably introduced with more recent patches to resolve variables.
The affected code is in JdbcDataSource.java
String bsz = initProps.getProperty(batchSize);
if (bsz != null) {
bsz = (String) context.getVariableResolver().resolve(bsz);
try {
batchSize = Integer.parseInt(bsz);
if (batchSize == -1)
batchSize = Integer.MIN_VALUE;
} catch (NumberFormatException e) {
LOG.warn(Invalid batch size: + bsz);
}
}
The call to context.getVariableResolver().resolve(bsz) is returning
null, leading to a NumberFormatException and the batchSize never being
set to Integer.MIN_VALUE. MySql won't use streaming result sets in
this case which can lead to the OOM we're seeing.
If your log file contains this entry like mine does, you're being
affected by this bug too.
Apr 15, 2009 1:21:58 PM
org.apache.solr.handler.dataimport.JdbcDataSource init
WARNING: Invalid batch size: null
-Bryan
On Apr 13, 2009, at Apr 13, 11:48 PM, Noble Paul നോബിള്
नोब्ळ् wrote:
DIH streams 1 row at a time.
DIH is just a component in Solr. Solr indexing also takes a lot of
memory
On Tue, Apr 14, 2009 at 12:02 PM, Mani Kumar manikumarchau...@gmail.com
wrote:
Yes its throwing the same OOM error and from same place...
yes i will try increasing the size ... just curious : how this
dataimport
works?
Does it loads the whole table into memory?
Is there any estimate about how much memory it needs to create
index for 1GB
of data.
thx
mani
On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar
shalinman...@gmail.com wrote:
On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar manikumarchau...@gmail.com
wrote:
Hi Shalin:
yes i tried with batchSize=-1 parameter as well
here the config i tried with
dataConfig
dataSource type=JdbcDataSource batchSize=-1 name=sp
driver=com.mysql.jdbc.Driver
url=jdbc:mysql://localhost/mydb_development
user=root password=** /
I hope i have used batchSize parameter @ right place.
Yes that is correct. Did it still throw OOM from the same place?
I'd suggest you increase the heap and see what works for you. Also
try
-server on the jvm.
--
Regards,
Shalin Shekhar Mangar.
--
--Noble Paul
Re: DataImporter : Java heap space
Hi Bryan,
Thanks a lot. It is invoking the wrong method
it should have been
bsz = context.getVariableResolver().replaceTokens(bsz);
it was a silly mistake
--Noble
On Thu, Apr 16, 2009 at 2:13 AM, Bryan Talbot btal...@aeriagames.com wrote:
I think there is a bug in the 1.4 daily builds of data import handler which
is causing the batchSize parameter to be ignored. This was probably
introduced with more recent patches to resolve variables.
The affected code is in JdbcDataSource.java
String bsz = initProps.getProperty(batchSize);
if (bsz != null) {
bsz = (String) context.getVariableResolver().resolve(bsz);
try {
batchSize = Integer.parseInt(bsz);
if (batchSize == -1)
batchSize = Integer.MIN_VALUE;
} catch (NumberFormatException e) {
LOG.warn(Invalid batch size: + bsz);
}
}
The call to context.getVariableResolver().resolve(bsz) is returning null,
leading to a NumberFormatException and the batchSize never being set to
Integer.MIN_VALUE. MySql won't use streaming result sets in this case which
can lead to the OOM we're seeing.
If your log file contains this entry like mine does, you're being affected
by this bug too.
Apr 15, 2009 1:21:58 PM org.apache.solr.handler.dataimport.JdbcDataSource
init
WARNING: Invalid batch size: null
-Bryan
On Apr 13, 2009, at Apr 13, 11:48 PM, Noble Paul നോബിള് नोब्ळ् wrote:
DIH streams 1 row at a time.
DIH is just a component in Solr. Solr indexing also takes a lot of memory
On Tue, Apr 14, 2009 at 12:02 PM, Mani Kumar manikumarchau...@gmail.com
wrote:
Yes its throwing the same OOM error and from same place...
yes i will try increasing the size ... just curious : how this dataimport
works?
Does it loads the whole table into memory?
Is there any estimate about how much memory it needs to create index for
1GB
of data.
thx
mani
On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar
shalinman...@gmail.com wrote:
On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar manikumarchau...@gmail.com
wrote:
Hi Shalin:
yes i tried with batchSize=-1 parameter as well
here the config i tried with
dataConfig
dataSource type=JdbcDataSource batchSize=-1 name=sp
driver=com.mysql.jdbc.Driver
url=jdbc:mysql://localhost/mydb_development
user=root password=** /
I hope i have used batchSize parameter @ right place.
Yes that is correct. Did it still throw OOM from the same place?
I'd suggest you increase the heap and see what works for you. Also try
-server on the jvm.
--
Regards,
Shalin Shekhar Mangar.
--
--Noble Paul
--
--Noble Paul
Re: DataImporter : Java heap space
Aah, Bryan you got it ... Thanks!
Noble: so i can hope that it'll be fixed soon :) thank you for fixing it ...
please lemme know when its done..
Thanks!
Mani Kumar
2009/4/16 Noble Paul നോബിള് नोब्ळ् noble.p...@gmail.com
Hi Bryan,
Thanks a lot. It is invoking the wrong method
it should have been
bsz = context.getVariableResolver().replaceTokens(bsz);
it was a silly mistake
--Noble
On Thu, Apr 16, 2009 at 2:13 AM, Bryan Talbot btal...@aeriagames.com
wrote:
I think there is a bug in the 1.4 daily builds of data import handler
which
is causing the batchSize parameter to be ignored. This was probably
introduced with more recent patches to resolve variables.
The affected code is in JdbcDataSource.java
String bsz = initProps.getProperty(batchSize);
if (bsz != null) {
bsz = (String) context.getVariableResolver().resolve(bsz);
try {
batchSize = Integer.parseInt(bsz);
if (batchSize == -1)
batchSize = Integer.MIN_VALUE;
} catch (NumberFormatException e) {
LOG.warn(Invalid batch size: + bsz);
}
}
The call to context.getVariableResolver().resolve(bsz) is returning null,
leading to a NumberFormatException and the batchSize never being set to
Integer.MIN_VALUE. MySql won't use streaming result sets in this case
which
can lead to the OOM we're seeing.
If your log file contains this entry like mine does, you're being
affected
by this bug too.
Apr 15, 2009 1:21:58 PM org.apache.solr.handler.dataimport.JdbcDataSource
init
WARNING: Invalid batch size: null
-Bryan
On Apr 13, 2009, at Apr 13, 11:48 PM, Noble Paul നോബിള് नोब्ळ् wrote:
DIH streams 1 row at a time.
DIH is just a component in Solr. Solr indexing also takes a lot of
memory
On Tue, Apr 14, 2009 at 12:02 PM, Mani Kumar
manikumarchau...@gmail.com
wrote:
Yes its throwing the same OOM error and from same place...
yes i will try increasing the size ... just curious : how this
dataimport
works?
Does it loads the whole table into memory?
Is there any estimate about how much memory it needs to create index
for
1GB
of data.
thx
mani
On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar
shalinman...@gmail.com wrote:
On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar
manikumarchau...@gmail.com
wrote:
Hi Shalin:
yes i tried with batchSize=-1 parameter as well
here the config i tried with
dataConfig
dataSource type=JdbcDataSource batchSize=-1 name=sp
driver=com.mysql.jdbc.Driver
url=jdbc:mysql://localhost/mydb_development
user=root password=** /
I hope i have used batchSize parameter @ right place.
Yes that is correct. Did it still throw OOM from the same place?
I'd suggest you increase the heap and see what works for you. Also try
-server on the jvm.
--
Regards,
Shalin Shekhar Mangar.
--
--Noble Paul
--
--Noble Paul
Re: DataImporter : Java heap space
Hi Shalin: yes i tried with batchSize=-1 parameter as well here the config i tried with dataConfig dataSource type=JdbcDataSource batchSize=-1 name=sp driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost/mydb_development user=root password=** / document name=items entity name=item dataSource=sp query=select * from items field column=id name=id / field column=title name=title / /entity /document /dataConfig I hope i have used batchSize parameter @ right place. Thanks! Mani Kumar On Tue, Apr 14, 2009 at 11:24 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Tue, Apr 14, 2009 at 11:18 AM, Mani Kumar manikumarchau...@gmail.com wrote: Here is the stack trace: notice in stack trace * at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)* It looks like that its trying to read whole table into memory at a time. n thts y getting OOM. Mani, the data-config.xml you posted does not have the batchSize=-1 attribute to your data source. Did you try that? This is a known bug in MySql jdbc driver. -- Regards, Shalin Shekhar Mangar.
Re: DataImporter : Java heap space
On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar manikumarchau...@gmail.comwrote: Hi Shalin: yes i tried with batchSize=-1 parameter as well here the config i tried with dataConfig dataSource type=JdbcDataSource batchSize=-1 name=sp driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost/mydb_development user=root password=** / I hope i have used batchSize parameter @ right place. Yes that is correct. Did it still throw OOM from the same place? I'd suggest you increase the heap and see what works for you. Also try -server on the jvm. -- Regards, Shalin Shekhar Mangar.
Re: DataImporter : Java heap space
Yes its throwing the same OOM error and from same place... yes i will try increasing the size ... just curious : how this dataimport works? Does it loads the whole table into memory? Is there any estimate about how much memory it needs to create index for 1GB of data. thx mani On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar manikumarchau...@gmail.com wrote: Hi Shalin: yes i tried with batchSize=-1 parameter as well here the config i tried with dataConfig dataSource type=JdbcDataSource batchSize=-1 name=sp driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost/mydb_development user=root password=** / I hope i have used batchSize parameter @ right place. Yes that is correct. Did it still throw OOM from the same place? I'd suggest you increase the heap and see what works for you. Also try -server on the jvm. -- Regards, Shalin Shekhar Mangar.
Re: DataImporter : Java heap space
DIH streams 1 row at a time. DIH is just a component in Solr. Solr indexing also takes a lot of memory On Tue, Apr 14, 2009 at 12:02 PM, Mani Kumar manikumarchau...@gmail.com wrote: Yes its throwing the same OOM error and from same place... yes i will try increasing the size ... just curious : how this dataimport works? Does it loads the whole table into memory? Is there any estimate about how much memory it needs to create index for 1GB of data. thx mani On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar manikumarchau...@gmail.com wrote: Hi Shalin: yes i tried with batchSize=-1 parameter as well here the config i tried with dataConfig dataSource type=JdbcDataSource batchSize=-1 name=sp driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost/mydb_development user=root password=** / I hope i have used batchSize parameter @ right place. Yes that is correct. Did it still throw OOM from the same place? I'd suggest you increase the heap and see what works for you. Also try -server on the jvm. -- Regards, Shalin Shekhar Mangar. -- --Noble Paul
DataImporter : Java heap space
Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this Apr 13, 2009 11:53:28 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call INFO: Creating a connection for entity slideshow with URL: jdbc:mysql://localhost/mydb_development Apr 13, 2009 11:53:29 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call INFO: Time taken for getConnection(): 319 Apr 13, 2009 11:53:32 PM org.apache.solr.handler.dataimport.DataImporter doFullImport SEVERE: Full Import failed org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.Buffer.init(Buffer.java:58) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840) My Java version $ java -version java version 1.5.0_16 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284) Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing) Is that i need to install a new java version? my db is also very huge ~15 GB please do the need full ... thanks mani kumar
Re: DataImporter : Java heap space
I am using Tomcat ... On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar manikumarchau...@gmail.comwrote: Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this Apr 13, 2009 11:53:28 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call INFO: Creating a connection for entity slideshow with URL: jdbc:mysql://localhost/mydb_development Apr 13, 2009 11:53:29 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call INFO: Time taken for getConnection(): 319 Apr 13, 2009 11:53:32 PM org.apache.solr.handler.dataimport.DataImporter doFullImport SEVERE: Full Import failed org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.Buffer.init(Buffer.java:58) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840) My Java version $ java -version java version 1.5.0_16 Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284) Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing) Is that i need to install a new java version? my db is also very huge ~15 GB please do the need full ... thanks mani kumar
Re: DataImporter : Java heap space
On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar manikumarchau...@gmail.comwrote: Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this java.lang.OutOfMemoryError: Java heap space at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.Buffer.init(Buffer.java:58) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840) How much heap size have you allocated to the jvm? Also see http://wiki.apache.org/solr/DataImportHandlerFaq -- Regards, Shalin Shekhar Mangar.
Re: DataImporter : Java heap space
Hi Shalin: Thanks for quick response! By defaults it was set to 1.93 MB. But i also tried it with following command: $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M I also tried tricks given on http://wiki.apache.org/solr/DataImportHandlerFaq page. what should i try next ? Thanks! Mani Kumar On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar manikumarchau...@gmail.com wrote: Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this java.lang.OutOfMemoryError: Java heap space at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.Buffer.init(Buffer.java:58) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840) How much heap size have you allocated to the jvm? Also see http://wiki.apache.org/solr/DataImportHandlerFaq -- Regards, Shalin Shekhar Mangar.
Re: DataImporter : Java heap space
Depending on your dataset and how your queries look you may very likely need to increase to a larger heap size. How many queries and rows are required for each of your documents to be generated? Ilan On 4/13/09 12:21 PM, Mani Kumar wrote: Hi Shalin: Thanks for quick response! By defaults it was set to 1.93 MB. But i also tried it with following command: $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M I also tried tricks given on http://wiki.apache.org/solr/DataImportHandlerFaq page. what should i try next ? Thanks! Mani Kumar On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumarmanikumarchau...@gmail.com wrote: Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this java.lang.OutOfMemoryError: Java heap space at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.Buffer.init(Buffer.java:58) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840) How much heap size have you allocated to the jvm? Also see http://wiki.apache.org/solr/DataImportHandlerFaq -- Regards, Shalin Shekhar Mangar. -- Ilan Rabinovitch i...@fonz.net --- SCALE 7x: 2009 Southern California Linux Expo Los Angeles, CA http://www.socallinuxexpo.org
Re: DataImporter : Java heap space
Hi ILAN: Only one query is required to generate a document ... Here is my data-config.xml dataConfig dataSource type=JdbcDataSource name=sp driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost/mydb_development user=root password=** / document name=items entity name=item dataSource=sp query=select * from items field column=id name=id / field column=title name=title / /entity /document /dataConfig and other useful info: mysql select * from items +--+ | count(*) | +--+ | 900051 | +--+ 1 row in set (0.00 sec) Each record consist of id and title. id is of type int(11) and title's avg. length is 50 chars. I am using tomcat with solr. here is the command i am using to start it ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M Thanks! for help. I appreciate it. -Mani Kumar On Tue, Apr 14, 2009 at 2:31 AM, Ilan Rabinovitch i...@fonz.net wrote: Depending on your dataset and how your queries look you may very likely need to increase to a larger heap size. How many queries and rows are required for each of your documents to be generated? Ilan On 4/13/09 12:21 PM, Mani Kumar wrote: Hi Shalin: Thanks for quick response! By defaults it was set to 1.93 MB. But i also tried it with following command: $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M I also tried tricks given on http://wiki.apache.org/solr/DataImportHandlerFaq page. what should i try next ? Thanks! Mani Kumar On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumarmanikumarchau...@gmail.com wrote: Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this java.lang.OutOfMemoryError: Java heap space at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.Buffer.init(Buffer.java:58) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840) How much heap size have you allocated to the jvm? Also see http://wiki.apache.org/solr/DataImportHandlerFaq -- Regards, Shalin Shekhar Mangar. -- Ilan Rabinovitch i...@fonz.net --- SCALE 7x: 2009 Southern California Linux Expo Los Angeles, CA http://www.socallinuxexpo.org
Re: DataImporter : Java heap space
DIH itself may not be consuming so much memory. It also includes the memory used by Solr. Do you have a hard limit on 400MB , is it not possible to increase it? On Tue, Apr 14, 2009 at 11:09 AM, Mani Kumar manikumarchau...@gmail.com wrote: Hi ILAN: Only one query is required to generate a document ... Here is my data-config.xml dataConfig dataSource type=JdbcDataSource name=sp driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost/mydb_development user=root password=** / document name=items entity name=item dataSource=sp query=select * from items field column=id name=id / field column=title name=title / /entity /document /dataConfig and other useful info: mysql select * from items +--+ | count(*) | +--+ | 900051 | +--+ 1 row in set (0.00 sec) Each record consist of id and title. id is of type int(11) and title's avg. length is 50 chars. I am using tomcat with solr. here is the command i am using to start it ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M Thanks! for help. I appreciate it. -Mani Kumar On Tue, Apr 14, 2009 at 2:31 AM, Ilan Rabinovitch i...@fonz.net wrote: Depending on your dataset and how your queries look you may very likely need to increase to a larger heap size. How many queries and rows are required for each of your documents to be generated? Ilan On 4/13/09 12:21 PM, Mani Kumar wrote: Hi Shalin: Thanks for quick response! By defaults it was set to 1.93 MB. But i also tried it with following command: $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M I also tried tricks given on http://wiki.apache.org/solr/DataImportHandlerFaq page. what should i try next ? Thanks! Mani Kumar On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumarmanikumarchau...@gmail.com wrote: Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this java.lang.OutOfMemoryError: Java heap space at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.Buffer.init(Buffer.java:58) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840) How much heap size have you allocated to the jvm? Also see http://wiki.apache.org/solr/DataImportHandlerFaq -- Regards, Shalin Shekhar Mangar. -- Ilan Rabinovitch i...@fonz.net --- SCALE 7x: 2009 Southern California Linux Expo Los Angeles, CA http://www.socallinuxexpo.org -- --Noble Paul
Re: DataImporter : Java heap space
Here is the stack trace: notice in stack trace * at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)* It looks like that its trying to read whole table into memory at a time. n thts y getting OOM. Apr 14, 2009 11:15:01 AM org.apache.solr.handler.dataimport.DataImporter doFullImport SEVERE: Full Import failed org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.Buffer.init(Buffer.java:58) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840) at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:468) at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2534) at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2159) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2548) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2477) at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:741) at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:587) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.init(JdbcDataSource.java:243) at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:207) at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:38) at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58) at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:335) ... 5 more Apr 14, 2009 11:15:01 AM org.apache.solr.update.DirectUpdateHandler2 rollback INFO: start rollback Apr 14, 2009 11:15:01 AM org.apache.solr.update.DirectUpdateHandler2 rollback INFO: end_rollback On Tue, Apr 14, 2009 at 11:09 AM, Mani Kumar manikumarchau...@gmail.comwrote: Hi ILAN: Only one query is required to generate a document ... Here is my data-config.xml dataConfig dataSource type=JdbcDataSource name=sp driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost/mydb_development user=root password=** / document name=items entity name=item dataSource=sp query=select * from items field column=id name=id / field column=title name=title / /entity /document /dataConfig and other useful info: mysql select * from items +--+ | count(*) | +--+ | 900051 | +--+ 1 row in set (0.00 sec) Each record consist of id and title. id is of type int(11) and title's avg. length is 50 chars. I am using tomcat with solr. here is the command i am using to start it ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M Thanks! for help. I appreciate it. -Mani Kumar On Tue, Apr 14, 2009 at 2:31 AM, Ilan Rabinovitch i...@fonz.net wrote: Depending on your dataset and how your queries look you may very likely need to increase to a larger heap size. How many queries and rows are required for each of your documents to be generated? Ilan On 4/13/09 12:21 PM, Mani Kumar wrote: Hi Shalin: Thanks for quick response! By defaults it was set to 1.93 MB. But i also tried it with following command: $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M I also tried tricks given on http://wiki.apache.org/solr/DataImportHandlerFaq page. what should i try next ? Thanks! Mani Kumar On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumarmanikumarchau...@gmail.com wrote: Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this java.lang.OutOfMemoryError: Java heap space at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164) at
Re: DataImporter : Java heap space
Hi Noble: But the question is how much memory? is there any rules or something like that? so that i can estimate the how much memory it requires? Yeah i can increase it upto 800MB max will try it and let you know Thanks! Mani 2009/4/14 Noble Paul നോബിള് नोब्ळ् noble.p...@gmail.com DIH itself may not be consuming so much memory. It also includes the memory used by Solr. Do you have a hard limit on 400MB , is it not possible to increase it? On Tue, Apr 14, 2009 at 11:09 AM, Mani Kumar manikumarchau...@gmail.com wrote: Hi ILAN: Only one query is required to generate a document ... Here is my data-config.xml dataConfig dataSource type=JdbcDataSource name=sp driver=com.mysql.jdbc.Driver url=jdbc:mysql://localhost/mydb_development user=root password=** / document name=items entity name=item dataSource=sp query=select * from items field column=id name=id / field column=title name=title / /entity /document /dataConfig and other useful info: mysql select * from items +--+ | count(*) | +--+ | 900051 | +--+ 1 row in set (0.00 sec) Each record consist of id and title. id is of type int(11) and title's avg. length is 50 chars. I am using tomcat with solr. here is the command i am using to start it ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M Thanks! for help. I appreciate it. -Mani Kumar On Tue, Apr 14, 2009 at 2:31 AM, Ilan Rabinovitch i...@fonz.net wrote: Depending on your dataset and how your queries look you may very likely need to increase to a larger heap size. How many queries and rows are required for each of your documents to be generated? Ilan On 4/13/09 12:21 PM, Mani Kumar wrote: Hi Shalin: Thanks for quick response! By defaults it was set to 1.93 MB. But i also tried it with following command: $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M I also tried tricks given on http://wiki.apache.org/solr/DataImportHandlerFaq page. what should i try next ? Thanks! Mani Kumar On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar manikumarchau...@gmail.com wrote: Hi All, I am trying to setup a Solr instance on my macbook. I get following errors when m trying to do a full db import ... please help me on this java.lang.OutOfMemoryError: Java heap space at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351) Caused by: java.lang.OutOfMemoryError: Java heap space at com.mysql.jdbc.Buffer.init(Buffer.java:58) at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444) at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840) How much heap size have you allocated to the jvm? Also see http://wiki.apache.org/solr/DataImportHandlerFaq -- Regards, Shalin Shekhar Mangar. -- Ilan Rabinovitch i...@fonz.net --- SCALE 7x: 2009 Southern California Linux Expo Los Angeles, CA http://www.socallinuxexpo.org -- --Noble Paul
Re: DataImporter : Java heap space
On Tue, Apr 14, 2009 at 11:18 AM, Mani Kumar manikumarchau...@gmail.comwrote: Here is the stack trace: notice in stack trace * at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)* It looks like that its trying to read whole table into memory at a time. n thts y getting OOM. Mani, the data-config.xml you posted does not have the batchSize=-1 attribute to your data source. Did you try that? This is a known bug in MySql jdbc driver. -- Regards, Shalin Shekhar Mangar.
