Re: 回复: Re: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows.
My guess the reason why local mode is OK while standalone cluster doesn't work is that in cluster mode, task results are serialized and sent to driver side. Driver need to deserialize the result, and thus occupies much more memory then local mode (where task result de/serialization is not performed). Cheng On 6/12/15 4:17 PM, Cheng, Hao wrote: Not sure if Spark Core will provide API to fetch the record one by one from the block manager, instead of the pulling them all into the driver memory. *From:*Cheng Lian [mailto:l...@databricks.com] *Sent:* Friday, June 12, 2015 3:51 PM *To:* 姜超才; Hester wang; user@spark.apache.org *Subject:* Re: 回复: Re: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. Thanks for the extra details and explanations Chaocai, will try to reproduce this when I got chance. Cheng On 6/12/15 3:44 PM, 姜超才 wrote: I said "OOM occurred on slave node", because I monitored memory utilization during the query task, on driver, very few memory was ocupied. And i remember i have ever seen the OOM stderr log on slave node. But recently there seems no OOM log on slave node. Follow the cmd 、data 、env and the code I gave you, the OOM can 100% repro on cluster mode. Thanks, SuperJ - 原始邮件信息 - *发件人**:* "Cheng Lian" <mailto:l...@databricks.com> *收件人**:* "姜超才" <mailto:jiangchao...@haiyisoft.com>, "Hester wang" <mailto:hester9...@gmail.com>, <mailto:user@spark.apache.org> *主题**:* Re: 回复: Re: 回 复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. *日期**:* 2015/06/12 15:30:08 (Fri) Hi Chaocai, Glad that 1.4 fixes your case. However, I'm a bit confused by your last comment saying "The OOM or lose heartbeat was occurred on slave node". Because from the log files you attached at first, those OOM actually happens on driver side (Thrift server log only contains log lines from driver side). Did you see OOM from executor stderr output? I ask this because there are still a large portion of users are using 1.3, and we may want deliver a fix if there does exist bugs that causes unexpected OOM. Cheng On 6/12/15 3:14 PM, 姜超才 wrote: Hi Lian, Today I update my spark to v1.4. This issue resolved. Thanks, SuperJ - 原始邮件信 息 - *发件人**:* "姜超才" *收件人**:* "Cheng Lian" , "Hester wang" , *主题**:* 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. *日期**:* 2015/06/11 08:56:28 (Thu) No problem on Local mode. I can get all rows. Select * from foo; The OOM or lose heartbeat was occured on slave node. Thanks, SuperJ - 原始邮件信 息 - *发件人**:* "Cheng Lian" *收件人**:* "姜超才" , "Hester wang" , *主题**:* Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. *日期**:* 2015/06/10 19:58:59 (Wed) Hm, I tried the following with 0.13.1 and 0.13.0 on my laptop (don't have access to a cluster for now) but couldn't reproduce this issue. Your program just executed smoothly... :-/ Command line used to start the Thrift server: ./sbin/start-thriftserver.sh --driver-memory 4g --master local SQL statements used to create the table with your data: create table foo(k string, v double); load data local inpath '/tmp/bar' into table foo; Tried this via Beeline: select * from foo limit 160; Also tried the Java program you provided. Could you also try to verify whether this single node local mode works for you? Will investigate this with a cluster when I get chance. Cheng On 6/10/15 5:19 PM, 姜超才 wrote: When set "spark.sql.thriftServer.incrementalCollect" and set driver memory to 7G, Things seems stable and simple: It can quickly run through the query line, but when traversal the result set ( while rs.hasNext ), it can quickly get the OOM: java heap space. See attachment. /usr/local/spark/spark-1.3.0/sbin/start-thriftserver.sh --master spark://cx-spark-001:7077 --conf spark.executor.memory=4g --conf spark.driver.memory=7g --conf spark.shuffle.consolidateFiles=true --conf spark.shuffle.manager=sort --conf "spark.executor.extraJavaOptions=-XX:-UseGCOverheadLimit" --conf spark.file.transferTo=false --conf spark.akka.timeout=2000 --conf spark.storage.memoryFraction=0.4 --conf spark.cores.max=8 --conf spark.kryoserializer.buff
RE: 回复: Re: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows.
Not sure if Spark RDD will provide API to fetch the record one by one from the final result set, instead of the pulling them all / (or whole partition data) and fit in the driver memory. Seems a big change. From: Cheng Lian [mailto:l...@databricks.com] Sent: Friday, June 12, 2015 3:51 PM To: 姜超才; Hester wang; user@spark.apache.org Subject: Re: 回复: Re: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. Thanks for the extra details and explanations Chaocai, will try to reproduce this when I got chance. Cheng On 6/12/15 3:44 PM, 姜超才 wrote: I said "OOM occurred on slave node", because I monitored memory utilization during the query task, on driver, very few memory was ocupied. And i remember i have ever seen the OOM stderr log on slave node. But recently there seems no OOM log on slave node. Follow the cmd 、data 、env and the code I gave you, the OOM can 100% repro on cluster mode. Thanks, SuperJ - 原始邮件信息 - 发件人: "Cheng Lian" <mailto:l...@databricks.com> 收件人: "姜超才" <mailto:jiangchao...@haiyisoft.com>, "Hester wang" <mailto:hester9...@gmail.com>, <mailto:user@spark.apache.org> 主题: Re: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. 日期: 2015/06/12 15:30:08 (Fri) Hi Chaocai, Glad that 1.4 fixes your case. However, I'm a bit confused by your last comment saying "The OOM or lose heartbeat was occurred on slave node". Because from the log files you attached at first, those OOM actually happens on driver side (Thrift server log only contains log lines from driver side). Did you see OOM from executor stderr output? I ask this because there are still a large portion of users are using 1.3, and we may want deliver a fix if there does exist bugs that causes unexpected OOM. Cheng On 6/12/15 3:14 PM, 姜超才 wrote: Hi Lian, Today I update my spark to v1.4. This issue resolved. Thanks, SuperJ - 原始邮件信息 - 发件人: "姜超才" 收件人: "Cheng Lian" , "Hester wang" , 主题: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. 日期: 2015/06/11 08:56:28 (Thu) No problem on Local mode. I can get all rows. Select * from foo; The OOM or lose heartbeat was occured on slave node. Thanks, SuperJ - 原始邮件信息 - 发件人: "Cheng Lian" 收件人: "姜超才" , "Hester wang" , 主题: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. 日期: 2015/06/10 19:58:59 (Wed) Hm, I tried the following with 0.13.1 and 0.13.0 on my laptop (don't have access to a cluster for now) but couldn't reproduce this issue. Your program just executed smoothly... :-/ Command line used to start the Thrift server: ./sbin/start-thriftserver.sh --driver-memory 4g --master local SQL statements used to create the table with your data: create table foo(k string, v double); load data local inpath '/tmp/bar' into table foo; Tried this via Beeline: select * from foo limit 160; Also tried the Java program you provided. Could you also try to verify whether this single node local mode works for you? Will investigate this with a cluster when I get chance. Cheng On 6/10/15 5:19 PM, 姜超才 wrote: When set "spark.sql.thriftServer.incrementalCollect" and set driver memory to 7G, Things seems stable and simple: It can quickly run through the query line, but when traversal the result set ( while rs.hasNext ), it can quickly get the OOM: java heap space. See attachment. /usr/local/spark/spark-1.3.0/sbin/start-thriftserver.sh --master spark://cx-spark-001:7077 --conf spark.executor.memory=4g --conf spark.driver.memory=7g --conf spark.shuffle.consolidateFiles=true --conf spark.shuffle.manager=sort --conf "spark.executor.extraJavaOptions=-XX:-UseGCOverheadLimit" --conf spark.file.transferTo=false --conf spark.akka.timeout=2000 --conf spark.storage.memoryFraction=0.4 --conf spark.cores.max=8 --conf spark.kryoserializer.buffer.mb=256 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.akka.frameSize=512 --driver-class-path /usr/local/hive/lib/classes12.jar --conf spark.sql.thriftServer.incrementalCollect=true Thanks, SuperJ - 原始邮件信息 - 发件人: "Cheng Lian" 收件人: "姜超才" , "Hester wang" , 主题: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. 日期: 2015/06/10 16:37:34 (Wed) Also, if the data isn't confidential, would you mind to send me a compressed copy (don't cc user@spark.apache.org<mailto:user@spark.apache.org>)? Cheng On 6/10/15 4:23 PM, 姜超才 wrote: Hi Lian, Thanks for your quick response. I forgot mention that I have tuned driver memory from 2G to 4G, seems got minor improvement, The dead way when fetching 1,400,000 rows changed from "OOM::GC overhead limit exceeded" to " lost worker heartbeat after 120s".
RE: 回复: Re: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows.
Not sure if Spark Core will provide API to fetch the record one by one from the block manager, instead of the pulling them all into the driver memory. From: Cheng Lian [mailto:l...@databricks.com] Sent: Friday, June 12, 2015 3:51 PM To: 姜超才; Hester wang; user@spark.apache.org Subject: Re: 回复: Re: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. Thanks for the extra details and explanations Chaocai, will try to reproduce this when I got chance. Cheng On 6/12/15 3:44 PM, 姜超才 wrote: I said "OOM occurred on slave node", because I monitored memory utilization during the query task, on driver, very few memory was ocupied. And i remember i have ever seen the OOM stderr log on slave node. But recently there seems no OOM log on slave node. Follow the cmd 、data 、env and the code I gave you, the OOM can 100% repro on cluster mode. Thanks, SuperJ - 原始邮件信息 - 发件人: "Cheng Lian" <mailto:l...@databricks.com> 收件人: "姜超才" <mailto:jiangchao...@haiyisoft.com>, "Hester wang" <mailto:hester9...@gmail.com>, <mailto:user@spark.apache.org> 主题: Re: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. 日期: 2015/06/12 15:30:08 (Fri) Hi Chaocai, Glad that 1.4 fixes your case. However, I'm a bit confused by your last comment saying "The OOM or lose heartbeat was occurred on slave node". Because from the log files you attached at first, those OOM actually happens on driver side (Thrift server log only contains log lines from driver side). Did you see OOM from executor stderr output? I ask this because there are still a large portion of users are using 1.3, and we may want deliver a fix if there does exist bugs that causes unexpected OOM. Cheng On 6/12/15 3:14 PM, 姜超才 wrote: Hi Lian, Today I update my spark to v1.4. This issue resolved. Thanks, SuperJ - 原始邮件信息 - 发件人: "姜超才" 收件人: "Cheng Lian" , "Hester wang" , 主题: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. 日期: 2015/06/11 08:56:28 (Thu) No problem on Local mode. I can get all rows. Select * from foo; The OOM or lose heartbeat was occured on slave node. Thanks, SuperJ - 原始邮件信息 - 发件人: "Cheng Lian" 收件人: "姜超才" , "Hester wang" , 主题: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. 日期: 2015/06/10 19:58:59 (Wed) Hm, I tried the following with 0.13.1 and 0.13.0 on my laptop (don't have access to a cluster for now) but couldn't reproduce this issue. Your program just executed smoothly... :-/ Command line used to start the Thrift server: ./sbin/start-thriftserver.sh --driver-memory 4g --master local SQL statements used to create the table with your data: create table foo(k string, v double); load data local inpath '/tmp/bar' into table foo; Tried this via Beeline: select * from foo limit 160; Also tried the Java program you provided. Could you also try to verify whether this single node local mode works for you? Will investigate this with a cluster when I get chance. Cheng On 6/10/15 5:19 PM, 姜超才 wrote: When set "spark.sql.thriftServer.incrementalCollect" and set driver memory to 7G, Things seems stable and simple: It can quickly run through the query line, but when traversal the result set ( while rs.hasNext ), it can quickly get the OOM: java heap space. See attachment. /usr/local/spark/spark-1.3.0/sbin/start-thriftserver.sh --master spark://cx-spark-001:7077 --conf spark.executor.memory=4g --conf spark.driver.memory=7g --conf spark.shuffle.consolidateFiles=true --conf spark.shuffle.manager=sort --conf "spark.executor.extraJavaOptions=-XX:-UseGCOverheadLimit" --conf spark.file.transferTo=false --conf spark.akka.timeout=2000 --conf spark.storage.memoryFraction=0.4 --conf spark.cores.max=8 --conf spark.kryoserializer.buffer.mb=256 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.akka.frameSize=512 --driver-class-path /usr/local/hive/lib/classes12.jar --conf spark.sql.thriftServer.incrementalCollect=true Thanks, SuperJ - 原始邮件信息 - 发件人: "Cheng Lian" 收件人: "姜超才" , "Hester wang" , 主题: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. 日期: 2015/06/10 16:37:34 (Wed) Also, if the data isn't confidential, would you mind to send me a compressed copy (don't cc user@spark.apache.org<mailto:user@spark.apache.org>)? Cheng On 6/10/15 4:23 PM, 姜超才 wrote: Hi Lian, Thanks for your quick response. I forgot mention that I have tuned driver memory from 2G to 4G, seems got minor improvement, The dead way when fetching 1,400,000 rows changed from "OOM::GC overhead limit exceeded" to " lost worker heartbeat after 120s". I will try to set "spark.sql.thriftSer
Re: 回复: Re: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows.
Thanks for the extra details and explanations Chaocai, will try to reproduce this when I got chance. Cheng On 6/12/15 3:44 PM, 姜超才 wrote: I said "OOM occurred on slave node", because I monitored memory utilization during the query task, on driver, very few memory was ocupied. And i remember i have ever seen the OOM stderr log on slave node. But recently there seems no OOM log on slave node. Follow the cmd 、data 、env and the code I gave you, the OOM can 100% repro on cluster mode. Thanks, SuperJ - 原始邮件信息 - *发件人:* "Cheng Lian" *收件人:* "姜超才" , "Hester wang" , *主题:* Re: 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. *日期:* 2015/06/12 15:30:08 (Fri) Hi Chaocai, Glad that 1.4 fixes your case. However, I'm a bit confused by your last comment saying "The OOM or lose heartbeat was occurred on slave node". Because from the log files you attached at first, those OOM actually happens on driver side (Thrift server log only contains log lines from driver side). Did you see OOM from executor stderr output? I ask this because there are still a large portion of users are using 1.3, and we may want deliver a fix if there does exist bugs that causes unexpected OOM. Cheng On 6/12/15 3:14 PM, 姜超才 wrote: Hi Lian, Today I update my spark to v1.4. This issue resolved. Thanks, SuperJ - 原始邮件信息 - *发件人:* "姜超才" *收件人:* "Cheng Lian" , "Hester wang" , *主题:* 回复: Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. *日期:* 2015/06/11 08:56:28 (Thu) No problem on Local mode. I can get all rows. Select * from foo; The OOM or lose heartbeat was occured on slave node. Thanks, SuperJ - 原始邮件信息 - *发件人:* "Cheng Lian" *收件人:* "姜超才" , "Hester wang" , *主题:* Re: 回复: Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. *日期:* 2015/06/10 19:58:59 (Wed) Hm, I tried the following with 0.13.1 and 0.13.0 on my laptop (don't have access to a cluster for now) but couldn't reproduce this issue. Your program just executed smoothly... :-/ Command line used to start the Thrift server: ./sbin/start-thriftserver.sh --driver-memory 4g --master local SQL statements used to create the table with your data: create table foo(k string, v double); load data local inpath '/tmp/bar' into table foo; Tried this via Beeline: select * from foo limit 160; Also tried the Java program you provided. Could you also try to verify whether this single node local mode works for you? Will investigate this with a cluster when I get chance. Cheng On 6/10/15 5:19 PM, 姜超才 wrote: When set "spark.sql.thriftServer.incrementalCollect" and set driver memory to 7G, Things seems stable and simple: It can quickly run through the query line, but when traversal the result set ( while rs.hasNext ), it can quickly get the OOM: java heap space. See attachment. /usr/local/spark/spark-1.3.0/sbin/start-thriftserver.sh --master spark://cx-spark-001:7077 --conf spark.executor.memory=4g --conf spark.driver.memory=7g --conf spark.shuffle.consolidateFiles=true --conf spark.shuffle.manager=sort --conf "spark.executor.extraJavaOptions=-XX:-UseGCOverheadLimit" --conf spark.file.transferTo=false --conf spark.akka.timeout=2000 --conf spark.storage.memoryFraction=0.4 --conf spark.cores.max=8 --conf spark.kryoserializer.buffer.mb=256 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.akka.frameSize=512 --driver-class-path /usr/local/hive/lib/classes12.jar --conf spark.sql.thriftServer.incrementalCollect=true Thanks, SuperJ - 原始邮件信息 - *发件人:* "Cheng Lian" *收件人:* "姜超才" , "Hester wang" , *主题:* Re: 回复: Re: Met OOM when fetching more than 1,000,000 rows. *日期:* 2015/06/10 16:37:34 (Wed) Also, if the data isn't confidential, would you mind to send me a compressed copy (don't cc user@spark.apache.org)? Cheng On 6/10/15 4:23 PM, 姜超才 wrote: Hi Lian, Thanks for your quick response. I forgot mention that I have tuned driver memory from 2G to 4G, seems got minor improvement, The dead way when fetching 1,400,000 rows changed from "OOM::GC overhead limit exceeded" to " lost worker heartbeat after 120s". I will try to set "spark.sql.thriftServer.incrementalCollect" and continue increase driver memory to 7G, and will send the result to you. Thanks, SuperJ - 原始邮件信息 - *发件人:* "Cheng Lian" *收件人:* "Hester wang" , *主题:* Re: Met OOM when fetching more than 1,000,000 rows. *日期:* 2015/06/10 16:15:47 (Wed) Hi Xiaohan, Would you please try to set "spark.sql.thriftServer.incrementalCollect" to "true" and increasing driver memory size? In this way, HiveThriftServer2 uses RDD.toLocalIterator rather than RDD.collect().iterator to return the result set. The key difference is that RDD.toLocalIterator retrieves a single partition at a time, thus avoid holding the whole result set on driver side. The memory issue happens on driver side rather than executor side, so tuning exe
