[jira] [Commented] (SPARK-12837) Spark driver requires large memory space for serialized results even there are no data collected to the driver
[ https://issues.apache.org/jira/browse/SPARK-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125726#comment-16125726 ] Todd Leo commented on SPARK-12837: -- [~cloud_fan] Is there a ticket tracking "driver-free broadcast" ? > Spark driver requires large memory space for serialized results even there > are no data collected to the driver > -- > > Key: SPARK-12837 > URL: https://issues.apache.org/jira/browse/SPARK-12837 > Project: Spark > Issue Type: Question > Components: SQL >Affects Versions: 1.5.2, 1.6.0 >Reporter: Tien-Dung LE >Assignee: Wenchen Fan >Priority: Critical > Fix For: 2.2.0 > > > Executing a sql statement with a large number of partitions requires a high > memory space for the driver even there are no requests to collect data back > to the driver. > Here are steps to re-produce the issue. > 1. Start spark shell with a spark.driver.maxResultSize setting > {code:java} > bin/spark-shell --driver-memory=1g --conf spark.driver.maxResultSize=1m > {code} > 2. Execute the code > {code:java} > case class Toto( a: Int, b: Int) > val df = sc.parallelize( 1 to 1e6.toInt).map( i => Toto( i, i)).toDF > sqlContext.setConf( "spark.sql.shuffle.partitions", "200" ) > df.groupBy("a").count().saveAsParquetFile( "toto1" ) // OK > sqlContext.setConf( "spark.sql.shuffle.partitions", 1e3.toInt.toString ) > df.repartition(1e3.toInt).groupBy("a").count().repartition(1e3.toInt).saveAsParquetFile( > "toto2" ) // ERROR > {code} > The error message is > {code:java} > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Total size of serialized results of 393 tasks (1025.9 KB) is bigger than > spark.driver.maxResultSize (1024.0 KB) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13298) DAG visualization does not render correctly for jobs
[ https://issues.apache.org/jira/browse/SPARK-13298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192324#comment-15192324 ] Todd Leo commented on SPARK-13298: -- Pls also see SPARK-13645: DAG Diagram not shown properly in Chrome > DAG visualization does not render correctly for jobs > > > Key: SPARK-13298 > URL: https://issues.apache.org/jira/browse/SPARK-13298 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.6.0 >Reporter: Lucas Woltmann >Assignee: Shixiong Zhu > Fix For: 1.6.1, 2.0.0 > > Attachments: dag_full.png, dag_viz.png, no-dag.png > > > Whenever I try to open the DAG for a job, I get something like this: > !dag_viz.png! > Obviously the svg doesn't get resized, but if I resize it manually, only the > first of four stages in the DAG is shown. > The js console says (variable v is null in peg$c34): > {code:javascript} > Uncaught TypeError: Cannot read property '3' of null > peg$c34 @ graphlib-dot.min.js:1 > peg$parseidDef @ graphlib-dot.min.js:1 > peg$parseaList @ graphlib-dot.min.js:1 > peg$parseattrListBlock @ graphlib-dot.min.js:1 > peg$parseattrList @ graphlib-dot.min.js:1 > peg$parsenodeStmt @ graphlib-dot.min.js:1 > peg$parsestmt @ graphlib-dot.min.js:1 > peg$parsestmtList @ graphlib-dot.min.js:1 > peg$parsesubgraphStmt @ graphlib-dot.min.js:1 > peg$parsenodeIdOrSubgraph @ graphlib-dot.min.js:1 > peg$parseedgeStmt @ graphlib-dot.min.js:1 > peg$parsestmt @ graphlib-dot.min.js:1 > peg$parsestmtList @ graphlib-dot.min.js:1 > peg$parsesubgraphStmt @ graphlib-dot.min.js:1 > peg$parsenodeIdOrSubgraph @ graphlib-dot.min.js:1 > peg$parseedgeStmt @ graphlib-dot.min.js:1 > peg$parsestmt @ graphlib-dot.min.js:1 > peg$parsestmtList @ graphlib-dot.min.js:1 > peg$parsegraphStmt @ graphlib-dot.min.js:1 > parse @ graphlib-dot.min.js:2 > readOne @ graphlib-dot.min.js:2 > renderDot @ spark-dag-viz.js:281 > (anonymous function) @ spark-dag-viz.js:248 > (anonymous function) @ d3.min.js: > 3Y @ d3.min.js:1 > _a.each @ d3.min.js:3 > renderDagVizForJob @ spark-dag-viz.js:207 > renderDagViz @ spark-dag-viz.js:163 > toggleDagViz @ spark-dag-viz.js:100 > onclick @ ?id=2:153 > {code} > (tested in FIrefox 44.0.1 and Chromium 48.0.2564.103) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13645) DAG Diagram not shown properly in Chrome
[ https://issues.apache.org/jira/browse/SPARK-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Leo updated SPARK-13645: - Description: In my Chrome 49, the execution DAG diagram can't be shown properly. Only a few grey dots lays there. Thought this is what I'm supposed to see at first. It works fine in Firefox, though. See the attachment below. was: In my Chrome 49, the execution DAG diagram can't be shown properly. Only a few grey dots lays there. Thought this is what I'm supposed to see at first. See the attachment below. > DAG Diagram not shown properly in Chrome > > > Key: SPARK-13645 > URL: https://issues.apache.org/jira/browse/SPARK-13645 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.6.0 > Environment: Chrome 49, 64-bit >Reporter: Todd Leo > Attachments: Slack for iOS Upload.jpg > > > In my Chrome 49, the execution DAG diagram can't be shown properly. Only a > few grey dots lays there. Thought this is what I'm supposed to see at first. > It works fine in Firefox, though. > See the attachment below. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13645) DAG Diagram not shown properly in Chrome
[ https://issues.apache.org/jira/browse/SPARK-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Leo updated SPARK-13645: - Description: In my Chrome 49, the execution DAG diagram can't be shown properly. Only a few grey dots lays there. Thought this is what I'm supposed to see at first. See the attachment below. was:In my Chrome 49, the execution DAG diagram can't be shown properly. Only a few grey dots lays there. Thought this is what I'm supposed to see at first. > DAG Diagram not shown properly in Chrome > > > Key: SPARK-13645 > URL: https://issues.apache.org/jira/browse/SPARK-13645 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.6.0 > Environment: Chrome 49, 64-bit >Reporter: Todd Leo > Attachments: Slack for iOS Upload.jpg > > > In my Chrome 49, the execution DAG diagram can't be shown properly. Only a > few grey dots lays there. Thought this is what I'm supposed to see at first. > See the attachment below. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13645) DAG Diagram not shown properly in Chrome
[ https://issues.apache.org/jira/browse/SPARK-13645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Leo updated SPARK-13645: - Attachment: Slack for iOS Upload.jpg > DAG Diagram not shown properly in Chrome > > > Key: SPARK-13645 > URL: https://issues.apache.org/jira/browse/SPARK-13645 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.6.0 > Environment: Chrome 49, 64-bit >Reporter: Todd Leo > Attachments: Slack for iOS Upload.jpg > > > In my Chrome 49, the execution DAG diagram can't be shown properly. Only a > few grey dots lays there. Thought this is what I'm supposed to see at first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-13645) DAG Diagram not shown properly in Chrome
Todd Leo created SPARK-13645: Summary: DAG Diagram not shown properly in Chrome Key: SPARK-13645 URL: https://issues.apache.org/jira/browse/SPARK-13645 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 1.6.0 Environment: Chrome 49, 64-bit Reporter: Todd Leo In my Chrome 49, the execution DAG diagram can't be shown properly. Only a few grey dots lays there. Thought this is what I'm supposed to see at first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org