[jira] [Commented] (SPARK-20213) DataFrameWriter operations do not show up in SQL tab
[ https://issues.apache.org/jira/browse/SPARK-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062534#comment-16062534 ] Apache Spark commented on SPARK-20213: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/18419 > DataFrameWriter operations do not show up in SQL tab > > > Key: SPARK-20213 > URL: https://issues.apache.org/jira/browse/SPARK-20213 > Project: Spark > Issue Type: Bug > Components: SQL, Web UI >Affects Versions: 2.0.2, 2.1.0 >Reporter: Ryan Blue >Assignee: Wenchen Fan > Fix For: 2.3.0 > > Attachments: Screen Shot 2017-05-03 at 5.00.19 PM.png > > > In 1.6.1, {{DataFrame}} writes started using {{DataFrameWriter}} actions like > {{insertInto}} would show up in the SQL tab. In 2.0.0 and later, they no > longer do. The problem is that 2.0.0 and later no longer wrap execution with > {{SQLExecution.withNewExecutionId}}, which emits > {{SparkListenerSQLExecutionStart}}. > Here are the relevant parts of the stack traces: > {code:title=Spark 1.6.1} > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) > org.apache.spark.sql.execution.QueryExecution$$anonfun$toRdd$1.apply(QueryExecution.scala:56) > org.apache.spark.sql.execution.QueryExecution$$anonfun$toRdd$1.apply(QueryExecution.scala:56) > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:53) > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:56) > => holding > Monitor(org.apache.spark.sql.hive.HiveContext$QueryExecution@424773807}) > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55) > org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:196) > {code} > {code:title=Spark 2.0.0} > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133) > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114) > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:86) > => holding Monitor(org.apache.spark.sql.execution.QueryExecution@490977924}) > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:86) > org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:301) > {code} > I think this was introduced by > [54d23599|https://github.com/apache/spark/commit/54d23599]. The fix should be > to add withNewExecutionId to > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L610 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20213) DataFrameWriter operations do not show up in SQL tab
[ https://issues.apache.org/jira/browse/SPARK-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020086#comment-16020086 ] Apache Spark commented on SPARK-20213: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/18064 > DataFrameWriter operations do not show up in SQL tab > > > Key: SPARK-20213 > URL: https://issues.apache.org/jira/browse/SPARK-20213 > Project: Spark > Issue Type: Bug > Components: SQL, Web UI >Affects Versions: 2.0.2, 2.1.0 >Reporter: Ryan Blue > Attachments: Screen Shot 2017-05-03 at 5.00.19 PM.png > > > In 1.6.1, {{DataFrame}} writes started using {{DataFrameWriter}} actions like > {{insertInto}} would show up in the SQL tab. In 2.0.0 and later, they no > longer do. The problem is that 2.0.0 and later no longer wrap execution with > {{SQLExecution.withNewExecutionId}}, which emits > {{SparkListenerSQLExecutionStart}}. > Here are the relevant parts of the stack traces: > {code:title=Spark 1.6.1} > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) > org.apache.spark.sql.execution.QueryExecution$$anonfun$toRdd$1.apply(QueryExecution.scala:56) > org.apache.spark.sql.execution.QueryExecution$$anonfun$toRdd$1.apply(QueryExecution.scala:56) > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:53) > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:56) > => holding > Monitor(org.apache.spark.sql.hive.HiveContext$QueryExecution@424773807}) > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55) > org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:196) > {code} > {code:title=Spark 2.0.0} > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133) > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114) > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:86) > => holding Monitor(org.apache.spark.sql.execution.QueryExecution@490977924}) > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:86) > org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:301) > {code} > I think this was introduced by > [54d23599|https://github.com/apache/spark/commit/54d23599]. The fix should be > to add withNewExecutionId to > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L610 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20213) DataFrameWriter operations do not show up in SQL tab
[ https://issues.apache.org/jira/browse/SPARK-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995928#comment-15995928 ] Ryan Blue commented on SPARK-20213: --- [~zsxwing], the PR adds a method that causes tests to fail if they aren't wrapped. If you remove the additional high-level calls to withNewExecutionId that I added, you can see all the test failures. > DataFrameWriter operations do not show up in SQL tab > > > Key: SPARK-20213 > URL: https://issues.apache.org/jira/browse/SPARK-20213 > Project: Spark > Issue Type: Bug > Components: SQL, Web UI >Affects Versions: 2.0.2, 2.1.0 >Reporter: Ryan Blue > Attachments: Screen Shot 2017-05-03 at 5.00.19 PM.png > > > In 1.6.1, {{DataFrame}} writes started using {{DataFrameWriter}} actions like > {{insertInto}} would show up in the SQL tab. In 2.0.0 and later, they no > longer do. The problem is that 2.0.0 and later no longer wrap execution with > {{SQLExecution.withNewExecutionId}}, which emits > {{SparkListenerSQLExecutionStart}}. > Here are the relevant parts of the stack traces: > {code:title=Spark 1.6.1} > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) > org.apache.spark.sql.execution.QueryExecution$$anonfun$toRdd$1.apply(QueryExecution.scala:56) > org.apache.spark.sql.execution.QueryExecution$$anonfun$toRdd$1.apply(QueryExecution.scala:56) > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:53) > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:56) > => holding > Monitor(org.apache.spark.sql.hive.HiveContext$QueryExecution@424773807}) > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55) > org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:196) > {code} > {code:title=Spark 2.0.0} > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133) > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114) > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:86) > => holding Monitor(org.apache.spark.sql.execution.QueryExecution@490977924}) > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:86) > org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:301) > {code} > I think this was introduced by > [54d23599|https://github.com/apache/spark/commit/54d23599]. The fix should be > to add withNewExecutionId to > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L610 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20213) DataFrameWriter operations do not show up in SQL tab
[ https://issues.apache.org/jira/browse/SPARK-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995923#comment-15995923 ] Shixiong Zhu commented on SPARK-20213: -- I tested the master branch, and I can see "insertInto" in SQL tab. Could you clarify the issue? It would be great if you can provide a reproducer. > DataFrameWriter operations do not show up in SQL tab > > > Key: SPARK-20213 > URL: https://issues.apache.org/jira/browse/SPARK-20213 > Project: Spark > Issue Type: Bug > Components: SQL, Web UI >Affects Versions: 2.0.2, 2.1.0 >Reporter: Ryan Blue > > In 1.6.1, {{DataFrame}} writes started using {{DataFrameWriter}} actions like > {{insertInto}} would show up in the SQL tab. In 2.0.0 and later, they no > longer do. The problem is that 2.0.0 and later no longer wrap execution with > {{SQLExecution.withNewExecutionId}}, which emits > {{SparkListenerSQLExecutionStart}}. > Here are the relevant parts of the stack traces: > {code:title=Spark 1.6.1} > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) > org.apache.spark.sql.execution.QueryExecution$$anonfun$toRdd$1.apply(QueryExecution.scala:56) > org.apache.spark.sql.execution.QueryExecution$$anonfun$toRdd$1.apply(QueryExecution.scala:56) > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:53) > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:56) > => holding > Monitor(org.apache.spark.sql.hive.HiveContext$QueryExecution@424773807}) > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55) > org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:196) > {code} > {code:title=Spark 2.0.0} > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133) > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114) > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:86) > => holding Monitor(org.apache.spark.sql.execution.QueryExecution@490977924}) > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:86) > org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:301) > {code} > I think this was introduced by > [54d23599|https://github.com/apache/spark/commit/54d23599]. The fix should be > to add withNewExecutionId to > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L610 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20213) DataFrameWriter operations do not show up in SQL tab
[ https://issues.apache.org/jira/browse/SPARK-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957292#comment-15957292 ] Apache Spark commented on SPARK-20213: -- User 'rdblue' has created a pull request for this issue: https://github.com/apache/spark/pull/17540 > DataFrameWriter operations do not show up in SQL tab > > > Key: SPARK-20213 > URL: https://issues.apache.org/jira/browse/SPARK-20213 > Project: Spark > Issue Type: Bug > Components: SQL, Web UI >Affects Versions: 2.0.2, 2.1.0 >Reporter: Ryan Blue > > In 1.6.1, {{DataFrame}} writes started using {{DataFrameWriter}} actions like > {{insertInto}} would show up in the SQL tab. In 2.0.0 and later, they no > longer do. The problem is that 2.0.0 and later no longer wrap execution with > {{SQLExecution.withNewExecutionId}}, which emits > {{SparkListenerSQLExecutionStart}}. > Here are the relevant parts of the stack traces: > {code:title=Spark 1.6.1} > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) > org.apache.spark.sql.execution.QueryExecution$$anonfun$toRdd$1.apply(QueryExecution.scala:56) > org.apache.spark.sql.execution.QueryExecution$$anonfun$toRdd$1.apply(QueryExecution.scala:56) > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:53) > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:56) > => holding > Monitor(org.apache.spark.sql.hive.HiveContext$QueryExecution@424773807}) > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55) > org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:196) > {code} > {code:title=Spark 2.0.0} > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133) > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114) > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:86) > => holding Monitor(org.apache.spark.sql.execution.QueryExecution@490977924}) > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:86) > org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:301) > {code} > I think this was introduced by > [54d23599|https://github.com/apache/spark/commit/54d23599]. The fix should be > to add withNewExecutionId to > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L610 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org