[jira] [Updated] (SPARK-49146) Move assertion errors related to watermark missing in append mode streaming queries to error framework
[ https://issues.apache.org/jira/browse/SPARK-49146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Gao updated SPARK-49146: --- Summary: Move assertion errors related to watermark missing in append mode streaming queries to error framework (was: Move assertion errors related to watermarks to error framework) > Move assertion errors related to watermark missing in append mode streaming > queries to error framework > -- > > Key: SPARK-49146 > URL: https://issues.apache.org/jira/browse/SPARK-49146 > Project: Spark > Issue Type: Task > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Bo Gao >Priority: Major > > This is a followup for https://issues.apache.org/jira/browse/SPARK-45539. The > errors added there should be classified as user errors. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-49146) Move assertion errors related to watermarks to error framework
[ https://issues.apache.org/jira/browse/SPARK-49146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Gao updated SPARK-49146: --- Description: This is a followup for https://issues.apache.org/jira/browse/SPARK-45539. The errors added there should be classified as user errors. > Move assertion errors related to watermarks to error framework > -- > > Key: SPARK-49146 > URL: https://issues.apache.org/jira/browse/SPARK-49146 > Project: Spark > Issue Type: Task > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Bo Gao >Priority: Major > > This is a followup for https://issues.apache.org/jira/browse/SPARK-45539. The > errors added there should be classified as user errors. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-49146) Move assertion errors related to watermarks to error framework
Bo Gao created SPARK-49146: -- Summary: Move assertion errors related to watermarks to error framework Key: SPARK-49146 URL: https://issues.apache.org/jira/browse/SPARK-49146 Project: Spark Issue Type: Task Components: Structured Streaming Affects Versions: 4.0.0 Reporter: Bo Gao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-49100) [Python State V2] Add verification for result iterator of transformWithState UDF
[ https://issues.apache.org/jira/browse/SPARK-49100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Gao updated SPARK-49100: --- Description: add verification that elements in result_iter for are indeed of type pd.DataFrame and confirm to assigned cols > [Python State V2] Add verification for result iterator of transformWithState > UDF > > > Key: SPARK-49100 > URL: https://issues.apache.org/jira/browse/SPARK-49100 > Project: Spark > Issue Type: Task > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Bo Gao >Priority: Major > > add verification that elements in result_iter for are indeed of type > pd.DataFrame and confirm to assigned cols -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48755) [Python State V2] Base implementation and ValueState support
Bo Gao created SPARK-48755: -- Summary: [Python State V2] Base implementation and ValueState support Key: SPARK-48755 URL: https://issues.apache.org/jira/browse/SPARK-48755 Project: Spark Issue Type: Task Components: Structured Streaming Affects Versions: 4.0.0 Reporter: Bo Gao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46963) Verify AQE is not enabled for Structured Streaming
[ https://issues.apache.org/jira/browse/SPARK-46963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Gao resolved SPARK-46963. Resolution: Won't Do > Verify AQE is not enabled for Structured Streaming > -- > > Key: SPARK-46963 > URL: https://issues.apache.org/jira/browse/SPARK-46963 > Project: Spark > Issue Type: Task > Components: Structured Streaming >Affects Versions: 4.0.0 >Reporter: Bo Gao >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46963) Verify AQE is not enabled for Structured Streaming
Bo Gao created SPARK-46963: -- Summary: Verify AQE is not enabled for Structured Streaming Key: SPARK-46963 URL: https://issues.apache.org/jira/browse/SPARK-46963 Project: Spark Issue Type: Task Components: Structured Streaming Affects Versions: 4.0.0 Reporter: Bo Gao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44877) Support python protobuf functions for Spark Connect
Bo Gao created SPARK-44877: -- Summary: Support python protobuf functions for Spark Connect Key: SPARK-44877 URL: https://issues.apache.org/jira/browse/SPARK-44877 Project: Spark Issue Type: Task Components: Connect Affects Versions: 3.5.0 Reporter: Bo Gao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44626) Followup on streaming query termination when client session is timed out for Spark Connect
Bo Gao created SPARK-44626: -- Summary: Followup on streaming query termination when client session is timed out for Spark Connect Key: SPARK-44626 URL: https://issues.apache.org/jira/browse/SPARK-44626 Project: Spark Issue Type: Task Components: Connect, Structured Streaming Affects Versions: 3.5.0 Reporter: Bo Gao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44434) Add more tests for Scala foreachBatch and streaming listeners
[ https://issues.apache.org/jira/browse/SPARK-44434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Gao updated SPARK-44434: --- Summary: Add more tests for Scala foreachBatch and streaming listeners (was: Add more tests for Scala foreachBatch and streaming listers ) > Add more tests for Scala foreachBatch and streaming listeners > -- > > Key: SPARK-44434 > URL: https://issues.apache.org/jira/browse/SPARK-44434 > Project: Spark > Issue Type: Task > Components: Connect, Structured Streaming >Affects Versions: 3.4.1 >Reporter: Raghu Angadi >Priority: Major > Fix For: 3.5.0 > > > Currently there are very few tests for Scala foreachBatch. Consider adding > more tests and covering more test scenarios (multiple queries etc). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44436) Session improvement for Scala foreachBatch
Bo Gao created SPARK-44436: -- Summary: Session improvement for Scala foreachBatch Key: SPARK-44436 URL: https://issues.apache.org/jira/browse/SPARK-44436 Project: Spark Issue Type: Task Components: Connect, Structured Streaming Affects Versions: 3.5.0 Reporter: Bo Gao Improve Scala foreachBatch to set up a Spark Connect session and use Spark Connect DataFrame instead of legacy DataFrame -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44400) Improve Scala StreamingQueryListener to provide users a way to access the Spark session for Spark Connect
[ https://issues.apache.org/jira/browse/SPARK-44400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Gao updated SPARK-44400: --- Description: Improve the Listener to provide users a way to access the Spark session and perform arbitrary actions inside the Listener. Right now users can use `val spark = SparkSession.builder.getOrCreate()` to create a Spark session inside the Listener, but this is a legacy session instead of a connect remote session. > Improve Scala StreamingQueryListener to provide users a way to access the > Spark session for Spark Connect > - > > Key: SPARK-44400 > URL: https://issues.apache.org/jira/browse/SPARK-44400 > Project: Spark > Issue Type: Task > Components: Connect, Structured Streaming >Affects Versions: 3.5.0 >Reporter: Bo Gao >Priority: Major > > Improve the Listener to provide users a way to access the Spark session and > perform arbitrary actions inside the Listener. Right now users can use `val > spark = SparkSession.builder.getOrCreate()` to create a Spark session inside > the Listener, but this is a legacy session instead of a connect remote > session. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44400) Improve Scala StreamingQueryListener to provide users a way to access the Spark session for Spark Connect
Bo Gao created SPARK-44400: -- Summary: Improve Scala StreamingQueryListener to provide users a way to access the Spark session for Spark Connect Key: SPARK-44400 URL: https://issues.apache.org/jira/browse/SPARK-44400 Project: Spark Issue Type: Task Components: Connect, Structured Streaming Affects Versions: 3.5.0 Reporter: Bo Gao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44201) Add support for Streaming Listener in Scala for Spark Connect
Bo Gao created SPARK-44201: -- Summary: Add support for Streaming Listener in Scala for Spark Connect Key: SPARK-44201 URL: https://issues.apache.org/jira/browse/SPARK-44201 Project: Spark Issue Type: Task Components: Structured Streaming Affects Versions: 3.5.0 Reporter: Bo Gao -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44136) StateManager may get materialized in executor instead of driver in FlatMapGroupsWithStateExec
Bo Gao created SPARK-44136: -- Summary: StateManager may get materialized in executor instead of driver in FlatMapGroupsWithStateExec Key: SPARK-44136 URL: https://issues.apache.org/jira/browse/SPARK-44136 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 3.3.0 Reporter: Bo Gao StateManager may get materialized in executor instead of driver in FlatMapGroupsWithStateExec because of a previous change https://issues.apache.org/jira/browse/SPARK-40411 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-43511) Implemented State APIs for Spark Connect Scala
[ https://issues.apache.org/jira/browse/SPARK-43511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722891#comment-17722891 ] Bo Gao edited comment on SPARK-43511 at 6/12/23 6:59 PM: - Created PR [https://github.com/apache/spark/pull/41558] was (Author: JIRAUSER300429): Created PR https://github.com/apache/spark/pull/40959 > Implemented State APIs for Spark Connect Scala > -- > > Key: SPARK-43511 > URL: https://issues.apache.org/jira/browse/SPARK-43511 > Project: Spark > Issue Type: Task > Components: Connect, Structured Streaming >Affects Versions: 3.5.0 >Reporter: Bo Gao >Priority: Major > > Implemented MapGroupsWithState and FlatMapGroupsWithState APIs for Spark > Connect Scala -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-43511) Implemented State APIs for Spark Connect Scala
[ https://issues.apache.org/jira/browse/SPARK-43511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722891#comment-17722891 ] Bo Gao commented on SPARK-43511: Created PR https://github.com/apache/spark/pull/40959 > Implemented State APIs for Spark Connect Scala > -- > > Key: SPARK-43511 > URL: https://issues.apache.org/jira/browse/SPARK-43511 > Project: Spark > Issue Type: Task > Components: Connect, Structured Streaming >Affects Versions: 3.5.0 >Reporter: Bo Gao >Priority: Major > > Implemented MapGroupsWithState and FlatMapGroupsWithState APIs for Spark > Connect Scala -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43511) Implemented State APIs for Spark Connect Scala
Bo Gao created SPARK-43511: -- Summary: Implemented State APIs for Spark Connect Scala Key: SPARK-43511 URL: https://issues.apache.org/jira/browse/SPARK-43511 Project: Spark Issue Type: Task Components: Connect, Structured Streaming Affects Versions: 3.5.0 Reporter: Bo Gao Implemented MapGroupsWithState and FlatMapGroupsWithState APIs for Spark Connect Scala -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
