[jira] [Created] (SPARK-48482) dropDuplicates and dropDuplicatesWithinWatermark should accept varargs

2024-05-30 Thread Wei Liu (Jira)
Wei Liu created SPARK-48482: --- Summary: dropDuplicates and dropDuplicatesWithinWatermark should accept varargs Key: SPARK-48482 URL: https://issues.apache.org/jira/browse/SPARK-48482 Project: Spark

[jira] [Created] (SPARK-48480) StreamingQueryListener thread should not be interruptable

2024-05-30 Thread Wei Liu (Jira)
Wei Liu created SPARK-48480: --- Summary: StreamingQueryListener thread should not be interruptable Key: SPARK-48480 URL: https://issues.apache.org/jira/browse/SPARK-48480 Project: Spark Issue Type:

[jira] [Commented] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

2024-05-24 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849348#comment-17849348 ] Wei Liu commented on SPARK-48411: - Sorry I tagged the wrong Yuchen > Add E2E test for

[jira] [Created] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

2024-05-24 Thread Wei Liu (Jira)
Wei Liu created SPARK-48411: --- Summary: Add E2E test for DropDuplicateWithinWatermark Key: SPARK-48411 URL: https://issues.apache.org/jira/browse/SPARK-48411 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-48411) Add E2E test for DropDuplicateWithinWatermark

2024-05-24 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849202#comment-17849202 ] Wei Liu commented on SPARK-48411: - [~liuyuchen777] is going to work on this > Add E2E test for

[jira] [Created] (SPARK-48181) Unify StreamingPythonRunner and PythonPlannerRunner

2024-05-07 Thread Wei Liu (Jira)
Wei Liu created SPARK-48181: --- Summary: Unify StreamingPythonRunner and PythonPlannerRunner Key: SPARK-48181 URL: https://issues.apache.org/jira/browse/SPARK-48181 Project: Spark Issue Type: New

[jira] [Created] (SPARK-48147) Remove all client listeners when local spark session is deleted

2024-05-06 Thread Wei Liu (Jira)
Wei Liu created SPARK-48147: --- Summary: Remove all client listeners when local spark session is deleted Key: SPARK-48147 URL: https://issues.apache.org/jira/browse/SPARK-48147 Project: Spark Issue

[jira] [Created] (SPARK-48093) Add config to switch between client side listener and server side listener

2024-05-02 Thread Wei Liu (Jira)
Wei Liu created SPARK-48093: --- Summary: Add config to switch between client side listener and server side listener Key: SPARK-48093 URL: https://issues.apache.org/jira/browse/SPARK-48093 Project: Spark

[jira] [Created] (SPARK-48002) Add Observed metrics test in PySpark StreamingQueryListeners

2024-04-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-48002: --- Summary: Add Observed metrics test in PySpark StreamingQueryListeners Key: SPARK-48002 URL: https://issues.apache.org/jira/browse/SPARK-48002 Project: Spark Issue

[jira] [Created] (SPARK-47877) Speed up test_parity_listener

2024-04-16 Thread Wei Liu (Jira)
Wei Liu created SPARK-47877: --- Summary: Speed up test_parity_listener Key: SPARK-47877 URL: https://issues.apache.org/jira/browse/SPARK-47877 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-47718) .sql() does not recognize watermark defined upstream

2024-04-09 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-47718: Labels: (was: pull-request-available) > .sql() does not recognize watermark defined upstream >

[jira] [Created] (SPARK-47722) Wait until RocksDB background work finish before closing

2024-04-03 Thread Wei Liu (Jira)
Wei Liu created SPARK-47722: --- Summary: Wait until RocksDB background work finish before closing Key: SPARK-47722 URL: https://issues.apache.org/jira/browse/SPARK-47722 Project: Spark Issue Type:

[jira] [Created] (SPARK-47332) StreamingPythonRunner don't need redundant logic for starting python process

2024-03-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-47332: --- Summary: StreamingPythonRunner don't need redundant logic for starting python process Key: SPARK-47332 URL: https://issues.apache.org/jira/browse/SPARK-47332 Project: Spark

[jira] [Created] (SPARK-47292) safeMapToJValue should consider when map is null

2024-03-05 Thread Wei Liu (Jira)
Wei Liu created SPARK-47292: --- Summary: safeMapToJValue should consider when map is null Key: SPARK-47292 URL: https://issues.apache.org/jira/browse/SPARK-47292 Project: Spark Issue Type: New

[jira] [Created] (SPARK-47277) PySpark util function assertDataFrameEqual should not support streaming DF

2024-03-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-47277: --- Summary: PySpark util function assertDataFrameEqual should not support streaming DF Key: SPARK-47277 URL: https://issues.apache.org/jira/browse/SPARK-47277 Project: Spark

[jira] [Created] (SPARK-47174) Client Side Listener - Server side implementation

2024-02-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-47174: --- Summary: Client Side Listener - Server side implementation Key: SPARK-47174 URL: https://issues.apache.org/jira/browse/SPARK-47174 Project: Spark Issue Type:

[jira] [Commented] (SPARK-47174) Client Side Listener - Server side implementation

2024-02-26 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820828#comment-17820828 ] Wei Liu commented on SPARK-47174: - im working on this > Client Side Listener - Server side

[jira] [Created] (SPARK-47173) fix typo in new streaming query listener explanation

2024-02-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-47173: --- Summary: fix typo in new streaming query listener explanation Key: SPARK-47173 URL: https://issues.apache.org/jira/browse/SPARK-47173 Project: Spark Issue Type:

[jira] [Created] (SPARK-46873) PySpark spark.streams should not recreate new StreamingQueryManager

2024-01-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-46873: --- Summary: PySpark spark.streams should not recreate new StreamingQueryManager Key: SPARK-46873 URL: https://issues.apache.org/jira/browse/SPARK-46873 Project: Spark

[jira] [Closed] (SPARK-44460) Pass user auth credential to Python workers for foreachBatch and listener

2024-01-23 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu closed SPARK-44460. --- > Pass user auth credential to Python workers for foreachBatch and listener >

[jira] [Updated] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46627: Attachment: Screenshot 2024-01-08 at 15.06.24.png > Streaming UI hover-over shows incorrect value >

[jira] [Commented] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804513#comment-17804513 ] Wei Liu commented on SPARK-46627: - Also batch percent doesn't add to 100% now: !Screenshot 2024-01-08

[jira] [Updated] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46627: Attachment: Screenshot 2024-01-08 at 1.55.57 PM.png > Streaming UI hover-over shows incorrect value >

[jira] [Commented] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804481#comment-17804481 ] Wei Liu commented on SPARK-46627: - Hi Kent : ) [~yao]  I was wondering if you have context in this

[jira] [Updated] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46627: Description: Running a simple streaming query: val df =

[jira] [Created] (SPARK-46627) Streaming UI hover-over shows incorrect value

2024-01-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-46627: --- Summary: Streaming UI hover-over shows incorrect value Key: SPARK-46627 URL: https://issues.apache.org/jira/browse/SPARK-46627 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-46384) Streaming UI doesn't display graph correctly

2023-12-12 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46384: Summary: Streaming UI doesn't display graph correctly (was: Streaming UI doesn't show graph) >

[jira] [Updated] (SPARK-46384) Structured Streaming UI doesn't display graph correctly

2023-12-12 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-46384: Summary: Structured Streaming UI doesn't display graph correctly (was: Streaming UI doesn't display

[jira] [Created] (SPARK-46384) Streaming UI doesn't show graph

2023-12-12 Thread Wei Liu (Jira)
Wei Liu created SPARK-46384: --- Summary: Streaming UI doesn't show graph Key: SPARK-46384 URL: https://issues.apache.org/jira/browse/SPARK-46384 Project: Spark Issue Type: Task Components:

[jira] [Created] (SPARK-46250) Deflake test_parity_listener

2023-12-04 Thread Wei Liu (Jira)
Wei Liu created SPARK-46250: --- Summary: Deflake test_parity_listener Key: SPARK-46250 URL: https://issues.apache.org/jira/browse/SPARK-46250 Project: Spark Issue Type: Task Components:

[jira] [Created] (SPARK-45845) Streaming UI add number of evicted state rows

2023-11-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-45845: --- Summary: Streaming UI add number of evicted state rows Key: SPARK-45845 URL: https://issues.apache.org/jira/browse/SPARK-45845 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-45637) Time window aggregation in separate streams followed by stream-stream join not returning results

2023-10-27 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-45637: Description: According to documentation update (SPARK-42591) resulting from SPARK-42376, Spark 3.5.0

[jira] [Created] (SPARK-45677) Observe API error logging

2023-10-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-45677: --- Summary: Observe API error logging Key: SPARK-45677 URL: https://issues.apache.org/jira/browse/SPARK-45677 Project: Spark Issue Type: Task Components:

[jira] [Updated] (SPARK-45053) Improve python version mismatch logging

2023-09-01 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-45053: Description: Currently the syntax of the python version mismatching is a little bit confusing, it uses

[jira] [Created] (SPARK-45056) Add process termination tests for Python foreachBatch and StreamingQueryListener

2023-09-01 Thread Wei Liu (Jira)
Wei Liu created SPARK-45056: --- Summary: Add process termination tests for Python foreachBatch and StreamingQueryListener Key: SPARK-45056 URL: https://issues.apache.org/jira/browse/SPARK-45056 Project:

[jira] [Created] (SPARK-45053) Improve python version mismatch logging

2023-09-01 Thread Wei Liu (Jira)
Wei Liu created SPARK-45053: --- Summary: Improve python version mismatch logging Key: SPARK-45053 URL: https://issues.apache.org/jira/browse/SPARK-45053 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-44971) [BUG Fix] PySpark StreamingQuerProgress fromJson

2023-08-25 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44971: Issue Type: Bug (was: New Feature) > [BUG Fix] PySpark StreamingQuerProgress fromJson >

[jira] [Created] (SPARK-44971) [BUG Fix] PySpark StreamingQuerProgress fromJson

2023-08-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-44971: --- Summary: [BUG Fix] PySpark StreamingQuerProgress fromJson Key: SPARK-44971 URL: https://issues.apache.org/jira/browse/SPARK-44971 Project: Spark Issue Type: New

[jira] [Resolved] (SPARK-44917) PySpark Streaming DataStreamWriter table API

2023-08-22 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-44917. - Resolution: Not A Problem > PySpark Streaming DataStreamWriter table API >

[jira] [Created] (SPARK-44917) PySpark Streaming DataStreamWriter table API

2023-08-22 Thread Wei Liu (Jira)
Wei Liu created SPARK-44917: --- Summary: PySpark Streaming DataStreamWriter table API Key: SPARK-44917 URL: https://issues.apache.org/jira/browse/SPARK-44917 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-44460) Pass user auth credential to Python workers for foreachBatch and listener

2023-08-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17757087#comment-17757087 ] Wei Liu commented on SPARK-44460: - [~rangadi] This seems to be a Databricks internal issue. See the

[jira] [Created] (SPARK-44839) Better error logging when user accesses spark session in foreachBatch and Listener

2023-08-16 Thread Wei Liu (Jira)
Wei Liu created SPARK-44839: --- Summary: Better error logging when user accesses spark session in foreachBatch and Listener Key: SPARK-44839 URL: https://issues.apache.org/jira/browse/SPARK-44839 Project:

[jira] [Commented] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754280#comment-17754280 ] Wei Liu commented on SPARK-44808: - This seems to be against the design principle of spark connect – We

[jira] [Resolved] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-44808. - Resolution: Won't Do > refreshListener() API on StreamingQueryManager for spark connect >

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for connect python listener and foreachBatch. Currently if

[jira] [Created] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
Wei Liu created SPARK-44808: --- Summary: refreshListener() API on StreamingQueryManager for spark connect Key: SPARK-44808 URL: https://issues.apache.org/jira/browse/SPARK-44808 Project: Spark

[jira] [Updated] (SPARK-44808) refreshListener() API on StreamingQueryManager for spark connect

2023-08-14 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44808: Description: I’m thinking of an improvement for python listener and foreachBatch. Currently if you

[jira] [Created] (SPARK-44764) Streaming process improvement

2023-08-10 Thread Wei Liu (Jira)
Wei Liu created SPARK-44764: --- Summary: Streaming process improvement Key: SPARK-44764 URL: https://issues.apache.org/jira/browse/SPARK-44764 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-44460) Pass user auth credential to Python workers for foreachBatch and listener

2023-07-28 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44460: Description: No user specific credentials are sent to Python worker that runs user functions like

[jira] [Updated] (SPARK-44516) Spark Connect Python StreamingQueryListener removeListener method actually shut down the listener process

2023-07-24 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44516: Summary: Spark Connect Python StreamingQueryListener removeListener method actually shut down the

[jira] [Created] (SPARK-44516) Spark Connect Python StreamingQueryListener removeListener method

2023-07-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-44516: --- Summary: Spark Connect Python StreamingQueryListener removeListener method Key: SPARK-44516 URL: https://issues.apache.org/jira/browse/SPARK-44516 Project: Spark

[jira] [Created] (SPARK-44515) Code Improvement: PySpark add util function to set python version

2023-07-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-44515: --- Summary: Code Improvement: PySpark add util function to set python version Key: SPARK-44515 URL: https://issues.apache.org/jira/browse/SPARK-44515 Project: Spark

[jira] [Created] (SPARK-44502) Add mission versionchanged field to docs

2023-07-20 Thread Wei Liu (Jira)
Wei Liu created SPARK-44502: --- Summary: Add mission versionchanged field to docs Key: SPARK-44502 URL: https://issues.apache.org/jira/browse/SPARK-44502 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-44484) Add missing json field batchDuration to StreamingQueryProgress

2023-07-19 Thread Wei Liu (Jira)
Wei Liu created SPARK-44484: --- Summary: Add missing json field batchDuration to StreamingQueryProgress Key: SPARK-44484 URL: https://issues.apache.org/jira/browse/SPARK-44484 Project: Spark Issue

[jira] [Commented] (SPARK-42962) Allow access to stopped streaming queries

2023-06-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735867#comment-17735867 ] Wei Liu commented on SPARK-42962: - Closing this as it's duplicate with SPARK-42940. Please kindly reopen

[jira] [Resolved] (SPARK-42962) Allow access to stopped streaming queries

2023-06-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-42962. - Resolution: Duplicate > Allow access to stopped streaming queries >

[jira] [Commented] (SPARK-42941) Add support for streaming listener in Python

2023-06-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735866#comment-17735866 ] Wei Liu commented on SPARK-42941: - Hmmm this is actually still in progress. I was thinking two PRs could

[jira] [Reopened] (SPARK-42941) Add support for streaming listener in Python

2023-06-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu reopened SPARK-42941: - > Add support for streaming listener in Python > > >

[jira] [Created] (SPARK-44046) Pyspark StreamingQueryListener listListener

2023-06-13 Thread Wei Liu (Jira)
Wei Liu created SPARK-44046: --- Summary: Pyspark StreamingQueryListener listListener Key: SPARK-44046 URL: https://issues.apache.org/jira/browse/SPARK-44046 Project: Spark Issue Type: New Feature

[jira] [Resolved] (SPARK-44029) Observation.py supports streaming dataframes

2023-06-12 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-44029. - Resolution: Not A Problem > Observation.py supports streaming dataframes >

[jira] [Created] (SPARK-44029) Observation.py supports streaming dataframes

2023-06-12 Thread Wei Liu (Jira)
Wei Liu created SPARK-44029: --- Summary: Observation.py supports streaming dataframes Key: SPARK-44029 URL: https://issues.apache.org/jira/browse/SPARK-44029 Project: Spark Issue Type: Documentation

[jira] [Created] (SPARK-44010) Python StreamingQueryProgress rowsPerSecond type fix

2023-06-08 Thread Wei Liu (Jira)
Wei Liu created SPARK-44010: --- Summary: Python StreamingQueryProgress rowsPerSecond type fix Key: SPARK-44010 URL: https://issues.apache.org/jira/browse/SPARK-44010 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-43796) Streaming ForeachWriter can't accept custom user defined class

2023-05-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-43796: --- Summary: Streaming ForeachWriter can't accept custom user defined class Key: SPARK-43796 URL: https://issues.apache.org/jira/browse/SPARK-43796 Project: Spark Issue

[jira] [Created] (SPARK-43761) Streaming ForeachWriter with UnboundRowEncoder

2023-05-23 Thread Wei Liu (Jira)
Wei Liu created SPARK-43761: --- Summary: Streaming ForeachWriter with UnboundRowEncoder Key: SPARK-43761 URL: https://issues.apache.org/jira/browse/SPARK-43761 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-42941) Add support for streaming listener in Python

2023-05-04 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-42941: Summary: Add support for streaming listener in Python (was: Add support for streaming listener.) > Add

[jira] [Created] (SPARK-43360) Scala Connect: Add StreamingQueryManager API

2023-05-03 Thread Wei Liu (Jira)
Wei Liu created SPARK-43360: --- Summary: Scala Connect: Add StreamingQueryManager API Key: SPARK-43360 URL: https://issues.apache.org/jira/browse/SPARK-43360 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-43132) Add foreach streaming API in Python

2023-04-27 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717347#comment-17717347 ] Wei Liu commented on SPARK-43132: - i'm working on this   > Add foreach streaming API in Python >

[jira] [Resolved] (SPARK-43054) Support foreach() in streaming spark connect

2023-04-27 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-43054. - Resolution: Duplicate duplicate with 43132 and 43133   > Support foreach() in streaming spark connect

[jira] [Created] (SPARK-43299) JVM Client throw StreamingQueryException when error handling is implemented

2023-04-26 Thread Wei Liu (Jira)
Wei Liu created SPARK-43299: --- Summary: JVM Client throw StreamingQueryException when error handling is implemented Key: SPARK-43299 URL: https://issues.apache.org/jira/browse/SPARK-43299 Project: Spark

[jira] [Updated] (SPARK-43287) Connect JVM client REPL not correctly shut down if killed

2023-04-25 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-43287: Description: How to reproduce: # Start a scala client

[jira] [Updated] (SPARK-43287) Connect JVM client REPL not correctly shut down if killed

2023-04-25 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-43287: Description: How to reproduce: # Start a scala client

[jira] [Updated] (SPARK-43287) Connect JVM client REPL not correctly shut down if killed

2023-04-25 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-43287: Description: How to reproduce: # Start a scala client

[jira] [Created] (SPARK-43287) Connect JVM client REPL not correctly shut down if killed

2023-04-25 Thread Wei Liu (Jira)
Wei Liu created SPARK-43287: --- Summary: Connect JVM client REPL not correctly shut down if killed Key: SPARK-43287 URL: https://issues.apache.org/jira/browse/SPARK-43287 Project: Spark Issue Type:

[jira] [Updated] (SPARK-43206) Connect Better StreamingQueryException

2023-04-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-43206: Summary: Connect Better StreamingQueryException (was: Streaming query exception() also include stack

[jira] [Commented] (SPARK-43206) Connect Better StreamingQueryException

2023-04-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715178#comment-17715178 ] Wei Liu commented on SPARK-43206: - Also cause, offsets, stack trace...   > Connect Better

[jira] [Commented] (SPARK-43206) Streaming query exception() also include stack trace

2023-04-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715171#comment-17715171 ] Wei Liu commented on SPARK-43206: - I'll work on this. To myself: don't forget jvm exceptions   >

[jira] [Commented] (SPARK-43032) Add StreamingQueryManager API

2023-04-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715147#comment-17715147 ] Wei Liu commented on SPARK-43032: - [https://github.com/apache/spark/pull/40861] still draft   > Add

[jira] [Commented] (SPARK-43143) Scala: Add StreamingQuery awaitTermination() API

2023-04-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715149#comment-17715149 ] Wei Liu commented on SPARK-43143: - I'm working on this   > Scala: Add StreamingQuery

[jira] [Commented] (SPARK-43134) Add streaming query exception API in Scala

2023-04-21 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715148#comment-17715148 ] Wei Liu commented on SPARK-43134: - I'm working on this   > Add streaming query exception API in Scala

[jira] [Updated] (SPARK-43206) Streaming query exception() also include stack trace

2023-04-19 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-43206: Description: [https://github.com/apache/spark/pull/40785#issuecomment-1515522281]     > Streaming

[jira] [Updated] (SPARK-43206) Streaming query exception() also include stack trace

2023-04-19 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-43206: Epic Link: SPARK-42938 > Streaming query exception() also include stack trace >

[jira] [Resolved] (SPARK-43167) Streaming Connect console output format support

2023-04-19 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-43167. - Resolution: Not A Problem automatically supported with existing Connect implementation   > Streaming

[jira] [Updated] (SPARK-43206) Streaming query exception() also include stack trace

2023-04-19 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-43206: Environment: (was: https://github.com/apache/spark/pull/40785#issuecomment-1515522281  ) >

[jira] [Created] (SPARK-43206) Streaming query exception() also include stack trace

2023-04-19 Thread Wei Liu (Jira)
Wei Liu created SPARK-43206: --- Summary: Streaming query exception() also include stack trace Key: SPARK-43206 URL: https://issues.apache.org/jira/browse/SPARK-43206 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-43167) Streaming Connect console output format support

2023-04-18 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713787#comment-17713787 ] Wei Liu commented on SPARK-43167: - Should be: ``` Welcome to                     __      / __/__ 

[jira] [Created] (SPARK-43167) Streaming Connect console output format support

2023-04-17 Thread Wei Liu (Jira)
Wei Liu created SPARK-43167: --- Summary: Streaming Connect console output format support Key: SPARK-43167 URL: https://issues.apache.org/jira/browse/SPARK-43167 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-43147) Python lint local config

2023-04-14 Thread Wei Liu (Jira)
Wei Liu created SPARK-43147: --- Summary: Python lint local config Key: SPARK-43147 URL: https://issues.apache.org/jira/browse/SPARK-43147 Project: Spark Issue Type: Task Components:

[jira] [Commented] (SPARK-42962) Allow access to stopped streaming queries

2023-04-11 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711155#comment-17711155 ] Wei Liu commented on SPARK-42962: - Please be aware of the TODOs in connect query.py and readwriter.py

[jira] [Comment Edited] (SPARK-43015) Python: await_termination & exception() for Streaming Connect

2023-04-06 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709070#comment-17709070 ] Wei Liu edited comment on SPARK-43015 at 4/6/23 10:05 PM: -- Duplicate with

[jira] [Created] (SPARK-43054) Support foreach() in streaming spark connect

2023-04-06 Thread Wei Liu (Jira)
Wei Liu created SPARK-43054: --- Summary: Support foreach() in streaming spark connect Key: SPARK-43054 URL: https://issues.apache.org/jira/browse/SPARK-43054 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-42951) Spark Connect: Streaming DataStreamReader API except table()

2023-04-06 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-42951: Issue Type: Task (was: Story) > Spark Connect: Streaming DataStreamReader API except table() >

[jira] [Updated] (SPARK-43042) Spark Connect: Streaming readerwriter table() API

2023-04-06 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-43042: Issue Type: Task (was: Story) > Spark Connect: Streaming readerwriter table() API >

[jira] [Updated] (SPARK-43042) Spark Connect: Streaming readerwriter table() API

2023-04-05 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-43042: Summary: Spark Connect: Streaming readerwriter table() API (was: Spark Connect: Streaming

[jira] [Updated] (SPARK-42951) Spark Connect: Streaming DataStreamReader API except table()

2023-04-05 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-42951: Affects Version/s: (was: 3.4.0) > Spark Connect: Streaming DataStreamReader API except table() >

[jira] [Created] (SPARK-43042) Spark Connect: Streaming DataStreamReader table() API

2023-04-05 Thread Wei Liu (Jira)
Wei Liu created SPARK-43042: --- Summary: Spark Connect: Streaming DataStreamReader table() API Key: SPARK-43042 URL: https://issues.apache.org/jira/browse/SPARK-43042 Project: Spark Issue Type:

[jira] [Updated] (SPARK-42951) Spark Connect: Streaming DataStreamReader API except table()

2023-04-05 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-42951: Epic Link: SPARK-42938 > Spark Connect: Streaming DataStreamReader API except table() >

[jira] [Updated] (SPARK-42960) Add remaining Streaming Query commands like await_termination()

2023-04-05 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-42960: Summary: Add remaining Streaming Query commands like await_termination() (was: Add remaining Streaming

  1   2   >