[jira] [Commented] (SPARK-43404) Filter current version while reusing sst files for RocksDB state store provider while uploading to DFS to prevent id mismatch

2023-05-07 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720390#comment-17720390 ] Anish Shrigondekar commented on SPARK-43404: PR here: [https://github.com/ap

[jira] [Created] (SPARK-43404) Filter current version while reusing sst files for RocksDB state store provider while uploading to DFS to prevent id mismatch

2023-05-07 Thread Anish Shrigondekar (Jira)
Anish Shrigondekar created SPARK-43404: -- Summary: Filter current version while reusing sst files for RocksDB state store provider while uploading to DFS to prevent id mismatch Key: SPARK-43404 URL: https://is

[jira] [Updated] (SPARK-43403) GET /history//1/jobs/ failed: java.lang.IllegalStateException: DB is closed

2023-05-07 Thread Zhou Yifan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhou Yifan updated SPARK-43403: --- Attachment: image-2023-05-08-11-33-13-634.png > GET /history//1/jobs/ failed: java.lang.IllegalState

[jira] [Updated] (SPARK-43403) GET /history//1/jobs/ failed: java.lang.IllegalStateException: DB is closed

2023-05-07 Thread Zhou Yifan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhou Yifan updated SPARK-43403: --- Description: !image-2023-05-08-11-33-13-634.png! (was: !image-2023-05-08-11-25-18-132.png!) > GET

[jira] [Created] (SPARK-43403) GET /history//1/jobs/ failed: java.lang.IllegalStateException: DB is closed

2023-05-07 Thread Zhou Yifan (Jira)
Zhou Yifan created SPARK-43403: -- Summary: GET /history//1/jobs/ failed: java.lang.IllegalStateException: DB is closed Key: SPARK-43403 URL: https://issues.apache.org/jira/browse/SPARK-43403 Project: Spar

[jira] [Created] (SPARK-43402) FileSourceScanExec supports push down data filter with scalar subquery

2023-05-07 Thread XiDuo You (Jira)
XiDuo You created SPARK-43402: - Summary: FileSourceScanExec supports push down data filter with scalar subquery Key: SPARK-43402 URL: https://issues.apache.org/jira/browse/SPARK-43402 Project: Spark

[jira] [Commented] (SPARK-38471) Use error classes in org.apache.spark.rdd

2023-05-07 Thread Bo Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720380#comment-17720380 ] Bo Zhang commented on SPARK-38471: -- Hi [~blindcat] , thanks for working on this! I don

[jira] [Assigned] (SPARK-43372) Use ; instead of && when enable set -ex

2023-05-07 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang reassigned SPARK-43372: --- Assignee: Yikun Jiang > Use ; instead of && when enable set -ex > -

[jira] [Resolved] (SPARK-43372) Use ; instead of && when enable set -ex

2023-05-07 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-43372. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 38 [https://github.

[jira] [Created] (SPARK-43401) Upgrade buf to v1.18.0

2023-05-07 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-43401: --- Summary: Upgrade buf to v1.18.0 Key: SPARK-43401 URL: https://issues.apache.org/jira/browse/SPARK-43401 Project: Spark Issue Type: Improvement Compon

[jira] [Updated] (SPARK-43354) Re-enable test_create_dataframe_from_pandas_with_day_time_interval

2023-05-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43354: - Description: This test fails with PyPy 3.8. > Re-enable test_create_dataframe_from_pandas_with_d

[jira] [Updated] (SPARK-43325) regexp_extract_all DataFrame API

2023-05-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43325: - Fix Version/s: (was: 3.4.1) > regexp_extract_all DataFrame API > --

[jira] [Updated] (SPARK-43384) Make `df.show` print a nice string for MapType

2023-05-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43384: - Fix Version/s: (was: 3.5.0) > Make `df.show` print a nice string for MapType > -

[jira] [Commented] (SPARK-43389) spark.read.csv throws NullPointerException when lineSep is set to None

2023-05-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720371#comment-17720371 ] Hyukjin Kwon commented on SPARK-43389: -- [~zach liu] you can simply just don't speci

[jira] [Updated] (SPARK-43386) Improve list of suggested column/attributes in `UNRESOLVED_COLUMN.WITH_SUGGESTION` error class

2023-05-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43386: - Fix Version/s: (was: 3.5.0) > Improve list of suggested column/attributes in > `UNRESOLVED_

[jira] [Updated] (SPARK-43385) The Generator's statistics should be ratio times greater than the child nodes

2023-05-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43385: - Fix Version/s: (was: 3.5.0) > The Generator's statistics should be ratio times greater than

[jira] [Resolved] (SPARK-43364) Add docs for rocksdb state store memory mgmt to structured streaming guide

2023-05-07 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-43364. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41042 [https://gi

[jira] [Assigned] (SPARK-43364) Add docs for rocksdb state store memory mgmt to structured streaming guide

2023-05-07 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-43364: Assignee: Anish Shrigondekar > Add docs for rocksdb state store memory mgmt to structured

[jira] [Commented] (SPARK-43388) Latest docker Spark image has critical CVE

2023-05-07 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720368#comment-17720368 ] Yikun Jiang commented on SPARK-43388: - Thanks for report. - The critical issue is re

[jira] [Created] (SPARK-43400) create table support the PRIMARY KEY keyword

2023-05-07 Thread melin (Jira)
melin created SPARK-43400: - Summary: create table support the PRIMARY KEY keyword Key: SPARK-43400 URL: https://issues.apache.org/jira/browse/SPARK-43400 Project: Spark Issue Type: Bug Comp

[jira] [Resolved] (SPARK-43362) Special handling of JSON type for MySQL connector

2023-05-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43362. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41040 [https://gi

[jira] [Assigned] (SPARK-43362) Special handling of JSON type for MySQL connector

2023-05-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43362: Assignee: Tianhan Hu > Special handling of JSON type for MySQL connector > --

[jira] [Updated] (SPARK-43398) Executor timeout should be max of idleTimeout rddTimeout shuffleTimeout

2023-05-07 Thread Zhongwei Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhongwei Zhu updated SPARK-43398: - Affects Version/s: (was: 3.4.0) > Executor timeout should be max of idleTimeout rddTimeout s

[jira] [Updated] (SPARK-43398) Executor timeout should be max of idleTimeout rddTimeout shuffleTimeout

2023-05-07 Thread Zhongwei Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhongwei Zhu updated SPARK-43398: - Affects Version/s: 3.0.0 > Executor timeout should be max of idleTimeout rddTimeout shuffleTimeo

[jira] [Updated] (SPARK-37829) An outer-join using joinWith on DataFrames returns Rows with null fields instead of null values

2023-05-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37829: -- Fix Version/s: 3.3.3 > An outer-join using joinWith on DataFrames returns Rows with null field

[jira] [Resolved] (SPARK-43181) spark-sql console should display the Spark WEB UI address

2023-05-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43181. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40844 [https://

[jira] [Assigned] (SPARK-43181) spark-sql console should display the Spark WEB UI address

2023-05-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43181: - Assignee: BingKun Pan > spark-sql console should display the Spark WEB UI address > ---

[jira] [Updated] (SPARK-43181) Show UI WebURL in spark-sql shell

2023-05-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-43181: -- Summary: Show UI WebURL in spark-sql shell (was: spark-sql console should display the Spark W

[jira] [Resolved] (SPARK-42780) Upgrade google Tink to 1.9.0

2023-05-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-42780. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40878 [https://

[jira] [Assigned] (SPARK-42780) Upgrade google Tink to 1.9.0

2023-05-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-42780: - Assignee: Bjørn Jørgensen > Upgrade google Tink to 1.9.0 >

[jira] [Created] (SPARK-43399) Add config to control threshold of unregister map output when fetch failed

2023-05-07 Thread Zhongwei Zhu (Jira)
Zhongwei Zhu created SPARK-43399: Summary: Add config to control threshold of unregister map output when fetch failed Key: SPARK-43399 URL: https://issues.apache.org/jira/browse/SPARK-43399 Project: S

[jira] [Commented] (SPARK-38471) Use error classes in org.apache.spark.rdd

2023-05-07 Thread xin chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720353#comment-17720353 ] xin chen commented on SPARK-38471: -- Hey [~bozhang] I'm currently working on this ticket

[jira] [Created] (SPARK-43398) Executor timeout should be max of idleTimeout rddTimeout shuffleTimeout

2023-05-07 Thread Zhongwei Zhu (Jira)
Zhongwei Zhu created SPARK-43398: Summary: Executor timeout should be max of idleTimeout rddTimeout shuffleTimeout Key: SPARK-43398 URL: https://issues.apache.org/jira/browse/SPARK-43398 Project: Spar

[jira] [Updated] (SPARK-39280) Speed up Timestamp type inference with user-provided format in JSON/CSV data source

2023-05-07 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-39280: Description: The optimization of {{DefaultTimestampFormatter}} has been implemented in [#36562|https://git

[jira] [Resolved] (SPARK-43394) Upgrade maven to 3.8.8

2023-05-07 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-43394. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41073 [https://gi

[jira] [Assigned] (SPARK-43394) Upgrade maven to 3.8.8

2023-05-07 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-43394: Assignee: Cheng Pan > Upgrade maven to 3.8.8 > -- > >

[jira] [Updated] (SPARK-39280) Speed up Timestamp type inference with user-provided format in JSON/CSV data source

2023-05-07 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-39280: - Summary: Speed up Timestamp type inference with user-provided format in JSON/CSV data source (w

[jira] [Updated] (SPARK-39280) Speed up Timestamp type inference with user-provided format in JSON/CSV data source

2023-05-07 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-39280: - Priority: Minor (was: Major) > Speed up Timestamp type inference with user-provided format in J

[jira] [Comment Edited] (SPARK-43382) Read and write csv and json files. Archive files such as zip or gz are supported

2023-05-07 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720110#comment-17720110 ] melin edited comment on SPARK-43382 at 5/7/23 8:35 AM: --- There is a

[jira] [Comment Edited] (SPARK-43382) Read and write csv and json files. Archive files such as zip or gz are supported

2023-05-07 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720110#comment-17720110 ] melin edited comment on SPARK-43382 at 5/7/23 8:34 AM: --- There is a

[jira] [Updated] (SPARK-43382) Read and write csv and json files. Archive files such as zip or gz are supported

2023-05-07 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-43382: -- Description: snowflake data import and export, support fixed files. For example:   {code:java} COPY INTO @mys