[jira] [Commented] (SPARK-42966) Make memory stream implement SupportsReportStatistics

2023-03-29 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706520#comment-17706520 ] Shixiong Zhu commented on SPARK-42966: -- cc [~kabhwan]  > Make memory stream implement

[jira] [Created] (SPARK-42966) Make memory stream implement SupportsReportStatistics

2023-03-29 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-42966: Summary: Make memory stream implement SupportsReportStatistics Key: SPARK-42966 URL: https://issues.apache.org/jira/browse/SPARK-42966 Project: Spark Issue

[jira] [Created] (SPARK-41885) --packages may not work on Windows 11

2023-01-04 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-41885: Summary: --packages may not work on Windows 11 Key: SPARK-41885 URL: https://issues.apache.org/jira/browse/SPARK-41885 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-41040) Self-union streaming query may fail when using readStream.table

2022-11-08 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-41040. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-41045) Pre-compute to eliminate ScalaReflection calls after deserializer is created

2022-11-08 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-41045. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-41045) Pre-compute to eliminate ScalaReflection calls after deserializer is created

2022-11-07 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-41045: Assignee: Shixiong Zhu > Pre-compute to eliminate ScalaReflection calls after

[jira] [Created] (SPARK-41045) Pre-compute to eliminate ScalaReflection calls after deserializer is created

2022-11-07 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-41045: Summary: Pre-compute to eliminate ScalaReflection calls after deserializer is created Key: SPARK-41045 URL: https://issues.apache.org/jira/browse/SPARK-41045

[jira] [Assigned] (SPARK-41040) Self-union streaming query may fail when using readStream.table

2022-11-07 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-41040: Assignee: Shixiong Zhu > Self-union streaming query may fail when using readStream.table

[jira] [Created] (SPARK-41040) Self-union streaming query may fail when using readStream.table

2022-11-07 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-41040: Summary: Self-union streaming query may fail when using readStream.table Key: SPARK-41040 URL: https://issues.apache.org/jira/browse/SPARK-41040 Project: Spark

[jira] [Commented] (SPARK-40937) Support programming APIs for update/delete/merge

2022-10-27 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625249#comment-17625249 ] Shixiong Zhu commented on SPARK-40937: -- cc [~cloud_fan]  > Support programming APIs for

[jira] [Created] (SPARK-40937) Support programming APIs for update/delete/merge

2022-10-27 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-40937: Summary: Support programming APIs for update/delete/merge Key: SPARK-40937 URL: https://issues.apache.org/jira/browse/SPARK-40937 Project: Spark Issue Type:

[jira] [Commented] (SPARK-39915) Dataset.repartition(N) may not create N partitions

2022-08-20 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17582457#comment-17582457 ] Shixiong Zhu commented on SPARK-39915: -- Yeah. I would consider this is a bug since the doc of

[jira] [Commented] (SPARK-39915) Dataset.repartition(N) may not create N partitions

2022-08-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576319#comment-17576319 ] Shixiong Zhu commented on SPARK-39915: -- {code:java} Welcome to __ /

[jira] [Commented] (SPARK-39915) Dataset.repartition(N) may not create N partitions

2022-07-28 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572685#comment-17572685 ] Shixiong Zhu commented on SPARK-39915: -- cc [~cloud_fan]  > Dataset.repartition(N) may not create N

[jira] [Created] (SPARK-39915) Dataset.repartition(N) may not create N partitions

2022-07-28 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-39915: Summary: Dataset.repartition(N) may not create N partitions Key: SPARK-39915 URL: https://issues.apache.org/jira/browse/SPARK-39915 Project: Spark Issue

[jira] [Resolved] (SPARK-34978) Support time-travel SQL syntax

2022-06-30 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-34978. -- Resolution: Duplicate This has been added by SPARK-37219 > Support time-travel SQL syntax >

[jira] [Updated] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

2022-01-31 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-38080: - Description: {code:java} [info] - awaitAnyTermination with timeout and resetTerminated ***

[jira] [Updated] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

2022-01-31 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-38080: - Description: {code:java} [info] - awaitAnyTermination with timeout and resetTerminated ***

[jira] [Created] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

2022-01-31 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-38080: Summary: Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated' Key: SPARK-38080 URL: https://issues.apache.org/jira/browse/SPARK-38080

[jira] [Assigned] (SPARK-37646) Avoid touching Scala reflection APIs in the lit function

2021-12-14 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-37646: Assignee: Shixiong Zhu > Avoid touching Scala reflection APIs in the lit function >

[jira] [Created] (SPARK-37646) Avoid touching Scala reflection APIs in the lit function

2021-12-14 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-37646: Summary: Avoid touching Scala reflection APIs in the lit function Key: SPARK-37646 URL: https://issues.apache.org/jira/browse/SPARK-37646 Project: Spark

[jira] [Created] (SPARK-36519) Store the RocksDB format in the checkpoint for a streaming query

2021-08-16 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-36519: Summary: Store the RocksDB format in the checkpoint for a streaming query Key: SPARK-36519 URL: https://issues.apache.org/jira/browse/SPARK-36519 Project: Spark

[jira] [Commented] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2021-04-12 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319631#comment-17319631 ] Shixiong Zhu commented on SPARK-31923: -- Do you have a reproduction? It's weird to see

[jira] [Commented] (SPARK-19256) Hive bucketing write support

2021-03-12 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300594#comment-17300594 ] Shixiong Zhu commented on SPARK-19256: -- [~chengsu] Thanks for the answers. Sounds like Metastore is

[jira] [Commented] (SPARK-19256) Hive bucketing write support

2021-03-12 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300578#comment-17300578 ] Shixiong Zhu commented on SPARK-19256: -- I read 

[jira] [Updated] (SPARK-34626) UnresolvedAttribute.sql may return an incorrect sql

2021-03-04 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-34626: - Description: UnresolvedAttribute("a" :: "b" :: Nil).sql returns {code:java} `a.b`{code} The

[jira] [Created] (SPARK-34626) UnresolvedAttribute.sql may return an incorrect sql

2021-03-04 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-34626: Summary: UnresolvedAttribute.sql may return an incorrect sql Key: SPARK-34626 URL: https://issues.apache.org/jira/browse/SPARK-34626 Project: Spark Issue

[jira] [Created] (SPARK-34599) INSERT INTO OVERWRITE doesn't support partition columns containing dot for DSv2

2021-03-02 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-34599: Summary: INSERT INTO OVERWRITE doesn't support partition columns containing dot for DSv2 Key: SPARK-34599 URL: https://issues.apache.org/jira/browse/SPARK-34599

[jira] [Created] (SPARK-34556) Checking duplicate static partition columns doesn't respect case sensitive conf

2021-02-26 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-34556: Summary: Checking duplicate static partition columns doesn't respect case sensitive conf Key: SPARK-34556 URL: https://issues.apache.org/jira/browse/SPARK-34556

[jira] [Commented] (SPARK-34034) "show create table" doesn't work for v2 table

2021-01-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260095#comment-17260095 ] Shixiong Zhu commented on SPARK-34034: -- NVM. I just realized `show create table` didn't return the

[jira] [Resolved] (SPARK-34034) "show create table" doesn't work for v2 table

2021-01-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-34034. -- Resolution: Not A Problem > "show create table" doesn't work for v2 table >

[jira] [Updated] (SPARK-34034) "show create table" doesn't work for v2 table

2021-01-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-34034: - Description: I was QAing Spark 3.1.0 RC1 and found one regression: "show create table" doesn't

[jira] [Commented] (SPARK-34034) "show create table" doesn't work for v2 table

2021-01-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260061#comment-17260061 ] Shixiong Zhu commented on SPARK-34034: -- [~cloud_fan] > "show create table" doesn't work for v2

[jira] [Updated] (SPARK-34034) "show create table" doesn't work for v2 table

2021-01-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-34034: - Labels: regression (was: regresion) > "show create table" doesn't work for v2 table >

[jira] [Updated] (SPARK-34034) "show create table" doesn't work for v2 table

2021-01-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-34034: - Labels: regresion (was: ) > "show create table" doesn't work for v2 table >

[jira] [Created] (SPARK-34034) "show create table" doesn't work for v2 table

2021-01-06 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-34034: Summary: "show create table" doesn't work for v2 table Key: SPARK-34034 URL: https://issues.apache.org/jira/browse/SPARK-34034 Project: Spark Issue Type:

[jira] [Commented] (SPARK-32896) Add DataStreamWriter.toTable API

2020-12-02 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242844#comment-17242844 ] Shixiong Zhu commented on SPARK-32896: -- The method name is changed to `toTable` in 

[jira] [Updated] (SPARK-32896) Add DataStreamWriter.toTable API

2020-12-02 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-32896: - Description: For now, there's no way to write to the table (especially catalog table) even the

[jira] [Updated] (SPARK-32896) Add DataStreamWriter.toTable API

2020-12-02 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-32896: - Summary: Add DataStreamWriter.toTable API (was: Add DataStreamWriter.table API) > Add

[jira] [Resolved] (SPARK-31953) Add Spark Structured Streaming History Server Support

2020-12-02 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-31953. -- Fix Version/s: 3.1.0 Resolution: Done > Add Spark Structured Streaming History Server

[jira] [Assigned] (SPARK-31953) Add Spark Structured Streaming History Server Support

2020-12-02 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-31953: Assignee: Genmao Yu > Add Spark Structured Streaming History Server Support >

[jira] [Created] (SPARK-33587) Kill the executor on nested fatal errors

2020-11-28 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-33587: Summary: Kill the executor on nested fatal errors Key: SPARK-33587 URL: https://issues.apache.org/jira/browse/SPARK-33587 Project: Spark Issue Type:

[jira] [Commented] (SPARK-21065) Spark Streaming concurrentJobs + StreamingJobProgressListener conflict

2020-10-18 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216375#comment-17216375 ] Shixiong Zhu commented on SPARK-21065: -- If you are seeing many active batches, it's likely your

[jira] [Commented] (SPARK-21065) Spark Streaming concurrentJobs + StreamingJobProgressListener conflict

2020-10-18 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216341#comment-17216341 ] Shixiong Zhu commented on SPARK-21065: -- `spark.streaming.concurrentJobs` is not safe. Fixing it

[jira] [Commented] (SPARK-32256) Hive may fail to detect Hadoop version when using isolated classloader

2020-07-09 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155018#comment-17155018 ] Shixiong Zhu commented on SPARK-32256: -- Yep. This doesn't happen in Hadoop 3.1.0 and above > Hive

[jira] [Updated] (SPARK-32256) Hive may fail to detect Hadoop version when using isolated classloader

2020-07-09 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-32256: - Description: Spark allows the user to set `spark.sql.hive.metastore.jars` to specify jars to

[jira] [Updated] (SPARK-32256) Hive may fail to detect Hadoop version when using isolated classloader

2020-07-09 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-32256: - Description: Spark allows the user to set `spark.sql.hive.metastore.jars` to specify jars to

[jira] [Assigned] (SPARK-32256) Hive may fail to detect Hadoop version when using isolated classloader

2020-07-09 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-32256: Assignee: Shixiong Zhu > Hive may fail to detect Hadoop version when using isolated

[jira] [Created] (SPARK-32256) Hive may fail to detect Hadoop version when using isolated classloader

2020-07-09 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-32256: Summary: Hive may fail to detect Hadoop version when using isolated classloader Key: SPARK-32256 URL: https://issues.apache.org/jira/browse/SPARK-32256 Project:

[jira] [Commented] (SPARK-28489) KafkaOffsetRangeCalculator.getRanges may drop offsets

2020-06-25 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145719#comment-17145719 ] Shixiong Zhu commented on SPARK-28489: -- [~klaustelenius] No. The "minPartitions" option for batch

[jira] [Updated] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-08 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-31923: - Affects Version/s: 2.3.0 2.3.1 2.3.2

[jira] [Updated] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-08 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-31923: - Affects Version/s: 2.4.0 2.4.1 2.4.2

[jira] [Issue Comment Deleted] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-08 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-31923: - Comment: was deleted (was: User 'zsxwing' has created a pull request for this issue:

[jira] [Resolved] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-08 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-31923. -- Assignee: Shixiong Zhu (was: Apache Spark) Resolution: Fixed > Event log cannot be

[jira] [Assigned] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-08 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-31923: Assignee: Shixiong Zhu > Event log cannot be generated when some internal accumulators

[jira] [Updated] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-08 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-31923: - Fix Version/s: 2.4.7 > Event log cannot be generated when some internal accumulators use

[jira] [Updated] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-08 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-31923: - Fix Version/s: 3.1.0 3.0.1 > Event log cannot be generated when some

[jira] [Updated] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-31923: - Description: A user may use internal accumulators by adding the "internal.metrics." prefix to

[jira] [Updated] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-31923: - Description: A user may use internal accumulators by adding the "internal.metrics." prefix to

[jira] [Updated] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-31923: - Description: A user may use internal accumulators by adding the "internal.metrics." prefix to

[jira] [Updated] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-06 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-31923: - Description: A user may use internal accumulators by adding the "internal.metrics." prefix to

[jira] [Created] (SPARK-31923) Event log cannot be generated when some internal accumulators use unexpected types

2020-06-06 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-31923: Summary: Event log cannot be generated when some internal accumulators use unexpected types Key: SPARK-31923 URL: https://issues.apache.org/jira/browse/SPARK-31923

[jira] [Resolved] (SPARK-30915) FileStreamSinkLog: Avoid reading the metadata log file when finding the latest batch ID

2020-05-22 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-30915. -- Fix Version/s: 3.1.0 Assignee: Jungtaek Lim Resolution: Fixed >

[jira] [Resolved] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite

2020-04-23 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-31542. -- Resolution: Not A Problem > Backport SPARK-25692 Remove static initialization of worker

[jira] [Commented] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuit

2020-04-23 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091060#comment-17091060 ] Shixiong Zhu commented on SPARK-31542: -- [~holden] The flaky test was caused by a new improvement in

[jira] [Created] (SPARK-31245) Document the backward incompatibility issue for Encoders.kryo and Encoders.javaSerialization

2020-03-25 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-31245: Summary: Document the backward incompatibility issue for Encoders.kryo and Encoders.javaSerialization Key: SPARK-31245 URL: https://issues.apache.org/jira/browse/SPARK-31245

[jira] [Commented] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2020-03-14 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059259#comment-17059259 ] Shixiong Zhu commented on SPARK-28556: -- This change has been reverted. See SPARK-31144 for the new

[jira] [Updated] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2020-03-14 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28556: - Docs Text: (was: In Spark 3.0, the type of "error" parameter in the

[jira] [Updated] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2020-03-14 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28556: - Labels: (was: release-notes) > Error should also be sent to QueryExecutionListener.onFailure

[jira] [Updated] (SPARK-31144) Wrap java.lang.Error with an exception for QueryExecutionListener.onFailure

2020-03-13 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-31144: - Issue Type: Bug (was: Test) > Wrap java.lang.Error with an exception for

[jira] [Created] (SPARK-31144) Wrap java.lang.Error with an exception for QueryExecutionListener.onFailure

2020-03-13 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-31144: Summary: Wrap java.lang.Error with an exception for QueryExecutionListener.onFailure Key: SPARK-31144 URL: https://issues.apache.org/jira/browse/SPARK-31144 Project:

[jira] [Created] (SPARK-30984) Add UI test for Structured Streaming UI

2020-02-28 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-30984: Summary: Add UI test for Structured Streaming UI Key: SPARK-30984 URL: https://issues.apache.org/jira/browse/SPARK-30984 Project: Spark Issue Type: Test

[jira] [Resolved] (SPARK-30943) Show "batch ID" in tool tip string for Structured Streaming UI graphs

2020-02-25 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-30943. -- Fix Version/s: 3.0.0 Assignee: Jungtaek Lim Resolution: Fixed > Show "batch

[jira] [Updated] (SPARK-30936) Forwards-compatibility in JsonProtocol in broken

2020-02-23 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-30936: - Summary: Forwards-compatibility in JsonProtocol in broken (was: Fix the broken

[jira] [Created] (SPARK-30936) Fix the broken forwards-compatibility in JsonProtocol

2020-02-23 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-30936: Summary: Fix the broken forwards-compatibility in JsonProtocol Key: SPARK-30936 URL: https://issues.apache.org/jira/browse/SPARK-30936 Project: Spark Issue

[jira] [Created] (SPARK-30927) StreamingQueryManager should avoid keeping reference to terminated StreamingQuery

2020-02-23 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-30927: Summary: StreamingQueryManager should avoid keeping reference to terminated StreamingQuery Key: SPARK-30927 URL: https://issues.apache.org/jira/browse/SPARK-30927

[jira] [Commented] (SPARK-15006) Generated JavaDoc should hide package private objects

2020-02-10 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033886#comment-17033886 ] Shixiong Zhu commented on SPARK-15006: -- cc [~jodersky] has this fixed in the upstream? > Generated

[jira] [Reopened] (SPARK-15006) Generated JavaDoc should hide package private objects

2020-02-10 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-15006: -- Reopening this since the issue still exists (See

[jira] [Created] (SPARK-30779) Review and fix issues in Structured Streaming API docs

2020-02-10 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-30779: Summary: Review and fix issues in Structured Streaming API docs Key: SPARK-30779 URL: https://issues.apache.org/jira/browse/SPARK-30779 Project: Spark Issue

[jira] [Created] (SPARK-30778) Javadoc of Java classes doesn't show up in Scala doc

2020-02-10 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-30778: Summary: Javadoc of Java classes doesn't show up in Scala doc Key: SPARK-30778 URL: https://issues.apache.org/jira/browse/SPARK-30778 Project: Spark Issue

[jira] [Commented] (SPARK-30657) Streaming limit after streaming dropDuplicates can throw error

2020-02-01 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028223#comment-17028223 ] Shixiong Zhu commented on SPARK-30657: -- [~tdas] Make sense. Agreed that the risk is high but the

[jira] [Updated] (SPARK-30658) Limit after on streaming dataframe before streaming agg returns wrong results

2020-01-31 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-30658: - Fix Version/s: 3.0.0 > Limit after on streaming dataframe before streaming agg returns wrong

[jira] [Updated] (SPARK-30657) Streaming limit after streaming dropDuplicates can throw error

2020-01-31 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-30657: - Fix Version/s: 3.0.0 > Streaming limit after streaming dropDuplicates can throw error >

[jira] [Resolved] (SPARK-30656) Support the "minPartitions" option in Kafka batch source and streaming source v1

2020-01-30 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-30656. -- Fix Version/s: 3.0.0 Resolution: Done > Support the "minPartitions" option in Kafka

[jira] [Resolved] (SPARK-29543) Support Structured Streaming UI

2020-01-29 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-29543. -- Fix Version/s: 3.0.0 Assignee: Genmao Yu Resolution: Done > Support

[jira] [Commented] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2020-01-29 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026098#comment-17026098 ] Shixiong Zhu commented on SPARK-28556: -- This also reminds me that we should also review all public

[jira] [Commented] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2020-01-28 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025655#comment-17025655 ] Shixiong Zhu commented on SPARK-28556: -- This is not deprecating the API. We just fixed an issue in

[jira] [Updated] (SPARK-30656) Support the "minPartitions" option in Kafka batch source and streaming source v1

2020-01-28 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-30656: - Issue Type: Improvement (was: Bug) > Support the "minPartitions" option in Kafka batch source

[jira] [Updated] (SPARK-26060) Track SparkConf entries and make SET command reject such entries.

2020-01-27 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26060: - Labels: release-notes (was: ) > Track SparkConf entries and make SET command reject such

[jira] [Updated] (SPARK-19287) JavaPairRDD flatMapValues requires function returning Iterable, not Iterator

2020-01-27 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19287: - Docs Text: JavaPairRDD/JavaPairDStream.flatMapValues() now requires a FlatMapFunction as an

[jira] [Created] (SPARK-30656) Support the "minPartitions" option in Kafka batch source and streaming source v1

2020-01-27 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-30656: Summary: Support the "minPartitions" option in Kafka batch source and streaming source v1 Key: SPARK-30656 URL: https://issues.apache.org/jira/browse/SPARK-30656

[jira] [Updated] (SPARK-25382) Remove ImageSchema.readImages in 3.0

2020-01-24 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25382: - Docs Text: In Spark 3.0.0, the deprecated ImageSchema class and its readImages methods have

[jira] [Updated] (SPARK-25382) Remove ImageSchema.readImages in 3.0

2020-01-24 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25382: - Docs Text: In Spark 3.0.0, the deprecated ImageSchema class and its readImages methods have

[jira] [Updated] (SPARK-26132) Remove support for Scala 2.11 in Spark 3.0.0

2020-01-22 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26132: - Docs Text: Scala 2.11 support is removed in Apache Spark 3.0.0. > Remove support for Scala 2.11

[jira] [Updated] (SPARK-26132) Remove support for Scala 2.11 in Spark 3.0.0

2020-01-22 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26132: - Labels: release-notes (was: ) > Remove support for Scala 2.11 in Spark 3.0.0 >

[jira] [Updated] (SPARK-24516) PySpark Bindings for K8S - make Python 3 the default

2020-01-22 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24516: - Labels: release-no (was: ) > PySpark Bindings for K8S - make Python 3 the default >

[jira] [Updated] (SPARK-24516) PySpark Bindings for K8S - make Python 3 the default

2020-01-22 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24516: - Labels: release-notes (was: release-no) > PySpark Bindings for K8S - make Python 3 the default

[jira] [Updated] (SPARK-25016) remove Support for hadoop 2.6

2020-01-22 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25016: - Docs Text: Hadoop 2.6.x support is removed. The default Hadoop version in Spark 3.0.0 in 2.7.x.

[jira] [Updated] (SPARK-25016) remove Support for hadoop 2.6

2020-01-22 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25016: - Labels: release_notes (was: ) > remove Support for hadoop 2.6 > -

[jira] [Updated] (SPARK-29930) Remove SQL configs declared to be removed in Spark 3.0

2020-01-22 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-29930: - Labels: release-notes (was: ) > Remove SQL configs declared to be removed in Spark 3.0 >

  1   2   3   4   5   6   7   8   9   10   >