[jira] [Created] (SPARK-31993) Generated code in 'concat_ws' fails to compile when splitting method is in effect

2020-06-14 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-31993: Summary: Generated code in 'concat_ws' fails to compile when splitting method is in effect Key: SPARK-31993 URL: https://issues.apache.org/jira/browse/SPARK-31993 Pro

[jira] [Commented] (SPARK-31992) Benchmark the EXCEPTION rebase mode

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135452#comment-17135452 ] Apache Spark commented on SPARK-31992: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-31992) Benchmark the EXCEPTION rebase mode

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31992: Assignee: Apache Spark > Benchmark the EXCEPTION rebase mode > --

[jira] [Assigned] (SPARK-31992) Benchmark the EXCEPTION rebase mode

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31992: Assignee: (was: Apache Spark) > Benchmark the EXCEPTION rebase mode > ---

[jira] [Created] (SPARK-31992) Benchmark the EXCEPTION rebase mode

2020-06-14 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31992: -- Summary: Benchmark the EXCEPTION rebase mode Key: SPARK-31992 URL: https://issues.apache.org/jira/browse/SPARK-31992 Project: Spark Issue Type: Test Co

[jira] [Assigned] (SPARK-31926) Fix concurrency issue for ThriftCLIService to getPortNumber

2020-06-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31926: --- Assignee: Kent Yao > Fix concurrency issue for ThriftCLIService to getPortNumber >

[jira] [Resolved] (SPARK-31926) Fix concurrency issue for ThriftCLIService to getPortNumber

2020-06-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31926. - Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull re

[jira] [Commented] (SPARK-31894) Introduce UnsafeRow format validation for streaming state store

2020-06-14 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135431#comment-17135431 ] Yuanjian Li commented on SPARK-31894: - This issue is blocking by SPARK-31990, see mo

[jira] [Commented] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135428#comment-17135428 ] Yuanjian Li commented on SPARK-31990: - [~maropu] Thanks for the quick fix! [~dongjo

[jira] [Commented] (SPARK-31968) write.partitionBy() creates duplicate subdirectories when user provides duplicate columns

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135377#comment-17135377 ] Dongjoon Hyun commented on SPARK-31968: --- Thanks, [~JinxinTang] and [~hyukjin.kwon]

[jira] [Assigned] (SPARK-31968) write.partitionBy() creates duplicate subdirectories when user provides duplicate columns

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31968: - Assignee: JinxinTang > write.partitionBy() creates duplicate subdirectories when user p

[jira] [Commented] (SPARK-31292) Replace toSet.toSeq with distinct for readability

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135374#comment-17135374 ] Apache Spark commented on SPARK-31292: -- User 'maropu' has created a pull request fo

[jira] [Commented] (SPARK-31968) write.partitionBy() creates duplicate subdirectories when user provides duplicate columns

2020-06-14 Thread JinxinTang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135342#comment-17135342 ] JinxinTang commented on SPARK-31968: [~qxzzxq] Thank you for this issue, it is my pl

[jira] [Commented] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135341#comment-17135341 ] Dongjoon Hyun commented on SPARK-31990: --- I linked SPARK-31292 according to the PR

[jira] [Commented] (SPARK-31968) write.partitionBy() creates duplicate subdirectories when user provides duplicate columns

2020-06-14 Thread JinxinTang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135340#comment-17135340 ] JinxinTang commented on SPARK-31968: Hi, [~dongjoon], [~hyukjin.kwon], I am  TJX2014

[jira] [Commented] (SPARK-31292) Replace toSet.toSeq with distinct for readability

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135338#comment-17135338 ] Dongjoon Hyun commented on SPARK-31292: --- Let's not backport improvement next time~

[jira] [Resolved] (SPARK-27633) Remove redundant aliases in NestedColumnAliasing

2020-06-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27633. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 24525 [https://gi

[jira] [Assigned] (SPARK-27633) Remove redundant aliases in NestedColumnAliasing

2020-06-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27633: Assignee: L. C. Hsieh > Remove redundant aliases in NestedColumnAliasing > --

[jira] [Updated] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31990: -- Labels: correctness (was: ) > Streaming's state store compatibility is broken > -

[jira] [Updated] (SPARK-29152) Spark Executor Plugin API shutdown is not proper when dynamic allocation enabled

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29152: -- Fix Version/s: (was: 2.4.7) > Spark Executor Plugin API shutdown is not proper when dynami

[jira] [Updated] (SPARK-31983) Tables of structured streaming tab show wrong result for duration column

2020-06-14 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-31983: --- Fix Version/s: 3.0.1 > Tables of structured streaming tab show wrong result for duration col

[jira] [Commented] (SPARK-31983) Tables of structured streaming tab show wrong result for duration column

2020-06-14 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135323#comment-17135323 ] Kousuke Saruta commented on SPARK-31983: Resolved for 3.0 by [https://github.com

[jira] [Commented] (SPARK-31968) write.partitionBy() creates duplicate subdirectories when user provides duplicate columns

2020-06-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135321#comment-17135321 ] Hyukjin Kwon commented on SPARK-31968: -- Oops, there's no confirmation here. Let me

[jira] [Assigned] (SPARK-31968) write.partitionBy() creates duplicate subdirectories when user provides duplicate columns

2020-06-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31968: Assignee: (was: Xuzhou Qin) > write.partitionBy() creates duplicate subdirectories wh

[jira] [Commented] (SPARK-31968) write.partitionBy() creates duplicate subdirectories when user provides duplicate columns

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135316#comment-17135316 ] Dongjoon Hyun commented on SPARK-31968: --- Hi, [~hyukjin.kwon]. [~qxzzxq] is TJX2014

[jira] [Commented] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135315#comment-17135315 ] Apache Spark commented on SPARK-31990: -- User 'maropu' has created a pull request fo

[jira] [Commented] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135314#comment-17135314 ] Apache Spark commented on SPARK-31990: -- User 'maropu' has created a pull request fo

[jira] [Assigned] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31990: Assignee: Apache Spark > Streaming's state store compatibility is broken > --

[jira] [Assigned] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31990: Assignee: (was: Apache Spark) > Streaming's state store compatibility is broken > ---

[jira] [Assigned] (SPARK-31968) write.partitionBy() creates duplicate subdirectories when user provides duplicate columns

2020-06-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31968: Assignee: Xuzhou Qin > write.partitionBy() creates duplicate subdirectories when user pro

[jira] [Commented] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135313#comment-17135313 ] Takeshi Yamamuro commented on SPARK-31990: -- Oh, nice catch, [~XuanYuan] ! I'll

[jira] [Updated] (SPARK-29152) Spark Executor Plugin API shutdown is not proper when dynamic allocation enabled

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29152: -- Fix Version/s: 2.4.7 > Spark Executor Plugin API shutdown is not proper when dynamic allocatio

[jira] [Commented] (SPARK-31983) Tables of structured streaming tab show wrong result for duration column

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135297#comment-17135297 ] Apache Spark commented on SPARK-31983: -- User 'iRakson' has created a pull request f

[jira] [Resolved] (SPARK-31983) Tables of structured streaming tab show wrong result for duration column

2020-06-14 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-31983. -- Fix Version/s: 3.1.0 Assignee: Rakesh Raushan Resolution: Fixed Resolved for 3

[jira] [Commented] (SPARK-24634) Add a new metric regarding number of rows later than watermark

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135295#comment-17135295 ] Apache Spark commented on SPARK-24634: -- User 'HeartSaVioR' has created a pull reque

[jira] [Assigned] (SPARK-31970) Make MDC configuration step be consistent between setLocalProperty and log4j.properties

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31970: - Assignee: wuyi > Make MDC configuration step be consistent between setLocalProperty and

[jira] [Resolved] (SPARK-31970) Make MDC configuration step be consistent between setLocalProperty and log4j.properties

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31970. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28801 [https://

[jira] [Updated] (SPARK-8981) Add MDC support in Executor

2020-06-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-8981: - Summary: Add MDC support in Executor (was: Set applicationId and appName in log4j MDC) > Add MD

[jira] [Commented] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135286#comment-17135286 ] Jungtaek Lim commented on SPARK-31990: -- Nice finding! One thing I'd like to add is

[jira] [Updated] (SPARK-31991) The SparkR regexp_replace function causes problems

2020-06-14 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-31991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oscar Brück updated SPARK-31991: Description: The SparkR regex functions in R are not working as they should. Here's a reprex.   {

[jira] [Created] (SPARK-31991) Regexp_replace causing problems

2020-06-14 Thread Jira
Oscar Brück created SPARK-31991: --- Summary: Regexp_replace causing problems Key: SPARK-31991 URL: https://issues.apache.org/jira/browse/SPARK-31991 Project: Spark Issue Type: Bug Compo

[jira] [Commented] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135272#comment-17135272 ] Xiao Li commented on SPARK-31990: - [~XuanYuan] just identified the problem when fixing t

[jira] [Commented] (SPARK-31967) Loading jobs UI page takes 40 seconds

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135271#comment-17135271 ] Xiao Li commented on SPARK-31967: - [~Gengliang.Wang] Please also change the fix versions

[jira] [Updated] (SPARK-31967) Loading jobs UI page takes 40 seconds

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31967: Fix Version/s: 2.4.7 > Loading jobs UI page takes 40 seconds > - > >

[jira] [Commented] (SPARK-31967) Loading jobs UI page takes 40 seconds

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135270#comment-17135270 ] Xiao Li commented on SPARK-31967: - nvm. This has been merged to 2.4 branch.  > Loading

[jira] [Commented] (SPARK-31967) Loading jobs UI page takes 40 seconds

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135269#comment-17135269 ] Xiao Li commented on SPARK-31967: - Do we need to resolve the  same issue in 2.4.6? > Lo

[jira] [Updated] (SPARK-31967) Loading jobs UI page takes 40 seconds

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31967: Affects Version/s: 2.4.6 > Loading jobs UI page takes 40 seconds > - >

[jira] [Commented] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135268#comment-17135268 ] Xiao Li commented on SPARK-31990: - cc [~maropu] [~sowen] [~sekikn] > Streaming's state

[jira] [Assigned] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-31990: --- Assignee: (was: Yuanjian Li) > Streaming's state store compatibility is broken > --

[jira] [Updated] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31990: Target Version/s: 3.0.1 > Streaming's state store compatibility is broken > --

[jira] [Updated] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31990: Priority: Blocker (was: Major) > Streaming's state store compatibility is broken > --

[jira] [Created] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Xiao Li (Jira)
Xiao Li created SPARK-31990: --- Summary: Streaming's state store compatibility is broken Key: SPARK-31990 URL: https://issues.apache.org/jira/browse/SPARK-31990 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31990: Environment: (was: [This line|https://github.com/apache/spark/pull/28062/files#diff-7a46f10c3cedbf013c

[jira] [Updated] (SPARK-31990) Streaming's state store compatibility is broken

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31990: Description: [This line|https://github.com/apache/spark/pull/28062/files#diff-7a46f10c3cedbf013cf255564d94

[jira] [Updated] (SPARK-31967) Loading jobs UI page takes 40 seconds

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31967: Fix Version/s: 3.1.0 > Loading jobs UI page takes 40 seconds > - > >

[jira] [Updated] (SPARK-31967) Loading jobs UI page takes 40 seconds

2020-06-14 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31967: Fix Version/s: 3.0.1 > Loading jobs UI page takes 40 seconds > - > >

[jira] [Assigned] (SPARK-31989) Generate JSON files with rebasing switch points using smaller steps

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31989: Assignee: Apache Spark > Generate JSON files with rebasing switch points using smaller st

[jira] [Assigned] (SPARK-31989) Generate JSON files with rebasing switch points using smaller steps

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31989: Assignee: (was: Apache Spark) > Generate JSON files with rebasing switch points using

[jira] [Commented] (SPARK-31989) Generate JSON files with rebasing switch points using smaller steps

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135261#comment-17135261 ] Apache Spark commented on SPARK-31989: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-31989) Generate JSON files with rebasing switch points using smaller steps

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135260#comment-17135260 ] Apache Spark commented on SPARK-31989: -- User 'MaxGekk' has created a pull request f

[jira] [Created] (SPARK-31989) Generate JSON files with rebasing switch points using smaller steps

2020-06-14 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31989: -- Summary: Generate JSON files with rebasing switch points using smaller steps Key: SPARK-31989 URL: https://issues.apache.org/jira/browse/SPARK-31989 Project: Spark

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: The function [getRootFields|#L120] doesn't preserve attribute metadata and may result

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: The function [getRootFields|#L120] doesn't preserve attribute metadata and may result

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: The function [getRootFields|#L120]] doesn't preserve attribute metadata and may result

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: The function [getRootFields|#L120] doesn't preserve attribute metadata and may result

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: The function [getRootFields|#L120]] doesn't preserve attribute metadata and may result

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: The function [getRootFields|#L120]] doesn't preserve attribute metadata and may result

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: The function [getRootFields|[https://github.com/apache/spark/blob/1e40bccf447dccad9d31

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: The function [getRootFields](https://github.com/apache/spark/blob/1e40bccf447dccad9d31

[jira] [Commented] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135206#comment-17135206 ] Apache Spark commented on SPARK-31988: -- User 'guykhazma' has created a pull request

[jira] [Assigned] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31988: Assignee: (was: Apache Spark) > Datasource V2 - schema pruning may discard attribute

[jira] [Commented] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135205#comment-17135205 ] Guy Khazma commented on SPARK-31988: PR https://github.com/apache/spark/pull/28826

[jira] [Assigned] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31988: Assignee: Apache Spark > Datasource V2 - schema pruning may discard attribute metadata >

[jira] [Commented] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135204#comment-17135204 ] Apache Spark commented on SPARK-31988: -- User 'guykhazma' has created a pull request

[jira] [Issue Comment Deleted] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Comment: was deleted (was: PR https://github.com/apache/spark/pull/28826) > Datasource V2 - schema

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: (was: SPARK-26837 added pruning of nested fields and specifically the function ) >

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: SPARK-26837 added pruning of nested fields and specifically the function  (was: SPARK-

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Summary: Datasource V2 - schema pruning may discard attribute metadata (was: Datasource V2 - Schema

[jira] [Updated] (SPARK-31988) Datasource V2 - schema pruning may discard attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: SPARK-26837 added  (was: S) > Datasource V2 - schema pruning may discard attribute met

[jira] [Updated] (SPARK-31988) Datasource V2 - Schema pruning discards attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guy Khazma updated SPARK-31988: --- Description: S > Datasource V2 - Schema pruning discards attribute metadata > --

[jira] [Created] (SPARK-31988) Datasource V2 - Schema pruning discards attribute metadata

2020-06-14 Thread Guy Khazma (Jira)
Guy Khazma created SPARK-31988: -- Summary: Datasource V2 - Schema pruning discards attribute metadata Key: SPARK-31988 URL: https://issues.apache.org/jira/browse/SPARK-31988 Project: Spark Issue

[jira] [Updated] (SPARK-31968) write.partitionBy() creates duplicate subdirectories when user provides duplicate columns

2020-06-14 Thread Xuzhou Qin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuzhou Qin updated SPARK-31968: --- Summary: write.partitionBy() creates duplicate subdirectories when user provides duplicate columns

[jira] [Commented] (SPARK-31968) write.partitionBy() creates duplicate subdirectories when user provide duplicate columns

2020-06-14 Thread Xuzhou Qin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135180#comment-17135180 ] Xuzhou Qin commented on SPARK-31968: Thank you TJX2014 for the PR and thank you all

[jira] [Updated] (SPARK-31987) Where to find Prometheus metrics path about executors in Spark 3

2020-06-14 Thread Smith Cruise (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Smith Cruise updated SPARK-31987: - Description: I know in Spark 3.0, it supports Prometheus Sink. But I don't know where to get me

[jira] [Updated] (SPARK-31987) Where to find Prometheus metrics path about executors in Spark 3

2020-06-14 Thread Smith Cruise (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Smith Cruise updated SPARK-31987: - Summary: Where to find Prometheus metrics path about executors in Spark 3 (was: Where to find P

[jira] [Created] (SPARK-31987) Where to find Prometheus metrics about executor in Spark 3

2020-06-14 Thread Smith Cruise (Jira)
Smith Cruise created SPARK-31987: Summary: Where to find Prometheus metrics about executor in Spark 3 Key: SPARK-31987 URL: https://issues.apache.org/jira/browse/SPARK-31987 Project: Spark Is

[jira] [Commented] (SPARK-26365) spark-submit for k8s cluster doesn't propagate exit code

2020-06-14 Thread Oscar Cassetti (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135130#comment-17135130 ] Oscar Cassetti commented on SPARK-26365: I think I identified the issue but I am

[jira] [Commented] (SPARK-31986) Test failure RebaseDateTimeSuite."optimization of micros rebasing - Julian to Gregorian"

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135125#comment-17135125 ] Apache Spark commented on SPARK-31986: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-31986) Test failure RebaseDateTimeSuite."optimization of micros rebasing - Julian to Gregorian"

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31986: Assignee: (was: Apache Spark) > Test failure RebaseDateTimeSuite."optimization of mic

[jira] [Assigned] (SPARK-31986) Test failure RebaseDateTimeSuite."optimization of micros rebasing - Julian to Gregorian"

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31986: Assignee: Apache Spark > Test failure RebaseDateTimeSuite."optimization of micros rebasin

[jira] [Commented] (SPARK-31986) Test failure RebaseDateTimeSuite."optimization of micros rebasing - Julian to Gregorian"

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135124#comment-17135124 ] Apache Spark commented on SPARK-31986: -- User 'MaxGekk' has created a pull request f

[jira] [Created] (SPARK-31986) Test failure RebaseDateTimeSuite."optimization of micros rebasing - Julian to Gregorian"

2020-06-14 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31986: -- Summary: Test failure RebaseDateTimeSuite."optimization of micros rebasing - Julian to Gregorian" Key: SPARK-31986 URL: https://issues.apache.org/jira/browse/SPARK-31986

[jira] [Commented] (SPARK-31950) Extract SQL keywords from the generated parser class in TableIdentifierParserSuite

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135113#comment-17135113 ] Apache Spark commented on SPARK-31950: -- User 'dilipbiswal' has created a pull reque

[jira] [Commented] (SPARK-31950) Extract SQL keywords from the generated parser class in TableIdentifierParserSuite

2020-06-14 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135114#comment-17135114 ] Apache Spark commented on SPARK-31950: -- User 'dilipbiswal' has created a pull reque

[jira] [Created] (SPARK-31985) Remove the incomplete code path on aggregation for continuous mode

2020-06-14 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-31985: Summary: Remove the incomplete code path on aggregation for continuous mode Key: SPARK-31985 URL: https://issues.apache.org/jira/browse/SPARK-31985 Project: Spark