[jira] [Created] (SPARK-31391) Add AdaptiveTestUtils to ease the test of AQE

wuyi created SPARK-31391: Summary: Add AdaptiveTestUtils to ease the test of AQE Key: SPARK-31391 URL: https://issues.apache.org/jira/browse/SPARK-31391 Project: Spark Issue Type: Test Comp

[jira] [Commented] (SPARK-31301) flatten the result dataframe of tests in stat

[ https://issues.apache.org/jira/browse/SPARK-31301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078958#comment-17078958 ] zhengruifeng commented on SPARK-31301: -- [~srowen] There are two methods now: {code:

[jira] [Commented] (SPARK-31301) flatten the result dataframe of tests in stat

[ https://issues.apache.org/jira/browse/SPARK-31301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078902#comment-17078902 ] Sean R. Owen commented on SPARK-31301: -- I guess so, but doesn't it become inconsist

[jira] [Commented] (SPARK-31301) flatten the result dataframe of tests in stat

[ https://issues.apache.org/jira/browse/SPARK-31301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078898#comment-17078898 ] zhengruifeng commented on SPARK-31301: -- [~srowen] How do you think about changing t

[jira] [Updated] (SPARK-31368) The query with the where condition failed,when the partition field is null

[ https://issues.apache.org/jira/browse/SPARK-31368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tanweihua updated SPARK-31368: -- Component/s: (was: Spark Shell) (was: Spark Core) (was: P

[jira] [Resolved] (SPARK-30818) Add LinearRegression wrapper to SparkR

[ https://issues.apache.org/jira/browse/SPARK-30818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30818. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 27593 [https://gi

[jira] [Assigned] (SPARK-30818) Add LinearRegression wrapper to SparkR

[ https://issues.apache.org/jira/browse/SPARK-30818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30818: Assignee: Maciej Szymkiewicz > Add LinearRegression wrapper to SparkR > -

[jira] [Resolved] (SPARK-31309) Migrate the ChiSquareTest from MLlib to ML

[ https://issues.apache.org/jira/browse/SPARK-31309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-31309. -- Resolution: Not A Problem > Migrate the ChiSquareTest from MLlib to ML > -

[jira] [Assigned] (SPARK-31382) Show a better error message for different python and pip installation mistake

[ https://issues.apache.org/jira/browse/SPARK-31382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31382: Assignee: Hyukjin Kwon > Show a better error message for different python and pip install

[jira] [Resolved] (SPARK-31382) Show a better error message for different python and pip installation mistake

[ https://issues.apache.org/jira/browse/SPARK-31382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31382. -- Fix Version/s: 3.0.0 2.4.6 Resolution: Fixed Issue resolved by pull

[jira] [Created] (SPARK-31390) Document Window Function

Huaxin Gao created SPARK-31390: -- Summary: Document Window Function Key: SPARK-31390 URL: https://issues.apache.org/jira/browse/SPARK-31390 Project: Spark Issue Type: Sub-task Component

[jira] [Resolved] (SPARK-29314) ProgressReporter.extractStateOperatorMetrics should not overwrite updated as 0 when it actually runs a batch even with no data

[ https://issues.apache.org/jira/browse/SPARK-29314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-29314. - Fix Version/s: 3.0.0 Resolution: Fixed Resolved by [https://github.com/apache/spark/pull/

[jira] [Assigned] (SPARK-29314) ProgressReporter.extractStateOperatorMetrics should not overwrite updated as 0 when it actually runs a batch even with no data

[ https://issues.apache.org/jira/browse/SPARK-29314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-29314: --- Assignee: Jungtaek Lim > ProgressReporter.extractStateOperatorMetrics should not overwrite

[jira] [Commented] (SPARK-31389) Ensure all tests in SQLMetricsSuite run with both codegen on and off

[ https://issues.apache.org/jira/browse/SPARK-31389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078803#comment-17078803 ] Srinivas Rishindra Pothireddi commented on SPARK-31389: --- I am work

[jira] [Updated] (SPARK-31389) Ensure all tests in SQLMetricsSuite run with both codegen on and off

[ https://issues.apache.org/jira/browse/SPARK-31389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-31389: -- Description: Many tests in SQLMetricsSuite run only with codeg

[jira] [Created] (SPARK-31389) Ensure all tests in SQLMetricsSuite run with both codegen on and off

Srinivas Rishindra Pothireddi created SPARK-31389: - Summary: Ensure all tests in SQLMetricsSuite run with both codegen on and off Key: SPARK-31389 URL: https://issues.apache.org/jira/browse/SPARK-3

[jira] [Commented] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer

[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078751#comment-17078751 ] Nick Afshartous commented on SPARK-27249: - [~enrush] Hi Everett, checking back o

[jira] [Updated] (SPARK-31386) Reading broadcast in UDF raises MemoryError when spark.executor.pyspark.memory is set

[ https://issues.apache.org/jira/browse/SPARK-31386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viacheslav Krot updated SPARK-31386: Description: Following code with udf causes MemoryError when `spark.executor.pyspark.memor

[jira] [Resolved] (SPARK-31009) Support json_object_keys function

[ https://issues.apache.org/jira/browse/SPARK-31009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31009. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 27836 [https://

[jira] [Assigned] (SPARK-31009) Support json_object_keys function

[ https://issues.apache.org/jira/browse/SPARK-31009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31009: - Assignee: Rakesh Raushan > Support json_object_keys function >

[jira] [Created] (SPARK-31388) org.apache.spark.sql.hive.thriftserver.CliSuite result matching is flaky

Juliusz Sompolski created SPARK-31388: - Summary: org.apache.spark.sql.hive.thriftserver.CliSuite result matching is flaky Key: SPARK-31388 URL: https://issues.apache.org/jira/browse/SPARK-31388 Pr

[jira] [Commented] (SPARK-31377) Add unit tests for "number of output rows" metric for joins in SQLMetricsSuite

[ https://issues.apache.org/jira/browse/SPARK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078524#comment-17078524 ] Srinivas Rishindra Pothireddi commented on SPARK-31377: --- I am work

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078520#comment-17078520 ] Venkata krishnan Sowrirajan commented on SPARK-22148: - Thanks for yo

[jira] [Updated] (SPARK-31387) HiveThriftServer2Listener update methods fail with unknown operation/session id

[ https://issues.apache.org/jira/browse/SPARK-31387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ali Smesseim updated SPARK-31387: - Description: HiveThriftServer2Listener update methods, such as  onSessionClosed and onOperationEr

[jira] [Created] (SPARK-31387) HiveThriftServer2Listener update methods fail with unknown operation/session id

Ali Smesseim created SPARK-31387: Summary: HiveThriftServer2Listener update methods fail with unknown operation/session id Key: SPARK-31387 URL: https://issues.apache.org/jira/browse/SPARK-31387 Proje

[jira] [Resolved] (SPARK-31362) Document Set Operators in SQL Reference

[ https://issues.apache.org/jira/browse/SPARK-31362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-31362. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28139 [https://gi

[jira] [Assigned] (SPARK-31362) Document Set Operators in SQL Reference

[ https://issues.apache.org/jira/browse/SPARK-31362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-31362: Assignee: Huaxin Gao > Document Set Operators in SQL Reference >

[jira] [Commented] (SPARK-31327) write spark version to avro file metadata

[ https://issues.apache.org/jira/browse/SPARK-31327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078384#comment-17078384 ] Dongjoon Hyun commented on SPARK-31327: --- This is backported to `branch-2.4` via [

[jira] [Updated] (SPARK-31327) write spark version to avro file metadata

[ https://issues.apache.org/jira/browse/SPARK-31327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31327: -- Fix Version/s: 2.4.6 > write spark version to avro file metadata > ---

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078283#comment-17078283 ] Wenchen Fan commented on SPARK-23128: - Yes, they are. https://issues.apache.org/jira

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078278#comment-17078278 ] Thomas Graves commented on SPARK-22148: --- so off the top of my head, I think the ma

[jira] [Created] (SPARK-31386) Reading broadcast in UDF raises MemoryError when spark.executor.pyspark.memory is set

Viacheslav Krot created SPARK-31386: --- Summary: Reading broadcast in UDF raises MemoryError when spark.executor.pyspark.memory is set Key: SPARK-31386 URL: https://issues.apache.org/jira/browse/SPARK-31386

[jira] [Comment Edited] (SPARK-31376) Non-global sort support for structured streaming

[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078219#comment-17078219 ] Adam Binford edited comment on SPARK-31376 at 4/8/20, 12:40 PM: --

[jira] [Commented] (SPARK-31376) Non-global sort support for structured streaming

[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078219#comment-17078219 ] Adam Binford commented on SPARK-31376: -- I tried multiple times to add myself to the

[jira] [Created] (SPARK-31385) Results of Julian-Gregorian rebasing don't match to Gregorian-Julian rebasing

Maxim Gekk created SPARK-31385: -- Summary: Results of Julian-Gregorian rebasing don't match to Gregorian-Julian rebasing Key: SPARK-31385 URL: https://issues.apache.org/jira/browse/SPARK-31385 Project: Sp

[jira] [Updated] (SPARK-31384) NPE in OptimizeSkewedJoin when there's a inputRDD of plan has 0 partition

[ https://issues.apache.org/jira/browse/SPARK-31384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-31384: - Summary: NPE in OptimizeSkewedJoin when there's a inputRDD of plan has 0 partition (was: Fix NPE in OptimizeSke

[jira] [Created] (SPARK-31384) Fix NPE in OptimizeSkewedJoin

wuyi created SPARK-31384: Summary: Fix NPE in OptimizeSkewedJoin Key: SPARK-31384 URL: https://issues.apache.org/jira/browse/SPARK-31384 Project: Spark Issue Type: Bug Components: SQL A

[jira] [Resolved] (SPARK-31379) Fix flaky test: o.a.s.scheduler.CoarseGrainedSchedulerBackendSuite.extra resources from executor

[ https://issues.apache.org/jira/browse/SPARK-31379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31379. -- Fix Version/s: 3.0.0 Assignee: wuyi Resolution: Fixed Fixed in https://github.

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077939#comment-17077939 ] Sandeep Katta commented on SPARK-23128: --- [~cloud_fan] [~carsonwang] any updates on

[jira] [Created] (SPARK-31383) Clean up the SQL documents in docs/sql-ref*

Takeshi Yamamuro created SPARK-31383: Summary: Clean up the SQL documents in docs/sql-ref* Key: SPARK-31383 URL: https://issues.apache.org/jira/browse/SPARK-31383 Project: Spark Issue Typ

[jira] [Created] (SPARK-31382) Show a better error message for different python and pip installation mistake

Hyukjin Kwon created SPARK-31382: Summary: Show a better error message for different python and pip installation mistake Key: SPARK-31382 URL: https://issues.apache.org/jira/browse/SPARK-31382 Project