[jira] [Created] (SPARK-32907) adaptively blockify instances

2020-09-17 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-32907: Summary: adaptively blockify instances Key: SPARK-32907 URL: https://issues.apache.org/jira/browse/SPARK-32907 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-32908) percentile_approx() returns incorrect results

2020-09-17 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-32908: -- Summary: percentile_approx() returns incorrect results Key: SPARK-32908 URL: https://issues.apache.org/jira/browse/SPARK-32908 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-32908) percentile_approx() returns incorrect results

2020-09-17 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197460#comment-17197460 ] Maxim Gekk commented on SPARK-32908: I am preparing a fix for the issue. > percenti

[jira] [Updated] (SPARK-32908) percentile_approx() returns incorrect results

2020-09-17 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-32908: --- Attachment: percentile_approx-input.csv > percentile_approx() returns incorrect results > --

[jira] [Updated] (SPARK-32908) percentile_approx() returns incorrect results

2020-09-17 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-32908: --- Description: Read input data from the attached CSV file: {code:scala} val df = spark.read.opti

[jira] [Assigned] (SPARK-32907) adaptively blockify instances

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32907: Assignee: (was: Apache Spark) > adaptively blockify instances > -

[jira] [Commented] (SPARK-32907) adaptively blockify instances

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197470#comment-17197470 ] Apache Spark commented on SPARK-32907: -- User 'zhengruifeng' has created a pull requ

[jira] [Assigned] (SPARK-32907) adaptively blockify instances

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32907: Assignee: Apache Spark > adaptively blockify instances > - >

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2020-09-17 Thread Lauri Koobas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197479#comment-17197479 ] Lauri Koobas commented on SPARK-29900: -- Sure. Using my own schema here as an exampl

[jira] [Created] (SPARK-32909) Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

2020-09-17 Thread Yang Jie (Jira)
Yang Jie created SPARK-32909: Summary: Pass all `sql/hive-thriftserver` module UTs in Scala 2.13 Key: SPARK-32909 URL: https://issues.apache.org/jira/browse/SPARK-32909 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-32909) Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32909: Assignee: (was: Apache Spark) > Pass all `sql/hive-thriftserver` module UTs in Scala

[jira] [Assigned] (SPARK-32909) Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32909: Assignee: Apache Spark > Pass all `sql/hive-thriftserver` module UTs in Scala 2.13 >

[jira] [Commented] (SPARK-32909) Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197524#comment-17197524 ] Apache Spark commented on SPARK-32909: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-32908) percentile_approx() returns incorrect results

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32908: Assignee: Apache Spark > percentile_approx() returns incorrect results >

[jira] [Assigned] (SPARK-32908) percentile_approx() returns incorrect results

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32908: Assignee: (was: Apache Spark) > percentile_approx() returns incorrect results > -

[jira] [Commented] (SPARK-32908) percentile_approx() returns incorrect results

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197592#comment-17197592 ] Apache Spark commented on SPARK-32908: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-32908) percentile_approx() returns incorrect results

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197593#comment-17197593 ] Apache Spark commented on SPARK-32908: -- User 'MaxGekk' has created a pull request f

[jira] [Resolved] (SPARK-32900) UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.

2020-09-17 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-32900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-32900. --- Fix Version/s: 3.1.0 3.0.2 2.4.8 Resolut

[jira] [Assigned] (SPARK-32900) UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.

2020-09-17 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-32900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell reassigned SPARK-32900: - Assignee: Tom van Bussel > UnsafeExternalSorter.SpillableIterator cannot spill

[jira] [Created] (SPARK-32910) Remove UninterruptibleThread usage from KafkaOffsetReader

2020-09-17 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-32910: - Summary: Remove UninterruptibleThread usage from KafkaOffsetReader Key: SPARK-32910 URL: https://issues.apache.org/jira/browse/SPARK-32910 Project: Spark I

[jira] [Commented] (SPARK-32910) Remove UninterruptibleThread usage from KafkaOffsetReader

2020-09-17 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197617#comment-17197617 ] Gabor Somogyi commented on SPARK-32910: --- Started to work on this. > Remove Uninte

[jira] [Updated] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2020-09-17 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-28367: -- Description: Spark uses an old and deprecated API named poll(long) which never returns and st

[jira] [Assigned] (SPARK-32901) UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thrown while spilling

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32901: Assignee: (was: Apache Spark) > UnsafeExternalSorter may cause a SparkOutOfMemoryErro

[jira] [Commented] (SPARK-32901) UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thrown while spilling

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197641#comment-17197641 ] Apache Spark commented on SPARK-32901: -- User 'tomvanbussel' has created a pull requ

[jira] [Assigned] (SPARK-32901) UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thrown while spilling

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32901: Assignee: Apache Spark > UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thr

[jira] [Commented] (SPARK-32901) UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thrown while spilling

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197642#comment-17197642 ] Apache Spark commented on SPARK-32901: -- User 'tomvanbussel' has created a pull requ

[jira] [Created] (SPARK-32911) UnsafeExternalSorter.SpillableIterator cannot spill after reading all records

2020-09-17 Thread Tom van Bussel (Jira)
Tom van Bussel created SPARK-32911: -- Summary: UnsafeExternalSorter.SpillableIterator cannot spill after reading all records Key: SPARK-32911 URL: https://issues.apache.org/jira/browse/SPARK-32911 Pro

[jira] [Created] (SPARK-32912) Sparksql authentication

2020-09-17 Thread fengchunfeng (Jira)
fengchunfeng created SPARK-32912: Summary: Sparksql authentication Key: SPARK-32912 URL: https://issues.apache.org/jira/browse/SPARK-32912 Project: Spark Issue Type: Improvement Com

[jira] [Commented] (SPARK-27589) Spark file source V2

2020-09-17 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197669#comment-17197669 ] Thomas Graves commented on SPARK-27589: --- I'm guessing my question got missed - doe

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

2020-09-17 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-32635: --- Affects Version/s: 2.4.7 > When pyspark.sql.functions.lit() function is used with dataframe cache, i

[jira] [Created] (SPARK-32913) Improve DecommissionInfo and DecommissionState for different use cases

2020-09-17 Thread wuyi (Jira)
wuyi created SPARK-32913: Summary: Improve DecommissionInfo and DecommissionState for different use cases Key: SPARK-32913 URL: https://issues.apache.org/jira/browse/SPARK-32913 Project: Spark Issue

[jira] [Updated] (SPARK-32913) Improve ExecutorDecommissionInfo and ExecutorDecommissionState for different use cases

2020-09-17 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-32913: - Summary: Improve ExecutorDecommissionInfo and ExecutorDecommissionState for different use cases (was: Improve D

[jira] [Commented] (SPARK-32908) percentile_approx() returns incorrect results

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197734#comment-17197734 ] Apache Spark commented on SPARK-32908: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-32908) percentile_approx() returns incorrect results

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197733#comment-17197733 ] Apache Spark commented on SPARK-32908: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-30283) V2 Command logical plan should use UnresolvedV2Relation for a table

2020-09-17 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197735#comment-17197735 ] Wenchen Fan commented on SPARK-30283: - Yea we should resume the work, and probably d

[jira] [Commented] (SPARK-32911) UnsafeExternalSorter.SpillableIterator cannot spill after reading all records

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197737#comment-17197737 ] Apache Spark commented on SPARK-32911: -- User 'tomvanbussel' has created a pull requ

[jira] [Assigned] (SPARK-32911) UnsafeExternalSorter.SpillableIterator cannot spill after reading all records

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32911: Assignee: (was: Apache Spark) > UnsafeExternalSorter.SpillableIterator cannot spill a

[jira] [Assigned] (SPARK-32911) UnsafeExternalSorter.SpillableIterator cannot spill after reading all records

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32911: Assignee: Apache Spark > UnsafeExternalSorter.SpillableIterator cannot spill after readin

[jira] [Assigned] (SPARK-32913) Improve ExecutorDecommissionInfo and ExecutorDecommissionState for different use cases

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32913: Assignee: Apache Spark > Improve ExecutorDecommissionInfo and ExecutorDecommissionState f

[jira] [Commented] (SPARK-32913) Improve ExecutorDecommissionInfo and ExecutorDecommissionState for different use cases

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197740#comment-17197740 ] Apache Spark commented on SPARK-32913: -- User 'Ngone51' has created a pull request f

[jira] [Assigned] (SPARK-32913) Improve ExecutorDecommissionInfo and ExecutorDecommissionState for different use cases

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32913: Assignee: (was: Apache Spark) > Improve ExecutorDecommissionInfo and ExecutorDecommis

[jira] [Commented] (SPARK-32913) Improve ExecutorDecommissionInfo and ExecutorDecommissionState for different use cases

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197741#comment-17197741 ] Apache Spark commented on SPARK-32913: -- User 'Ngone51' has created a pull request f

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2020-09-17 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197742#comment-17197742 ] Wenchen Fan commented on SPARK-29900: - I think it was due to Spark wanting to make S

[jira] [Commented] (SPARK-32898) totalExecutorRunTimeMs is too big

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197762#comment-17197762 ] Apache Spark commented on SPARK-32898: -- User 'Ngone51' has created a pull request f

[jira] [Assigned] (SPARK-32898) totalExecutorRunTimeMs is too big

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32898: Assignee: Apache Spark > totalExecutorRunTimeMs is too big >

[jira] [Assigned] (SPARK-32898) totalExecutorRunTimeMs is too big

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32898: Assignee: (was: Apache Spark) > totalExecutorRunTimeMs is too big > -

[jira] [Commented] (SPARK-32898) totalExecutorRunTimeMs is too big

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197763#comment-17197763 ] Apache Spark commented on SPARK-32898: -- User 'Ngone51' has created a pull request f

[jira] [Created] (SPARK-32914) Avoid calling dataType multiple times for each expression

2020-09-17 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-32914: --- Summary: Avoid calling dataType multiple times for each expression Key: SPARK-32914 URL: https://issues.apache.org/jira/browse/SPARK-32914 Project: Spark Issue

[jira] [Assigned] (SPARK-32914) Avoid calling dataType multiple times for each expression

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32914: Assignee: Apache Spark > Avoid calling dataType multiple times for each expression >

[jira] [Commented] (SPARK-32914) Avoid calling dataType multiple times for each expression

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1719#comment-1719 ] Apache Spark commented on SPARK-32914: -- User 'wangyum' has created a pull request f

[jira] [Commented] (SPARK-32914) Avoid calling dataType multiple times for each expression

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197778#comment-17197778 ] Apache Spark commented on SPARK-32914: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-32914) Avoid calling dataType multiple times for each expression

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32914: Assignee: (was: Apache Spark) > Avoid calling dataType multiple times for each expres

[jira] [Resolved] (SPARK-32887) Example command in https://spark.apache.org/docs/latest/sql-ref-syntax-aux-show-table.html to be changed

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32887. --- Fix Version/s: 3.0.2 3.1.0 Resolution: Fixed Issue resolved by pul

[jira] [Assigned] (SPARK-32887) Example command in https://spark.apache.org/docs/latest/sql-ref-syntax-aux-show-table.html to be changed

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32887: - Assignee: Udbhav Agrawal > Example command in > https://spark.apache.org/docs/latest/s

[jira] [Commented] (SPARK-32914) Avoid calling dataType multiple times for each expression

2020-09-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197796#comment-17197796 ] Yuming Wang commented on SPARK-32914: - After this change: {noformat} --- Execution p

[jira] [Created] (SPARK-32915) RPC implementation to support pushing and merging shuffle blocks

2020-09-17 Thread Min Shen (Jira)
Min Shen created SPARK-32915: Summary: RPC implementation to support pushing and merging shuffle blocks Key: SPARK-32915 URL: https://issues.apache.org/jira/browse/SPARK-32915 Project: Spark Iss

[jira] [Created] (SPARK-32916) Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle

2020-09-17 Thread Min Shen (Jira)
Min Shen created SPARK-32916: Summary: Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle Key: SPARK-32916 URL: https://issues.apache.org/jira/browse/SPARK-32916

[jira] [Created] (SPARK-32917) Add support for executors to push shuffle blocks after successful map task completion

2020-09-17 Thread Min Shen (Jira)
Min Shen created SPARK-32917: Summary: Add support for executors to push shuffle blocks after successful map task completion Key: SPARK-32917 URL: https://issues.apache.org/jira/browse/SPARK-32917 Project

[jira] [Updated] (SPARK-32738) thread safe endpoints may hang due to fatal error

2020-09-17 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32738: Fix Version/s: 3.0.2 > thread safe endpoints may hang due to fatal error > ---

[jira] [Created] (SPARK-32918) RPC implementation to support control plane coordination for push-based shuffle

2020-09-17 Thread Min Shen (Jira)
Min Shen created SPARK-32918: Summary: RPC implementation to support control plane coordination for push-based shuffle Key: SPARK-32918 URL: https://issues.apache.org/jira/browse/SPARK-32918 Project: Spar

[jira] [Created] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions

2020-09-17 Thread Min Shen (Jira)
Min Shen created SPARK-32919: Summary: Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions Key: SPARK-32919 URL:

[jira] [Created] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

2020-09-17 Thread Min Shen (Jira)
Min Shen created SPARK-32920: Summary: Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage Key: SPARK-32920

[jira] [Created] (SPARK-32921) Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario

2020-09-17 Thread Min Shen (Jira)
Min Shen created SPARK-32921: Summary: Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario Key: SPARK-32921 URL

[jira] [Created] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures

2020-09-17 Thread Min Shen (Jira)
Min Shen created SPARK-32922: Summary: Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures Key: SPARK-32922 URL: http

[jira] [Created] (SPARK-32923) Add support to properly handle different type of stage retries

2020-09-17 Thread Min Shen (Jira)
Min Shen created SPARK-32923: Summary: Add support to properly handle different type of stage retries Key: SPARK-32923 URL: https://issues.apache.org/jira/browse/SPARK-32923 Project: Spark Issue

[jira] [Created] (SPARK-32924) Web UI sort on duration is wrong

2020-09-17 Thread t oo (Jira)
t oo created SPARK-32924: Summary: Web UI sort on duration is wrong Key: SPARK-32924 URL: https://issues.apache.org/jira/browse/SPARK-32924 Project: Spark Issue Type: Bug Components: Web UI

[jira] [Updated] (SPARK-32924) Web UI sort on duration is wrong

2020-09-17 Thread t oo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-32924: - Attachment: ui_sort.png > Web UI sort on duration is wrong > > >

[jira] [Updated] (SPARK-32858) UnwrapCastInBinaryComparison: support other numeric types

2020-09-17 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-32858: - Summary: UnwrapCastInBinaryComparison: support other numeric types (was: UnwrapCastInBinaryComparison:

[jira] [Created] (SPARK-32925) Support push-based shuffle in multiple deployment environments

2020-09-17 Thread Min Shen (Jira)
Min Shen created SPARK-32925: Summary: Support push-based shuffle in multiple deployment environments Key: SPARK-32925 URL: https://issues.apache.org/jira/browse/SPARK-32925 Project: Spark Issue

[jira] [Commented] (SPARK-32925) Support push-based shuffle in multiple deployment environments

2020-09-17 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197893#comment-17197893 ] Min Shen commented on SPARK-32925: -- cc [~dongjoon], [~holden], [~dbtsai] for comments o

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32635: -- Component/s: (was: PySpark) > When pyspark.sql.functions.lit() function is used with dataf

[jira] [Commented] (SPARK-32858) UnwrapCastInBinaryComparison: support other numeric types

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197897#comment-17197897 ] Apache Spark commented on SPARK-32858: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-32858) UnwrapCastInBinaryComparison: support other numeric types

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32858: Assignee: (was: Apache Spark) > UnwrapCastInBinaryComparison: support other numeric t

[jira] [Assigned] (SPARK-32858) UnwrapCastInBinaryComparison: support other numeric types

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32858: Assignee: Apache Spark > UnwrapCastInBinaryComparison: support other numeric types >

[jira] [Updated] (SPARK-32926) Add Scala 2.13 build test in GitHub Action

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32926: -- Component/s: (was: Spark Core) > Add Scala 2.13 build test in GitHub Action >

[jira] [Created] (SPARK-32926) Add Scala 2.13 build test in GitHub Action

2020-09-17 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-32926: - Summary: Add Scala 2.13 build test in GitHub Action Key: SPARK-32926 URL: https://issues.apache.org/jira/browse/SPARK-32926 Project: Spark Issue Type: Test

[jira] [Assigned] (SPARK-32926) Add Scala 2.13 build test in GitHub Action

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32926: Assignee: Apache Spark > Add Scala 2.13 build test in GitHub Action > ---

[jira] [Commented] (SPARK-32926) Add Scala 2.13 build test in GitHub Action

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197909#comment-17197909 ] Apache Spark commented on SPARK-32926: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-32926) Add Scala 2.13 build test in GitHub Action

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32926: Assignee: (was: Apache Spark) > Add Scala 2.13 build test in GitHub Action >

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32635: -- Affects Version/s: 2.3.4 > When pyspark.sql.functions.lit() function is used with dataframe ca

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32635: -- Affects Version/s: 2.2.3 > When pyspark.sql.functions.lit() function is used with dataframe ca

[jira] [Updated] (SPARK-32635) When pyspark.sql.functions.lit() function is used with dataframe cache, it returns wrong result

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32635: -- Affects Version/s: 2.1.3 > When pyspark.sql.functions.lit() function is used with dataframe ca

[jira] [Created] (SPARK-32927) Bitwise OR, AND and XOR should have similar canonicalization rules to boolean OR and AND

2020-09-17 Thread Tanel Kiis (Jira)
Tanel Kiis created SPARK-32927: -- Summary: Bitwise OR, AND and XOR should have similar canonicalization rules to boolean OR and AND Key: SPARK-32927 URL: https://issues.apache.org/jira/browse/SPARK-32927

[jira] [Updated] (SPARK-32927) Bitwise OR, AND and XOR should have similar canonicalization rules to boolean OR and AND

2020-09-17 Thread Tanel Kiis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanel Kiis updated SPARK-32927: --- Priority: Minor (was: Major) > Bitwise OR, AND and XOR should have similar canonicalization rules t

[jira] [Updated] (SPARK-32927) Bitwise OR, AND and XOR should have similar canonicalization rules to boolean OR and AND

2020-09-17 Thread Tanel Kiis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanel Kiis updated SPARK-32927: --- Priority: Major (was: Minor) > Bitwise OR, AND and XOR should have similar canonicalization rules t

[jira] [Assigned] (SPARK-32927) Bitwise OR, AND and XOR should have similar canonicalization rules to boolean OR and AND

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32927: Assignee: Apache Spark > Bitwise OR, AND and XOR should have similar canonicalization rul

[jira] [Commented] (SPARK-32927) Bitwise OR, AND and XOR should have similar canonicalization rules to boolean OR and AND

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197927#comment-17197927 ] Apache Spark commented on SPARK-32927: -- User 'tanelk' has created a pull request fo

[jira] [Assigned] (SPARK-32927) Bitwise OR, AND and XOR should have similar canonicalization rules to boolean OR and AND

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32927: Assignee: (was: Apache Spark) > Bitwise OR, AND and XOR should have similar canonical

[jira] [Commented] (SPARK-32927) Bitwise OR, AND and XOR should have similar canonicalization rules to boolean OR and AND

2020-09-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197928#comment-17197928 ] Apache Spark commented on SPARK-32927: -- User 'tanelk' has created a pull request fo

[jira] [Created] (SPARK-32928) Non-deterministic expressions should not be reordered inside AND and OR

2020-09-17 Thread Tanel Kiis (Jira)
Tanel Kiis created SPARK-32928: -- Summary: Non-deterministic expressions should not be reordered inside AND and OR Key: SPARK-32928 URL: https://issues.apache.org/jira/browse/SPARK-32928 Project: Spark

[jira] [Updated] (SPARK-32928) Non-deterministic expressions should not be reordered inside AND and OR

2020-09-17 Thread Tanel Kiis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanel Kiis updated SPARK-32928: --- Description: Using the splitDisjunctivePredicates and splitConjunctivePredicates helper methods can

[jira] [Created] (SPARK-32929) StreamSuite failure on IBM Z: - SPARK-20432: union one stream with itself

2020-09-17 Thread Michael Munday (Jira)
Michael Munday created SPARK-32929: -- Summary: StreamSuite failure on IBM Z: - SPARK-20432: union one stream with itself Key: SPARK-32929 URL: https://issues.apache.org/jira/browse/SPARK-32929 Project

[jira] [Resolved] (SPARK-32926) Add Scala 2.13 build test in GitHub Action

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32926. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29793 [https://

[jira] [Assigned] (SPARK-32926) Add Scala 2.13 build test in GitHub Action

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32926: - Assignee: Dongjoon Hyun > Add Scala 2.13 build test in GitHub Action >

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2020-09-17 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197965#comment-17197965 ] Terry Kim commented on SPARK-29900: --- These are the outputs from postgres and mysql. I

[jira] [Assigned] (SPARK-32909) Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32909: - Assignee: Yang Jie > Pass all `sql/hive-thriftserver` module UTs in Scala 2.13 > --

[jira] [Resolved] (SPARK-32909) Pass all `sql/hive-thriftserver` module UTs in Scala 2.13

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32909. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29783 [https://

[jira] [Updated] (SPARK-32926) Add Scala 2.13 build test in GitHub Action

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32926: -- Parent: SPARK-25075 Issue Type: Sub-task (was: Test) > Add Scala 2.13 build test in G

[jira] [Resolved] (SPARK-32889) orc table column name doesn't support special characters.

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32889. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29761 [https://

[jira] [Assigned] (SPARK-32889) orc table column name doesn't support special characters.

2020-09-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32889: - Assignee: jason jin > orc table column name doesn't support special characters. > -

  1   2   >