[jira] [Updated] (SPARK-28519) Tests failed on aarch64 due the value of math.log and power function is different

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28519: - Description: Sorry to disturb again, we ran unit tests on arm64 instance, and there are other s

[jira] [Commented] (SPARK-28519) Tests failed on aarch64 due the value of math.log and power function is different

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894302#comment-16894302 ] Hyukjin Kwon commented on SPARK-28519: -- There are multiple issues related with it.

[jira] [Resolved] (SPARK-28522) Pass dynamic parameters to custom file input format

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28522. -- Resolution: Invalid > Pass dynamic parameters to custom file input format > --

[jira] [Commented] (SPARK-28522) Pass dynamic parameters to custom file input format

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894301#comment-16894301 ] Hyukjin Kwon commented on SPARK-28522: -- You can use Hadoop configuration for hadoop

[jira] [Resolved] (SPARK-28526) Kinesis Metric Sink

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28526. -- Resolution: Won't Fix > Kinesis Metric Sink > --- > > Key: SPA

[jira] [Commented] (SPARK-28526) Kinesis Metric Sink

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894300#comment-16894300 ] Hyukjin Kwon commented on SPARK-28526: -- I think that should better be existed as a

[jira] [Resolved] (SPARK-28523) Cant Select from a partitioned table using the Hive Warehouse Connector

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28523. -- Resolution: Invalid This looks an issue in HWC > Cant Select from a partitioned table using t

[jira] [Resolved] (SPARK-28533) PySpark datatype casting error

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28533. -- Resolution: Invalid > PySpark datatype casting error > -- > >

[jira] [Commented] (SPARK-28533) PySpark datatype casting error

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894294#comment-16894294 ] Hyukjin Kwon commented on SPARK-28533: -- That's because you specified the output typ

[jira] [Updated] (SPARK-28533) PySpark datatype casting error

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28533: - Description: Hello, I have faced an issue while casting the datatype of a column in pyspark 2.4

[jira] [Updated] (SPARK-28533) PySpark datatype casting error

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28533: - Description: Hello, I have faced an issue while casting the datatype of a column in pyspark 2.4

[jira] [Updated] (SPARK-28533) PySpark datatype casting error

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28533: - Description: Hello, I have faced an issue while casting the datatype of a column in pyspark 2.4

[jira] [Resolved] (SPARK-28530) Fix Join Reorder optimizer rule to enforce idempotence

2019-07-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-28530. - Resolution: Fixed Assignee: Yesheng Ma Fix Version/s: 3.0.0 > Fix Join Reorder optimizer

[jira] [Assigned] (SPARK-28532) Fix subquery optimizer rule to enforce idempotence

2019-07-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-28532: --- Assignee: Yesheng Ma > Fix subquery optimizer rule to enforce idempotence > ---

[jira] [Resolved] (SPARK-28532) Fix subquery optimizer rule to enforce idempotence

2019-07-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-28532. - Resolution: Fixed Fix Version/s: 3.0.0 > Fix subquery optimizer rule to enforce idempotence > ---

[jira] [Comment Edited] (SPARK-28528) Fix Idempotence for Once batches in Catalyst optimizer

2019-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894269#comment-16894269 ] Dongjoon Hyun edited comment on SPARK-28528 at 7/27/19 5:04 AM: --

[jira] [Commented] (SPARK-28528) Fix Idempotence for Once batches in Catalyst optimizer

2019-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894269#comment-16894269 ] Dongjoon Hyun commented on SPARK-28528: --- [~manifoldQAQ]. All issues seems to be `I

[jira] [Commented] (SPARK-28450) When scan hive data of a not existed partition, it return an error

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894264#comment-16894264 ] Hyukjin Kwon commented on SPARK-28450: -- [~angerszhuuu], please provide the self-con

[jira] [Resolved] (SPARK-28450) When scan hive data of a not existed partition, it return an error

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28450. -- Resolution: Incomplete > When scan hive data of a not existed partition, it return an error >

[jira] [Resolved] (SPARK-28480) Types of input parameters of a UDF affect the ability to cache the result

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28480. -- Resolution: Cannot Reproduce > Types of input parameters of a UDF affect the ability to cache

[jira] [Resolved] (SPARK-28442) Potentially persist data without leadership

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28442. -- Resolution: Incomplete No feedback. > Potentially persist data without leadership > -

[jira] [Created] (SPARK-28536) Reduce shuffle partitions in Python UDF tests in SQLQueryTestSuite

2019-07-26 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-28536: Summary: Reduce shuffle partitions in Python UDF tests in SQLQueryTestSuite Key: SPARK-28536 URL: https://issues.apache.org/jira/browse/SPARK-28536 Project: Spark

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

2019-07-26 Thread Nasir Ali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894238#comment-16894238 ] Nasir Ali commented on SPARK-28502: --- I tried to set timzone to UTC as suggested by [~i

[jira] [Assigned] (SPARK-28441) PythonUDF used in correlated scalar subquery causes UnsupportedOperationException

2019-07-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28441: --- Assignee: Liang-Chi Hsieh > PythonUDF used in correlated scalar subquery causes > Unsuppor

[jira] [Resolved] (SPARK-28441) PythonUDF used in correlated scalar subquery causes UnsupportedOperationException

2019-07-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28441. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25204 [https://gith

[jira] [Updated] (SPARK-28535) Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"

2019-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28535: -- Affects Version/s: 2.3.3 2.4.3 > Flaky test: JobCancellationSuite."inte

[jira] [Created] (SPARK-28535) Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"

2019-07-26 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-28535: -- Summary: Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader" Key: SPARK-28535 URL: https://issues.apache.org/jira/browse/SPARK-28535 Proje

[jira] [Created] (SPARK-28534) Update node affinity for DockerForDesktop backend in PVTestsSuite

2019-07-26 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-28534: - Summary: Update node affinity for DockerForDesktop backend in PVTestsSuite Key: SPARK-28534 URL: https://issues.apache.org/jira/browse/SPARK-28534 Project: Spark

[jira] [Updated] (SPARK-28533) PySpark datatype casting error

2019-07-26 Thread RoopTeja Muppalla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] RoopTeja Muppalla updated SPARK-28533: -- Summary: PySpark datatype casting error (was: Spark datatype error) > PySpark datatyp

[jira] [Created] (SPARK-28533) Spark datatype error

2019-07-26 Thread RoopTeja Muppalla (JIRA)
RoopTeja Muppalla created SPARK-28533: - Summary: Spark datatype error Key: SPARK-28533 URL: https://issues.apache.org/jira/browse/SPARK-28533 Project: Spark Issue Type: Bug Comp

[jira] [Resolved] (SPARK-28518) Fix StatisticsCollectionTestBase#getDataSize refer to ChecksumFileSystem#isChecksumFile

2019-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28518. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25259 [https://

[jira] [Assigned] (SPARK-28518) Fix StatisticsCollectionTestBase#getDataSize refer to ChecksumFileSystem#isChecksumFile

2019-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28518: - Assignee: Yuming Wang > Fix StatisticsCollectionTestBase#getDataSize refer to > Checks

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

2019-07-26 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894134#comment-16894134 ] Li Jin commented on SPARK-28502: Hmm.. I think this has sth to do with timezone, can you

[jira] [Assigned] (SPARK-25285) Add executor task metrics to track the number of tasks started and of tasks successfully completed

2019-07-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25285: -- Assignee: Luca Canali > Add executor task metrics to track the number of tasks starte

[jira] [Resolved] (SPARK-25285) Add executor task metrics to track the number of tasks started and of tasks successfully completed

2019-07-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25285. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22290 [https:

[jira] [Assigned] (SPARK-27889) Make development scripts under dev/ support Python 3

2019-07-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-27889: --- Assignee: Weichen Xu (was: Xiao Li) > Make development scripts under dev/ support Python 3 > -

[jira] [Issue Comment Deleted] (SPARK-28476) Support ALTER DATABASE SET LOCATION

2019-07-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-28476: Comment: was deleted (was: [~ivy.weichan.xu]  ) > Support ALTER DATABASE SET LOCATION >

[jira] [Created] (SPARK-28532) Fix subquery optimizer rule to enforce idempotence

2019-07-26 Thread Yesheng Ma (JIRA)
Yesheng Ma created SPARK-28532: -- Summary: Fix subquery optimizer rule to enforce idempotence Key: SPARK-28532 URL: https://issues.apache.org/jira/browse/SPARK-28532 Project: Spark Issue Type: Su

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

2019-07-26 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894025#comment-16894025 ] Bryan Cutler commented on SPARK-28502: -- I'm not sure, but I don't think you can use

[jira] [Created] (SPARK-28531) Fix Extract Python UDFs optimizer rule to enforce idempotence

2019-07-26 Thread Yesheng Ma (JIRA)
Yesheng Ma created SPARK-28531: -- Summary: Fix Extract Python UDFs optimizer rule to enforce idempotence Key: SPARK-28531 URL: https://issues.apache.org/jira/browse/SPARK-28531 Project: Spark Is

[jira] [Created] (SPARK-28530) Fix Join Reorder optimizer rule to enforce idempotence

2019-07-26 Thread Yesheng Ma (JIRA)
Yesheng Ma created SPARK-28530: -- Summary: Fix Join Reorder optimizer rule to enforce idempotence Key: SPARK-28530 URL: https://issues.apache.org/jira/browse/SPARK-28530 Project: Spark Issue Type

[jira] [Created] (SPARK-28529) Fix PullupCorrelatedPredicates optimizer rule to enforce idempotence

2019-07-26 Thread Yesheng Ma (JIRA)
Yesheng Ma created SPARK-28529: -- Summary: Fix PullupCorrelatedPredicates optimizer rule to enforce idempotence Key: SPARK-28529 URL: https://issues.apache.org/jira/browse/SPARK-28529 Project: Spark

[jira] [Created] (SPARK-28528) Fix Idempotence for Once batches in Catalyst optimizer

2019-07-26 Thread Yesheng Ma (JIRA)
Yesheng Ma created SPARK-28528: -- Summary: Fix Idempotence for Once batches in Catalyst optimizer Key: SPARK-28528 URL: https://issues.apache.org/jira/browse/SPARK-28528 Project: Spark Issue Type

[jira] [Created] (SPARK-28527) Build a Test Framework for Thriftserver

2019-07-26 Thread Xiao Li (JIRA)
Xiao Li created SPARK-28527: --- Summary: Build a Test Framework for Thriftserver Key: SPARK-28527 URL: https://issues.apache.org/jira/browse/SPARK-28527 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-28527) Build a Test Framework for Thriftserver

2019-07-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894015#comment-16894015 ] Xiao Li commented on SPARK-28527: - [~yumwang]  > Build a Test Framework for Thriftserve

[jira] [Assigned] (SPARK-28507) remove deprecated API context(self, sqlContext) from pyspark/ml/util.py

2019-07-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28507: - Assignee: Huaxin Gao > remove deprecated API context(self, sqlContext) from pyspark/ml/util.py

[jira] [Created] (SPARK-28526) Kinesis Metric Sink

2019-07-26 Thread Nick Karpov (JIRA)
Nick Karpov created SPARK-28526: --- Summary: Kinesis Metric Sink Key: SPARK-28526 URL: https://issues.apache.org/jira/browse/SPARK-28526 Project: Spark Issue Type: New Feature Component

[jira] [Resolved] (SPARK-28507) remove deprecated API context(self, sqlContext) from pyspark/ml/util.py

2019-07-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28507. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25246 [https://github.c

[jira] [Updated] (SPARK-28526) Kinesis Metric Sink

2019-07-26 Thread Nick Karpov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Karpov updated SPARK-28526: Description: Spark currently supports ConsoleSink, CSVSink, JmxSink, MetricsServlet, GraphiteSink

[jira] [Created] (SPARK-28525) Allow Launcher to be applied Java options

2019-07-26 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-28525: -- Summary: Allow Launcher to be applied Java options Key: SPARK-28525 URL: https://issues.apache.org/jira/browse/SPARK-28525 Project: Spark Issue Type: Imp

[jira] [Resolved] (SPARK-28524) ThriftServerTab lost error message

2019-07-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-28524. - Resolution: Fixed Assignee: Yuming Wang Fix Version/s: 3.0.0 > ThriftServerTab lost erro

[jira] [Updated] (SPARK-28260) Add CLOSED state to ExecutionState

2019-07-26 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28260: Issue Type: Sub-task (was: Improvement) Parent: SPARK-28426 > Add CLOSED state to Executi

[jira] [Updated] (SPARK-28524) ThriftServerTab lost error message

2019-07-26 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28524: Description: !3.0.0.png! !2.4.3.png! > ThriftServerTab lost error message > ---

[jira] [Updated] (SPARK-28524) ThriftServerTab lost error message

2019-07-26 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28524: Attachment: 2.4.3.png 3.0.0.png > ThriftServerTab lost error message > ---

[jira] [Created] (SPARK-28524) ThriftServerTab lost error message

2019-07-26 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28524: --- Summary: ThriftServerTab lost error message Key: SPARK-28524 URL: https://issues.apache.org/jira/browse/SPARK-28524 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-28523) Cant Select from a partitioned table using the Hive Warehouse Connector

2019-07-26 Thread Varun Rao (JIRA)
Varun Rao created SPARK-28523: - Summary: Cant Select from a partitioned table using the Hive Warehouse Connector Key: SPARK-28523 URL: https://issues.apache.org/jira/browse/SPARK-28523 Project: Spark

[jira] [Resolved] (SPARK-28499) Optimize MinMaxScaler

2019-07-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28499. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25244 [https://github.c

[jira] [Assigned] (SPARK-28499) Optimize MinMaxScaler

2019-07-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28499: - Assignee: zhengruifeng > Optimize MinMaxScaler > - > > Key:

[jira] [Commented] (SPARK-16996) Hive ACID delta files not seen

2019-07-26 Thread Abhishek Somani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893807#comment-16893807 ] Abhishek Somani commented on SPARK-16996: - We have worked on and open sourced a

[jira] [Commented] (SPARK-15348) Hive ACID

2019-07-26 Thread Abhishek Somani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893806#comment-16893806 ] Abhishek Somani commented on SPARK-15348: - We have worked on and open sourced a

[jira] [Commented] (SPARK-28505) Add data source option for omitting partitioned columns when saving to file

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893698#comment-16893698 ] Hyukjin Kwon commented on SPARK-28505: -- I don't quite understand. Hive also reads p

[jira] [Resolved] (SPARK-28505) Add data source option for omitting partitioned columns when saving to file

2019-07-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28505. -- Resolution: Invalid > Add data source option for omitting partitioned columns when saving to f

[jira] [Issue Comment Deleted] (SPARK-28521) Fix error message for built-in functions

2019-07-26 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28521: Comment: was deleted (was: I'm working on.) > Fix error message for built-in functions >

[jira] [Created] (SPARK-28522) Pass dynamic parameters to custom file input format

2019-07-26 Thread Ayan Mukherjee (JIRA)
Ayan Mukherjee created SPARK-28522: -- Summary: Pass dynamic parameters to custom file input format Key: SPARK-28522 URL: https://issues.apache.org/jira/browse/SPARK-28522 Project: Spark Issue

[jira] [Assigned] (SPARK-25809) Support additional K8S cluster types for integration tests

2019-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25809: - Assignee: Rob Vesse > Support additional K8S cluster types for integration tests >

[jira] [Updated] (SPARK-25809) Support additional K8S cluster types for integration tests

2019-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25809: -- Fix Version/s: 3.0.0 > Support additional K8S cluster types for integration tests > --

[jira] [Commented] (SPARK-28484) spark-submit uses wrong SPARK_HOME with deploy-mode "cluster"

2019-07-26 Thread Kalle Jepsen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893448#comment-16893448 ] Kalle Jepsen commented on SPARK-28484: -- thank you > spark-submit uses wrong SPARK_

[jira] [Commented] (SPARK-28521) Fix error message for built-in functions

2019-07-26 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893438#comment-16893438 ] Yuming Wang commented on SPARK-28521: - [https://github.com/apache/spark/blob/832ff87

[jira] [Commented] (SPARK-28521) Fix error message for built-in functions

2019-07-26 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893429#comment-16893429 ] Yuming Wang commented on SPARK-28521: - I'm working on. > Fix error message for buil

[jira] [Created] (SPARK-28521) Fix error message for built-in functions

2019-07-26 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28521: --- Summary: Fix error message for built-in functions Key: SPARK-28521 URL: https://issues.apache.org/jira/browse/SPARK-28521 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-28520) WholeStageCodegen does not work property for LocalTableScanExec

2019-07-26 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-28520: -- Summary: WholeStageCodegen does not work property for LocalTableScanExec Key: SPARK-28520 URL: https://issues.apache.org/jira/browse/SPARK-28520 Project: Spark

[jira] [Resolved] (SPARK-28489) KafkaOffsetRangeCalculator.getRanges may drop offsets

2019-07-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28489. --- Resolution: Fixed Fix Version/s: 3.0.0 2.4.4 This is resolved via

[jira] [Updated] (SPARK-28519) Tests failed on aarch64 due the value of math.log and power function is different

2019-07-26 Thread huangtianhua (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangtianhua updated SPARK-28519: - Description: Sorry to disturb again, we ran unit tests on arm64 instance, and there are other s

[jira] [Created] (SPARK-28519) Tests failed on aarch64 due the value of math.log and power function is different

2019-07-26 Thread huangtianhua (JIRA)
huangtianhua created SPARK-28519: Summary: Tests failed on aarch64 due the value of math.log and power function is different Key: SPARK-28519 URL: https://issues.apache.org/jira/browse/SPARK-28519 Pro