[jira] [Assigned] (SPARK-25418) The metadata of DataSource table should not include Hive-generated storage properties.

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25418: Assignee: (was: Apache Spark) > The metadata of DataSource table should not include

[jira] [Assigned] (SPARK-25418) The metadata of DataSource table should not include Hive-generated storage properties.

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25418: Assignee: Apache Spark > The metadata of DataSource table should not include

[jira] [Commented] (SPARK-25418) The metadata of DataSource table should not include Hive-generated storage properties.

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613040#comment-16613040 ] Apache Spark commented on SPARK-25418: -- User 'ueshin' has created a pull request for this issue:

[jira] [Created] (SPARK-25418) The metadata of DataSource table should not include Hive-generated storage properties.

2018-09-12 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-25418: - Summary: The metadata of DataSource table should not include Hive-generated storage properties. Key: SPARK-25418 URL: https://issues.apache.org/jira/browse/SPARK-25418

[jira] [Commented] (SPARK-25352) Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613033#comment-16613033 ] Apache Spark commented on SPARK-25352: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-25352) Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613032#comment-16613032 ] Apache Spark commented on SPARK-25352: -- User 'viirya' has created a pull request for this issue:

[jira] [Resolved] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

2018-09-12 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li resolved SPARK-25295. -- Resolution: Fixed Fix Version/s: 2.4.0 > Pod names conflicts in client mode, if previous

[jira] [Updated] (SPARK-25415) Make plan change log in RuleExecutor configurable by SQLConf

2018-09-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25415: Priority: Major (was: Minor) > Make plan change log in RuleExecutor configurable by SQLConf >

[jira] [Resolved] (SPARK-25415) Make plan change log in RuleExecutor configurable by SQLConf

2018-09-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25415. - Resolution: Fixed Assignee: Maryann Xue Fix Version/s: 3.0.0 > Make plan change log in

[jira] [Commented] (SPARK-25397) SparkSession.conf fails when given default value with Python 3

2018-09-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612994#comment-16612994 ] Hyukjin Kwon commented on SPARK-25397: -- [~josephkb], do you want to backport this bit or just

[jira] [Commented] (SPARK-25378) ArrayData.toArray(StringType) assume UTF8String in 2.4

2018-09-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612965#comment-16612965 ] Hyukjin Kwon commented on SPARK-25378: -- {quote} If it is not pubic, why didn't we hide it in the

[jira] [Commented] (SPARK-25378) ArrayData.toArray(StringType) assume UTF8String in 2.4

2018-09-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612947#comment-16612947 ] Wenchen Fan commented on SPARK-25378: - [~viirya] Can you take a look and see how hard it is to fix

[jira] [Assigned] (SPARK-25357) Add metadata to SparkPlanInfo to dump more information like file path to event log

2018-09-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25357: --- Assignee: Lantao Jin > Add metadata to SparkPlanInfo to dump more information like file

[jira] [Resolved] (SPARK-25402) Null handling in BooleanSimplification

2018-09-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25402. - Resolution: Fixed Fix Version/s: 2.4.0 2.3.2 > Null handling in

[jira] [Resolved] (SPARK-25357) Add metadata to SparkPlanInfo to dump more information like file path to event log

2018-09-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25357. - Resolution: Fixed Fix Version/s: 2.3.2 2.4.0 Issue resolved by pull

[jira] [Resolved] (SPARK-25387) Malformed CSV causes NPE

2018-09-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25387. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22374

[jira] [Assigned] (SPARK-25387) Malformed CSV causes NPE

2018-09-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25387: --- Assignee: Maxim Gekk > Malformed CSV causes NPE > > >

[jira] [Resolved] (SPARK-23820) Allow the long form of call sites to be recorded in the log

2018-09-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23820. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22398

[jira] [Comment Edited] (SPARK-25291) Flakiness of tests in terms of executor memory (SecretsTestSuite)

2018-09-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612894#comment-16612894 ] Stavros Kontopoulos edited comment on SPARK-25291 at 9/13/18 12:41 AM:

[jira] [Comment Edited] (SPARK-25291) Flakiness of tests in terms of executor memory (SecretsTestSuite)

2018-09-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612894#comment-16612894 ] Stavros Kontopoulos edited comment on SPARK-25291 at 9/13/18 12:41 AM:

[jira] [Commented] (SPARK-25291) Flakiness of tests in terms of executor memory (SecretsTestSuite)

2018-09-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612894#comment-16612894 ] Stavros Kontopoulos commented on SPARK-25291: - [~ifilonenko] I can have a look a bit weird.

[jira] [Commented] (SPARK-23012) Support for predicate pushdown and partition pruning when left joining large Hive tables

2018-09-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612881#comment-16612881 ] Yuming Wang commented on SPARK-23012: - It seems the following PR resolves your issue:

[jira] [Assigned] (SPARK-25416) ArrayPosition function may return incorrect result when right expression is implicitly downcasted.

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25416: Assignee: Apache Spark > ArrayPosition function may return incorrect result when right

[jira] [Commented] (SPARK-25416) ArrayPosition function may return incorrect result when right expression is implicitly downcasted.

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612817#comment-16612817 ] Apache Spark commented on SPARK-25416: -- User 'dilipbiswal' has created a pull request for this

[jira] [Commented] (SPARK-25416) ArrayPosition function may return incorrect result when right expression is implicitly downcasted.

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612818#comment-16612818 ] Apache Spark commented on SPARK-25416: -- User 'dilipbiswal' has created a pull request for this

[jira] [Assigned] (SPARK-25416) ArrayPosition function may return incorrect result when right expression is implicitly downcasted.

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25416: Assignee: (was: Apache Spark) > ArrayPosition function may return incorrect result

[jira] [Assigned] (SPARK-25417) ArrayContains function may return incorrect result when right expression is implicitly down casted

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25417: Assignee: Apache Spark > ArrayContains function may return incorrect result when right

[jira] [Commented] (SPARK-25417) ArrayContains function may return incorrect result when right expression is implicitly down casted

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612816#comment-16612816 ] Apache Spark commented on SPARK-25417: -- User 'dilipbiswal' has created a pull request for this

[jira] [Assigned] (SPARK-25417) ArrayContains function may return incorrect result when right expression is implicitly down casted

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25417: Assignee: (was: Apache Spark) > ArrayContains function may return incorrect result

[jira] [Created] (SPARK-25417) ArrayContains function may return incorrect result when right expression is implicitly down casted

2018-09-12 Thread Dilip Biswal (JIRA)
Dilip Biswal created SPARK-25417: Summary: ArrayContains function may return incorrect result when right expression is implicitly down casted Key: SPARK-25417 URL:

[jira] [Commented] (SPARK-25415) Make plan change log in RuleExecutor configurable by SQLConf

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612804#comment-16612804 ] Apache Spark commented on SPARK-25415: -- User 'dilipbiswal' has created a pull request for this

[jira] [Commented] (SPARK-25415) Make plan change log in RuleExecutor configurable by SQLConf

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612803#comment-16612803 ] Apache Spark commented on SPARK-25415: -- User 'dilipbiswal' has created a pull request for this

[jira] [Assigned] (SPARK-25415) Make plan change log in RuleExecutor configurable by SQLConf

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25415: Assignee: (was: Apache Spark) > Make plan change log in RuleExecutor configurable by

[jira] [Commented] (SPARK-25415) Make plan change log in RuleExecutor configurable by SQLConf

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612766#comment-16612766 ] Apache Spark commented on SPARK-25415: -- User 'maryannxue' has created a pull request for this

[jira] [Assigned] (SPARK-25415) Make plan change log in RuleExecutor configurable by SQLConf

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25415: Assignee: Apache Spark > Make plan change log in RuleExecutor configurable by SQLConf >

[jira] [Updated] (SPARK-25416) ArrayPosition function may return incorrect result when right expression is implicitly downcasted.

2018-09-12 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dilip Biswal updated SPARK-25416: - Summary: ArrayPosition function may return incorrect result when right expression is implicitly

[jira] [Created] (SPARK-25416) ArrayPosition may return incorrect result when right expression is downcasted.

2018-09-12 Thread Dilip Biswal (JIRA)
Dilip Biswal created SPARK-25416: Summary: ArrayPosition may return incorrect result when right expression is downcasted. Key: SPARK-25416 URL: https://issues.apache.org/jira/browse/SPARK-25416

[jira] [Created] (SPARK-25415) Make plan change log in RuleExecutor configurable by SQLConf

2018-09-12 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-25415: --- Summary: Make plan change log in RuleExecutor configurable by SQLConf Key: SPARK-25415 URL: https://issues.apache.org/jira/browse/SPARK-25415 Project: Spark

[jira] [Commented] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612662#comment-16612662 ] Apache Spark commented on SPARK-25295: -- User 'skonto' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25295: Assignee: Apache Spark > Pod names conflicts in client mode, if previous submission was

[jira] [Assigned] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25295: Assignee: (was: Apache Spark) > Pod names conflicts in client mode, if previous

[jira] [Updated] (SPARK-20350) Apply Complementation Laws during boolean expression simplification

2018-09-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20350: -- Component/s: (was: Optimizer) SQL > Apply Complementation Laws during

[jira] [Closed] (SPARK-20799) Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL

2018-09-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-20799. - > Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL >

[jira] [Resolved] (SPARK-20799) Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL

2018-09-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-20799. --- Resolution: Won't Fix > Unable to infer schema for ORC/Parquet on S3N when secrets are in

[jira] [Commented] (SPARK-20799) Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL

2018-09-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612642#comment-16612642 ] Dongjoon Hyun commented on SPARK-20799: --- +1 for closing this as a WONTFIX. > Unable to infer

[jira] [Commented] (SPARK-25402) Null handling in BooleanSimplification

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612530#comment-16612530 ] Apache Spark commented on SPARK-25402: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-25402) Null handling in BooleanSimplification

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612529#comment-16612529 ] Apache Spark commented on SPARK-25402: -- User 'gatorsmile' has created a pull request for this

[jira] [Resolved] (SPARK-25363) Schema pruning doesn't work if nested column is used in where clause

2018-09-12 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-25363. - Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Issue resolved by pull request

[jira] [Commented] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

2018-09-12 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612436#comment-16612436 ] Joseph K. Bradley commented on SPARK-25321: --- You're right; these are breaking changes. If

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

2018-09-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612433#comment-16612433 ] Marcelo Vanzin commented on SPARK-25380: Yep. That's a 200MB plan description string... >

[jira] [Commented] (SPARK-25378) ArrayData.toArray(StringType) assume UTF8String in 2.4

2018-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612399#comment-16612399 ] Xiangrui Meng commented on SPARK-25378: --- Comments from [~vomjom] at

[jira] [Commented] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

2018-09-12 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612337#comment-16612337 ] Stavros Kontopoulos commented on SPARK-25295: - guys I started working on a short fix. > Pod

[jira] [Assigned] (SPARK-25352) Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold

2018-09-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25352: --- Assignee: Liang-Chi Hsieh > Perform ordered global limit when limit number is bigger than

[jira] [Resolved] (SPARK-25352) Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold

2018-09-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25352. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22344

[jira] [Comment Edited] (SPARK-24627) [Spark2.3.0] After HDFS Token expire kinit not able to submit job using beeline

2018-09-12 Thread Ayush Anubhava (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612085#comment-16612085 ] Ayush Anubhava edited comment on SPARK-24627 at 9/12/18 12:53 PM: --

[jira] [Commented] (SPARK-24627) [Spark2.3.0] After HDFS Token expire kinit not able to submit job using beeline

2018-09-12 Thread Ayush Anubhava (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612085#comment-16612085 ] Ayush Anubhava commented on SPARK-24627: Check the the principal name given in spark-default

[jira] [Resolved] (SPARK-25371) Vector Assembler with no input columns leads to opaque error

2018-09-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25371. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0

[jira] [Commented] (SPARK-25150) Joining DataFrames derived from the same source yields confusing/incorrect results

2018-09-12 Thread Evelyn Bayes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612026#comment-16612026 ] Evelyn Bayes commented on SPARK-25150: -- Hey Peter, don't stress it. I'm new to the community as

[jira] [Commented] (SPARK-20153) Support Multiple aws credentials in order to access multiple Hive on S3 table in spark application

2018-09-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612001#comment-16612001 ] Steve Loughran commented on SPARK-20153: bq. Amazon EMR does not currently support use of the

[jira] [Commented] (SPARK-20799) Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL

2018-09-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611995#comment-16611995 ] Steve Loughran commented on SPARK-20799: Update: Hadoop 3.3+ will remove all support for

[jira] [Assigned] (SPARK-25414) The numInputRows metrics can be incorrect for streaming self-join

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25414: Assignee: Wenchen Fan (was: Apache Spark) > The numInputRows metrics can be incorrect

[jira] [Assigned] (SPARK-25414) The numInputRows metrics can be incorrect for streaming self-join

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25414: Assignee: Apache Spark (was: Wenchen Fan) > The numInputRows metrics can be incorrect

[jira] [Commented] (SPARK-25414) The numInputRows metrics can be incorrect for streaming self-join

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611932#comment-16611932 ] Apache Spark commented on SPARK-25414: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-25414) The numInputRows metrics can be incorrect for streaming self-join

2018-09-12 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25414: --- Summary: The numInputRows metrics can be incorrect for streaming self-join Key: SPARK-25414 URL: https://issues.apache.org/jira/browse/SPARK-25414 Project: Spark

[jira] [Updated] (SPARK-25413) [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done

2018-09-12 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeep katta updated SPARK-25413: -- Attachment: decimalBoundaryDataHive.csv > [2.3.2.rc5 Blocker] Precision Value is going for

[jira] [Updated] (SPARK-25413) [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done

2018-09-12 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeep katta updated SPARK-25413: -- Attachment: (was: decimalBoundaryDataHive.csv) > [2.3.2.rc5 Blocker] Precision Value is

[jira] [Updated] (SPARK-25413) [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done

2018-09-12 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeep katta updated SPARK-25413: -- Description: sql("create table if not exists hiveBigDecimal(ID Int, date Timestamp, country

[jira] [Commented] (SPARK-25413) [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done

2018-09-12 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611759#comment-16611759 ] Ajith S commented on SPARK-25413: - Thank you for reporting the issue sandeep. I think the problem is

[jira] [Assigned] (SPARK-25413) [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25413: Assignee: (was: Apache Spark) > [2.3.2.rc5 Blocker] Precision Value is going for

[jira] [Commented] (SPARK-25413) [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611758#comment-16611758 ] Apache Spark commented on SPARK-25413: -- User 'ajithme' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25413) [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done

2018-09-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25413: Assignee: Apache Spark > [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg

[jira] [Updated] (SPARK-25413) [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done

2018-09-12 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeep katta updated SPARK-25413: -- Priority: Blocker (was: Minor) > [2.3.2.rc5 Blocker] Precision Value is going for toss when

[jira] [Updated] (SPARK-25413) [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done

2018-09-12 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeep katta updated SPARK-25413: -- Attachment: decimalBoundaryDataHive.csv > [2.3.2.rc5 Blocker] Precision Value is going for

[jira] [Created] (SPARK-25413) [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done

2018-09-12 Thread sandeep katta (JIRA)
sandeep katta created SPARK-25413: - Summary: [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done Key: SPARK-25413 URL: https://issues.apache.org/jira/browse/SPARK-25413 Project:

[jira] [Commented] (SPARK-21542) Helper functions for custom Python Persistence

2018-09-12 Thread Peter Knight (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611733#comment-16611733 ] Peter Knight commented on SPARK-21542: -- Thanks for the replay [~JohnHBauer]. Yes I am usingĀ 

[jira] [Commented] (SPARK-20184) performance regression for complex/long sql when enable whole stage codegen

2018-09-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611717#comment-16611717 ] Kazuaki Ishizaki commented on SPARK-20184: -- In {{branch-2.4}}, we still see the performance