[jira] [Resolved] (SPARK-27182) Move the conflict source code of the sql/core module to sql/core/v1.2.1

2019-03-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-27182. - Resolution: Fixed Assignee: Yuming Wang Fix Version/s: 3.0.0 > Move the conflict source

[jira] [Commented] (SPARK-25464) Dropping database can remove the hive warehouse directory contents

2019-03-26 Thread Sandeep Katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802442#comment-16802442 ] Sandeep Katta commented on SPARK-25464: --- as per the discussion in

[jira] [Commented] (SPARK-27289) spark-submit explicit configuration does not take effect but Spark UI shows it's effective

2019-03-26 Thread Udbhav Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802441#comment-16802441 ] Udbhav Agrawal commented on SPARK-27289: thanks, i will check this issue  > spark-submit

[jira] [Updated] (SPARK-27289) spark-submit explicit configuration does not take effect but Spark UI shows it's effective

2019-03-26 Thread KaiXu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-27289: -- Attachment: Capture.PNG > spark-submit explicit configuration does not take effect but Spark UI shows > it's

[jira] [Created] (SPARK-27289) spark-submit explicit configuration does not take effect but Spark UI shows it's effective

2019-03-26 Thread KaiXu (JIRA)
KaiXu created SPARK-27289: - Summary: spark-submit explicit configuration does not take effect but Spark UI shows it's effective Key: SPARK-27289 URL: https://issues.apache.org/jira/browse/SPARK-27289

[jira] [Commented] (SPARK-27224) Spark to_json parses UTC timestamp incorrectly

2019-03-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802346#comment-16802346 ] Hyukjin Kwon commented on SPARK-27224: -- Can you try to set {{timestampFormat}}? > Spark to_json

[jira] [Commented] (SPARK-27282) Spark incorrect results when using UNION with GROUP BY clause

2019-03-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802338#comment-16802338 ] Hyukjin Kwon commented on SPARK-27282: -- What's Toolkit? can you make the code self-reproducible? >

[jira] [Created] (SPARK-27288) Pruning nested field in complex map key from object serializers

2019-03-26 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-27288: --- Summary: Pruning nested field in complex map key from object serializers Key: SPARK-27288 URL: https://issues.apache.org/jira/browse/SPARK-27288 Project: Spark

[jira] [Assigned] (SPARK-27285) Support describing output of a CTE

2019-03-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27285: --- Assignee: Dilip Biswal > Support describing output of a CTE >

[jira] [Resolved] (SPARK-27285) Support describing output of a CTE

2019-03-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27285. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24224

[jira] [Resolved] (SPARK-27269) File source v2 should validate data schema only

2019-03-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27269. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24203

[jira] [Assigned] (SPARK-27269) File source v2 should validate data schema only

2019-03-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27269: Assignee: Gengliang Wang > File source v2 should validate data schema only >

[jira] [Resolved] (SPARK-27275) Potential corruption in EncryptedMessage.transferTo

2019-03-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27275. - Resolution: Fixed Fix Version/s: 3.0.0 > Potential corruption in

[jira] [Resolved] (SPARK-27242) Avoid using default time zone in formatting TIMESTAMP/DATE literals

2019-03-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27242. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24181

[jira] [Assigned] (SPARK-27242) Avoid using default time zone in formatting TIMESTAMP/DATE literals

2019-03-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27242: --- Assignee: Maxim Gekk > Avoid using default time zone in formatting TIMESTAMP/DATE literals

[jira] [Created] (SPARK-27287) PCAModel.load() does not honor spark configs

2019-03-26 Thread Dharmesh Kakadia (JIRA)
Dharmesh Kakadia created SPARK-27287: Summary: PCAModel.load() does not honor spark configs Key: SPARK-27287 URL: https://issues.apache.org/jira/browse/SPARK-27287 Project: Spark Issue

[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

2019-03-26 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802202#comment-16802202 ] Imran Rashid commented on SPARK-27220: -- sorry for the delay. i don't have a really strong opinion

[jira] [Assigned] (SPARK-24793) Make spark-submit more useful with k8s

2019-03-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24793: -- Assignee: Stavros Kontopoulos > Make spark-submit more useful with k8s >

[jira] [Resolved] (SPARK-24793) Make spark-submit more useful with k8s

2019-03-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24793. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23599

[jira] [Created] (SPARK-27286) Handles exceptions on proceeding to next record in FilePartitionReader

2019-03-26 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-27286: -- Summary: Handles exceptions on proceeding to next record in FilePartitionReader Key: SPARK-27286 URL: https://issues.apache.org/jira/browse/SPARK-27286 Project:

[jira] [Assigned] (SPARK-24432) Add support for dynamic resource allocation

2019-03-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24432: -- Assignee: Marcelo Vanzin > Add support for dynamic resource allocation >

[jira] [Assigned] (SPARK-24432) Add support for dynamic resource allocation

2019-03-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24432: -- Assignee: (was: Marcelo Vanzin) > Add support for dynamic resource allocation >

[jira] [Updated] (SPARK-27276) Increase the minimum pyarrow version to 0.12.0

2019-03-26 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-27276: - Description: The current minimum version is 0.8.0, which is pretty ancient since Arrow has

[jira] [Updated] (SPARK-27269) File source v2 should validate data schema only

2019-03-26 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-27269: --- Issue Type: Bug (was: Task) > File source v2 should validate data schema only >

[jira] [Commented] (SPARK-21097) Dynamic allocation will preserve cached data

2019-03-26 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801960#comment-16801960 ] Imran Rashid commented on SPARK-21097: -- {quote} I'm wondering if this is going to be subsumed by

[jira] [Commented] (SPARK-27282) Spark incorrect results when using UNION with GROUP BY clause

2019-03-26 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801930#comment-16801930 ] Marco Gaido commented on SPARK-27282: - Please do not use Blocker/Critical as they are reserved for

[jira] [Updated] (SPARK-27282) Spark incorrect results when using UNION with GROUP BY clause

2019-03-26 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-27282: Priority: Major (was: Blocker) > Spark incorrect results when using UNION with GROUP BY clause >

[jira] [Updated] (SPARK-27285) Support describing output of a CTE

2019-03-26 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dilip Biswal updated SPARK-27285: - Priority: Minor (was: Major) > Support describing output of a CTE >

[jira] [Created] (SPARK-27285) Support describing output of a CTE

2019-03-26 Thread Dilip Biswal (JIRA)
Dilip Biswal created SPARK-27285: Summary: Support describing output of a CTE Key: SPARK-27285 URL: https://issues.apache.org/jira/browse/SPARK-27285 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-27224) Spark to_json parses UTC timestamp incorrectly

2019-03-26 Thread Jurriaan Pruis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801802#comment-16801802 ] Jurriaan Pruis commented on SPARK-27224: [~hyukjin.kwon] is this the same as

[jira] [Commented] (SPARK-17914) Spark SQL casting to TimestampType with nanosecond results in incorrect timestamp

2019-03-26 Thread Jurriaan Pruis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801795#comment-16801795 ] Jurriaan Pruis commented on SPARK-17914: I'm also seeing this issue where the millisecond part

[jira] [Commented] (SPARK-27283) BigDecimal arithmetic losing precision

2019-03-26 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801778#comment-16801778 ] Marco Gaido commented on SPARK-27283: - [~Mats_SX] another issue which could happen using Decimal

[jira] [Commented] (SPARK-27248) REFRESH TABLE should recreate cache with same cache name and storage level

2019-03-26 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801702#comment-16801702 ] William Wong commented on SPARK-27248: -- https://github.com/apache/spark/pull/24221 Hi @Hyukjin,

[jira] [Created] (SPARK-27284) Spark Standalone aggregated logs in 1 file per appid (ala yarn -logs -applicationId)

2019-03-26 Thread t oo (JIRA)
t oo created SPARK-27284: Summary: Spark Standalone aggregated logs in 1 file per appid (ala yarn -logs -applicationId) Key: SPARK-27284 URL: https://issues.apache.org/jira/browse/SPARK-27284 Project: Spark

[jira] [Resolved] (SPARK-27277) Recover from setting fix version failure in merge script

2019-03-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27277. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24213

[jira] [Assigned] (SPARK-27277) Recover from setting fix version failure in merge script

2019-03-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27277: Assignee: Hyukjin Kwon > Recover from setting fix version failure in merge script >

[jira] [Updated] (SPARK-27282) Spark incorrect results when using UNION with GROUP BY clause

2019-03-26 Thread Sofia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sofia updated SPARK-27282: -- Priority: Blocker (was: Major) > Spark incorrect results when using UNION with GROUP BY clause >

[jira] [Commented] (SPARK-27283) BigDecimal arithmetic losing precision

2019-03-26 Thread Mats (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801532#comment-16801532 ] Mats commented on SPARK-27283: -- I tried this in MySQL, which does not produce SparkSQL's behaviour: (foo

[jira] [Resolved] (SPARK-27245) Optimizer repeat Python UDF calls

2019-03-26 Thread Andrea Rota (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrea Rota resolved SPARK-27245. - Resolution: Duplicate Duplicated issue, the behavior is by design as the optimizer might choose

[jira] [Updated] (SPARK-27283) BigDecimal arithmetic losing precision

2019-03-26 Thread Mats (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mats updated SPARK-27283: - Description: When performing arithmetics between doubles and decimals, the resulting value is always a double.

[jira] [Updated] (SPARK-27283) BigDecimal arithmetic losing precision

2019-03-26 Thread Mats (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mats updated SPARK-27283: - Issue Type: Bug (was: Question) > BigDecimal arithmetic losing precision >

[jira] [Created] (SPARK-27283) BigDecimal arithmetic losing precision

2019-03-26 Thread Mats (JIRA)
Mats created SPARK-27283: Summary: BigDecimal arithmetic losing precision Key: SPARK-27283 URL: https://issues.apache.org/jira/browse/SPARK-27283 Project: Spark Issue Type: Question

[jira] [Created] (SPARK-27282) Spark incorrect results when using UNION with GROUP BY clause

2019-03-26 Thread Sofia (JIRA)
Sofia created SPARK-27282: - Summary: Spark incorrect results when using UNION with GROUP BY clause Key: SPARK-27282 URL: https://issues.apache.org/jira/browse/SPARK-27282 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27281) Wrong latest offsets returned by DirectKafkaInputDStream#latestOffsets

2019-03-26 Thread Viacheslav Krot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viacheslav Krot updated SPARK-27281: Description: I have a very strange and hard to reproduce issue when using kafka direct

[jira] [Commented] (SPARK-15544) Bouncing Zookeeper node causes Active spark master to exit

2019-03-26 Thread Moein Hosseini (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801484#comment-16801484 ] Moein Hosseini commented on SPARK-15544: [~kabhwan] I did some on this issue, but it's not well

[jira] [Created] (SPARK-27281) Wrong latest offsets returned by DirectKafkaInputDStream#latestOffsets

2019-03-26 Thread Viacheslav Krot (JIRA)
Viacheslav Krot created SPARK-27281: --- Summary: Wrong latest offsets returned by DirectKafkaInputDStream#latestOffsets Key: SPARK-27281 URL: https://issues.apache.org/jira/browse/SPARK-27281

[jira] [Reopened] (SPARK-27204) First time Loading application page from History Server is taking time when event log size is huge

2019-03-26 Thread ABHISHEK KUMAR GUPTA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA reopened SPARK-27204: -- > First time Loading application page from History Server is taking time when >

[jira] [Updated] (SPARK-27267) spark 2.4 use 1.1.7.x snappy-java, but its behavior is different from 1.1.2.x

2019-03-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27267: - Target Version/s: (was: 2.4.0) > spark 2.4 use 1.1.7.x snappy-java, but its behavior is

[jira] [Commented] (SPARK-27239) Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" from spark 2.2

2019-03-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801456#comment-16801456 ] Hyukjin Kwon commented on SPARK-27239: -- Doesn't matter which one is resolved as a duplicate. >

[jira] [Resolved] (SPARK-27239) Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" from spark 2.2.

2019-03-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27239. -- Resolution: Duplicate > Processing Compressed HDFS files with spark failing with error: >

[jira] [Created] (SPARK-27280) infer filters from Join's OR condition

2019-03-26 Thread Song Jun (JIRA)
Song Jun created SPARK-27280: Summary: infer filters from Join's OR condition Key: SPARK-27280 URL: https://issues.apache.org/jira/browse/SPARK-27280 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-27239) Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" from spar

2019-03-26 Thread Rajesh Kumar K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801449#comment-16801449 ] Rajesh Kumar K edited comment on SPARK-27239 at 3/26/19 7:13 AM: -

[jira] [Commented] (SPARK-27239) Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" from spark 2.2

2019-03-26 Thread Rajesh Kumar K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801449#comment-16801449 ] Rajesh Kumar K commented on SPARK-27239: [~hyukjin.kwon] you resolved my Jira instead of the

[jira] [Resolved] (SPARK-27264) spark sql released all executor but the job is not done

2019-03-26 Thread Mike Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Chan resolved SPARK-27264. --- Resolution: Invalid Before making further attempt, I want to ensure broadcast join is enabled on

[jira] [Commented] (SPARK-27264) spark sql released all executor but the job is not done

2019-03-26 Thread Mike Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801437#comment-16801437 ] Mike Chan commented on SPARK-27264: --- I have repartition the data before transformation and still found

[jira] [Updated] (SPARK-27239) Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" from spark 2.2.X

2019-03-26 Thread Rajesh Kumar K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Kumar K updated SPARK-27239: --- Priority: Major (was: Blocker) > Processing Compressed HDFS files with spark failing with

[jira] [Reopened] (SPARK-27239) Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" from spark 2.2.

2019-03-26 Thread Rajesh Kumar K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Kumar K reopened SPARK-27239: Reopening this Jira, since Jira : SPARK-27259 is duplicate and it exactly cloned from this