[jira] [Assigned] (SPARK-24624) Can not mix vectorized and non-vectorized UDFs

2018-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24624: Assignee: Li Jin > Can not mix vectorized and non-vectorized UDFs >

[jira] [Resolved] (SPARK-24624) Can not mix vectorized and non-vectorized UDFs

2018-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24624. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21650

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560624#comment-16560624 ] Apache Spark commented on SPARK-24924: -- User 'gatorsmile' has created a pull request for this

[jira] [Updated] (SPARK-24957) Decimal arithmetic can lead to wrong values using codegen

2018-07-27 Thread David Vogelbacher (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Vogelbacher updated SPARK-24957: -- Description: I noticed a bug when doing arithmetic on a dataframe containing decimal

[jira] [Created] (SPARK-24957) Decimal arithmetic can lead to wrong values using codegen

2018-07-27 Thread David Vogelbacher (JIRA)
David Vogelbacher created SPARK-24957: - Summary: Decimal arithmetic can lead to wrong values using codegen Key: SPARK-24957 URL: https://issues.apache.org/jira/browse/SPARK-24957 Project: Spark

[jira] [Commented] (SPARK-24895) Spark 2.4.0 Snapshot artifacts has broken metadata due to mismatched filenames

2018-07-27 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560600#comment-16560600 ] Kazuaki Ishizaki commented on SPARK-24895: -- [~ericfchang] Thank you very much for your

[jira] [Assigned] (SPARK-24956) Upgrade maven from 3.3.9 to 3.5.4

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24956: Assignee: (was: Apache Spark) > Upgrade maven from 3.3.9 to 3.5.4 >

[jira] [Assigned] (SPARK-24956) Upgrade maven from 3.3.9 to 3.5.4

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24956: Assignee: Apache Spark > Upgrade maven from 3.3.9 to 3.5.4 >

[jira] [Commented] (SPARK-24956) Upgrade maven from 3.3.9 to 3.5.4

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560595#comment-16560595 ] Apache Spark commented on SPARK-24956: -- User 'kiszk' has created a pull request for this issue:

[jira] [Created] (SPARK-24956) Upgrade maven from 3.3.9 to 3.5.4

2018-07-27 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-24956: Summary: Upgrade maven from 3.3.9 to 3.5.4 Key: SPARK-24956 URL: https://issues.apache.org/jira/browse/SPARK-24956 Project: Spark Issue Type:

[jira] [Updated] (SPARK-24954) Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled

2018-07-27 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-24954: - Priority: Blocker (was: Major) > Fail fast on job submit if run a barrier stage with dynamic

[jira] [Updated] (SPARK-24809) Serializing LongHashedRelation in executor may result in data error

2018-07-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24809: Labels: correctness (was: ) > Serializing LongHashedRelation in executor may result in data

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560544#comment-16560544 ] shane knapp commented on SPARK-24950: - the spark-master-test-sbt build failed on ubuntu, but the the

[jira] [Commented] (SPARK-24721) Failed to call PythonUDF whose input is the output of another PythonUDF

2018-07-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560537#comment-16560537 ] Wenchen Fan commented on SPARK-24721: - good catch! I think we should filter out python UDFs when

[jira] [Updated] (SPARK-24955) spark continuing to execute on a task despite not reading all data from a downed machine

2018-07-27 Thread San Tung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] San Tung updated SPARK-24955: - Description: We've recently run into a few instances where a downed node has led to incomplete data,

[jira] [Created] (SPARK-24955) spark continuing to execute on a task despite not reading all data from a downed machine

2018-07-27 Thread San Tung (JIRA)
San Tung created SPARK-24955: Summary: spark continuing to execute on a task despite not reading all data from a downed machine Key: SPARK-24955 URL: https://issues.apache.org/jira/browse/SPARK-24955

[jira] [Created] (SPARK-24954) Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled

2018-07-27 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-24954: Summary: Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled Key: SPARK-24954 URL: https://issues.apache.org/jira/browse/SPARK-24954

[jira] [Commented] (SPARK-24953) Prune a branch in `CaseWhen` if previously seen

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560498#comment-16560498 ] Apache Spark commented on SPARK-24953: -- User 'dbtsai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24953) Prune a branch in `CaseWhen` if previously seen

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24953: Assignee: (was: Apache Spark) > Prune a branch in `CaseWhen` if previously seen >

[jira] [Assigned] (SPARK-24953) Prune a branch in `CaseWhen` if previously seen

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24953: Assignee: Apache Spark > Prune a branch in `CaseWhen` if previously seen >

[jira] [Created] (SPARK-24953) Prune a branch in `CaseWhen` if previously seen

2018-07-27 Thread DB Tsai (JIRA)
DB Tsai created SPARK-24953: --- Summary: Prune a branch in `CaseWhen` if previously seen Key: SPARK-24953 URL: https://issues.apache.org/jira/browse/SPARK-24953 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24724) Discuss necessary info and access in barrier mode + Kubernetes

2018-07-27 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560436#comment-16560436 ] Yinan Li commented on SPARK-24724: -- Sorry haven't got a chance to look into this. What pieces of info

[jira] [Updated] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24950: -- Affects Version/s: 2.2.2 > scala DateTimeUtilsSuite daysToMillis and millisToDays fails

[jira] [Updated] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24950: -- Affects Version/s: 2.3.1 > scala DateTimeUtilsSuite daysToMillis and millisToDays fails

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560411#comment-16560411 ] Apache Spark commented on SPARK-24950: -- User 'srowen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24952) Support LZMA2 compression by Avro datasource

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24952: Assignee: Apache Spark > Support LZMA2 compression by Avro datasource >

[jira] [Commented] (SPARK-24952) Support LZMA2 compression by Avro datasource

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560357#comment-16560357 ] Apache Spark commented on SPARK-24952: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24952) Support LZMA2 compression by Avro datasource

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24952: Assignee: (was: Apache Spark) > Support LZMA2 compression by Avro datasource >

[jira] [Created] (SPARK-24952) Support LZMA2 compression by Avro datasource

2018-07-27 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-24952: -- Summary: Support LZMA2 compression by Avro datasource Key: SPARK-24952 URL: https://issues.apache.org/jira/browse/SPARK-24952 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-24721) Failed to call PythonUDF whose input is the output of another PythonUDF

2018-07-27 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560337#comment-16560337 ] Li Jin edited comment on SPARK-24721 at 7/27/18 9:18 PM: - I think the issue is

[jira] [Commented] (SPARK-24721) Failed to call PythonUDF whose input is the output of another PythonUDF

2018-07-27 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560337#comment-16560337 ] Li Jin commented on SPARK-24721: I think the issue is the UDF is being pushed down to the

[jira] [Updated] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

2018-07-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23243: -- Priority: Blocker (was: Major) > Shuffle+Repartition on an RDD could lead to incorrect

[jira] [Created] (SPARK-24951) Table valued functions should throw AnalysisException instead of IllegalArgumentException

2018-07-27 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-24951: --- Summary: Table valued functions should throw AnalysisException instead of IllegalArgumentException Key: SPARK-24951 URL: https://issues.apache.org/jira/browse/SPARK-24951

[jira] [Issue Comment Deleted] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-24950: Comment: was deleted (was:

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560318#comment-16560318 ] shane knapp commented on SPARK-24950: - testing this manually:

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread Chris Martin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560312#comment-16560312 ] Chris Martin commented on SPARK-24950: -- Hi,   just to say that I looked at this and came to the

[jira] [Comment Edited] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread Chris Martin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560312#comment-16560312 ] Chris Martin edited comment on SPARK-24950 at 7/27/18 8:48 PM: --- Hi,  

[jira] [Assigned] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24950: Assignee: (was: Apache Spark) > scala DateTimeUtilsSuite daysToMillis and

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560310#comment-16560310 ] shane knapp commented on SPARK-24950: -

[jira] [Assigned] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24950: Assignee: Apache Spark > scala DateTimeUtilsSuite daysToMillis and millisToDays fails

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560308#comment-16560308 ] Apache Spark commented on SPARK-24950: -- User 'd80tb7' has created a pull request for this issue:

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560306#comment-16560306 ] shane knapp commented on SPARK-24950: - sgtm i also dug through the java release notes WRT timezone

[jira] [Commented] (SPARK-24721) Failed to call PythonUDF whose input is the output of another PythonUDF

2018-07-27 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560283#comment-16560283 ] Li Jin commented on SPARK-24721: {code:java} from pyspark.sql.functions import udf, lit, col

[jira] [Commented] (SPARK-23146) Support client mode for Kubernetes cluster backend

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560279#comment-16560279 ] Apache Spark commented on SPARK-23146: -- User 'mccheah' has created a pull request for this issue:

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560249#comment-16560249 ] Sean Owen commented on SPARK-24950: --- It's pretty clear this is down to differences in how time zones

[jira] [Updated] (SPARK-24922) Iterative rdd union + reduceByKey operations on small dataset leads to "No space left on device" error on account of lot of shuffle spill.

2018-07-27 Thread Dinesh Dharme (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Dharme updated SPARK-24922: -- Description: I am trying to do few (union + reduceByKey) operations on a hiearchical dataset

[jira] [Updated] (SPARK-24702) Unable to cast to calendar interval in spark sql.

2018-07-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24702: Target Version/s: 3.0.0 > Unable to cast to calendar interval in spark sql. >

[jira] [Assigned] (SPARK-24912) Broadcast join OutOfMemory stack trace obscures actual cause of OOM

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24912: Assignee: Apache Spark > Broadcast join OutOfMemory stack trace obscures actual cause of

[jira] [Assigned] (SPARK-24912) Broadcast join OutOfMemory stack trace obscures actual cause of OOM

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24912: Assignee: (was: Apache Spark) > Broadcast join OutOfMemory stack trace obscures

[jira] [Commented] (SPARK-24912) Broadcast join OutOfMemory stack trace obscures actual cause of OOM

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560155#comment-16560155 ] Apache Spark commented on SPARK-24912: -- User 'bersprockets' has created a pull request for this

[jira] [Commented] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560126#comment-16560126 ] shane knapp commented on SPARK-24950: - one solution, of course, is to pin the java version on the

[jira] [Updated] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-24950: Description: during my travails to port the spark builds to run on ubuntu 16.04LTS, i have

[jira] [Created] (SPARK-24950) scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13

2018-07-27 Thread shane knapp (JIRA)
shane knapp created SPARK-24950: --- Summary: scala DateTimeUtilsSuite daysToMillis and millisToDays fails w/java 8 181-b13 Key: SPARK-24950 URL: https://issues.apache.org/jira/browse/SPARK-24950 Project:

[jira] [Assigned] (SPARK-24817) Implement BarrierTaskContext.barrier()

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24817: Assignee: Apache Spark > Implement BarrierTaskContext.barrier() >

[jira] [Commented] (SPARK-24817) Implement BarrierTaskContext.barrier()

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560096#comment-16560096 ] Apache Spark commented on SPARK-24817: -- User 'jiangxb1987' has created a pull request for this

[jira] [Assigned] (SPARK-24817) Implement BarrierTaskContext.barrier()

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24817: Assignee: (was: Apache Spark) > Implement BarrierTaskContext.barrier() >

[jira] [Commented] (SPARK-18381) Wrong date conversion between spark and python for dates before 1583

2018-07-27 Thread Stephen Brennan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560068#comment-16560068 ] Stephen Brennan commented on SPARK-18381: - Just encountered this issue myself and came here to

[jira] [Commented] (SPARK-24865) Remove AnalysisBarrier

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560052#comment-16560052 ] Apache Spark commented on SPARK-24865: -- User 'rxin' has created a pull request for this issue:

[jira] [Resolved] (SPARK-13343) speculative tasks that didn't commit shouldn't be marked as success

2018-07-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13343. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 >

[jira] [Commented] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

2018-07-27 Thread Satyajit varma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560036#comment-16560036 ] Satyajit varma commented on SPARK-20597: [~jlaskowski] will do submit a PR today >

[jira] [Commented] (SPARK-21216) Streaming DataFrames fail to join with Hive tables

2018-07-27 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560034#comment-16560034 ] Russell Spitzer commented on SPARK-21216: - For anyone else searching, this also fixes custom

[jira] [Assigned] (SPARK-21960) Spark Streaming Dynamic Allocation should respect spark.executor.instances

2018-07-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21960: - Assignee: Karthik Palaniappan > Spark Streaming Dynamic Allocation should respect

[jira] [Resolved] (SPARK-21960) Spark Streaming Dynamic Allocation should respect spark.executor.instances

2018-07-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21960. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 19183

[jira] [Resolved] (SPARK-17984) Add support for numa aware feature

2018-07-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17984. --- Resolution: Won't Fix See pull requests. > Add support for numa aware feature >

[jira] [Created] (SPARK-24949) pyspark.sql.Column breaks the iterable contract

2018-07-27 Thread Daniel Shields (JIRA)
Daniel Shields created SPARK-24949: -- Summary: pyspark.sql.Column breaks the iterable contract Key: SPARK-24949 URL: https://issues.apache.org/jira/browse/SPARK-24949 Project: Spark Issue

[jira] [Comment Edited] (SPARK-24895) Spark 2.4.0 Snapshot artifacts has broken metadata due to mismatched filenames

2018-07-27 Thread Eric Chang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559996#comment-16559996 ] Eric Chang edited comment on SPARK-24895 at 7/27/18 5:00 PM: - [~kiszk] for

[jira] [Commented] (SPARK-24895) Spark 2.4.0 Snapshot artifacts has broken metadata due to mismatched filenames

2018-07-27 Thread Eric Chang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559996#comment-16559996 ] Eric Chang commented on SPARK-24895: [~kiszk] for maven, you may need 3.5.2 which includes this fix:

[jira] [Commented] (SPARK-24895) Spark 2.4.0 Snapshot artifacts has broken metadata due to mismatched filenames

2018-07-27 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559987#comment-16559987 ] Kazuaki Ishizaki commented on SPARK-24895: -- I see. Thank you very much. At first, I will try to

[jira] [Commented] (SPARK-24895) Spark 2.4.0 Snapshot artifacts has broken metadata due to mismatched filenames

2018-07-27 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559977#comment-16559977 ] Yin Huai commented on SPARK-24895: -- [https://github.com/spotbugs/spotbugs-maven-plugin/issues/21] has

[jira] [Commented] (SPARK-24895) Spark 2.4.0 Snapshot artifacts has broken metadata due to mismatched filenames

2018-07-27 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559974#comment-16559974 ] Kazuaki Ishizaki commented on SPARK-24895: -- [~yhuai] Thank you. BTW, how can I re-enable

[jira] [Commented] (SPARK-24925) input bytesRead metrics fluctuate from time to time

2018-07-27 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559972#comment-16559972 ] Kazuaki Ishizaki commented on SPARK-24925: -- Do we need a test case or which test case covers

[jira] [Assigned] (SPARK-24948) SHS filters wrongly some applications due to permission check

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24948: Assignee: Apache Spark > SHS filters wrongly some applications due to permission check >

[jira] [Resolved] (SPARK-24881) New options - compression and compressionLevel

2018-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24881. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21837

[jira] [Assigned] (SPARK-24881) New options - compression and compressionLevel

2018-07-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24881: Assignee: Maxim Gekk > New options - compression and compressionLevel >

[jira] [Commented] (SPARK-24948) SHS filters wrongly some applications due to permission check

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559951#comment-16559951 ] Apache Spark commented on SPARK-24948: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24948) SHS filters wrongly some applications due to permission check

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24948: Assignee: (was: Apache Spark) > SHS filters wrongly some applications due to

[jira] [Resolved] (SPARK-24921) SparkStreaming steadily increasing job generation delay due to apparent URLClassLoader contention

2018-07-27 Thread Tommy S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy S resolved SPARK-24921. - Resolution: Not A Bug > SparkStreaming steadily increasing job generation delay due to apparent >

[jira] [Commented] (SPARK-24921) SparkStreaming steadily increasing job generation delay due to apparent URLClassLoader contention

2018-07-27 Thread Tommy S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559943#comment-16559943 ] Tommy S commented on SPARK-24921: - Fair point. I'll close this ticket and reopen if I confirm that it is

[jira] [Created] (SPARK-24948) SHS filters wrongly some applications due to permission check

2018-07-27 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-24948: --- Summary: SHS filters wrongly some applications due to permission check Key: SPARK-24948 URL: https://issues.apache.org/jira/browse/SPARK-24948 Project: Spark

[jira] [Updated] (SPARK-24927) The hadoop-provided profile doesn't play well with Snappy-compressed Parquet files

2018-07-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24927: Fix Version/s: 2.4.0 2.2.3 > The hadoop-provided profile doesn't play well with

[jira] [Resolved] (SPARK-24927) The hadoop-provided profile doesn't play well with Snappy-compressed Parquet files

2018-07-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24927. - Resolution: Fixed Fix Version/s: 2.3.2 > The hadoop-provided profile doesn't play well with

[jira] [Commented] (SPARK-24882) separate responsibilities of the data source v2 read API

2018-07-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559918#comment-16559918 ] Wenchen Fan commented on SPARK-24882: - Hi [~rdblue] , I like your naming changes and will update

[jira] [Commented] (SPARK-24940) Coalesce Hint for SQL Queries

2018-07-27 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559909#comment-16559909 ] John Zhuge commented on SPARK-24940: Thx [~hyukjin.kwon] > Coalesce Hint for SQL Queries >

[jira] [Updated] (SPARK-24947) aggregateAsync and foldAsync for RDD

2018-07-27 Thread Cody Allen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Allen updated SPARK-24947: --- Description: {{AsyncRDDActions}} contains {{collectAsync}}, {{countAsync}}, {{foreachAsync}}, etc;

[jira] [Created] (SPARK-24947) aggregateAsync and foldAsync for RDD

2018-07-27 Thread Cody Allen (JIRA)
Cody Allen created SPARK-24947: -- Summary: aggregateAsync and foldAsync for RDD Key: SPARK-24947 URL: https://issues.apache.org/jira/browse/SPARK-24947 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-24935) Problem with Executing Hive UDF's from Spark 2.2 Onwards

2018-07-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559828#comment-16559828 ] Wenchen Fan commented on SPARK-24935: - seems like Hive UDAF can reject partial aggregate? > Problem

[jira] [Updated] (SPARK-24942) Improve cluster resource management with jobs containing barrier stage

2018-07-27 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xingbo updated SPARK-24942: - Target Version/s: 3.0.0 > Improve cluster resource management with jobs containing barrier

[jira] [Updated] (SPARK-24941) Add RDDBarrier.coalesce() function

2018-07-27 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xingbo updated SPARK-24941: - Target Version/s: 3.0.0 > Add RDDBarrier.coalesce() function >

[jira] [Created] (SPARK-24946) PySpark - Allow np.Arrays and pd.Series in df.approxQuantile

2018-07-27 Thread Paul Westenthanner (JIRA)
Paul Westenthanner created SPARK-24946: -- Summary: PySpark - Allow np.Arrays and pd.Series in df.approxQuantile Key: SPARK-24946 URL: https://issues.apache.org/jira/browse/SPARK-24946 Project:

[jira] [Resolved] (SPARK-23928) High-order function: shuffle(x) → array

2018-07-27 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23928. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21802

[jira] [Assigned] (SPARK-23928) High-order function: shuffle(x) → array

2018-07-27 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23928: - Assignee: H Lu > High-order function: shuffle(x) → array >

[jira] [Commented] (SPARK-24944) SparkUi build problem

2018-07-27 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559790#comment-16559790 ] Marco Gaido commented on SPARK-24944: - This seems more a problem in your project and your

[jira] [Commented] (SPARK-24945) Switch to uniVocity 2.7.2

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559783#comment-16559783 ] Apache Spark commented on SPARK-24945: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24945) Switch to uniVocity 2.7.2

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24945: Assignee: Apache Spark > Switch to uniVocity 2.7.2 > - > >

[jira] [Assigned] (SPARK-24945) Switch to uniVocity 2.7.2

2018-07-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24945: Assignee: (was: Apache Spark) > Switch to uniVocity 2.7.2 >

[jira] [Updated] (SPARK-24945) Switch to uniVocity 2.7.2

2018-07-27 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-24945: --- Summary: Switch to uniVocity 2.7.2 (was: Switch to unoVocity 2.7.2) > Switch to uniVocity 2.7.2 >

[jira] [Created] (SPARK-24945) Switch to unoVocity 2.7.2

2018-07-27 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-24945: -- Summary: Switch to unoVocity 2.7.2 Key: SPARK-24945 URL: https://issues.apache.org/jira/browse/SPARK-24945 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-24944) SparkUi build problem

2018-07-27 Thread Fabio (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabio updated SPARK-24944: -- Attachment: build-test.zip > SparkUi build problem > - > > Key:

[jira] [Created] (SPARK-24944) SparkUi build problem

2018-07-27 Thread Fabio (JIRA)
Fabio created SPARK-24944: - Summary: SparkUi build problem Key: SPARK-24944 URL: https://issues.apache.org/jira/browse/SPARK-24944 Project: Spark Issue Type: Bug Components: Build

[jira] [Commented] (SPARK-24492) Endless attempted task when TaskCommitDenied exception writing to S3A

2018-07-27 Thread Dmitry Bugaychenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559560#comment-16559560 ] Dmitry Bugaychenko commented on SPARK-24492: Seen the same problem with Spark 2.3.1 on YARN

  1   2   >