[jira] [Created] (SPARK-26951) Should not throw KryoException when root cause is IOexception

2019-02-20 Thread zhoukang (JIRA)
zhoukang created SPARK-26951: Summary: Should not throw KryoException when root cause is IOexception Key: SPARK-26951 URL: https://issues.apache.org/jira/browse/SPARK-26951 Project: Spark Issue

[jira] [Created] (SPARK-26950) Make RandomDataGenerator use Float.NaN or Double.NaN for all NaN values

2019-02-20 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-26950: - Summary: Make RandomDataGenerator use Float.NaN or Double.NaN for all NaN values Key: SPARK-26950 URL: https://issues.apache.org/jira/browse/SPARK-26950 Project:

[jira] [Assigned] (SPARK-26950) Make RandomDataGenerator use Float.NaN or Double.NaN for all NaN values

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26950: Assignee: Apache Spark > Make RandomDataGenerator use Float.NaN or Double.NaN for all

[jira] [Assigned] (SPARK-26950) Make RandomDataGenerator use Float.NaN or Double.NaN for all NaN values

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26950: Assignee: (was: Apache Spark) > Make RandomDataGenerator use Float.NaN or Double.NaN

[jira] [Commented] (SPARK-26825) Spark Structure Streaming job failing when submitted in cluster mode

2019-02-20 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773727#comment-16773727 ] Jungtaek Lim commented on SPARK-26825: -- Similar issue was reported (SPARK-19909) which root reason

[jira] [Commented] (SPARK-24818) Ensure all the barrier tasks in the same stage are launched together

2019-02-20 Thread luzengxiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773729#comment-16773729 ] luzengxiang commented on SPARK-24818: - "cannot fulfill task locality requirements" keeps happening!

[jira] [Resolved] (SPARK-26919) change maven default compile java home

2019-02-20 Thread daile (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] daile resolved SPARK-26919. --- Resolution: Done Fix Version/s: 2.4.0 > change maven default compile java home >

[jira] [Commented] (SPARK-26425) Add more constraint checks in file streaming source to avoid checkpoint corruption

2019-02-20 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773705#comment-16773705 ] Jungtaek Lim commented on SPARK-26425: -- Seems like no work was done in 2 months while observations

[jira] [Assigned] (SPARK-26949) Prevent "purge" to remove needed batch files in CompactibleFileStreamLog

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26949: Assignee: Apache Spark > Prevent "purge" to remove needed batch files in

[jira] [Assigned] (SPARK-26949) Prevent "purge" to remove needed batch files in CompactibleFileStreamLog

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26949: Assignee: (was: Apache Spark) > Prevent "purge" to remove needed batch files in

[jira] [Created] (SPARK-26949) Prevent "purge" to remove needed batch files in CompactibleFileStreamLog

2019-02-20 Thread Jungtaek Lim (JIRA)
Jungtaek Lim created SPARK-26949: Summary: Prevent "purge" to remove needed batch files in CompactibleFileStreamLog Key: SPARK-26949 URL: https://issues.apache.org/jira/browse/SPARK-26949 Project:

[jira] [Commented] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread Thincrs (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773632#comment-16773632 ] Thincrs commented on SPARK-26946: - A user of thincrs has selected this issue. Deadline: Thu, Feb 28,

[jira] [Resolved] (SPARK-26643) Spark Hive throw an AnalysisException,when set table properties.But this AnalysisException contains one typo and one unsuited word.

2019-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26643. --- Resolution: Not A Problem > Spark Hive throw an AnalysisException,when set table properties.But

[jira] [Updated] (SPARK-26643) Spark Hive throw an AnalysisException,when set table properties.But this AnalysisException contains one typo and one unsuited word.

2019-02-20 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows: {code:java}

[jira] [Updated] (SPARK-26643) Spark Hive throw an AnalysisException,when set table properties.But this AnalysisException contains one typo and one unsuited word.

2019-02-20 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Summary: Spark Hive throw an AnalysisException,when set table properties.But this AnalysisException

[jira] [Updated] (SPARK-26643) Spark Hive throw an AnalysisException,when set table properties.But this AnalysisException contains two typo.

2019-02-20 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows: {code:java}

[jira] [Updated] (SPARK-26643) Spark Hive throw an AnalysisException,when set table properties.But this AnalysisException contains two typo.

2019-02-20 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Summary: Spark Hive throw an AnalysisException,when set table properties.But this AnalysisException

[jira] [Updated] (SPARK-26643) Spark Hive throw an analysis exception,when set table properties.But this

2019-02-20 Thread jiaan.geng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Summary: Spark Hive throw an analysis exception,when set table properties.But this (was: Spark

[jira] [Commented] (SPARK-24935) Problem with Executing Hive UDF's from Spark 2.2 Onwards

2019-02-20 Thread gavin hu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773574#comment-16773574 ] gavin hu commented on SPARK-24935: -- Hi [~pgandhi]  "A user of sketches library..." That's me! I'm so

[jira] [Commented] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773556#comment-16773556 ] Apache Spark commented on SPARK-26946: -- User 'jzhuge' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26946: Assignee: (was: Apache Spark) > Identifiers for multi-catalog Spark >

[jira] [Assigned] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26946: Assignee: Apache Spark > Identifiers for multi-catalog Spark >

[jira] [Commented] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773553#comment-16773553 ] Apache Spark commented on SPARK-26946: -- User 'jzhuge' has created a pull request for this issue:

[jira] [Created] (SPARK-26948) vertex and edge rowkey upgrade and support multiple types?

2019-02-20 Thread daile (JIRA)
daile created SPARK-26948: - Summary: vertex and edge rowkey upgrade and support multiple types? Key: SPARK-26948 URL: https://issues.apache.org/jira/browse/SPARK-26948 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24935) Problem with Executing Hive UDF's from Spark 2.2 Onwards

2019-02-20 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773541#comment-16773541 ] Parth Gandhi commented on SPARK-24935: -- Hi [~zanderl], thank you for your comment. Will do my best

[jira] [Updated] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-26946: --- Component/s: (was: Spark Core) > Identifiers for multi-catalog Spark >

[jira] [Commented] (SPARK-24935) Problem with Executing Hive UDF's from Spark 2.2 Onwards

2019-02-20 Thread Zander Lichstein (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773533#comment-16773533 ] Zander Lichstein commented on SPARK-24935: -- Glad to see this has been fixed!  What are the

[jira] [Resolved] (SPARK-26824) Streaming queries may store checkpoint data in a wrong directory

2019-02-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26824. -- Resolution: Fixed Fix Version/s: 3.0.0 > Streaming queries may store checkpoint data

[jira] [Comment Edited] (SPARK-26947) Pyspark KMeans Clustering job fails on large values of k

2019-02-20 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773457#comment-16773457 ] Parth Gandhi edited comment on SPARK-26947 at 2/20/19 10:43 PM: I am

[jira] [Assigned] (SPARK-26892) saveAsTextFile throws NullPointerException when null row present

2019-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26892: - Assignee: liupengcheng > saveAsTextFile throws NullPointerException when null row present >

[jira] [Resolved] (SPARK-26892) saveAsTextFile throws NullPointerException when null row present

2019-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26892. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23799

[jira] [Commented] (SPARK-26947) Pyspark KMeans Clustering job fails on large values of k

2019-02-20 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773457#comment-16773457 ] Parth Gandhi commented on SPARK-26947: -- I am unable to attach the dummy dataset as the size of the

[jira] [Updated] (SPARK-26947) Pyspark KMeans Clustering job fails on large values of k

2019-02-20 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Gandhi updated SPARK-26947: - Attachment: clustering_app.py > Pyspark KMeans Clustering job fails on large values of k >

[jira] [Created] (SPARK-26947) Pyspark KMeans Clustering job fails on large values of k

2019-02-20 Thread Parth Gandhi (JIRA)
Parth Gandhi created SPARK-26947: Summary: Pyspark KMeans Clustering job fails on large values of k Key: SPARK-26947 URL: https://issues.apache.org/jira/browse/SPARK-26947 Project: Spark

[jira] [Updated] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-26946: --- Description: Propose semantics for identifiers and a listing API to support multiple catalogs.

[jira] [Commented] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

2019-02-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773381#comment-16773381 ] Hyukjin Kwon commented on SPARK-26858: -- Oh, I see. Sorry there was misunderstanding. I think you're

[jira] [Resolved] (SPARK-26877) Support user-level app staging directory in yarn mode when spark.yarn.stagingDir specified

2019-02-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26877. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23786

[jira] [Assigned] (SPARK-26877) Support user-level app staging directory in yarn mode when spark.yarn.stagingDir specified

2019-02-20 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26877: -- Assignee: liupengcheng > Support user-level app staging directory in yarn mode when

[jira] [Assigned] (SPARK-26729) Spark on Kubernetes tooling hardcodes default image names

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26729: Assignee: (was: Apache Spark) > Spark on Kubernetes tooling hardcodes default image

[jira] [Assigned] (SPARK-26729) Spark on Kubernetes tooling hardcodes default image names

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26729: Assignee: Apache Spark > Spark on Kubernetes tooling hardcodes default image names >

[jira] [Commented] (SPARK-26884) Let task acquire memory accurately when using spilled memory

2019-02-20 Thread Thincrs (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773286#comment-16773286 ] Thincrs commented on SPARK-26884: - A user of thincrs has selected this issue. Deadline: Wed, Feb 27,

[jira] [Commented] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

2019-02-20 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773284#comment-16773284 ] Bryan Cutler commented on SPARK-26858: -- {quote} (One other possibility I was thinking about batches

[jira] [Created] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread John Zhuge (JIRA)
John Zhuge created SPARK-26946: -- Summary: Identifiers for multi-catalog Spark Key: SPARK-26946 URL: https://issues.apache.org/jira/browse/SPARK-26946 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-22709) move config related infrastructure from Spark Core to a new module

2019-02-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773111#comment-16773111 ] Wenchen Fan commented on SPARK-22709: - it's not about where we define configs, it's about whether we

[jira] [Created] (SPARK-26945) Python streaming tests flaky while cleaning temp directories after StreamingQuery.stop

2019-02-20 Thread Alessandro Bellina (JIRA)
Alessandro Bellina created SPARK-26945: -- Summary: Python streaming tests flaky while cleaning temp directories after StreamingQuery.stop Key: SPARK-26945 URL:

[jira] [Updated] (SPARK-26903) Remove the TimeZone cache

2019-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-26903: -- Docs Text: Prior to Spark 3, if an invalid timezone was specified to to_utc_timestamp and

[jira] [Resolved] (SPARK-26900) Simplify truncation to quarter of year

2019-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26900. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23808

[jira] [Created] (SPARK-26944) Python unit-tests.log not available in artifacts for a build in Jenkins

2019-02-20 Thread Alessandro Bellina (JIRA)
Alessandro Bellina created SPARK-26944: -- Summary: Python unit-tests.log not available in artifacts for a build in Jenkins Key: SPARK-26944 URL: https://issues.apache.org/jira/browse/SPARK-26944

[jira] [Assigned] (SPARK-26900) Simplify truncation to quarter of year

2019-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26900: - Assignee: Maxim Gekk > Simplify truncation to quarter of year >

[jira] [Resolved] (SPARK-22798) Add multiple column support to PySpark StringIndexer

2019-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22798. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23741

[jira] [Assigned] (SPARK-22798) Add multiple column support to PySpark StringIndexer

2019-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-22798: - Assignee: Huaxin Gao > Add multiple column support to PySpark StringIndexer >

[jira] [Commented] (SPARK-22709) move config related infrastructure from Spark Core to a new module

2019-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773072#comment-16773072 ] Sean Owen commented on SPARK-22709: --- Saw this late, but I'm not sure about it. This means a new module

[jira] [Commented] (SPARK-22000) org.codehaus.commons.compiler.CompileException: toString method is not declared

2019-02-20 Thread Sergey Derugo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773064#comment-16773064 ] Sergey Derugo commented on SPARK-22000: --- I've got similar issue. My sample code is attached. >

[jira] [Commented] (SPARK-9135) Filter fails when filtering with a method reference to overloaded method

2019-02-20 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773067#comment-16773067 ] Valeria Vasylieva commented on SPARK-9135: -- I would like to work on it > Filter fails when

[jira] [Updated] (SPARK-22000) org.codehaus.commons.compiler.CompileException: toString method is not declared

2019-02-20 Thread Sergey Derugo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Derugo updated SPARK-22000: -- Attachment: testcase.zip > org.codehaus.commons.compiler.CompileException: toString method is

[jira] [Updated] (SPARK-26943) Weird behaviour with `.cache()`

2019-02-20 Thread Will Uto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Uto updated SPARK-26943: - Description:   {code:java} sdf.count(){code}   works fine. However:   {code:java} sdf = sdf.cache()

[jira] [Commented] (SPARK-25810) Spark structured streaming logs auto.offset.reset=earliest even though startingOffsets is set to latest

2019-02-20 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773044#comment-16773044 ] Valeria Vasylieva commented on SPARK-25810: --- I suppose the cause is here: 

[jira] [Resolved] (SPARK-26859) Fix field writer index bug in non-vectorized ORC deserializer

2019-02-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26859. - Resolution: Fixed Fix Version/s: 2.4.1 3.0.0 Issue resolved by pull

[jira] [Assigned] (SPARK-26859) Fix field writer index bug in non-vectorized ORC deserializer

2019-02-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26859: --- Assignee: Ivan Vergiliev > Fix field writer index bug in non-vectorized ORC deserializer >

[jira] [Updated] (SPARK-26943) Weird behaviour with `.cache()`

2019-02-20 Thread Will Uto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Will Uto updated SPARK-26943: - Description:   {code:java} sdf.count(){code}   works fine. However:   {code:java} sdf = sdf.cache()

[jira] [Created] (SPARK-26943) Weird behaviour with `.cache()`

2019-02-20 Thread Will Uto (JIRA)
Will Uto created SPARK-26943: Summary: Weird behaviour with `.cache()` Key: SPARK-26943 URL: https://issues.apache.org/jira/browse/SPARK-26943 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-22601) Data load is getting displayed successful on providing non existing hdfs file path

2019-02-20 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-22601: - Assignee: Sujith > Data load is getting displayed successful on providing non existing hdfs

[jira] [Commented] (SPARK-26869) UDF with struct requires to have _1 and _2 as struct field names

2019-02-20 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772988#comment-16772988 ] Valeria Vasylieva commented on SPARK-26869: --- [~anddonram] you are trying to treat Struct as

[jira] [Resolved] (SPARK-24211) Flaky test: StreamingOuterJoinSuite

2019-02-20 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-24211. -- Resolution: Fixed Assignee: Jungtaek Lim Fix Version/s: 2.3.4

[jira] [Commented] (SPARK-24211) Flaky test: StreamingOuterJoinSuite

2019-02-20 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772980#comment-16772980 ] Takeshi Yamamuro commented on SPARK-24211: -- I closed this cuz it seems these test failures

[jira] [Updated] (SPARK-26927) Race condition may cause dynamic allocation not working

2019-02-20 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26927: - Issue Type: Bug (was: Improvement) > Race condition may cause dynamic allocation not working >

[jira] [Commented] (SPARK-24239) Flaky test: KafkaContinuousSourceSuite.subscribing topic by name from earliest offsets

2019-02-20 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772984#comment-16772984 ] Takeshi Yamamuro commented on SPARK-24239: -- I closed this cuz it seems these test failures

[jira] [Resolved] (SPARK-24239) Flaky test: KafkaContinuousSourceSuite.subscribing topic by name from earliest offsets

2019-02-20 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-24239. -- Resolution: Fixed Assignee: Jungtaek Lim Fix Version/s: 2.3.4

[jira] [Created] (SPARK-26942) spark v 2.3.2 test failure in hive module

2019-02-20 Thread ketan kunde (JIRA)
ketan kunde created SPARK-26942: --- Summary: spark v 2.3.2 test failure in hive module Key: SPARK-26942 URL: https://issues.apache.org/jira/browse/SPARK-26942 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-26941) incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming

2019-02-20 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26941: - Summary: incorrect computation of maxNumExecutorFailures in ApplicationMaster for streaming

[jira] [Updated] (SPARK-26941) maxNumExecutorFailures should be computed with spark.streaming.dynamicAllocation.maxExecutors in ApplicationMaster for streaming

2019-02-20 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26941: - Component/s: YARN Summary: maxNumExecutorFailures should be computed with

[jira] [Assigned] (SPARK-26941) maxNumExecutorFailures should be computed with spark.streaming.dynamicAllocation.maxExecutors in streaming

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26941: Assignee: (was: Apache Spark) > maxNumExecutorFailures should be computed with >

[jira] [Assigned] (SPARK-26941) maxNumExecutorFailures should be computed with spark.streaming.dynamicAllocation.maxExecutors in streaming

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26941: Assignee: Apache Spark > maxNumExecutorFailures should be computed with >

[jira] [Created] (SPARK-26941) maxNumExecutorFailures should be computed with spark.streaming.dynamicAllocation.maxExecutors in streaming

2019-02-20 Thread liupengcheng (JIRA)
liupengcheng created SPARK-26941: Summary: maxNumExecutorFailures should be computed with spark.streaming.dynamicAllocation.maxExecutors in streaming Key: SPARK-26941 URL:

[jira] [Commented] (SPARK-22601) Data load is getting displayed successful on providing non existing hdfs file path

2019-02-20 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772921#comment-16772921 ] Sujith commented on SPARK-22601: *[gatorsmile|https://github.com/gatorsmile] [~srowen]  please assign

[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

2019-02-20 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772875#comment-16772875 ] Prashant Sharma commented on SPARK-24432: - Any update on this work? > Add support for dynamic

[jira] [Assigned] (SPARK-26901) Vectorized gapply should not prune columns

2019-02-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26901: --- Assignee: Hyukjin Kwon > Vectorized gapply should not prune columns >

[jira] [Resolved] (SPARK-26901) Vectorized gapply should not prune columns

2019-02-20 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26901. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23810

[jira] [Updated] (SPARK-26940) Observed greater deviation on big endian platform for SingletonReplSuite test case

2019-02-20 Thread Anuja Jakhade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-26940: -- Summary: Observed greater deviation on big endian platform for SingletonReplSuite test case

[jira] [Updated] (SPARK-26940) Observed greater deviation on big endian platform for SingletonReplSuite test case

2019-02-20 Thread Anuja Jakhade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-26940: -- Description: I have built Apache Spark v2.3.2 on Big Endian with AdoptJDK OpenJ9 1.8.0_202.

[jira] [Updated] (SPARK-26940) Observed greater deviation on big endian platform for SingletonReplSuite test case

2019-02-20 Thread Anuja Jakhade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-26940: -- Description: I have built Apache Spark v2.3.2 on Big Endian platform with AdoptJDK OpenJ9 

[jira] [Updated] (SPARK-26940) Observed greater deviation on big endian for SingletonReplSuite test case

2019-02-20 Thread Anuja Jakhade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-26940: -- Attachment: (was: failure_log) > Observed greater deviation on big endian for

[jira] [Updated] (SPARK-26940) Observed greater deviation on big endian for SingletonReplSuite test case

2019-02-20 Thread Anuja Jakhade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-26940: -- Attachment: failure_log.txt > Observed greater deviation on big endian for SingletonReplSuite

[jira] [Updated] (SPARK-26940) Observed greater deviation on big endian for SingletonReplSuite test case

2019-02-20 Thread Anuja Jakhade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-26940: -- Attachment: failure_log > Observed greater deviation on big endian for SingletonReplSuite

[jira] [Updated] (SPARK-26940) Observed greater deviation on big endian for SingletonReplSuite test case

2019-02-20 Thread Anuja Jakhade (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuja Jakhade updated SPARK-26940: -- Description: I have built Apache Spark v2.3.2 on Big Endian with AdoptJDK OpenJ9 1.8.0_202.

[jira] [Commented] (SPARK-26930) Tests in ParquetFilterSuite don't verify filter class

2019-02-20 Thread Nandor Kollar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772829#comment-16772829 ] Nandor Kollar commented on SPARK-26930: --- I don't know either, but I feel that the second option

[jira] [Created] (SPARK-26940) Observed greater deviation Big Endian for SingletonReplSuite test case

2019-02-20 Thread Anuja Jakhade (JIRA)
Anuja Jakhade created SPARK-26940: - Summary: Observed greater deviation Big Endian for SingletonReplSuite test case Key: SPARK-26940 URL: https://issues.apache.org/jira/browse/SPARK-26940 Project:

[jira] [Commented] (SPARK-26930) Tests in ParquetFilterSuite don't verify filter class

2019-02-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772826#comment-16772826 ] Hyukjin Kwon commented on SPARK-26930: -- I am not sure which way will be minimised and simple way.

[jira] [Commented] (SPARK-26930) Tests in ParquetFilterSuite don't verify filter class

2019-02-20 Thread Nandor Kollar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772823#comment-16772823 ] Nandor Kollar commented on SPARK-26930: --- What do you think is the better approach? Test for the

[jira] [Commented] (SPARK-26930) Tests in ParquetFilterSuite don't verify filter class

2019-02-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772812#comment-16772812 ] Hyukjin Kwon commented on SPARK-26930: -- Yea, {{IsNotNull}} will be inserted (see also

[jira] [Commented] (SPARK-26930) Tests in ParquetFilterSuite don't verify filter class

2019-02-20 Thread Nandor Kollar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772806#comment-16772806 ] Nandor Kollar commented on SPARK-26930: --- Thanks [~hyukjin.kwon] for taking a look at this Jira.

[jira] [Comment Edited] (SPARK-26930) Tests in ParquetFilterSuite don't verify filter class

2019-02-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772795#comment-16772795 ] Hyukjin Kwon edited comment on SPARK-26930 at 2/20/19 9:15 AM: --- Ah, gotya

[jira] [Commented] (SPARK-26930) Tests in ParquetFilterSuite don't verify filter class

2019-02-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772795#comment-16772795 ] Hyukjin Kwon commented on SPARK-26930: -- Ah, gotya {{maybeFilter.exists(_.getClass ===

[jira] [Assigned] (SPARK-26939) Fix some outdated comments about task schedulers

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26939: Assignee: Apache Spark > Fix some outdated comments about task schedulers >

[jira] [Assigned] (SPARK-26939) Fix some outdated comments about task schedulers

2019-02-20 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26939: Assignee: (was: Apache Spark) > Fix some outdated comments about task schedulers >

[jira] [Created] (SPARK-26939) Fix some outdated comments about task schedulers

2019-02-20 Thread Chenxiao Mao (JIRA)
Chenxiao Mao created SPARK-26939: Summary: Fix some outdated comments about task schedulers Key: SPARK-26939 URL: https://issues.apache.org/jira/browse/SPARK-26939 Project: Spark Issue Type:

[jira] [Updated] (SPARK-26930) Tests in ParquetFilterSuite don't verify filter class

2019-02-20 Thread Nandor Kollar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated SPARK-26930: -- Summary: Tests in ParquetFilterSuite don't verify filter class (was: Several test cases in