[jira] [Created] (SPARK-25268) runParallelPersonalizedPageRank throws serialization Exception

2018-08-28 Thread Bago Amirbekian (JIRA)
Bago Amirbekian created SPARK-25268: --- Summary: runParallelPersonalizedPageRank throws serialization Exception Key: SPARK-25268 URL: https://issues.apache.org/jira/browse/SPARK-25268 Project: Spark

[jira] [Resolved] (SPARK-25235) Merge the REPL code in Scala 2.11 and 2.12 branches

2018-08-28 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-25235. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22246

[jira] [Commented] (SPARK-25248) Audit barrier APIs for Spark 2.4

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595891#comment-16595891 ] Apache Spark commented on SPARK-25248: -- User 'mengxr' has created a pull request for this issue:

[jira] [Resolved] (SPARK-25112) Spark history cleaner does not work.

2018-08-28 Thread Gu Chao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gu Chao resolved SPARK-25112. - Resolution: Invalid Sorry, It's not a problem. > Spark history cleaner does not work. >

[jira] [Comment Edited] (SPARK-19490) Hive partition columns are case-sensitive

2018-08-28 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595852#comment-16595852 ] Harish edited comment on SPARK-19490 at 8/29/18 2:19 AM: - I see the same issue

[jira] [Commented] (SPARK-19490) Hive partition columns are case-sensitive

2018-08-28 Thread Harish (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595852#comment-16595852 ] Harish commented on SPARK-19490: I > Hive partition columns are case-sensitive >

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

2018-08-28 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595835#comment-16595835 ] Bryan Cutler commented on SPARK-23874: -- [~smilegator] I linked the most relevant pyarrow issues

[jira] [Assigned] (SPARK-25253) Refactor pyspark connection & authentication

2018-08-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25253: Assignee: Imran Rashid > Refactor pyspark connection & authentication >

[jira] [Resolved] (SPARK-25253) Refactor pyspark connection & authentication

2018-08-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25253. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22247

[jira] [Updated] (SPARK-25266) Fix memory leak in Barrier Execution Mode

2018-08-28 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-25266: --- Description: BarrierCoordinator uses Timer and TimerTask. `TimerTask#cancel()` is invoked

[jira] [Updated] (SPARK-25266) Fix memory leak in Barrier Execution Mode

2018-08-28 Thread Kousuke Saruta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-25266: --- Summary: Fix memory leak in Barrier Execution Mode (was: Fix memory leak vulnerability in

[jira] [Assigned] (SPARK-25260) Fix namespace handling in SchemaConverters.toAvroType

2018-08-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25260: Assignee: Arun Mahadevan > Fix namespace handling in SchemaConverters.toAvroType >

[jira] [Resolved] (SPARK-25260) Fix namespace handling in SchemaConverters.toAvroType

2018-08-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25260. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22251

[jira] [Assigned] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2018-08-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-22357: - Assignee: Bo Meng > SparkContext.binaryFiles ignore minPartitions parameter >

[jira] [Resolved] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2018-08-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22357. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21638

[jira] [Resolved] (SPARK-25212) Support Filter in ConvertToLocalRelation

2018-08-28 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25212. - Resolution: Fixed Assignee: Bogdan Raducanu Fix Version/s: 2.4.0 > Support Filter in

[jira] [Commented] (SPARK-25258) Upgrade kryo package to version 4.0.2+

2018-08-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595650#comment-16595650 ] Sean Owen commented on SPARK-25258: --- Both of these suggest kryo 4.  The issue is, does it cause any

[jira] [Commented] (SPARK-25044) Address translation of LMF closure primitive args to Object in Scala 2.12

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595647#comment-16595647 ] Apache Spark commented on SPARK-25044: -- User 'srowen' has created a pull request for this issue:

[jira] [Commented] (SPARK-20799) Unable to infer schema for ORC/Parquet on S3N when secrets are in the URL

2018-08-28 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595633#comment-16595633 ] Steve Loughran commented on SPARK-20799: [~jzijlstra] yes, the final listing. Note that in

[jira] [Resolved] (SPARK-25004) Add spark.executor.pyspark.memory config to set resource.RLIMIT_AS

2018-08-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25004. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21977

[jira] [Assigned] (SPARK-25004) Add spark.executor.pyspark.memory config to set resource.RLIMIT_AS

2018-08-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25004: -- Assignee: Ryan Blue > Add spark.executor.pyspark.memory config to set

[jira] [Commented] (SPARK-25267) Disable ConvertToLocalRelation in the test cases of sql/core and sql/hive

2018-08-28 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595450#comment-16595450 ] Dilip Biswal commented on SPARK-25267: -- [~smilegator] Thanks Sean.. will take a look. > Disable

[jira] [Resolved] (SPARK-25240) A deadlock in ALTER TABLE RECOVER PARTITIONS

2018-08-28 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25240. - Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 2.4.0 > A deadlock in ALTER TABLE

[jira] [Commented] (SPARK-25267) Disable ConvertToLocalRelation in the test cases of sql/core and sql/hive

2018-08-28 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595388#comment-16595388 ] Xiao Li commented on SPARK-25267: - cc [~dkbiswal] > Disable ConvertToLocalRelation in the test cases of

[jira] [Created] (SPARK-25267) Disable ConvertToLocalRelation in the test cases of sql/core and sql/hive

2018-08-28 Thread Xiao Li (JIRA)
Xiao Li created SPARK-25267: --- Summary: Disable ConvertToLocalRelation in the test cases of sql/core and sql/hive Key: SPARK-25267 URL: https://issues.apache.org/jira/browse/SPARK-25267 Project: Spark

[jira] [Commented] (SPARK-25266) Fix memory leak vulnerability in Barrier Execution Mode

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595367#comment-16595367 ] Apache Spark commented on SPARK-25266: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25266) Fix memory leak vulnerability in Barrier Execution Mode

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25266: Assignee: Apache Spark (was: Kousuke Saruta) > Fix memory leak vulnerability in Barrier

[jira] [Assigned] (SPARK-25266) Fix memory leak vulnerability in Barrier Execution Mode

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25266: Assignee: Kousuke Saruta (was: Apache Spark) > Fix memory leak vulnerability in Barrier

[jira] [Commented] (SPARK-25219) KMeans Clustering - Text Data - Results are incorrect

2018-08-28 Thread Vasanthkumar Velayudham (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595363#comment-16595363 ] Vasanthkumar Velayudham commented on SPARK-25219: - Hi [~mgaido] - Thanks for your

[jira] [Updated] (SPARK-25219) KMeans Clustering - Text Data - Results are incorrect

2018-08-28 Thread Vasanthkumar Velayudham (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasanthkumar Velayudham updated SPARK-25219: Attachment: SKLearn_Kmeans.txt Apache_Logs_Results.xlsx

[jira] [Updated] (SPARK-24704) The order of stages in the DAG graph is incorrect

2018-08-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24704: --- Fix Version/s: 2.3.2 > The order of stages in the DAG graph is incorrect >

[jira] [Created] (SPARK-25266) Fix memory leak vulnerability in Barrier Execution Mode

2018-08-28 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-25266: -- Summary: Fix memory leak vulnerability in Barrier Execution Mode Key: SPARK-25266 URL: https://issues.apache.org/jira/browse/SPARK-25266 Project: Spark

[jira] [Created] (SPARK-25265) Fix memory leak vulnerability in Barrier Execution Mode

2018-08-28 Thread Kousuke Saruta (JIRA)
Kousuke Saruta created SPARK-25265: -- Summary: Fix memory leak vulnerability in Barrier Execution Mode Key: SPARK-25265 URL: https://issues.apache.org/jira/browse/SPARK-25265 Project: Spark

[jira] [Resolved] (SPARK-25119) stages in wrong order within job page DAG chart

2018-08-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25119. Resolution: Duplicate > stages in wrong order within job page DAG chart >

[jira] [Assigned] (SPARK-23679) uiWebUrl show inproper URL when running on YARN

2018-08-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23679: -- Assignee: Saisai Shao > uiWebUrl show inproper URL when running on YARN >

[jira] [Resolved] (SPARK-23679) uiWebUrl show inproper URL when running on YARN

2018-08-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23679. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22164

[jira] [Resolved] (SPARK-23997) Configurable max number of buckets

2018-08-28 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23997. - Resolution: Fixed Assignee: Fernando Pereira Fix Version/s: 2.4.0 > Configurable max

[jira] [Assigned] (SPARK-25264) Fix comma-delineated arguments passed into PythonRunner and RRunner

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25264: Assignee: Apache Spark > Fix comma-delineated arguments passed into PythonRunner and

[jira] [Assigned] (SPARK-25264) Fix comma-delineated arguments passed into PythonRunner and RRunner

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25264: Assignee: (was: Apache Spark) > Fix comma-delineated arguments passed into

[jira] [Commented] (SPARK-25264) Fix comma-delineated arguments passed into PythonRunner and RRunner

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595306#comment-16595306 ] Apache Spark commented on SPARK-25264: -- User 'ifilonenko' has created a pull request for this

[jira] [Comment Edited] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases

2018-08-28 Thread yucai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595298#comment-16595298 ] yucai edited comment on SPARK-25206 at 8/28/18 5:06 PM:  Do you want to simulate

[jira] [Comment Edited] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases

2018-08-28 Thread yucai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595298#comment-16595298 ] yucai edited comment on SPARK-25206 at 8/28/18 5:05 PM:  Do you want to simulate

[jira] [Commented] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases

2018-08-28 Thread yucai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595298#comment-16595298 ] yucai commented on SPARK-25206: ---   Do you want to simulate an Exception in Spark?  Backporting 

[jira] [Updated] (SPARK-25264) Fix comma-delineated arguments passed into PythonRunner and RRunner

2018-08-28 Thread Ilan Filonenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilan Filonenko updated SPARK-25264: --- Description: The arguments passed into the PythonRunner and RRunner are comma-delineated. 

[jira] [Created] (SPARK-25264) Fix comma-delineated arguments passed into PythonRunner and RRunner

2018-08-28 Thread Ilan Filonenko (JIRA)
Ilan Filonenko created SPARK-25264: -- Summary: Fix comma-delineated arguments passed into PythonRunner and RRunner Key: SPARK-25264 URL: https://issues.apache.org/jira/browse/SPARK-25264 Project:

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595254#comment-16595254 ] Apache Spark commented on SPARK-25262: -- User 'rvesse' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25262: Assignee: Apache Spark > Make Spark local dir volumes configurable with Spark on

[jira] [Assigned] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25262: Assignee: (was: Apache Spark) > Make Spark local dir volumes configurable with Spark

[jira] [Commented] (SPARK-25175) Field resolution should fail if there is ambiguity for ORC data source native implementation

2018-08-28 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595240#comment-16595240 ] Chenxiao Mao commented on SPARK-25175: -- [~cloud_fan] Does it make sense? > Field resolution should

[jira] [Updated] (SPARK-25175) Field resolution should fail if there is ambiguity for ORC data source native implementation

2018-08-28 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-25175: - Summary: Field resolution should fail if there is ambiguity for ORC data source native

[jira] [Commented] (SPARK-25175) Field resolution should fail if there is ambiguity in case-insensitive mode when reading from ORC

2018-08-28 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595225#comment-16595225 ] Chenxiao Mao commented on SPARK-25175: -- After a deep dive into ORC file read paths (data source

[jira] [Updated] (SPARK-25175) Field resolution should fail if there is ambiguity in case-insensitive mode when reading from ORC

2018-08-28 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-25175: - Summary: Field resolution should fail if there is ambiguity in case-insensitive mode when

[jira] [Comment Edited] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-28 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593194#comment-16593194 ] Chenxiao Mao edited comment on SPARK-25175 at 8/28/18 4:00 PM: ---

[jira] [Updated] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-28 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-25175: - Description: SPARK-25132 adds support for case-insensitive field resolution when reading from

[jira] [Commented] (SPARK-24918) Executor Plugin API

2018-08-28 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595182#comment-16595182 ] Imran Rashid commented on SPARK-24918: -- Yes, that's right, its to avoid putting a reference to the

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2018-08-28 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595178#comment-16595178 ] Rob Vesse commented on SPARK-25262: --- I have changes for this almost ready and plan to open a PR

[jira] [Resolved] (SPARK-25005) Structured streaming doesn't support kafka transaction (creating empty offset with abort & markers)

2018-08-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25005. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.4.0 > Structured

[jira] [Resolved] (SPARK-25218) Potential resource leaks in TransportServer and SocketAuthHelper

2018-08-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-25218. -- Resolution: Fixed Fix Version/s: 2.4.0 > Potential resource leaks in TransportServer

[jira] [Comment Edited] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-28 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593185#comment-16593185 ] Chenxiao Mao edited comment on SPARK-25175 at 8/28/18 3:27 PM: ---

[jira] [Comment Edited] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-28 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593194#comment-16593194 ] Chenxiao Mao edited comment on SPARK-25175 at 8/28/18 3:22 PM: ---

[jira] [Commented] (SPARK-23622) Flaky Test: HiveClientSuites

2018-08-28 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595068#comment-16595068 ] Marco Gaido commented on SPARK-23622: - This failure became permanent in the last build (at least

[jira] [Commented] (SPARK-23663) Spark Streaming Kafka 010 , fails with "java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access"

2018-08-28 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595044#comment-16595044 ] Gabor Somogyi commented on SPARK-23663: --- [~sipeti] from the stacktrace the root cause looks the

[jira] [Commented] (SPARK-23663) Spark Streaming Kafka 010 , fails with "java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access"

2018-08-28 Thread Peter Simon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595035#comment-16595035 ] Peter Simon commented on SPARK-23663: - [~gsomogyi] isn't this the same as SPARK-19185? > Spark

[jira] [Commented] (SPARK-24918) Executor Plugin API

2018-08-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595020#comment-16595020 ] Sean Owen commented on SPARK-24918: --- Wait, why doesn't a static init "run everywhere"? is "everywhere"

[jira] [Created] (SPARK-25263) Add scheduler integration test for SPARK-24909

2018-08-28 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-25263: - Summary: Add scheduler integration test for SPARK-24909 Key: SPARK-25263 URL: https://issues.apache.org/jira/browse/SPARK-25263 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-25091) UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory

2018-08-28 Thread Chao Fang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594981#comment-16594981 ] Chao Fang edited comment on SPARK-25091 at 8/28/18 1:39 PM: Yes, I think

[jira] [Commented] (SPARK-25091) UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory

2018-08-28 Thread Chao Fang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594981#comment-16594981 ] Chao Fang commented on SPARK-25091: --- Yes, I think it's UI issue. Today I run the CACHE/UNCACHE TABLE

[jira] [Updated] (SPARK-25091) UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory

2018-08-28 Thread Chao Fang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Fang updated SPARK-25091: -- Attachment: 4.png > UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory >

[jira] [Updated] (SPARK-25091) UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory

2018-08-28 Thread Chao Fang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Fang updated SPARK-25091: -- Attachment: 3.png 2.png 1.png 0.png > UNCACHE TABLE,

[jira] [Commented] (SPARK-25213) DataSourceV2 doesn't seem to produce unsafe rows

2018-08-28 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594973#comment-16594973 ] Li Jin commented on SPARK-25213: This is resolved by https://github.com/apache/spark/pull/22104 >

[jira] [Commented] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases

2018-08-28 Thread yucai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594958#comment-16594958 ] yucai commented on SPARK-25206: --- [~smilegator] , 2.1's exception is from parquet. {code:java}

[jira] [Created] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2018-08-28 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-25262: - Summary: Make Spark local dir volumes configurable with Spark on Kubernetes Key: SPARK-25262 URL: https://issues.apache.org/jira/browse/SPARK-25262 Project: Spark

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

2018-08-28 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594933#comment-16594933 ] Rob Vesse commented on SPARK-25024: --- [~tgraves] Attempting to answer your questions: * We never used

[jira] [Assigned] (SPARK-25102) Write Spark version information to Parquet file footers

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25102: Assignee: Apache Spark > Write Spark version information to Parquet file footers >

[jira] [Commented] (SPARK-25102) Write Spark version information to Parquet file footers

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594914#comment-16594914 ] Apache Spark commented on SPARK-25102: -- User 'npoberezkin' has created a pull request for this

[jira] [Assigned] (SPARK-25102) Write Spark version information to Parquet file footers

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25102: Assignee: (was: Apache Spark) > Write Spark version information to Parquet file

[jira] [Assigned] (SPARK-24411) Adding native Java tests for `isInCollection`

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24411: Assignee: (was: Apache Spark) > Adding native Java tests for `isInCollection` >

[jira] [Commented] (SPARK-24411) Adding native Java tests for `isInCollection`

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594855#comment-16594855 ] Apache Spark commented on SPARK-24411: -- User 'aai95' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24411) Adding native Java tests for `isInCollection`

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24411: Assignee: Apache Spark > Adding native Java tests for `isInCollection` >

[jira] [Comment Edited] (SPARK-25102) Write Spark version information to Parquet file footers

2018-08-28 Thread Nikita Poberezkin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594809#comment-16594809 ] Nikita Poberezkin edited comment on SPARK-25102 at 8/28/18 10:43 AM: -

[jira] [Commented] (SPARK-25102) Write Spark version information to Parquet file footers

2018-08-28 Thread Nikita Poberezkin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594809#comment-16594809 ] Nikita Poberezkin commented on SPARK-25102: --- Hi, Zoltan. I've tried to override getName method

[jira] [Assigned] (SPARK-25261) Update configuration.md, correct the default units of spark.driver|executor.memory

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25261: Assignee: (was: Apache Spark) > Update configuration.md, correct the default units

[jira] [Commented] (SPARK-25261) Update configuration.md, correct the default units of spark.driver|executor.memory

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594716#comment-16594716 ] Apache Spark commented on SPARK-25261: -- User 'ivoson' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25261) Update configuration.md, correct the default units of spark.driver|executor.memory

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25261: Assignee: Apache Spark > Update configuration.md, correct the default units of >

[jira] [Created] (SPARK-25261) Update configuration.md, correct the default units of spark.driver|executor.memory

2018-08-28 Thread huangtengfei (JIRA)
huangtengfei created SPARK-25261: Summary: Update configuration.md, correct the default units of spark.driver|executor.memory Key: SPARK-25261 URL: https://issues.apache.org/jira/browse/SPARK-25261

[jira] [Assigned] (SPARK-25260) Fix namespace handling in SchemaConverters.toAvroType

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25260: Assignee: Apache Spark > Fix namespace handling in SchemaConverters.toAvroType >

[jira] [Commented] (SPARK-25260) Fix namespace handling in SchemaConverters.toAvroType

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594682#comment-16594682 ] Apache Spark commented on SPARK-25260: -- User 'arunmahadevan' has created a pull request for this

[jira] [Assigned] (SPARK-25260) Fix namespace handling in SchemaConverters.toAvroType

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25260: Assignee: (was: Apache Spark) > Fix namespace handling in

[jira] [Created] (SPARK-25260) Fix namespace handling in SchemaConverters.toAvroType

2018-08-28 Thread Arun Mahadevan (JIRA)
Arun Mahadevan created SPARK-25260: -- Summary: Fix namespace handling in SchemaConverters.toAvroType Key: SPARK-25260 URL: https://issues.apache.org/jira/browse/SPARK-25260 Project: Spark

[jira] [Commented] (SPARK-20977) NPE in CollectionAccumulator

2018-08-28 Thread howie yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594659#comment-16594659 ] howie yu commented on SPARK-20977: -- I use pyspark 2.3.1 also have this problem , but in different line

[jira] [Resolved] (SPARK-25226) Extend functionality of from_json to support arrays of differently-typed elements

2018-08-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25226. -- Resolution: Won't Fix Please don't reopen until we are clear that this is going to be fixed.

[jira] [Commented] (SPARK-25226) Extend functionality of from_json to support arrays of differently-typed elements

2018-08-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594648#comment-16594648 ] Hyukjin Kwon commented on SPARK-25226: -- {quote} 1. What exactly is a catalog string? {quote} Hive

[jira] [Reopened] (SPARK-25226) Extend functionality of from_json to support arrays of differently-typed elements

2018-08-28 Thread Yuriy Davygora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy Davygora reopened SPARK-25226: > Extend functionality of from_json to support arrays of differently-typed > elements >

[jira] [Commented] (SPARK-25226) Extend functionality of from_json to support arrays of differently-typed elements

2018-08-28 Thread Yuriy Davygora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594643#comment-16594643 ] Yuriy Davygora commented on SPARK-25226: [~hyukjin.kwon] 1. What exactly is a catalog string?

[jira] [Issue Comment Deleted] (SPARK-25259) Left/Right join support push down during-join predicates

2018-08-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-25259: Comment: was deleted (was: I'm working on.) > Left/Right join support push down during-join

[jira] [Assigned] (SPARK-25259) Left/Right join support push down during-join predicates

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25259: Assignee: (was: Apache Spark) > Left/Right join support push down during-join

[jira] [Assigned] (SPARK-25259) Left/Right join support push down during-join predicates

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25259: Assignee: Apache Spark > Left/Right join support push down during-join predicates >

[jira] [Commented] (SPARK-25259) Left/Right join support push down during-join predicates

2018-08-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594606#comment-16594606 ] Apache Spark commented on SPARK-25259: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-25259) Left/Right join support push down during-join predicates

2018-08-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594563#comment-16594563 ] Yuming Wang commented on SPARK-25259: - I'm working on. > Left/Right join support push down

[jira] [Created] (SPARK-25259) Left/Right join support push down during-join predicates

2018-08-28 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-25259: --- Summary: Left/Right join support push down during-join predicates Key: SPARK-25259 URL: https://issues.apache.org/jira/browse/SPARK-25259 Project: Spark Issue

  1   2   >