[jira] [Comment Edited] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

2018-09-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606638#comment-16606638 ] Yuming Wang edited comment on SPARK-25330 at 9/7/18 3:04 AM: - [~srowen] It

[jira] [Commented] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

2018-09-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606659#comment-16606659 ] Yuming Wang commented on SPARK-25330: - It affects Spark enable Hive support with a proxy user. >

[jira] [Updated] (SPARK-25271) Creating parquet table with all the column null throws exception

2018-09-06 Thread shivusondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shivusondur updated SPARK-25271: Attachment: image-2018-09-07-09-12-34-944.png > Creating parquet table with all the column null

[jira] [Resolved] (SPARK-25343) Extend CSV parsing to Dataset[List[String]]

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25343. -- Resolution: Won't Fix Let me leave this as {{Won't Fix}} for now. > Extend CSV parsing to

[jira] [Commented] (SPARK-25344) Break large tests.py files into smaller files

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606679#comment-16606679 ] Hyukjin Kwon commented on SPARK-25344: -- I actually roughly tried this and then quit before since

[jira] [Commented] (SPARK-25359) Incorporate pyspark test output into jenkins test report

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606683#comment-16606683 ] Hyukjin Kwon commented on SPARK-25359: -- Adding [~shaneknapp] as well. > Incorporate pyspark test

[jira] [Updated] (SPARK-25361) Support for Kinesis Client Library 2.0

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25361: - Target Version/s: (was: 3.0.0) > Support for Kinesis Client Library 2.0 >

[jira] [Commented] (SPARK-25361) Support for Kinesis Client Library 2.0

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606685#comment-16606685 ] Hyukjin Kwon commented on SPARK-25361: -- (please avoid to set the target version which is usually

[jira] [Updated] (SPARK-25271) Creating parquet table with all the column null throws exception

2018-09-06 Thread shivusondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shivusondur updated SPARK-25271: Attachment: image-2018-09-07-09-29-52-899.png > Creating parquet table with all the column null

[jira] [Updated] (SPARK-25271) Creating parquet table with all the column null throws exception

2018-09-06 Thread shivusondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shivusondur updated SPARK-25271: Attachment: image-2018-09-07-09-29-33-370.png > Creating parquet table with all the column null

[jira] [Updated] (SPARK-25271) Creating parquet table with all the column null throws exception

2018-09-06 Thread shivusondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shivusondur updated SPARK-25271: Attachment: image-2018-09-07-09-33-03-095.png > Creating parquet table with all the column null

[jira] [Updated] (SPARK-25271) Creating parquet table with all the column null throws exception

2018-09-06 Thread shivusondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shivusondur updated SPARK-25271: Attachment: image-2018-09-07-09-32-43-892.png > Creating parquet table with all the column null

[jira] [Commented] (SPARK-25271) Creating parquet table with all the column null throws exception

2018-09-06 Thread shivusondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606693#comment-16606693 ] shivusondur commented on SPARK-25271: - As [~S71955] told, The Behaviour changed form above 

[jira] [Updated] (SPARK-25358) MutableProjection supports fallback to an interpreted mode

2018-09-06 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-25358: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-23580 >

[jira] [Created] (SPARK-25359) Incorporate pyspark test output into jenkins test report

2018-09-06 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-25359: Summary: Incorporate pyspark test output into jenkins test report Key: SPARK-25359 URL: https://issues.apache.org/jira/browse/SPARK-25359 Project: Spark

[jira] [Updated] (SPARK-25108) Dataset.show() generates incorrect padding for Unicode Character

2018-09-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25108: -- Fix Version/s: 3.0.0 > Dataset.show() generates incorrect padding for Unicode Character >

[jira] [Commented] (SPARK-24771) Upgrade AVRO version from 1.7.7 to 1.8

2018-09-06 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605308#comment-16605308 ] Gengliang Wang commented on SPARK-24771: [~vanzin] I am OK with either way. Shading Avro 1.8 in

[jira] [Commented] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605335#comment-16605335 ] Apache Spark commented on SPARK-25313: -- User 'gengliangwang' has created a pull request for this

[jira] [Resolved] (SPARK-25301) When a view uses an UDF from a non default database, Spark analyser throws AnalysisException

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25301. -- Resolution: Not A Problem I am resolving this since for the current status it works as

[jira] [Commented] (SPARK-12321) JSON format for logical/physical execution plans

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605317#comment-16605317 ] Apache Spark commented on SPARK-12321: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-12321) JSON format for logical/physical execution plans

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605316#comment-16605316 ] Apache Spark commented on SPARK-12321: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606633#comment-16606633 ] Hyukjin Kwon commented on SPARK-25036: -- There are already too many warnings and I assume it's

[jira] [Commented] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

2018-09-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606638#comment-16606638 ] Yuming Wang commented on SPARK-25330: - [~srowen] It affects Hive. {{spark-sql}}, {{spark-shell}} and

[jira] [Commented] (SPARK-23098) Migrate Kafka batch source to v2

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606642#comment-16606642 ] Hyukjin Kwon commented on SPARK-23098: -- ping [~joseph.torres] > Migrate Kafka batch source to v2 >

[jira] [Commented] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

2018-09-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606644#comment-16606644 ] Sean Owen commented on SPARK-25330: --- For clarity, you mean none of those things work with a proxy

[jira] [Resolved] (SPARK-25356) Add Parquet block size (row group size) option to SparkSQL configuration

2018-09-06 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian resolved SPARK-25356. - Resolution: Invalid > Add Parquet block size (row group size) option to SparkSQL configuration >

[jira] [Comment Edited] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606633#comment-16606633 ] Hyukjin Kwon edited comment on SPARK-25036 at 9/7/18 2:15 AM: -- There are

[jira] [Updated] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

2018-09-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23243: Fix Version/s: 2.3.2 > Shuffle+Repartition on an RDD could lead to incorrect answers >

[jira] [Updated] (SPARK-25157) Streaming of image files from directory

2018-09-06 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-25157: Target Version/s: 3.0.0 > Streaming of image files from directory >

[jira] [Created] (SPARK-25355) Support --proxy-user for Spark on K8s

2018-09-06 Thread Stavros Kontopoulos (JIRA)
Stavros Kontopoulos created SPARK-25355: --- Summary: Support --proxy-user for Spark on K8s Key: SPARK-25355 URL: https://issues.apache.org/jira/browse/SPARK-25355 Project: Spark Issue

[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI

2018-09-06 Thread Michael Spector (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605642#comment-16605642 ] Michael Spector commented on SPARK-23670: - The bug still happens in Apache Spark 2.3.1:

[jira] [Commented] (SPARK-25208) Loosen Cast.forceNullable for DecimalType.

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605680#comment-16605680 ] Apache Spark commented on SPARK-25208: -- User 'ueshin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25354) Parquet vectorized record reader has unneeded operation in several methods

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25354: Assignee: Apache Spark > Parquet vectorized record reader has unneeded operation in

[jira] [Assigned] (SPARK-25354) Parquet vectorized record reader has unneeded operation in several methods

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25354: Assignee: (was: Apache Spark) > Parquet vectorized record reader has unneeded

[jira] [Commented] (SPARK-25343) Extend CSV parsing to Dataset[List[String]]

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605439#comment-16605439 ] Hyukjin Kwon commented on SPARK-25343: -- The workaround should be like: {code} val ds:

[jira] [Updated] (SPARK-25343) Extend CSV parsing to Dataset[List[String]]

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25343: - Component/s: (was: Spark Core) SQL > Extend CSV parsing to

[jira] [Commented] (SPARK-25343) Extend CSV parsing to Dataset[List[String]]

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605443#comment-16605443 ] Hyukjin Kwon commented on SPARK-25343: -- I wouldn't add more APIs but try to work around if the

[jira] [Commented] (SPARK-14220) Build and test Spark against Scala 2.12

2018-09-06 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605552#comment-16605552 ] Stavros Kontopoulos commented on SPARK-14220: - Great! > Build and test Spark against Scala

[jira] [Updated] (SPARK-25353) executeTake in SparkPlan could decode rows more than necessary.

2018-09-06 Thread Dooyoung Hwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dooyoung Hwang updated SPARK-25353: --- Description: In some cases, executeTake in SparkPlan could decode more than necessary. For

[jira] [Updated] (SPARK-25353) executeTake in SparkPlan could decode rows more than necessary.

2018-09-06 Thread Dooyoung Hwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dooyoung Hwang updated SPARK-25353: --- Description: In some cases, executeTake in SparkPlan could decode more than necessary. For

[jira] [Updated] (SPARK-25345) Deprecate public APIs from ImageSchema

2018-09-06 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25345: --- Description: After SPARK-22666, we can deprecate the public APIs in ImageSchema (Scala/Python) and

[jira] [Commented] (SPARK-14220) Build and test Spark against Scala 2.12

2018-09-06 Thread Darcy Shen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605457#comment-16605457 ] Darcy Shen commented on SPARK-14220: Congrats!   First green one:  

[jira] [Commented] (SPARK-25271) Creating parquet table with all the column null throws exception

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605453#comment-16605453 ] Hyukjin Kwon commented on SPARK-25271: -- will take a look later but mind if I ask to elaborate

[jira] [Commented] (SPARK-25354) Parquet vectorized record reader has unneeded operation in several methods

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605468#comment-16605468 ] Apache Spark commented on SPARK-25354: -- User 'SongYadong' has created a pull request for this

[jira] [Commented] (SPARK-25354) Parquet vectorized record reader has unneeded operation in several methods

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605469#comment-16605469 ] Apache Spark commented on SPARK-25354: -- User 'SongYadong' has created a pull request for this

[jira] [Updated] (SPARK-25353) executeTake in SparkPlan could decode rows more than necessary.

2018-09-06 Thread Dooyoung Hwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dooyoung Hwang updated SPARK-25353: --- Summary: executeTake in SparkPlan could decode rows more than necessary. (was: executeTake

[jira] [Updated] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI

2018-09-06 Thread Michael Spector (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Spector updated SPARK-23670: Attachment: heapdump_OOM.png > Memory leak of SparkPlanGraphWrapper in sparkUI >

[jira] [Created] (SPARK-25354) Parquet vectorized record reader has unneeded operation in several methods

2018-09-06 Thread SongYadong (JIRA)
SongYadong created SPARK-25354: -- Summary: Parquet vectorized record reader has unneeded operation in several methods Key: SPARK-25354 URL: https://issues.apache.org/jira/browse/SPARK-25354 Project:

[jira] [Resolved] (SPARK-25333) Ability to add new columns in Dataset in a user-defined position

2018-09-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25333. -- Resolution: Won't Fix > Ability to add new columns in Dataset in a user-defined position >

[jira] [Assigned] (SPARK-25345) Deprecate public APIs from ImageSchema

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25345: Assignee: Apache Spark > Deprecate public APIs from ImageSchema >

[jira] [Commented] (SPARK-25345) Deprecate public APIs from ImageSchema

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605577#comment-16605577 ] Apache Spark commented on SPARK-25345: -- User 'WeichenXu123' has created a pull request for this

[jira] [Assigned] (SPARK-25345) Deprecate public APIs from ImageSchema

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25345: Assignee: (was: Apache Spark) > Deprecate public APIs from ImageSchema >

[jira] [Commented] (SPARK-25345) Deprecate public APIs from ImageSchema

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605575#comment-16605575 ] Apache Spark commented on SPARK-25345: -- User 'WeichenXu123' has created a pull request for this

[jira] [Created] (SPARK-25356) Add Parquet block size (row group size) option to SparkSQL configuration

2018-09-06 Thread liuxian (JIRA)
liuxian created SPARK-25356: --- Summary: Add Parquet block size (row group size) option to SparkSQL configuration Key: SPARK-25356 URL: https://issues.apache.org/jira/browse/SPARK-25356 Project: Spark

[jira] [Commented] (SPARK-25356) Add Parquet block size (row group size) option to SparkSQL configuration

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605619#comment-16605619 ] Apache Spark commented on SPARK-25356: -- User '10110346' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25356) Add Parquet block size (row group size) option to SparkSQL configuration

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25356: Assignee: Apache Spark > Add Parquet block size (row group size) option to SparkSQL

[jira] [Assigned] (SPARK-25356) Add Parquet block size (row group size) option to SparkSQL configuration

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25356: Assignee: (was: Apache Spark) > Add Parquet block size (row group size) option to

[jira] [Updated] (SPARK-25353) executeTake in SparkPlan could deserialize more than necessary.

2018-09-06 Thread Dooyoung Hwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dooyoung Hwang updated SPARK-25353: --- Summary: executeTake in SparkPlan could deserialize more than necessary. (was: executeTake

[jira] [Updated] (SPARK-25353) executeTake has been modified to avoid unnecessary deserialization.

2018-09-06 Thread Dooyoung Hwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dooyoung Hwang updated SPARK-25353: --- Summary: executeTake has been modified to avoid unnecessary deserialization. (was:

[jira] [Commented] (SPARK-17732) ALTER TABLE DROP PARTITION should support comparators

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605749#comment-16605749 ] Apache Spark commented on SPARK-17732: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Updated] (SPARK-24838) Support uncorrelated IN/EXISTS subqueries for more operators

2018-09-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24838: Target Version/s: 3.0.0 (was: 2.4.0) > Support uncorrelated IN/EXISTS subqueries for more

[jira] [Commented] (SPARK-24838) Support uncorrelated IN/EXISTS subqueries for more operators

2018-09-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605775#comment-16605775 ] Wenchen Fan commented on SPARK-24838: - It's too late for 2.4, I'm retargeting it to 3.0, thanks! >

[jira] [Assigned] (SPARK-25108) Dataset.show() generates incorrect padding for Unicode Character

2018-09-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25108: - Assignee: xuejianbest > Dataset.show() generates incorrect padding for Unicode Character >

[jira] [Resolved] (SPARK-25108) Dataset.show() generates incorrect padding for Unicode Character

2018-09-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25108. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22048

[jira] [Created] (SPARK-25358) MutableProjection supports fallback to an interpreted mode

2018-09-06 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-25358: Summary: MutableProjection supports fallback to an interpreted mode Key: SPARK-25358 URL: https://issues.apache.org/jira/browse/SPARK-25358 Project: Spark

[jira] [Updated] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

2018-09-06 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-25128: --- Target Version/s: 3.0.0 (was: 2.4.0, 2.3.3) Priority: Minor (was: Major) >

[jira] [Resolved] (SPARK-14220) Build and test Spark against Scala 2.12

2018-09-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-14220. --- Resolution: Fixed Fix Version/s: 2.4.0 Heh, OK looks like the 2.12 build really might work

[jira] [Resolved] (SPARK-25027) LegacyAccumulatorWrapper test fails in Scala 2.12

2018-09-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25027. --- Resolution: Duplicate Target Version/s: (was: 2.4.0) Oops, no a duplicate >

[jira] [Assigned] (SPARK-25328) Add an example for having two columns as the grouping key in group aggregate pandas UDF

2018-09-06 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-25328: Assignee: Hyukjin Kwon > Add an example for having two columns as the grouping key in

[jira] [Commented] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

2018-09-06 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605916#comment-16605916 ] Erik Erlandson commented on SPARK-25128: Retargeting to next release sounds good. There has been

[jira] [Resolved] (SPARK-25328) Add an example for having two columns as the grouping key in group aggregate pandas UDF

2018-09-06 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-25328. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22329

[jira] [Commented] (SPARK-25036) Scala 2.12 issues: Compilation error with sbt

2018-09-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605773#comment-16605773 ] Wenchen Fan commented on SPARK-25036: - Have we resolved all the problems for this ticket? > Scala

[jira] [Updated] (SPARK-25357) Abbreviated metadata in DataSourceScanExec results in incomplete information in event log

2018-09-06 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-25357: --- Summary: Abbreviated metadata in DataSourceScanExec results in incomplete information in event log

[jira] [Comment Edited] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

2018-09-06 Thread Brahma Reddy Battula (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605738#comment-16605738 ] Brahma Reddy Battula edited comment on SPARK-25330 at 9/6/18 12:56 PM:

[jira] [Created] (SPARK-25357) Abbreviated metadata in DataSourceScanExec results in incomplete location in event log

2018-09-06 Thread Lantao Jin (JIRA)
Lantao Jin created SPARK-25357: -- Summary: Abbreviated metadata in DataSourceScanExec results in incomplete location in event log Key: SPARK-25357 URL: https://issues.apache.org/jira/browse/SPARK-25357

[jira] [Updated] (SPARK-25108) Dataset.show() generates incorrect padding for Unicode Character

2018-09-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25108: -- Fix Version/s: (was: 2.4.0) > Dataset.show() generates incorrect padding for Unicode Character >

[jira] [Assigned] (SPARK-25358) MutableProjection supports fallback to an interpreted mode

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25358: Assignee: Apache Spark > MutableProjection supports fallback to an interpreted mode >

[jira] [Assigned] (SPARK-25358) MutableProjection supports fallback to an interpreted mode

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25358: Assignee: (was: Apache Spark) > MutableProjection supports fallback to an

[jira] [Commented] (SPARK-25358) MutableProjection supports fallback to an interpreted mode

2018-09-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605888#comment-16605888 ] Apache Spark commented on SPARK-25358: -- User 'maropu' has created a pull request for this issue:

[jira] [Updated] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-25313: Fix Version/s: 2.3.2 > Fix regression in FileFormatWriter output schema >

[jira] [Commented] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

2018-09-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605906#comment-16605906 ] Yuming Wang commented on SPARK-25330: - [~brahmareddy] Sorry. I didn't have script because we need a

[jira] [Commented] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

2018-09-06 Thread Brahma Reddy Battula (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605738#comment-16605738 ] Brahma Reddy Battula commented on SPARK-25330: -- [~yumwang] is it possible to share debug

[jira] [Comment Edited] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

2018-09-06 Thread Brahma Reddy Battula (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605738#comment-16605738 ] Brahma Reddy Battula edited comment on SPARK-25330 at 9/6/18 12:56 PM:

[jira] [Commented] (SPARK-25128) multiple simultaneous job submissions against k8s backend cause driver pods to hang

2018-09-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605762#comment-16605762 ] Wenchen Fan commented on SPARK-25128: - Do we have a solution for this issue? or a workaround? Shall

[jira] [Updated] (SPARK-25108) Dataset.show() generates incorrect padding for Unicode Character

2018-09-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25108: -- Priority: Minor (was: Critical) > Dataset.show() generates incorrect padding for Unicode Character >

[jira] [Updated] (SPARK-25357) Abbreviated simpleString in DataSourceScanExec results in incomplete information in event log

2018-09-06 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-25357: --- Summary: Abbreviated simpleString in DataSourceScanExec results in incomplete information in event

[jira] [Updated] (SPARK-25357) Abbreviated metadata in DataSourceScanExec results in incomplete information in event log

2018-09-06 Thread Lantao Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-25357: --- Description: Field {{metadata}} removed from {{SparkPlanInfo}} in SPARK-17701. Corresponding, this

[jira] [Commented] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

2018-09-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605917#comment-16605917 ] Sean Owen commented on SPARK-25330: --- [~yumwang] does this affect basically anyone using spark-sql with

[jira] [Created] (SPARK-25361) Support for Kinesis Client Library 2.0

2018-09-06 Thread Cory Locklear (JIRA)
Cory Locklear created SPARK-25361: - Summary: Support for Kinesis Client Library 2.0 Key: SPARK-25361 URL: https://issues.apache.org/jira/browse/SPARK-25361 Project: Spark Issue Type:

[jira] [Created] (SPARK-25360) Parallelized RDDs of Ranges could have known partitioner

2018-09-06 Thread holdenk (JIRA)
holdenk created SPARK-25360: --- Summary: Parallelized RDDs of Ranges could have known partitioner Key: SPARK-25360 URL: https://issues.apache.org/jira/browse/SPARK-25360 Project: Spark Issue Type:

[jira] [Issue Comment Deleted] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

2018-09-06 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-25295: - Comment: was deleted (was: We made it clear in the documentation of the Kubernetes mode at

[jira] [Updated] (SPARK-25328) Add an example for having two columns as the grouping key in group aggregate pandas UDF

2018-09-06 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-25328: - Fix Version/s: 3.0.0 > Add an example for having two columns as the grouping key in group

[jira] [Resolved] (SPARK-25072) PySpark custom Row class can be given extra parameters

2018-09-06 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-25072. -- Resolution: Fixed Fix Version/s: 2.3.2 2.4.0

[jira] [Assigned] (SPARK-25072) PySpark custom Row class can be given extra parameters

2018-09-06 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-25072: Assignee: Li Yuanjian > PySpark custom Row class can be given extra parameters >

[jira] [Commented] (SPARK-25330) Permission issue after upgrade hadoop version to 2.7.7

2018-09-06 Thread Eric Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606052#comment-16606052 ] Eric Yang commented on SPARK-25330: --- {quote} user.getRealUser(): ad...@kerberos.mycom.com

[jira] [Resolved] (SPARK-25268) runParallelPersonalizedPageRank throws serialization Exception

2018-09-06 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-25268. --- Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Issue

[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI

2018-09-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606343#comment-16606343 ] Marcelo Vanzin commented on SPARK-23670: Do you mind listing the {{SparkPlanGraphNodeWrapper}}

[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI

2018-09-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606355#comment-16606355 ] Marcelo Vanzin commented on SPARK-23670: Yep, looks like just huge plans. Do you mind opening a

[jira] [Updated] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI

2018-09-06 Thread Michael Spector (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Spector updated SPARK-23670: Attachment: Screen Shot 2018-09-06 at 23.19.56.png > Memory leak of SparkPlanGraphWrapper

[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI

2018-09-06 Thread Michael Spector (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606354#comment-16606354 ] Michael Spector commented on SPARK-23670: - !Screen Shot 2018-09-06 at

  1   2   >