[jira] [Resolved] (SPARK-25736) add tests to verify the behavior of multi-column count

2018-10-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25736. -- Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Issue resolved by pull

[jira] [Created] (SPARK-25740) Set some configuration need invalidateStatsCache

2018-10-16 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-25740: --- Summary: Set some configuration need invalidateStatsCache Key: SPARK-25740 URL: https://issues.apache.org/jira/browse/SPARK-25740 Project: Spark Issue Type: Bu

[jira] [Assigned] (SPARK-25740) Set some configuration need invalidateStatsCache

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25740: Assignee: Apache Spark > Set some configuration need invalidateStatsCache > -

[jira] [Commented] (SPARK-25740) Set some configuration need invalidateStatsCache

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651264#comment-16651264 ] Apache Spark commented on SPARK-25740: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-25740) Set some configuration need invalidateStatsCache

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25740: Assignee: (was: Apache Spark) > Set some configuration need invalidateStatsCache > --

[jira] [Updated] (SPARK-25740) Set some configuration need invalidateStatsCache

2018-10-16 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-25740: Description: How to reproduce: {code:sql} # spark-sql create table t1 (a int) stored as parquet; c

[jira] [Updated] (SPARK-25740) Set some configuration need invalidateStatsCache

2018-10-16 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-25740: Description: How to reproduce: {code:sql} # spark-sql create table t1 (a int) stored as parquet; c

[jira] [Updated] (SPARK-22386) Data Source V2 improvements

2018-10-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-22386: - Target Version/s: 3.0.0 (was: 2.5.0) > Data Source V2 improvements > --

[jira] [Commented] (SPARK-24882) data source v2 API improvement

2018-10-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651303#comment-16651303 ] Hyukjin Kwon commented on SPARK-24882: -- [~cloud_fan], should we put this JIRA under

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651301#comment-16651301 ] Mridul Muralidharan commented on SPARK-25732: - [~vanzin] With long running a

[jira] [Updated] (SPARK-25729) It is better to replace `minPartitions` with `defaultParallelism` , when `minPartitions` is less than `defaultParallelism`

2018-10-16 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-25729: Description: In ‘WholeTextFileRDD’,when `minPartitions` is less than `defaultParallelism`, it is better t

[jira] [Commented] (SPARK-10816) EventTime based sessionization

2018-10-16 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651404#comment-16651404 ] Jungtaek Lim commented on SPARK-10816: -- Update: I've crafted another performance te

[jira] [Commented] (SPARK-18127) Add hooks and extension points to Spark

2018-10-16 Thread Ayush Nigam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651427#comment-16651427 ] Ayush Nigam commented on SPARK-18127: - Is there any documentation on how to use Hook

[jira] [Updated] (SPARK-25741) Long URLs are not rendered properly in web UI

2018-10-16 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-25741: --- Summary: Long URLs are not rendered properly in web UI (was: Long URL are not rendered prop

[jira] [Created] (SPARK-25741) Long URL are not rendered properly in web UI

2018-10-16 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-25741: -- Summary: Long URL are not rendered properly in web UI Key: SPARK-25741 URL: https://issues.apache.org/jira/browse/SPARK-25741 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-25741) Long URLs are not rendered properly in web UI

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25741: Assignee: Apache Spark > Long URLs are not rendered properly in web UI >

[jira] [Commented] (SPARK-25741) Long URLs are not rendered properly in web UI

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651438#comment-16651438 ] Apache Spark commented on SPARK-25741: -- User 'gengliangwang' has created a pull req

[jira] [Commented] (SPARK-25741) Long URLs are not rendered properly in web UI

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651437#comment-16651437 ] Apache Spark commented on SPARK-25741: -- User 'gengliangwang' has created a pull req

[jira] [Assigned] (SPARK-25741) Long URLs are not rendered properly in web UI

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25741: Assignee: (was: Apache Spark) > Long URLs are not rendered properly in web UI > -

[jira] [Commented] (SPARK-25728) SPIP: Structured Intermediate Representation (Tungsten IR) for generating Java code

2018-10-16 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651480#comment-16651480 ] Marco Gaido commented on SPARK-25728: - Thanks [~kiszk]. I will check it ASAP, thanks

[jira] [Created] (SPARK-25742) Is there a way to pass the Azure blob storage credentials to the spark for k8s init-container?

2018-10-16 Thread Oscar Bonilla (JIRA)
Oscar Bonilla created SPARK-25742: - Summary: Is there a way to pass the Azure blob storage credentials to the spark for k8s init-container? Key: SPARK-25742 URL: https://issues.apache.org/jira/browse/SPARK-25742

[jira] [Created] (SPARK-25743) New executors are not launched for kubernetes spark thrift on deleting existing executors

2018-10-16 Thread neenu (JIRA)
neenu created SPARK-25743: - Summary: New executors are not launched for kubernetes spark thrift on deleting existing executors Key: SPARK-25743 URL: https://issues.apache.org/jira/browse/SPARK-25743 Project:

[jira] [Updated] (SPARK-25743) New executors are not launched for kubernetes spark thrift on deleting existing executors

2018-10-16 Thread neenu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] neenu updated SPARK-25743: -- Attachment: query_0_correct.sql > New executors are not launched for kubernetes spark thrift on deleting > ex

[jira] [Commented] (SPARK-25743) New executors are not launched for kubernetes spark thrift on deleting existing executors

2018-10-16 Thread neenu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651514#comment-16651514 ] neenu commented on SPARK-25743: --- Also attached the tpcds query list executed [^query_0_cor

[jira] [Updated] (SPARK-25743) New executors are not launched for kubernetes spark thrift on deleting existing executors

2018-10-16 Thread neenu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] neenu updated SPARK-25743: -- Attachment: driver.log > New executors are not launched for kubernetes spark thrift on deleting > existing ex

[jira] [Created] (SPARK-25744) Allow kubernetes integration tests to be run against a real cluster.

2018-10-16 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-25744: --- Summary: Allow kubernetes integration tests to be run against a real cluster. Key: SPARK-25744 URL: https://issues.apache.org/jira/browse/SPARK-25744 Project: S

[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651546#comment-16651546 ] Apache Spark commented on SPARK-21402: -- User 'vofque' has created a pull request fo

[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651550#comment-16651550 ] Apache Spark commented on SPARK-21402: -- User 'vofque' has created a pull request fo

[jira] [Commented] (SPARK-18832) Spark SQL: Thriftserver unable to run a registered Hive UDTF

2018-10-16 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651565#comment-16651565 ] Yuming Wang commented on SPARK-18832: - [~roadster11x] I can't reproduce. > Spark SQ

[jira] [Updated] (SPARK-24882) data source v2 API improvement

2018-10-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24882: Issue Type: Sub-task (was: Improvement) Parent: SPARK-22386 > data source v2 API improvem

[jira] [Updated] (SPARK-24882) data source v2 API improvement

2018-10-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24882: Fix Version/s: (was: 2.4.0) 3.0.0 > data source v2 API improvement > --

[jira] [Assigned] (SPARK-24499) Documentation improvement of Spark core and SQL

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24499: Assignee: Apache Spark > Documentation improvement of Spark core and SQL > --

[jira] [Commented] (SPARK-24499) Documentation improvement of Spark core and SQL

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651598#comment-16651598 ] Apache Spark commented on SPARK-24499: -- User 'xuanyuanking' has created a pull requ

[jira] [Assigned] (SPARK-24499) Documentation improvement of Spark core and SQL

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24499: Assignee: (was: Apache Spark) > Documentation improvement of Spark core and SQL > ---

[jira] [Resolved] (SPARK-25579) Use quoted attribute names if needed in pushed ORC predicates

2018-10-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25579. -- Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Issue resolved by pull

[jira] [Created] (SPARK-25745) docker-image-tool.sh ignores errors from Docker

2018-10-16 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-25745: - Summary: docker-image-tool.sh ignores errors from Docker Key: SPARK-25745 URL: https://issues.apache.org/jira/browse/SPARK-25745 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-24902) Add integration tests for PVs

2018-10-16 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651621#comment-16651621 ] Stavros Kontopoulos commented on SPARK-24902: - This is blocked by this issue

[jira] [Comment Edited] (SPARK-24902) Add integration tests for PVs

2018-10-16 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651621#comment-16651621 ] Stavros Kontopoulos edited comment on SPARK-24902 at 10/16/18 12:50 PM: --

[jira] [Comment Edited] (SPARK-24902) Add integration tests for PVs

2018-10-16 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651621#comment-16651621 ] Stavros Kontopoulos edited comment on SPARK-24902 at 10/16/18 12:50 PM: --

[jira] [Commented] (SPARK-24499) Documentation improvement of Spark core and SQL

2018-10-16 Thread Yuanjian Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651628#comment-16651628 ] Yuanjian Li commented on SPARK-24499: - {quote} Is this just about splitting up the d

[jira] [Commented] (SPARK-25745) docker-image-tool.sh ignores errors from Docker

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651630#comment-16651630 ] Apache Spark commented on SPARK-25745: -- User 'rvesse' has created a pull request fo

[jira] [Assigned] (SPARK-25745) docker-image-tool.sh ignores errors from Docker

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25745: Assignee: (was: Apache Spark) > docker-image-tool.sh ignores errors from Docker > ---

[jira] [Assigned] (SPARK-25745) docker-image-tool.sh ignores errors from Docker

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25745: Assignee: Apache Spark > docker-image-tool.sh ignores errors from Docker > --

[jira] [Commented] (SPARK-25745) docker-image-tool.sh ignores errors from Docker

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651631#comment-16651631 ] Apache Spark commented on SPARK-25745: -- User 'rvesse' has created a pull request fo

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651657#comment-16651657 ] Thomas Graves commented on SPARK-25732: --- So like Marcelo mentioned can't you re-us

[jira] [Assigned] (SPARK-25705) Remove Kafka 0.8 support in Spark 3.0.0

2018-10-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25705: - Assignee: Sean Owen > Remove Kafka 0.8 support in Spark 3.0.0 > ---

[jira] [Resolved] (SPARK-25705) Remove Kafka 0.8 support in Spark 3.0.0

2018-10-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25705. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22703 [https://github.c

[jira] [Updated] (SPARK-25739) Double quote coming in as empty value even when emptyValue set as null

2018-10-16 Thread Brian Jones (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Jones updated SPARK-25739: Flags: Important (was: Patch,Important) > Double quote coming in as empty value even when emptyVa

[jira] [Updated] (SPARK-25739) Double quote coming in as empty value even when emptyValue set as null

2018-10-16 Thread Brian Jones (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Jones updated SPARK-25739: Flags: Patch,Important (was: Important) > Double quote coming in as empty value even when emptyVa

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651800#comment-16651800 ] Marco Gaido commented on SPARK-25732: - [~tgraves] I think they can be reused, the po

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651827#comment-16651827 ] Thomas Graves commented on SPARK-25732: --- yeah I understand the concern, we don't w

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651840#comment-16651840 ] Thomas Graves commented on SPARK-25732: --- sorry just realized I misread the second

[jira] [Comment Edited] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651840#comment-16651840 ] Thomas Graves edited comment on SPARK-25732 at 10/16/18 2:49 PM: -

[jira] [Comment Edited] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651840#comment-16651840 ] Thomas Graves edited comment on SPARK-25732 at 10/16/18 2:53 PM: -

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651860#comment-16651860 ] Marco Gaido commented on SPARK-25732: - [~tgraves] yes, exactly it is what I am refer

[jira] [Created] (SPARK-25746) Refactoring ExpressionEncoder

2018-10-16 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-25746: --- Summary: Refactoring ExpressionEncoder Key: SPARK-25746 URL: https://issues.apache.org/jira/browse/SPARK-25746 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-25746) Refactoring ExpressionEncoder

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25746: Assignee: (was: Apache Spark) > Refactoring ExpressionEncoder > -

[jira] [Commented] (SPARK-25746) Refactoring ExpressionEncoder

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651927#comment-16651927 ] Apache Spark commented on SPARK-25746: -- User 'viirya' has created a pull request fo

[jira] [Assigned] (SPARK-25746) Refactoring ExpressionEncoder

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25746: Assignee: Apache Spark > Refactoring ExpressionEncoder > - >

[jira] [Created] (SPARK-25747) remove ColumnarBatchScan.needsUnsafeRowConversion

2018-10-16 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25747: --- Summary: remove ColumnarBatchScan.needsUnsafeRowConversion Key: SPARK-25747 URL: https://issues.apache.org/jira/browse/SPARK-25747 Project: Spark Issue Type: I

[jira] [Assigned] (SPARK-25747) remove ColumnarBatchScan.needsUnsafeRowConversion

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25747: Assignee: Apache Spark (was: Wenchen Fan) > remove ColumnarBatchScan.needsUnsafeRowConve

[jira] [Commented] (SPARK-25747) remove ColumnarBatchScan.needsUnsafeRowConversion

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652007#comment-16652007 ] Apache Spark commented on SPARK-25747: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-25747) remove ColumnarBatchScan.needsUnsafeRowConversion

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25747: Assignee: Wenchen Fan (was: Apache Spark) > remove ColumnarBatchScan.needsUnsafeRowConve

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652029#comment-16652029 ] Marcelo Vanzin commented on SPARK-25732: bq. livy server would not know which da

[jira] [Commented] (SPARK-25643) Performance issues querying wide rows

2018-10-16 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652039#comment-16652039 ] Ruslan Dautkhanov commented on SPARK-25643: --- [~viirya] we confirm this problem

[jira] [Commented] (SPARK-25742) Is there a way to pass the Azure blob storage credentials to the spark for k8s init-container?

2018-10-16 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652064#comment-16652064 ] Yinan Li commented on SPARK-25742: -- The k8s secrets you add through the {{spark.kuberne

[jira] [Created] (SPARK-25748) Upgrade net.razorvine:pyrolite version

2018-10-16 Thread Ankur Gupta (JIRA)
Ankur Gupta created SPARK-25748: --- Summary: Upgrade net.razorvine:pyrolite version Key: SPARK-25748 URL: https://issues.apache.org/jira/browse/SPARK-25748 Project: Spark Issue Type: Bug

[jira] [Reopened] (SPARK-17875) Remove unneeded direct dependence on Netty 3.x

2018-10-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-17875: --- > Remove unneeded direct dependence on Netty 3.x > -- > >

[jira] [Commented] (SPARK-25748) Upgrade net.razorvine:pyrolite version

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652109#comment-16652109 ] Marcelo Vanzin commented on SPARK-25748: That CVE seems completely unrelated to

[jira] [Created] (SPARK-25749) Exception thrown while reading avro file with large schema

2018-10-16 Thread Raj (JIRA)
Raj created SPARK-25749: --- Summary: Exception thrown while reading avro file with large schema Key: SPARK-25749 URL: https://issues.apache.org/jira/browse/SPARK-25749 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-25749) Exception thrown while reading avro file with large schema

2018-10-16 Thread Raj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raj updated SPARK-25749: Attachment: MainCC.scala EncoderExample.scala > Exception thrown while reading avro file with larg

[jira] [Updated] (SPARK-25749) Exception thrown while reading avro file with large schema

2018-10-16 Thread Raj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raj updated SPARK-25749: Attachment: build.sbt > Exception thrown while reading avro file with large schema > -

[jira] [Updated] (SPARK-25749) Exception thrown while reading avro file with large schema

2018-10-16 Thread Raj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raj updated SPARK-25749: Attachment: exception > Exception thrown while reading avro file with large schema > -

[jira] [Updated] (SPARK-25749) Exception thrown while reading avro file with large schema

2018-10-16 Thread Raj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raj updated SPARK-25749: Description: Hi, We are migrating our jobs from Spark 2.2.0 to Spark 2.3.1. One of the job reads avro source that

[jira] [Resolved] (SPARK-25748) Upgrade net.razorvine:pyrolite version

2018-10-16 Thread Ankur Gupta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Gupta resolved SPARK-25748. - Resolution: Invalid Yeah, it seems this jar was tagged incorrectly in the internal security scan

[jira] [Commented] (SPARK-20327) Add CLI support for YARN custom resources, like GPUs

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652393#comment-16652393 ] Apache Spark commented on SPARK-20327: -- User 'vanzin' has created a pull request fo

[jira] [Commented] (SPARK-20327) Add CLI support for YARN custom resources, like GPUs

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652395#comment-16652395 ] Apache Spark commented on SPARK-20327: -- User 'vanzin' has created a pull request fo

[jira] [Created] (SPARK-25750) Secure HDFS Integration Testing

2018-10-16 Thread Ilan Filonenko (JIRA)
Ilan Filonenko created SPARK-25750: -- Summary: Secure HDFS Integration Testing Key: SPARK-25750 URL: https://issues.apache.org/jira/browse/SPARK-25750 Project: Spark Issue Type: Test

[jira] [Assigned] (SPARK-25750) Secure HDFS Integration Testing

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25750: Assignee: Apache Spark > Secure HDFS Integration Testing > --

[jira] [Commented] (SPARK-25750) Secure HDFS Integration Testing

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652400#comment-16652400 ] Apache Spark commented on SPARK-25750: -- User 'ifilonenko' has created a pull reques

[jira] [Assigned] (SPARK-25750) Secure HDFS Integration Testing

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25750: Assignee: (was: Apache Spark) > Secure HDFS Integration Testing > ---

[jira] [Commented] (SPARK-20202) Remove references to org.spark-project.hive

2018-10-16 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652440#comment-16652440 ] t oo commented on SPARK-20202: -- bump > Remove references to org.spark-project.hive > -

[jira] [Created] (SPARK-25751) Unit Testing for Kerberos Support for Spark on Kubernetes

2018-10-16 Thread Ilan Filonenko (JIRA)
Ilan Filonenko created SPARK-25751: -- Summary: Unit Testing for Kerberos Support for Spark on Kubernetes Key: SPARK-25751 URL: https://issues.apache.org/jira/browse/SPARK-25751 Project: Spark

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-10-16 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652449#comment-16652449 ] t oo commented on SPARK-18673: -- bump > Dataframes doesn't work on Hadoop 3.x; Hive rejects

[jira] [Updated] (SPARK-25750) Integration Testing for Kerberos Support for Spark on Kubernetes

2018-10-16 Thread Ilan Filonenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilan Filonenko updated SPARK-25750: --- Summary: Integration Testing for Kerberos Support for Spark on Kubernetes (was: Secure HDFS

[jira] [Created] (SPARK-25752) Add trait to easily whitelist logical operators that produce named output from CleanupAliases

2018-10-16 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-25752: - Summary: Add trait to easily whitelist logical operators that produce named output from CleanupAliases Key: SPARK-25752 URL: https://issues.apache.org/jira/browse/SPARK-25752

[jira] [Assigned] (SPARK-25394) Expose App status metrics as Source

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25394: -- Assignee: Stavros Kontopoulos > Expose App status metrics as Source > ---

[jira] [Resolved] (SPARK-25394) Expose App status metrics as Source

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25394. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22381 [https:

[jira] [Resolved] (SPARK-25631) KafkaRDDSuite: basic usage 2 min 4 sec

2018-10-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25631. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22670 [https://github.c

[jira] [Assigned] (SPARK-25631) KafkaRDDSuite: basic usage 2 min 4 sec

2018-10-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25631: - Assignee: Dilip Biswal > KafkaRDDSuite: basic usage2 min 4 sec > --

[jira] [Resolved] (SPARK-25632) KafkaRDDSuite: compacted topic 2 min 5 sec.

2018-10-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25632. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22670 [https://github.c

[jira] [Assigned] (SPARK-25632) KafkaRDDSuite: compacted topic 2 min 5 sec.

2018-10-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25632: - Assignee: Dilip Biswal > KafkaRDDSuite: compacted topic 2 min 5 sec. >

[jira] [Resolved] (SPARK-25693) Fix the multiple "manager" classes in org.apache.spark.deploy.security

2018-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25693. Resolution: Duplicate I started looking at this while waiting for reviews of SPARK-23781,

[jira] [Commented] (SPARK-24787) Events being dropped at an alarming rate due to hsync being slow for eventLogging

2018-10-16 Thread Devaraj K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652687#comment-16652687 ] Devaraj K commented on SPARK-24787: --- It seems here the overhead is coming due the forc

[jira] [Assigned] (SPARK-24787) Events being dropped at an alarming rate due to hsync being slow for eventLogging

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24787: Assignee: Apache Spark > Events being dropped at an alarming rate due to hsync being slow

[jira] [Assigned] (SPARK-24787) Events being dropped at an alarming rate due to hsync being slow for eventLogging

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24787: Assignee: (was: Apache Spark) > Events being dropped at an alarming rate due to hsync

[jira] [Commented] (SPARK-24787) Events being dropped at an alarming rate due to hsync being slow for eventLogging

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652691#comment-16652691 ] Apache Spark commented on SPARK-24787: -- User 'devaraj-kavali' has created a pull re

[jira] [Commented] (SPARK-24787) Events being dropped at an alarming rate due to hsync being slow for eventLogging

2018-10-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652690#comment-16652690 ] Apache Spark commented on SPARK-24787: -- User 'devaraj-kavali' has created a pull re

[jira] [Commented] (SPARK-25733) The method toLocalIterator() with dataframe doesn't work

2018-10-16 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652746#comment-16652746 ] Bryan Cutler commented on SPARK-25733: -- Is this a duplicate of SPARK-23961? > The

[jira] [Created] (SPARK-25753) binaryFiles broken for small files

2018-10-16 Thread liuxian (JIRA)
liuxian created SPARK-25753: --- Summary: binaryFiles broken for small files Key: SPARK-25753 URL: https://issues.apache.org/jira/browse/SPARK-25753 Project: Spark Issue Type: Bug Components

  1   2   >