[jira] [Commented] (SPARK-22565) Session-based windowing

2018-09-27 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631370#comment-16631370 ] Li Yuanjian commented on SPARK-22565: - [~kabhwan] Great thanks for noticing me, sorry for only

[jira] [Comment Edited] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

2018-09-27 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631295#comment-16631295 ] Jungtaek Lim edited comment on SPARK-25380 at 9/28/18 3:41 AM: --- IMHO it

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

2018-09-27 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631313#comment-16631313 ] Jungtaek Lim commented on SPARK-25380: -- Btw, reproducer still helps when we tackle it with only UI

[jira] [Updated] (SPARK-25563) Spark application hangs If container allocate on lost Nodemanager

2018-09-27 Thread devinduan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] devinduan updated SPARK-25563: -- Summary: Spark application hangs If container allocate on lost Nodemanager (was: Spark application

[jira] [Created] (SPARK-25563) Spark application hangs

2018-09-27 Thread devinduan (JIRA)
devinduan created SPARK-25563: - Summary: Spark application hangs Key: SPARK-25563 URL: https://issues.apache.org/jira/browse/SPARK-25563 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-22565) Session-based windowing

2018-09-27 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631301#comment-16631301 ] Jungtaek Lim commented on SPARK-22565: -- [~XuanYuan] Hello, I've been initiating supporting this

[jira] [Comment Edited] (SPARK-22565) Session-based windowing

2018-09-27 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631298#comment-16631298 ] Li Yuanjian edited comment on SPARK-22565 at 9/28/18 3:23 AM: -- Thanks for

[jira] [Commented] (SPARK-22565) Session-based windowing

2018-09-27 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631300#comment-16631300 ] Li Yuanjian commented on SPARK-22565: - Also cc [~zsxwing] [~tdas], we are translating the design doc

[jira] [Commented] (SPARK-22565) Session-based windowing

2018-09-27 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631298#comment-16631298 ] Li Yuanjian commented on SPARK-22565: - Thanks for reporting this. Actually we also met this problem

[jira] [Updated] (SPARK-22565) Session-based windowing

2018-09-27 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Yuanjian updated SPARK-22565: Attachment: screenshot-1.png > Session-based windowing > --- > >

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

2018-09-27 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631295#comment-16631295 ] Jungtaek Lim commented on SPARK-25380: -- IMHO it depends on how we see the issue and how we would

[jira] [Created] (SPARK-25562) The Spark add audit log

2018-09-27 Thread yinghua_zh (JIRA)
yinghua_zh created SPARK-25562: -- Summary: The Spark add audit log Key: SPARK-25562 URL: https://issues.apache.org/jira/browse/SPARK-25562 Project: Spark Issue Type: New Feature

[jira] [Assigned] (SPARK-25560) Allow Function Injection in SparkSessionExtensions

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25560: Assignee: Apache Spark > Allow Function Injection in SparkSessionExtensions >

[jira] [Assigned] (SPARK-25560) Allow Function Injection in SparkSessionExtensions

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25560: Assignee: (was: Apache Spark) > Allow Function Injection in SparkSessionExtensions >

[jira] [Commented] (SPARK-25560) Allow Function Injection in SparkSessionExtensions

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631257#comment-16631257 ] Apache Spark commented on SPARK-25560: -- User 'RussellSpitzer' has created a pull request for this

[jira] [Assigned] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24630: Assignee: Apache Spark > SPIP: Support SQLStreaming in Spark >

[jira] [Assigned] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24630: Assignee: (was: Apache Spark) > SPIP: Support SQLStreaming in Spark >

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631253#comment-16631253 ] Apache Spark commented on SPARK-24630: -- User 'stczwd' has created a pull request for this issue:

[jira] [Commented] (SPARK-25380) Generated plans occupy over 50% of Spark driver memory

2018-09-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631236#comment-16631236 ] Marcelo Vanzin commented on SPARK-25380: If all you want is see this live, why do you need a

[jira] [Commented] (SPARK-25459) Add viewOriginalText back to CatalogTable

2018-09-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631232#comment-16631232 ] Dongjoon Hyun commented on SPARK-25459: --- [~chrisz28]. It seems that JIRA is unable to accept your

[jira] [Updated] (SPARK-25459) Add viewOriginalText back to CatalogTable

2018-09-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25459: -- Attachment: error_message.png > Add viewOriginalText back to CatalogTable >

[jira] [Commented] (SPARK-10816) EventTime based sessionization

2018-09-27 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631206#comment-16631206 ] Jungtaek Lim commented on SPARK-10816: -- To avoid any concerns/wonders, I believe my proposal and

[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql

2018-09-27 Thread Karthik Manamcheri (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631193#comment-16631193 ] Karthik Manamcheri commented on SPARK-25561: The root cause was from SPARK-17992 ping

[jira] [Created] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql

2018-09-27 Thread Karthik Manamcheri (JIRA)
Karthik Manamcheri created SPARK-25561: -- Summary: HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql Key: SPARK-25561 URL: https://issues.apache.org/jira/browse/SPARK-25561

[jira] [Comment Edited] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-09-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631186#comment-16631186 ] Hyukjin Kwon edited comment on SPARK-18112 at 9/27/18 11:29 PM: To avoid

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-09-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631186#comment-16631186 ] Hyukjin Kwon commented on SPARK-18112: -- To avoid Hive fork 1.2.1 in Spark itself at all, please

[jira] [Commented] (SPARK-25559) Just remove the unsupported predicates in Parquet

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631184#comment-16631184 ] Apache Spark commented on SPARK-25559: -- User 'dbtsai' has created a pull request for this issue:

[jira] [Commented] (SPARK-25559) Just remove the unsupported predicates in Parquet

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631183#comment-16631183 ] Apache Spark commented on SPARK-25559: -- User 'dbtsai' has created a pull request for this issue:

[jira] [Commented] (SPARK-25559) Just remove the unsupported predicates in Parquet

2018-09-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631182#comment-16631182 ] DB Tsai commented on SPARK-25559: - https://github.com/apache/spark/pull/22574 > Just remove the

[jira] [Issue Comment Deleted] (SPARK-25556) Predicate Pushdown for Nested fields

2018-09-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-25556: Comment: was deleted (was: User 'dbtsai' has created a pull request for this issue:

[jira] [Issue Comment Deleted] (SPARK-25556) Predicate Pushdown for Nested fields

2018-09-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-25556: Comment: was deleted (was: User 'dbtsai' has created a pull request for this issue:

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-09-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631159#comment-16631159 ] Hyukjin Kwon commented on SPARK-18112: -- {quote} If I use that 1.2.1 fork I was getting some query

[jira] [Commented] (SPARK-25556) Predicate Pushdown for Nested fields

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1663#comment-1663 ] Apache Spark commented on SPARK-25556: -- User 'dbtsai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25556) Predicate Pushdown for Nested fields

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25556: Assignee: Apache Spark (was: DB Tsai) > Predicate Pushdown for Nested fields >

[jira] [Commented] (SPARK-25556) Predicate Pushdown for Nested fields

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631108#comment-16631108 ] Apache Spark commented on SPARK-25556: -- User 'dbtsai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25556) Predicate Pushdown for Nested fields

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25556: Assignee: DB Tsai (was: Apache Spark) > Predicate Pushdown for Nested fields >

[jira] [Created] (SPARK-25560) Allow Function Injection in SparkSessionExtensions

2018-09-27 Thread Russell Spitzer (JIRA)
Russell Spitzer created SPARK-25560: --- Summary: Allow Function Injection in SparkSessionExtensions Key: SPARK-25560 URL: https://issues.apache.org/jira/browse/SPARK-25560 Project: Spark

[jira] [Updated] (SPARK-25559) Just remove the unsupported predicates in Parquet

2018-09-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-25559: Description: Currently, in *ParquetFilters*, if one of the children predicates is not supported by

[jira] [Comment Edited] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-09-27 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630947#comment-16630947 ] Stavros Kontopoulos edited comment on SPARK-24579 at 9/27/18 8:12 PM:

[jira] [Comment Edited] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-09-27 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630947#comment-16630947 ] Stavros Kontopoulos edited comment on SPARK-24579 at 9/27/18 7:37 PM:

[jira] [Comment Edited] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-09-27 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630947#comment-16630947 ] Stavros Kontopoulos edited comment on SPARK-24579 at 9/27/18 7:36 PM:

[jira] [Comment Edited] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-09-27 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630947#comment-16630947 ] Stavros Kontopoulos edited comment on SPARK-24579 at 9/27/18 7:35 PM:

[jira] [Commented] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-09-27 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630947#comment-16630947 ] Stavros Kontopoulos commented on SPARK-24579: - Nice work. Since I cant comment in the design

[jira] [Comment Edited] (SPARK-24724) Discuss necessary info and access in barrier mode + Kubernetes

2018-09-27 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630912#comment-16630912 ] Stavros Kontopoulos edited comment on SPARK-24724 at 9/27/18 7:06 PM:

[jira] [Commented] (SPARK-24724) Discuss necessary info and access in barrier mode + Kubernetes

2018-09-27 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630912#comment-16630912 ] Stavros Kontopoulos commented on SPARK-24724: - AFAIK in standalone mode passwordless

[jira] [Commented] (SPARK-25334) Default SessionCatalog should support UDFs

2018-09-27 Thread Swapnil Chougule (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630890#comment-16630890 ] Swapnil Chougule commented on SPARK-25334: -- +1 My work has also been impacted by same.

[jira] [Commented] (SPARK-25557) ORC predicate pushdown for nested fields

2018-09-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630879#comment-16630879 ] Dongjoon Hyun commented on SPARK-25557: --- Got it, [~dbtsai] . > ORC predicate pushdown for nested

[jira] [Commented] (SPARK-25558) Pushdown predicates for nested fields in DataSource Strategy

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630871#comment-16630871 ] Apache Spark commented on SPARK-25558: -- User 'dbtsai' has created a pull request for this issue:

[jira] [Commented] (SPARK-25558) Pushdown predicates for nested fields in DataSource Strategy

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630870#comment-16630870 ] Apache Spark commented on SPARK-25558: -- User 'dbtsai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25558) Pushdown predicates for nested fields in DataSource Strategy

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25558: Assignee: DB Tsai (was: Apache Spark) > Pushdown predicates for nested fields in

[jira] [Assigned] (SPARK-25558) Pushdown predicates for nested fields in DataSource Strategy

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25558: Assignee: Apache Spark (was: DB Tsai) > Pushdown predicates for nested fields in

[jira] [Updated] (SPARK-25558) Pushdown predicates for nested fields in DataSource Strategy

2018-09-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-25558: Summary: Pushdown predicates for nested fields in DataSource Strategy (was: Creating predicates for

[jira] [Created] (SPARK-25559) Just remove the unsupported predicates in Parquet

2018-09-27 Thread DB Tsai (JIRA)
DB Tsai created SPARK-25559: --- Summary: Just remove the unsupported predicates in Parquet Key: SPARK-25559 URL: https://issues.apache.org/jira/browse/SPARK-25559 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-25558) Creating predicates for nested fields in DataSource Strategy

2018-09-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-25558: Issue Type: Sub-task (was: New Feature) Parent: SPARK-25556 > Creating predicates for nested

[jira] [Assigned] (SPARK-25558) Creating predicates for nested fields in DataSource Strategy

2018-09-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-25558: --- Assignee: DB Tsai > Creating predicates for nested fields in DataSource Strategy >

[jira] [Created] (SPARK-25558) Creating predicates for nested fields in DataSource Strategy

2018-09-27 Thread DB Tsai (JIRA)
DB Tsai created SPARK-25558: --- Summary: Creating predicates for nested fields in DataSource Strategy Key: SPARK-25558 URL: https://issues.apache.org/jira/browse/SPARK-25558 Project: Spark Issue

[jira] [Created] (SPARK-25557) ORC predicate pushdown for nested fields

2018-09-27 Thread DB Tsai (JIRA)
DB Tsai created SPARK-25557: --- Summary: ORC predicate pushdown for nested fields Key: SPARK-25557 URL: https://issues.apache.org/jira/browse/SPARK-25557 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-25556) Predicate Pushdown for Nested fields

2018-09-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-25556: --- Assignee: DB Tsai > Predicate Pushdown for Nested fields > >

[jira] [Updated] (SPARK-17636) Parquet predicate pushdown for nested fields

2018-09-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-17636: Summary: Parquet predicate pushdown for nested fields (was: Parquet filter push down doesn't handle

[jira] [Created] (SPARK-25556) Predicate Pushdown for Nested fields

2018-09-27 Thread DB Tsai (JIRA)
DB Tsai created SPARK-25556: --- Summary: Predicate Pushdown for Nested fields Key: SPARK-25556 URL: https://issues.apache.org/jira/browse/SPARK-25556 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-25461) PySpark Pandas UDF outputs incorrect results when input columns contain None

2018-09-27 Thread Christopher Said (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630830#comment-16630830 ] Christopher Said commented on SPARK-25461: -- For what it's worth, converting all floats to False

[jira] [Updated] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2018-09-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-17636: Issue Type: Sub-task (was: Bug) Parent: SPARK-25556 > Parquet filter push down doesn't handle

[jira] [Commented] (SPARK-25521) Job id showing null when insert into command Job is finished.

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630827#comment-16630827 ] Apache Spark commented on SPARK-25521: -- User 'sujith71955' has created a pull request for this

[jira] [Assigned] (SPARK-25521) Job id showing null when insert into command Job is finished.

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25521: Assignee: Apache Spark > Job id showing null when insert into command Job is finished. >

[jira] [Assigned] (SPARK-25521) Job id showing null when insert into command Job is finished.

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25521: Assignee: (was: Apache Spark) > Job id showing null when insert into command Job is

[jira] [Resolved] (SPARK-25533) Inconsistent message for Completed Jobs in the JobUI, when there are failed jobs, compared to spark2.2

2018-09-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25533. Resolution: Fixed Fix Version/s: 2.4.0 2.3.3 > Inconsistent

[jira] [Commented] (SPARK-25392) [Spark Job History]Inconsistent behaviour for pool details in spark web UI and history server page

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630781#comment-16630781 ] Apache Spark commented on SPARK-25392: -- User 'sandeep-katta' has created a pull request for this

[jira] [Assigned] (SPARK-25392) [Spark Job History]Inconsistent behaviour for pool details in spark web UI and history server page

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25392: Assignee: Apache Spark > [Spark Job History]Inconsistent behaviour for pool details in

[jira] [Commented] (SPARK-25392) [Spark Job History]Inconsistent behaviour for pool details in spark web UI and history server page

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630779#comment-16630779 ] Apache Spark commented on SPARK-25392: -- User 'sandeep-katta' has created a pull request for this

[jira] [Assigned] (SPARK-25392) [Spark Job History]Inconsistent behaviour for pool details in spark web UI and history server page

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25392: Assignee: (was: Apache Spark) > [Spark Job History]Inconsistent behaviour for pool

[jira] [Assigned] (SPARK-25454) Division between operands with negative scale can cause precision loss

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25454: Assignee: Apache Spark > Division between operands with negative scale can cause

[jira] [Assigned] (SPARK-25454) Division between operands with negative scale can cause precision loss

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25454: Assignee: (was: Apache Spark) > Division between operands with negative scale can

[jira] [Commented] (SPARK-10816) EventTime based sessionization

2018-09-27 Thread Arun Mahadevan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630697#comment-16630697 ] Arun Mahadevan commented on SPARK-10816: Reiterating earlier point. "map/flapMapGroupsWithState"

[jira] [Assigned] (SPARK-25546) RDDInfo uses SparkEnv before it may have been initialized

2018-09-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25546: - Assignee: Marcelo Vanzin > RDDInfo uses SparkEnv before it may have been initialized >

[jira] [Resolved] (SPARK-25546) RDDInfo uses SparkEnv before it may have been initialized

2018-09-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25546. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22558

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-09-27 Thread Eugeniu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630689#comment-16630689 ] Eugeniu commented on SPARK-18112: - _"The problem here looks, you guys completely replaced the jars into

[jira] [Created] (SPARK-25555) Generic constructs for windowing and support for custom windows

2018-09-27 Thread Arun Mahadevan (JIRA)
Arun Mahadevan created SPARK-2: -- Summary: Generic constructs for windowing and support for custom windows Key: SPARK-2 URL: https://issues.apache.org/jira/browse/SPARK-2 Project: Spark

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-09-27 Thread Mingjie Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630672#comment-16630672 ] Mingjie Tang commented on SPARK-25501: -- [~gsomogyi] thanks so much for your hard work, let me look

[jira] [Updated] (SPARK-25415) Make plan change log in RuleExecutor configurable by SQLConf

2018-09-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25415: - Fix Version/s: (was: 3.0.0) 2.5.0 > Make plan change log in RuleExecutor

[jira] [Created] (SPARK-25554) Avro logical types get ignored in SchemaConverters.toSqlType

2018-09-27 Thread Yanan Li (JIRA)
Yanan Li created SPARK-25554: Summary: Avro logical types get ignored in SchemaConverters.toSqlType Key: SPARK-25554 URL: https://issues.apache.org/jira/browse/SPARK-25554 Project: Spark Issue

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-09-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630543#comment-16630543 ] Hyukjin Kwon commented on SPARK-18112: -- Also, take a look for the codes, open JIRAs and PRs before

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-09-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630542#comment-16630542 ] Hyukjin Kwon commented on SPARK-18112: -- To be more specific, the code you guys pointed out is

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-09-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630485#comment-16630485 ] Hyukjin Kwon commented on SPARK-18112: -- At that code you pointed out, Spark's Hive fork should be

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-09-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630487#comment-16630487 ] Hyukjin Kwon commented on SPARK-18112: -- I asked questions because i'm pretty sure it's

[jira] [Updated] (SPARK-25553) Add EmptyInterpolatedStringChecker to scalastyle-config.xml

2018-09-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25553: -- Priority: Minor (was: Major) > Add EmptyInterpolatedStringChecker to scalastyle-config.xml >

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-09-27 Thread Tavis Barr (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630472#comment-16630472 ] Tavis Barr commented on SPARK-18112: Why are you even asking these questions?  I have already

[jira] [Commented] (SPARK-25523) Multi thread execute sparkSession.read().jdbc(url, table, properties) problem

2018-09-27 Thread huanghuai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630421#comment-16630421 ] huanghuai commented on SPARK-25523: --- Sorry, I know this question is hard to understand, but my project

[jira] [Commented] (SPARK-21291) R bucketBy partitionBy API

2018-09-27 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630410#comment-16630410 ] Felix Cheung commented on SPARK-21291: -- Wait. I don’t think saveAsTable is the same thing? > R

[jira] [Assigned] (SPARK-25553) Add EmptyInterpolatedStringChecker to scalastyle-config.xml

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25553: Assignee: (was: Apache Spark) > Add EmptyInterpolatedStringChecker to

[jira] [Commented] (SPARK-25553) Add EmptyInterpolatedStringChecker to scalastyle-config.xml

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630401#comment-16630401 ] Apache Spark commented on SPARK-25553: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25553) Add EmptyInterpolatedStringChecker to scalastyle-config.xml

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25553: Assignee: Apache Spark > Add EmptyInterpolatedStringChecker to scalastyle-config.xml >

[jira] [Commented] (SPARK-25553) Add EmptyInterpolatedStringChecker to scalastyle-config.xml

2018-09-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630400#comment-16630400 ] Apache Spark commented on SPARK-25553: -- User 'wangyum' has created a pull request for this issue:

[jira] [Resolved] (SPARK-21743) top-most limit should not cause memory leak

2018-09-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21743. - Resolution: Fixed Fix Version/s: 2.4.0 > top-most limit should not cause memory leak >

[jira] [Resolved] (SPARK-21436) Take advantage of known partioner for distinct on RDDs

2018-09-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21436. - Resolution: Fixed Fix Version/s: 2.5.0 Issue resolved by pull request 22010

[jira] [Assigned] (SPARK-21436) Take advantage of known partioner for distinct on RDDs

2018-09-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21436: --- Assignee: holdenk > Take advantage of known partioner for distinct on RDDs >

[jira] [Created] (SPARK-25553) Add EmptyInterpolatedStringChecker to scalastyle-config.xml

2018-09-27 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-25553: --- Summary: Add EmptyInterpolatedStringChecker to scalastyle-config.xml Key: SPARK-25553 URL: https://issues.apache.org/jira/browse/SPARK-25553 Project: Spark

[jira] [Updated] (SPARK-25522) Improve type promotion for input arguments of elementAt function

2018-09-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-25522: Fix Version/s: (was: 2.5.0) 2.4.0 > Improve type promotion for input

[jira] [Resolved] (SPARK-25551) Remove unused InSubquery expression

2018-09-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25551. - Resolution: Fixed Fix Version/s: 2.5.0 Issue resolved by pull request 22556

[jira] [Assigned] (SPARK-25551) Remove unused InSubquery expression

2018-09-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25551: --- Assignee: Marco Gaido > Remove unused InSubquery expression >

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2018-09-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630236#comment-16630236 ] Hyukjin Kwon commented on SPARK-18112: -- Such things should be asked to mailing list really. It

  1   2   >