[jira] [Commented] (SPARK-23589) Add interpreted execution for ExternalMapToCatalyst expression

2018-03-08 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392536#comment-16392536 ] Takeshi Yamamuro commented on SPARK-23589: -- I'll make a pr after I finish other sub-tickets:

[jira] [Updated] (SPARK-22246) UnsafeRow, UnsafeArrayData, and UnsafeMapData use MemoryBlock

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22246: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-10399 > UnsafeRow,

[jira] [Commented] (SPARK-23632) sparkR.session() error with spark packages - JVM is not ready after 10 seconds

2018-03-08 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392507#comment-16392507 ] Felix Cheung commented on SPARK-23632: -- To clarify, are you running into problem because the package

[jira] [Created] (SPARK-23638) Spark on k8s: spark.kubernetes.initContainer.image has no effect

2018-03-08 Thread maheshvra (JIRA)
maheshvra created SPARK-23638: - Summary: Spark on k8s: spark.kubernetes.initContainer.image has no effect Key: SPARK-23638 URL: https://issues.apache.org/jira/browse/SPARK-23638 Project: Spark

[jira] [Commented] (SPARK-23627) Provide isEmpty() function in DataSet

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392468#comment-16392468 ] Apache Spark commented on SPARK-23627: -- User 'goungoun' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23627) Provide isEmpty() function in DataSet

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23627: Assignee: (was: Apache Spark) > Provide isEmpty() function in DataSet >

[jira] [Assigned] (SPARK-23627) Provide isEmpty() function in DataSet

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23627: Assignee: Apache Spark > Provide isEmpty() function in DataSet >

[jira] [Assigned] (SPARK-23637) Yarn might allocate more resource if a same executor is killed multiple times.

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23637: Assignee: Apache Spark > Yarn might allocate more resource if a same executor is killed

[jira] [Assigned] (SPARK-23637) Yarn might allocate more resource if a same executor is killed multiple times.

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23637: Assignee: (was: Apache Spark) > Yarn might allocate more resource if a same executor

[jira] [Commented] (SPARK-23637) Yarn might allocate more resource if a same executor is killed multiple times.

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392458#comment-16392458 ] Apache Spark commented on SPARK-23637: -- User 'jinxing64' has created a pull request for this issue:

[jira] [Commented] (SPARK-23637) Yarn might allocate more resource if a same executor is killed multiple times.

2018-03-08 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392457#comment-16392457 ] jin xing commented on SPARK-23637: -- PR here: https://github.com/apache/spark/pull/20781 > Yarn might

[jira] [Updated] (SPARK-23637) Yarn might allocate more resource if a same executor is killed multiple times.

2018-03-08 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-23637: - Description: *{{YarnAllocator}}* uses *{{numExecutorsRunning}}* to track the number of running

[jira] [Updated] (SPARK-23637) Yarn might allocate more resource if a same executor is killed multiple times.

2018-03-08 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-23637: - Description: YarnAllocator}} uses {{numExecutorsRunning to track the number of running

[jira] [Updated] (SPARK-23637) Yarn might allocate more resource if a same executor is killed multiple times.

2018-03-08 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-23637: - Description: {{YarnAllocator }}uses {{numExecutorsRunning}} to track the number of running executor.

[jira] [Created] (SPARK-23637) Yarn might allocate more resource if a same executor is killed multiple times.

2018-03-08 Thread jin xing (JIRA)
jin xing created SPARK-23637: Summary: Yarn might allocate more resource if a same executor is killed multiple times. Key: SPARK-23637 URL: https://issues.apache.org/jira/browse/SPARK-23637 Project:

[jira] [Updated] (SPARK-23637) Yarn might allocate more resource if a same executor is killed multiple times.

2018-03-08 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-23637: - Description: {{YarnAllocator}} uses {{numExecutorsRunning}} to track the number of running executor.

[jira] [Updated] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak updated SPARK-23636: --- Description: h2.   h2. Summary   While using the KafkaUtils.createRDD API - we receive below listed error, 

[jira] [Updated] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak updated SPARK-23636: --- Description: h2.   h2. Summary   While using the KafkaUtils.createRDD API - we receive below listed error, 

[jira] [Updated] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak updated SPARK-23636: --- Description: h2.   h2. Summary   While using the KafkaUtils.createRDD API - we receive below listed error, 

[jira] [Updated] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak updated SPARK-23636: --- Description: h2.   h2. Summary   While using the KafkaUtils.createRDD API - we receive below listed error, 

[jira] [Updated] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak updated SPARK-23636: --- Description: h2.   h2. Summary   While using the KafkaUtils.createRDD API - we receive below listed error, 

[jira] [Updated] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak updated SPARK-23636: --- Description: While using the KafkaUtils.createRDD API - we receive below listed error, especially when 1

[jira] [Updated] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak updated SPARK-23636: --- Description: While using the KafkaUtils.createRDD API - we receive below listed error, especially when 1

[jira] [Assigned] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23598: Assignee: (was: Apache Spark) > WholeStageCodegen can lead to IllegalAccessError

[jira] [Assigned] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23598: Assignee: Apache Spark > WholeStageCodegen can lead to IllegalAccessError calling append

[jira] [Commented] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392360#comment-16392360 ] Apache Spark commented on SPARK-23598: -- User 'kiszk' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391834#comment-16391834 ] Kazuaki Ishizaki edited comment on SPARK-23598 at 3/9/18 3:20 AM: --

[jira] [Updated] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak updated SPARK-23636: --- Description: While using the KafkaUtils.createRDD API - we receive below listed error, especially when 1

[jira] [Updated] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak updated SPARK-23636: --- Description: While using the KafkaUtils.createRDD API - we receive below listed error, especially when 1

[jira] [Updated] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak updated SPARK-23636: --- Description: While using the KafkaUtils.createRDD API - we receive below listed error, especially when 1

[jira] [Created] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-03-08 Thread Deepak (JIRA)
Deepak created SPARK-23636: -- Summary: [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access Key: SPARK-23636

[jira] [Commented] (SPARK-23600) conda_panda_example test fails to import panda lib with Spark 2.3

2018-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392301#comment-16392301 ] Hyukjin Kwon commented on SPARK-23600: -- ping [~ssha...@hortonworks.com] > conda_panda_example test

[jira] [Commented] (SPARK-23613) Different Analyzed logical plan data types for the same table in different queries

2018-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392298#comment-16392298 ] Hyukjin Kwon commented on SPARK-23613: -- Let's avoid to set a blocker which is usually reserved for

[jira] [Updated] (SPARK-23613) Different Analyzed logical plan data types for the same table in different queries

2018-03-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23613: - Priority: Major (was: Blocker) > Different Analyzed logical plan data types for the same table

[jira] [Created] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

2018-03-08 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-23635: --- Summary: Spark executor env variable is overwritten by same name AM env variable Key: SPARK-23635 URL: https://issues.apache.org/jira/browse/SPARK-23635 Project: Spark

[jira] [Updated] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

2018-03-08 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23635: Description: In the current Spark on YARN code, AM always will copy and overwrite its env

[jira] [Assigned] (SPARK-10884) Support prediction on single instance for regression and classification related models

2018-03-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-10884: - Assignee: Weichen Xu (was: Yanbo Liang) > Support prediction on single

[jira] [Commented] (SPARK-21568) ConsoleProgressBar should only be enabled in shells

2018-03-08 Thread Matthias Boehm (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392196#comment-16392196 ] Matthias Boehm commented on SPARK-21568: I just upgraded to Spark 2.3 and was about to file a bug

[jira] [Commented] (SPARK-23584) Add interpreted execution to NewInstance expression

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392155#comment-16392155 ] Apache Spark commented on SPARK-23584: -- User 'maropu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23584) Add interpreted execution to NewInstance expression

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23584: Assignee: Apache Spark > Add interpreted execution to NewInstance expression >

[jira] [Assigned] (SPARK-23584) Add interpreted execution to NewInstance expression

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23584: Assignee: (was: Apache Spark) > Add interpreted execution to NewInstance expression >

[jira] [Commented] (SPARK-23162) PySpark ML LinearRegressionSummary missing r2adj

2018-03-08 Thread kevin yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392097#comment-16392097 ] kevin yu commented on SPARK-23162: -- Currently testing the code.. will open an pr soon. Kevin > PySpark

[jira] [Created] (SPARK-23634) AttributeReferences may be too conservative wrt nullability after optimization

2018-03-08 Thread Henry Robinson (JIRA)
Henry Robinson created SPARK-23634: -- Summary: AttributeReferences may be too conservative wrt nullability after optimization Key: SPARK-23634 URL: https://issues.apache.org/jira/browse/SPARK-23634

[jira] [Resolved] (SPARK-23271) Parquet output contains only "_SUCCESS" file after empty DataFrame saving

2018-03-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23271. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20525

[jira] [Assigned] (SPARK-23271) Parquet output contains only "_SUCCESS" file after empty DataFrame saving

2018-03-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23271: --- Assignee: Dilip Biswal > Parquet output contains only "_SUCCESS" file after empty DataFrame

[jira] [Commented] (SPARK-21030) extend hint syntax to support any expression for Python and R

2018-03-08 Thread Dylan Guedes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392049#comment-16392049 ] Dylan Guedes commented on SPARK-21030: -- So, I started, and here is my progress: 

[jira] [Commented] (SPARK-23325) DataSourceV2 readers should always produce InternalRow.

2018-03-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392048#comment-16392048 ] Wenchen Fan commented on SPARK-23325: - It's hard to stabilize the binary format like `UnsafeRow` and

[jira] [Comment Edited] (SPARK-21030) extend hint syntax to support any expression for Python and R

2018-03-08 Thread Dylan Guedes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392049#comment-16392049 ] Dylan Guedes edited comment on SPARK-21030 at 3/8/18 10:50 PM: --- So, I

[jira] [Commented] (SPARK-23325) DataSourceV2 readers should always produce InternalRow.

2018-03-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392046#comment-16392046 ] Wenchen Fan commented on SPARK-23325: - I think it's mostly document work. We need to add document for

[jira] [Assigned] (SPARK-23615) Add maxDF Parameter to Python CountVectorizer

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23615: Assignee: Apache Spark > Add maxDF Parameter to Python CountVectorizer >

[jira] [Assigned] (SPARK-23615) Add maxDF Parameter to Python CountVectorizer

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23615: Assignee: (was: Apache Spark) > Add maxDF Parameter to Python CountVectorizer >

[jira] [Commented] (SPARK-23615) Add maxDF Parameter to Python CountVectorizer

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392029#comment-16392029 ] Apache Spark commented on SPARK-23615: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Created] (SPARK-23633) Update Pandas UDFs section in sql-programming-guide

2018-03-08 Thread Li Jin (JIRA)
Li Jin created SPARK-23633: -- Summary: Update Pandas UDFs section in sql-programming-guide Key: SPARK-23633 URL: https://issues.apache.org/jira/browse/SPARK-23633 Project: Spark Issue Type:

[jira] [Created] (SPARK-23632) sparkR.session() error with spark packages - JVM is not ready after 10 seconds

2018-03-08 Thread Jaehyeon Kim (JIRA)
Jaehyeon Kim created SPARK-23632: Summary: sparkR.session() error with spark packages - JVM is not ready after 10 seconds Key: SPARK-23632 URL: https://issues.apache.org/jira/browse/SPARK-23632

[jira] [Assigned] (SPARK-23630) Spark-on-YARN missing user customizations of hadoop config

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23630: Assignee: (was: Apache Spark) > Spark-on-YARN missing user customizations of hadoop

[jira] [Commented] (SPARK-23630) Spark-on-YARN missing user customizations of hadoop config

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391938#comment-16391938 ] Apache Spark commented on SPARK-23630: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23630) Spark-on-YARN missing user customizations of hadoop config

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23630: Assignee: Apache Spark > Spark-on-YARN missing user customizations of hadoop config >

[jira] [Updated] (SPARK-23630) Spark-on-YARN missing user customizations of hadoop config

2018-03-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-23630: --- Description: In my change to fix SPARK-22372, I removed some code that allowed user

[jira] [Resolved] (SPARK-23602) PrintToStderr should behave the same in interpreted mode

2018-03-08 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-23602. --- Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 >

[jira] [Commented] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391877#comment-16391877 ] Dongjoon Hyun commented on SPARK-23549: --- Thank you for reporting and making a patch for this,

[jira] [Created] (SPARK-23631) Add summary to RandomForestClassificationModel

2018-03-08 Thread Evan Zamir (JIRA)
Evan Zamir created SPARK-23631: -- Summary: Add summary to RandomForestClassificationModel Key: SPARK-23631 URL: https://issues.apache.org/jira/browse/SPARK-23631 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23549: -- Affects Version/s: 1.6.3 > Spark SQL unexpected behavior when comparing timestamp to date >

[jira] [Updated] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23549: -- Affects Version/s: 2.0.2 > Spark SQL unexpected behavior when comparing timestamp to date >

[jira] [Updated] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23549: -- Affects Version/s: 2.1.2 > Spark SQL unexpected behavior when comparing timestamp to date >

[jira] [Updated] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23549: -- Affects Version/s: 2.3.0 > Spark SQL unexpected behavior when comparing timestamp to date >

[jira] [Commented] (SPARK-23325) DataSourceV2 readers should always produce InternalRow.

2018-03-08 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391861#comment-16391861 ] Michael Armbrust commented on SPARK-23325: -- It does seem like it would be that hard to stabilize

[jira] [Commented] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391834#comment-16391834 ] Kazuaki Ishizaki commented on SPARK-23598: -- Thanks, I confirmed that I can reproduce this issue

[jira] [Commented] (SPARK-23325) DataSourceV2 readers should always produce InternalRow.

2018-03-08 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391768#comment-16391768 ] Ryan Blue commented on SPARK-23325: --- By exposing an interface that uses UnsafeRow, don't we already

[jira] [Created] (SPARK-23630) Spark-on-YARN missing user customizations of hadoop config

2018-03-08 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-23630: -- Summary: Spark-on-YARN missing user customizations of hadoop config Key: SPARK-23630 URL: https://issues.apache.org/jira/browse/SPARK-23630 Project: Spark

[jira] [Commented] (SPARK-14681) Provide label/impurity stats for spark.ml decision tree nodes

2018-03-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391762#comment-16391762 ] Joseph K. Bradley commented on SPARK-14681: --- [~WeichenXu123] Thanks for the PR! I'll comment

[jira] [Comment Edited] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-08 Thread David Vogelbacher (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391709#comment-16391709 ] David Vogelbacher edited comment on SPARK-23598 at 3/8/18 6:41 PM: ---

[jira] [Commented] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-08 Thread David Vogelbacher (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391709#comment-16391709 ] David Vogelbacher commented on SPARK-23598: --- [~mgaido] {{HashAggregateExec}} calls

[jira] [Assigned] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23549: Assignee: (was: Apache Spark) > Spark SQL unexpected behavior when comparing

[jira] [Assigned] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23549: Assignee: Apache Spark > Spark SQL unexpected behavior when comparing timestamp to date >

[jira] [Commented] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391686#comment-16391686 ] Apache Spark commented on SPARK-23549: -- User 'kiszk' has created a pull request for this issue:

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

2018-03-08 Thread Attila Zsolt Piros (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391649#comment-16391649 ] Attila Zsolt Piros commented on SPARK-16630: I am working on this issue. > Blacklist a node

[jira] [Comment Edited] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391386#comment-16391386 ] Kazuaki Ishizaki edited comment on SPARK-23549 at 3/8/18 5:33 PM: -- I

[jira] [Commented] (SPARK-23615) Add maxDF Parameter to Python CountVectorizer

2018-03-08 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391577#comment-16391577 ] Bryan Cutler commented on SPARK-23615: -- Sure, go ahead > Add maxDF Parameter to Python

[jira] [Commented] (SPARK-18165) Kinesis support in Structured Streaming

2018-03-08 Thread Vikram Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391557#comment-16391557 ] Vikram Agrawal commented on SPARK-18165: [~gaurav24] - yeah I saw that. Nonetheless, I have spent

[jira] [Commented] (SPARK-18165) Kinesis support in Structured Streaming

2018-03-08 Thread Gaurav Shah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391501#comment-16391501 ] Gaurav Shah commented on SPARK-18165: - Databricks have it implemented not sure why is it exclusive

[jira] [Commented] (SPARK-23625) spark sql long-running mission will be dead

2018-03-08 Thread Yu Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391447#comment-16391447 ] Yu Wang commented on SPARK-23625: - [~hvanhovell] thank you for your answer, but it doesn't happen

[jira] [Created] (SPARK-23629) Building streaming-kafka-0-8-assembly or streaming-flume-assembly adds incompatible jline jar to assembly

2018-03-08 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-23629: - Summary: Building streaming-kafka-0-8-assembly or streaming-flume-assembly adds incompatible jline jar to assembly Key: SPARK-23629 URL:

[jira] [Commented] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391386#comment-16391386 ] Kazuaki Ishizaki commented on SPARK-23549: -- I see. Make sense. It would be good to cast

[jira] [Commented] (SPARK-23598) WholeStageCodegen can lead to IllegalAccessError calling append for HashAggregateExec

2018-03-08 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391373#comment-16391373 ] Marco Gaido commented on SPARK-23598: - [~dvogelbacher] the parameter you are talking about is taken

[jira] [Assigned] (SPARK-23602) PrintToStderr should behave the same in interpreted mode

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23602: Assignee: (was: Apache Spark) > PrintToStderr should behave the same in interpreted

[jira] [Assigned] (SPARK-23602) PrintToStderr should behave the same in interpreted mode

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23602: Assignee: Apache Spark > PrintToStderr should behave the same in interpreted mode >

[jira] [Commented] (SPARK-23602) PrintToStderr should behave the same in interpreted mode

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391354#comment-16391354 ] Apache Spark commented on SPARK-23602: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Commented] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Dong Jiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391351#comment-16391351 ] Dong Jiang commented on SPARK-23549: [~kiszk], I expect your query to return false, as presto/Athena

[jira] [Commented] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391344#comment-16391344 ] Kazuaki Ishizaki commented on SPARK-23549: -- I think that this is a problem in Spark. My question

[jira] [Commented] (SPARK-18165) Kinesis support in Structured Streaming

2018-03-08 Thread Vikram Agrawal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391305#comment-16391305 ] Vikram Agrawal commented on SPARK-18165: I have worked on an implementation of Kinesis

[jira] [Commented] (SPARK-23513) java.io.IOException: Expected 12 fields, but got 5 for row :Spark submit error

2018-03-08 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391293#comment-16391293 ] zhoukang commented on SPARK-23513: -- Could you please post some more details?[~Fray] >

[jira] [Assigned] (SPARK-23628) WholeStageCodegen can generate methods with too many params

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23628: Assignee: Apache Spark > WholeStageCodegen can generate methods with too many params >

[jira] [Commented] (SPARK-23628) WholeStageCodegen can generate methods with too many params

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391286#comment-16391286 ] Apache Spark commented on SPARK-23628: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23628) WholeStageCodegen can generate methods with too many params

2018-03-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23628: Assignee: (was: Apache Spark) > WholeStageCodegen can generate methods with too many

[jira] [Comment Edited] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391282#comment-16391282 ] zhoukang edited comment on SPARK-23549 at 3/8/18 2:07 PM: -- I think this is a

[jira] [Commented] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-08 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391282#comment-16391282 ] zhoukang commented on SPARK-23549: -- I think this is a bug.Which may caused by rule below: {code:java}

[jira] [Resolved] (SPARK-22751) Improve ML RandomForest shuffle performance

2018-03-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22751. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20472

[jira] [Assigned] (SPARK-22751) Improve ML RandomForest shuffle performance

2018-03-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-22751: - Assignee: lucio35 > Improve ML RandomForest shuffle performance >

[jira] [Created] (SPARK-23628) WholeStageCodegen can generate methods with too many params

2018-03-08 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-23628: --- Summary: WholeStageCodegen can generate methods with too many params Key: SPARK-23628 URL: https://issues.apache.org/jira/browse/SPARK-23628 Project: Spark

[jira] [Resolved] (SPARK-23592) Add interpreted execution for DecodeUsingSerializer expression

2018-03-08 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-23592. --- Resolution: Fixed Fix Version/s: 2.4.0 > Add interpreted execution for

  1   2   >