[jira] [Commented] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

2018-04-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453521#comment-16453521 ] Saisai Shao commented on SPARK-24062: - Issue resolved by pull request 21138

[jira] [Resolved] (SPARK-24062) SASL encryption cannot be worked in ThriftServer

2018-04-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24062. - Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.4.0

[jira] [Resolved] (SPARK-23916) High-order function: array_join(x, delimiter, null_replacement) → varchar

2018-04-25 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23916. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21011

[jira] [Assigned] (SPARK-23916) High-order function: array_join(x, delimiter, null_replacement) → varchar

2018-04-25 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23916: - Assignee: Marco Gaido > High-order function: array_join(x, delimiter, null_replacement)

[jira] [Resolved] (SPARK-23902) Provide an option in months_between UDF to disable rounding-off

2018-04-25 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23902. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21008

[jira] [Assigned] (SPARK-23902) Provide an option in months_between UDF to disable rounding-off

2018-04-25 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23902: - Assignee: Marco Gaido > Provide an option in months_between UDF to disable rounding-off

[jira] [Commented] (SPARK-21645) SparkSQL Left outer join get the error result when use phoenix spark plugin

2018-04-25 Thread shining (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453410#comment-16453410 ] shining commented on SPARK-21645: - When left outer join occurs between table a and table b, the filter of

[jira] [Assigned] (SPARK-21645) SparkSQL Left outer join get the error result when use phoenix spark plugin

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21645: Assignee: (was: Apache Spark) > SparkSQL Left outer join get the error result when

[jira] [Assigned] (SPARK-21645) SparkSQL Left outer join get the error result when use phoenix spark plugin

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21645: Assignee: Apache Spark > SparkSQL Left outer join get the error result when use phoenix

[jira] [Commented] (SPARK-21645) SparkSQL Left outer join get the error result when use phoenix spark plugin

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453405#comment-16453405 ] Apache Spark commented on SPARK-21645: -- User 'shining1989' has created a pull request for this

[jira] [Commented] (SPARK-21661) SparkSQL can't merge load table from Hadoop

2018-04-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453394#comment-16453394 ] Hyukjin Kwon commented on SPARK-21661: -- Another note: we now have

[jira] [Commented] (SPARK-23151) Provide a distribution of Spark with Hadoop 3.0

2018-04-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453368#comment-16453368 ] Hyukjin Kwon commented on SPARK-23151: -- [~ste...@apache.org], is it an exact duplicate of

[jira] [Commented] (SPARK-24094) Change description strings of v2 streaming sources to reflect the change

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453348#comment-16453348 ] Apache Spark commented on SPARK-24094: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24094) Change description strings of v2 streaming sources to reflect the change

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24094: Assignee: Tathagata Das (was: Apache Spark) > Change description strings of v2 streaming

[jira] [Comment Edited] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453345#comment-16453345 ] kumar edited comment on SPARK-24089 at 4/26/18 1:50 AM: UNION just joins the

[jira] [Assigned] (SPARK-24094) Change description strings of v2 streaming sources to reflect the change

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24094: Assignee: Apache Spark (was: Tathagata Das) > Change description strings of v2 streaming

[jira] [Commented] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453345#comment-16453345 ] kumar commented on SPARK-24089: --- UNION just joins the DataFrames, i am not looking for that. I want

[jira] [Commented] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453336#comment-16453336 ] Hyukjin Kwon commented on SPARK-24089: -- (Critical+ is also usually reserved for committers,

[jira] [Updated] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24089: - Priority: Major (was: Critical) > DataFrame.write().mode(SaveMode.Append).insertInto(TABLE) >

[jira] [Resolved] (SPARK-24086) Exception while executing spark streaming examples

2018-04-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24086. -- Resolution: Invalid >From a quick look, that sounds because you didn't provide a profile for

[jira] [Resolved] (SPARK-24092) spark.python.worker.reuse does not work?

2018-04-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24092. -- Resolution: Invalid Questions should go to mailing list. You could have a better and quicker

[jira] [Assigned] (SPARK-24069) Add array_max / array_min functions

2018-04-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24069: Assignee: Hyukjin Kwon > Add array_max / array_min functions >

[jira] [Resolved] (SPARK-24069) Add array_max / array_min functions

2018-04-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24069. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21142

[jira] [Commented] (SPARK-24036) Stateful operators in continuous processing

2018-04-25 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453290#comment-16453290 ] Jungtaek Lim commented on SPARK-24036: -- Btw, I would like to say the idea for iterator hack and

[jira] [Created] (SPARK-24094) Change description strings of v2 streaming sources to reflect the change

2018-04-25 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-24094: - Summary: Change description strings of v2 streaming sources to reflect the change Key: SPARK-24094 URL: https://issues.apache.org/jira/browse/SPARK-24094 Project:

[jira] [Commented] (SPARK-24070) TPC-DS Performance Tests for Parquet 1.10.0 Upgrade

2018-04-25 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453258#comment-16453258 ] Takeshi Yamamuro commented on SPARK-24070: -- Sure, I checked the numbers and see:

[jira] [Commented] (SPARK-22210) Online LDA variationalTopicInference should use random seed to have stable behavior

2018-04-25 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453228#comment-16453228 ] Joseph K. Bradley commented on SPARK-22210: --- [~lu.DB] Would you like to do this? It should be

[jira] [Comment Edited] (SPARK-24036) Stateful operators in continuous processing

2018-04-25 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453209#comment-16453209 ] Jungtaek Lim edited comment on SPARK-24036 at 4/25/18 10:54 PM: Maybe

[jira] [Commented] (SPARK-24036) Stateful operators in continuous processing

2018-04-25 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453209#comment-16453209 ] Jungtaek Lim commented on SPARK-24036: -- Maybe better to share what I've observed from continuous

[jira] [Comment Edited] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-25 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452790#comment-16452790 ] Bruce Robbins edited comment on SPARK-23715 at 4/25/18 10:00 PM: -

[jira] [Resolved] (SPARK-23824) Make inpurityStats publicly accessible in ml.tree.Node

2018-04-25 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-23824. --- Resolution: Duplicate > Make inpurityStats publicly accessible in ml.tree.Node >

[jira] [Commented] (SPARK-24057) put the real data type in the AssertionError message

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453092#comment-16453092 ] Apache Spark commented on SPARK-24057: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24057) put the real data type in the AssertionError message

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24057: Assignee: Apache Spark > put the real data type in the AssertionError message >

[jira] [Assigned] (SPARK-24057) put the real data type in the AssertionError message

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24057: Assignee: (was: Apache Spark) > put the real data type in the AssertionError message

[jira] [Updated] (SPARK-24057) put the real data type in the AssertionError message

2018-04-25 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao updated SPARK-24057: --- Issue Type: Improvement (was: Bug) > put the real data type in the AssertionError message >

[jira] [Updated] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

2018-04-25 Thread Weiqing Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated SPARK-24093: - Description: To make third parties able to get the information of streaming writer, for example,

[jira] [Commented] (SPARK-24036) Stateful operators in continuous processing

2018-04-25 Thread Arun Mahadevan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453058#comment-16453058 ] Arun Mahadevan commented on SPARK-24036: Hi [~joseph.torres], I am also interested to contribute

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

2018-04-25 Thread Tavis Barr (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453054#comment-16453054 ] Tavis Barr commented on SPARK-13446: I do not believe the issue causing the above stack trace has

[jira] [Created] (SPARK-24093) Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes

2018-04-25 Thread Weiqing Yang (JIRA)
Weiqing Yang created SPARK-24093: Summary: Make some fields of KafkaStreamWriter/InternalRowMicroBatchWriter visible to outside of the classes Key: SPARK-24093 URL:

[jira] [Updated] (SPARK-24092) spark.python.worker.reuse does not work?

2018-04-25 Thread David Figueroa (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Figueroa updated SPARK-24092: --- Description: {{spark.python.worker.reuse is true by default but even after explicitly

[jira] [Updated] (SPARK-24092) spark.python.worker.reuse does not work?

2018-04-25 Thread David Figueroa (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Figueroa updated SPARK-24092: --- Description: {{spark.python.worker.reuse is true by default but even after explicitly

[jira] [Created] (SPARK-24092) spark.python.worker.reuse does not work?

2018-04-25 Thread David Figueroa (JIRA)
David Figueroa created SPARK-24092: -- Summary: spark.python.worker.reuse does not work? Key: SPARK-24092 URL: https://issues.apache.org/jira/browse/SPARK-24092 Project: Spark Issue Type:

[jira] [Commented] (SPARK-22239) User-defined window functions with pandas udf

2018-04-25 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452977#comment-16452977 ] Li Jin commented on SPARK-22239: [~hvanhovell], I have done a bit further research of UDF over rolling

[jira] [Commented] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452942#comment-16452942 ] Marco Gaido commented on SPARK-24089: - Anyway, for what I can see from your post on stackoverflow,

[jira] [Commented] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452937#comment-16452937 ] Marco Gaido commented on SPARK-24089: - Blocker can be set only by commiters, I moved to Critical. >

[jira] [Updated] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-24089: Priority: Critical (was: Blocker) > DataFrame.write().mode(SaveMode.Append).insertInto(TABLE) >

[jira] [Resolved] (SPARK-24050) StreamingQuery does not calculate input / processing rates in some cases

2018-04-25 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24050. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21126

[jira] [Comment Edited] (SPARK-23933) High-order function: map(array, array) → map<K,V>

2018-04-25 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452404#comment-16452404 ] Kazuaki Ishizaki edited comment on SPARK-23933 at 4/25/18 6:48 PM: ---

[jira] [Commented] (SPARK-23715) from_utc_timestamp returns incorrect results for some UTC date/time values

2018-04-25 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452790#comment-16452790 ] Bruce Robbins commented on SPARK-23715: --- [~cloud_fan] I'll give separate answers for String input

[jira] [Updated] (SPARK-24091) Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files

2018-04-25 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-24091: - Affects Version/s: (was: 2.3.0) 2.4.0 > Internally used ConfigMap prevents

[jira] [Created] (SPARK-24091) Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files

2018-04-25 Thread Yinan Li (JIRA)
Yinan Li created SPARK-24091: Summary: Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files Key: SPARK-24091 URL: https://issues.apache.org/jira/browse/SPARK-24091

[jira] [Commented] (SPARK-23850) We should not redact username|user|url from UI by default

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452729#comment-16452729 ] Apache Spark commented on SPARK-23850: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23850) We should not redact username|user|url from UI by default

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23850: Assignee: Apache Spark > We should not redact username|user|url from UI by default >

[jira] [Assigned] (SPARK-23850) We should not redact username|user|url from UI by default

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23850: Assignee: (was: Apache Spark) > We should not redact username|user|url from UI by

[jira] [Created] (SPARK-24090) Kubernetes Backend Hotlist for Spark 2.4

2018-04-25 Thread Anirudh Ramanathan (JIRA)
Anirudh Ramanathan created SPARK-24090: -- Summary: Kubernetes Backend Hotlist for Spark 2.4 Key: SPARK-24090 URL: https://issues.apache.org/jira/browse/SPARK-24090 Project: Spark Issue

[jira] [Updated] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar updated SPARK-24089: -- Component/s: Java API > DataFrame.write().mode(SaveMode.Append).insertInto(TABLE) >

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

2018-04-25 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452666#comment-16452666 ] Bryan Cutler commented on SPARK-23874: -- [~smilegator] the Arrow community decided to put their

[jira] [Updated] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar updated SPARK-24089: -- Description: I am completely stuck with this issue, unable to progress further. For more info pls refer this

[jira] [Updated] (SPARK-24089) DataFrame.write().mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar updated SPARK-24089: -- Summary: DataFrame.write().mode(SaveMode.Append).insertInto(TABLE) (was:

[jira] [Created] (SPARK-24089) DataFrame.write.mode(SaveMode.Append).insertInto(TABLE)

2018-04-25 Thread kumar (JIRA)
kumar created SPARK-24089: - Summary: DataFrame.write.mode(SaveMode.Append).insertInto(TABLE) Key: SPARK-24089 URL: https://issues.apache.org/jira/browse/SPARK-24089 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-24067) Backport SPARK-17147 to 2.3 (Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction))

2018-04-25 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452603#comment-16452603 ] Cody Koeninger commented on SPARK-24067: The original PR

[jira] [Commented] (SPARK-24036) Stateful operators in continuous processing

2018-04-25 Thread Jose Torres (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452576#comment-16452576 ] Jose Torres commented on SPARK-24036: - The broader Spark community is of course always welcome to

[jira] [Commented] (SPARK-24070) TPC-DS Performance Tests for Parquet 1.10.0 Upgrade

2018-04-25 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452527#comment-16452527 ] Xiao Li commented on SPARK-24070: - [~mswit] Thank you for your suggestions! This is very helpful! >

[jira] [Created] (SPARK-24088) only HadoopRDD leverage HDFS Cache as preferred location

2018-04-25 Thread Xiaoju Wu (JIRA)
Xiaoju Wu created SPARK-24088: - Summary: only HadoopRDD leverage HDFS Cache as preferred location Key: SPARK-24088 URL: https://issues.apache.org/jira/browse/SPARK-24088 Project: Spark Issue

[jira] [Assigned] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22674: Assignee: Apache Spark > PySpark breaks serialization of namedtuple subclasses >

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452462#comment-16452462 ] Apache Spark commented on SPARK-22674: -- User 'superbobry' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22674: Assignee: (was: Apache Spark) > PySpark breaks serialization of namedtuple subclasses

[jira] [Commented] (SPARK-23929) pandas_udf schema mapped by position and not by name

2018-04-25 Thread Tr3wory (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452459#comment-16452459 ] Tr3wory commented on SPARK-23929: - Yes, but that's not simpler than using "columns=[...]": {code:python}

[jira] [Commented] (SPARK-23933) High-order function: map(array, array) → map<K,V>

2018-04-25 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452404#comment-16452404 ] Kazuaki Ishizaki commented on SPARK-23933: -- Thank you for your comment. The current map can take

[jira] [Commented] (SPARK-24087) Avoid shuffle when join keys are a super-set of bucket keys

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452399#comment-16452399 ] Apache Spark commented on SPARK-24087: -- User 'yucai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24087) Avoid shuffle when join keys are a super-set of bucket keys

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24087: Assignee: (was: Apache Spark) > Avoid shuffle when join keys are a super-set of

[jira] [Assigned] (SPARK-24087) Avoid shuffle when join keys are a super-set of bucket keys

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24087: Assignee: Apache Spark > Avoid shuffle when join keys are a super-set of bucket keys >

[jira] [Commented] (SPARK-24067) Backport SPARK-17147 to 2.3 (Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction))

2018-04-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452396#comment-16452396 ] Sean Owen commented on SPARK-24067: --- It seems like a clear bug fix. Granted it's not trivial, but it is

[jira] [Created] (SPARK-24087) Avoid shuffle when join keys are a super-set of bucket keys

2018-04-25 Thread yucai (JIRA)
yucai created SPARK-24087: - Summary: Avoid shuffle when join keys are a super-set of bucket keys Key: SPARK-24087 URL: https://issues.apache.org/jira/browse/SPARK-24087 Project: Spark Issue Type:

[jira] [Commented] (SPARK-20087) Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd listeners

2018-04-25 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452369#comment-16452369 ] Imran Rashid commented on SPARK-20087: -- Sound good to me, I'm in favor of the change > Include

[jira] [Commented] (SPARK-23929) pandas_udf schema mapped by position and not by name

2018-04-25 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452355#comment-16452355 ] Li Jin commented on SPARK-23929: [~tr3w] does using OrderedDict help in your case? > pandas_udf schema

[jira] [Commented] (SPARK-23933) High-order function: map(array, array) → map<K,V>

2018-04-25 Thread Alex Wajda (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452350#comment-16452350 ] Alex Wajda commented on SPARK-23933: Why can't {{map}} be overloaded? So that if you pass one array

[jira] [Commented] (SPARK-24067) Backport SPARK-17147 to 2.3 (Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction))

2018-04-25 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452337#comment-16452337 ] Cody Koeninger commented on SPARK-24067: Given the response on the dev list about criteria for

[jira] [Updated] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB

2018-04-25 Thread fengchaoge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-21337: --- Attachment: (was: 1tes.zip) > SQL which has large ‘case when’ expressions may cause code

[jira] [Created] (SPARK-24086) Exception while executing spark streaming examples

2018-04-25 Thread Chandra Hasan (JIRA)
Chandra Hasan created SPARK-24086: - Summary: Exception while executing spark streaming examples Key: SPARK-24086 URL: https://issues.apache.org/jira/browse/SPARK-24086 Project: Spark Issue

[jira] [Commented] (SPARK-23929) pandas_udf schema mapped by position and not by name

2018-04-25 Thread Tr3wory (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452282#comment-16452282 ] Tr3wory commented on SPARK-23929: - I think the problem is even more nuanced: in python the order of the

[jira] [Commented] (SPARK-23927) High-order function: sequence

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452246#comment-16452246 ] Apache Spark commented on SPARK-23927: -- User 'wajda' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23927) High-order function: sequence

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23927: Assignee: (was: Apache Spark) > High-order function: sequence >

[jira] [Assigned] (SPARK-23927) High-order function: sequence

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23927: Assignee: Apache Spark > High-order function: sequence > - >

[jira] [Created] (SPARK-24085) Scalar subquery error

2018-04-25 Thread Alexey Baturin (JIRA)
Alexey Baturin created SPARK-24085: -- Summary: Scalar subquery error Key: SPARK-24085 URL: https://issues.apache.org/jira/browse/SPARK-24085 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB

2018-04-25 Thread fengchaoge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-21337: --- Attachment: 1tes.zip > SQL which has large ‘case when’ expressions may cause code generation beyond

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-04-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452215#comment-16452215 ] Steve Loughran commented on SPARK-18673: looking @ our local commit logs, the HDP version

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-04-25 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452196#comment-16452196 ] Steve Loughran commented on SPARK-18673: HIVE-16081 commit 93db527f47 contains the one-line

[jira] [Updated] (SPARK-24084) Add job group id for query through spark-sql

2018-04-25 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-24084: - Description: For spark-sql we can add job group id for the same statement. (was: For thrift server we

[jira] [Updated] (SPARK-24084) Add job group id for query through spark-sql

2018-04-25 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-24084: - Summary: Add job group id for query through spark-sql (was: Add job group id for query through Thrift

[jira] [Created] (SPARK-24084) Add job group id for query through Thrift Server

2018-04-25 Thread zhoukang (JIRA)
zhoukang created SPARK-24084: Summary: Add job group id for query through Thrift Server Key: SPARK-24084 URL: https://issues.apache.org/jira/browse/SPARK-24084 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-24070) TPC-DS Performance Tests for Parquet 1.10.0 Upgrade

2018-04-25 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-24070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452096#comment-16452096 ] Michał Świtakowski edited comment on SPARK-24070 at 4/25/18 11:43 AM:

[jira] [Commented] (SPARK-24070) TPC-DS Performance Tests for Parquet 1.10.0 Upgrade

2018-04-25 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-24070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452096#comment-16452096 ] Michał Świtakowski commented on SPARK-24070: [~maropu] I think you can just use the existing

[jira] [Resolved] (SPARK-23880) table cache should be lazy and don't trigger any jobs.

2018-04-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23880. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21018

[jira] [Assigned] (SPARK-23880) table cache should be lazy and don't trigger any jobs.

2018-04-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23880: --- Assignee: Takeshi Yamamuro > table cache should be lazy and don't trigger any jobs. >

[jira] [Commented] (SPARK-24012) Union of map and other compatible column

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452059#comment-16452059 ] Apache Spark commented on SPARK-24012: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-20894) Error while checkpointing to HDFS

2018-04-25 Thread Aydin Kocas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452041#comment-16452041 ] Aydin Kocas commented on SPARK-20894: - removing the checkpoint location along with the 

[jira] [Commented] (SPARK-20894) Error while checkpointing to HDFS

2018-04-25 Thread Aydin Kocas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452030#comment-16452030 ] Aydin Kocas commented on SPARK-20894: - having the same issue on 2.3 - what's the solution? Am

[jira] [Resolved] (SPARK-24012) Union of map and other compatible column

2018-04-25 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24012. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21100

[jira] [Assigned] (SPARK-24058) Default Params in ML should be saved separately: Python API

2018-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24058: Assignee: Apache Spark > Default Params in ML should be saved separately: Python API >

  1   2   >