[jira] [Updated] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23265: --- Description: SPARK-22397 added support for multiple columns to {{QuantileDiscretizer}}. If b

[jira] [Commented] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344604#comment-16344604 ] Nick Pentreath commented on SPARK-23265: cc [~huaxing]  > Update multi-column er

[jira] [Updated] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23265: --- Issue Type: Improvement (was: Documentation) > Update multi-column error handling logic in Q

[jira] [Created] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-01-29 Thread Nick Pentreath (JIRA)
Nick Pentreath created SPARK-23265: -- Summary: Update multi-column error handling logic in QuantileDiscretizer Key: SPARK-23265 URL: https://issues.apache.org/jira/browse/SPARK-23265 Project: Spark

[jira] [Updated] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23265: --- Description: SPARK-22397 added support for multiple columns to {{QuantileDiscretizer}}. If b

[jira] [Resolved] (SPARK-23138) Add user guide example for multiclass logistic regression summary

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-23138. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20332 [https:/

[jira] [Assigned] (SPARK-23138) Add user guide example for multiclass logistic regression summary

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath reassigned SPARK-23138: -- Assignee: Seth Hendrickson > Add user guide example for multiclass logistic regression

[jira] [Commented] (SPARK-20928) SPIP: Continuous Processing Mode for Structured Streaming

2018-01-29 Thread liweisheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344565#comment-16344565 ] liweisheng commented on SPARK-20928: What about introducing a new way of non-block sh

[jira] [Assigned] (SPARK-23264) Support interval values without INTERVAL clauses

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23264: Assignee: (was: Apache Spark) > Support interval values without INTERVAL clauses > ---

[jira] [Assigned] (SPARK-23264) Support interval values without INTERVAL clauses

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23264: Assignee: Apache Spark > Support interval values without INTERVAL clauses > --

[jira] [Commented] (SPARK-23264) Support interval values without INTERVAL clauses

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344561#comment-16344561 ] Apache Spark commented on SPARK-23264: -- User 'maropu' has created a pull request for

[jira] [Resolved] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23157. - Resolution: Invalid > withColumn fails for a column that is a result of mapped DataSet >

[jira] [Created] (SPARK-23264) Support interval values without INTERVAL clauses

2018-01-29 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-23264: Summary: Support interval values without INTERVAL clauses Key: SPARK-23264 URL: https://issues.apache.org/jira/browse/SPARK-23264 Project: Spark Issu

[jira] [Commented] (SPARK-23174) Fix pep8 to latest official version

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344553#comment-16344553 ] Apache Spark commented on SPARK-23174: -- User 'ueshin' has created a pull request for

[jira] [Assigned] (SPARK-23222) Flaky test: DataFrameRangeSuite

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23222: Assignee: Apache Spark > Flaky test: DataFrameRangeSuite > ---

[jira] [Assigned] (SPARK-23222) Flaky test: DataFrameRangeSuite

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23222: Assignee: (was: Apache Spark) > Flaky test: DataFrameRangeSuite >

[jira] [Commented] (SPARK-23222) Flaky test: DataFrameRangeSuite

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344548#comment-16344548 ] Apache Spark commented on SPARK-23222: -- User 'viirya' has created a pull request for

[jira] [Issue Comment Deleted] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-29 Thread Gaurav Garg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurav Garg updated SPARK-18016: Comment: was deleted (was: [~kiszk], this programs also gives the Constant pool error in my enviro

[jira] [Comment Edited] (SPARK-23252) When NodeManager and CoarseGrainedExecutorBackend processes are killed, the job will be blocked

2018-01-29 Thread Bang Xiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344484#comment-16344484 ] Bang Xiao edited comment on SPARK-23252 at 1/30/18 4:30 AM: A

[jira] [Commented] (SPARK-23252) When NodeManager and CoarseGrainedExecutorBackend processes are killed, the job will be blocked

2018-01-29 Thread Bang Xiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344484#comment-16344484 ] Bang Xiao commented on SPARK-23252: --- After the executor and NodeManager is killed, fail

[jira] [Assigned] (SPARK-23263) create table stored as parquet should update table size if automatic update table size is enabled

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23263: Assignee: (was: Apache Spark) > create table stored as parquet should update table siz

[jira] [Commented] (SPARK-23263) create table stored as parquet should update table size if automatic update table size is enabled

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344460#comment-16344460 ] Apache Spark commented on SPARK-23263: -- User 'wangyum' has created a pull request fo

[jira] [Assigned] (SPARK-23263) create table stored as parquet should update table size if automatic update table size is enabled

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23263: Assignee: Apache Spark > create table stored as parquet should update table size if automa

[jira] [Created] (SPARK-23263) create table stored as parquet should update table size if automatic update table size is enabled

2018-01-29 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-23263: --- Summary: create table stored as parquet should update table size if automatic update table size is enabled Key: SPARK-23263 URL: https://issues.apache.org/jira/browse/SPARK-23263

[jira] [Updated] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-29 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MBA Learns to Code updated SPARK-23246: --- Description: I am having consistent OOM crashes when trying to use PySpark for iterat

[jira] [Resolved] (SPARK-23088) History server not showing incomplete/running applications

2018-01-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23088. - Resolution: Fixed Assignee: paul mackles Fix Version/s: 2.4.0 Issue resolved by p

[jira] [Commented] (SPARK-23237) Add UI / endpoint for threaddumps for executors with active tasks

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344434#comment-16344434 ] Imran Rashid commented on SPARK-23237: -- Can you expand a bit about what you are worr

[jira] [Comment Edited] (SPARK-23236) Make it easier to find the rest API, especially in local mode

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344433#comment-16344433 ] Imran Rashid edited comment on SPARK-23236 at 1/30/18 3:09 AM:

[jira] [Commented] (SPARK-23236) Make it easier to find the rest API, especially in local mode

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344433#comment-16344433 ] Imran Rashid commented on SPARK-23236: -- bq. 1. /api and /api/v1 to give the same res

[jira] [Commented] (SPARK-23262) mix-in interface should extend the interface it aimed to mix in

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344396#comment-16344396 ] Apache Spark commented on SPARK-23262: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-23262) mix-in interface should extend the interface it aimed to mix in

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23262: Assignee: Wenchen Fan (was: Apache Spark) > mix-in interface should extend the interface

[jira] [Assigned] (SPARK-23262) mix-in interface should extend the interface it aimed to mix in

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23262: Assignee: Apache Spark (was: Wenchen Fan) > mix-in interface should extend the interface

[jira] [Created] (SPARK-23262) mix-in interface should extend the interface it aimed to mix in

2018-01-29 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-23262: --- Summary: mix-in interface should extend the interface it aimed to mix in Key: SPARK-23262 URL: https://issues.apache.org/jira/browse/SPARK-23262 Project: Spark

[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2018-01-29 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344339#comment-16344339 ] Alex Bozarth commented on SPARK-18085: -- [~vanzin] since this is complete and going i

[jira] [Closed] (SPARK-21664) Use the column name as the file name.

2018-01-29 Thread jifei_yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jifei_yang closed SPARK-21664. -- We can use the partition to save the column names, such as: {code:java} case class UserInfo(name:String,fa

[jira] [Resolved] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23246. --- Resolution: Not A Problem Yes, did you have a look? It's dominated by things like {{class org.apache

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

2018-01-29 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344168#comment-16344168 ] Alex Bozarth commented on SPARK-23235: -- Your discussion clarified my concern for me.

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344140#comment-16344140 ] Imran Rashid commented on SPARK-23235: -- [~ajbozarth] can you explain your concern?

[jira] [Assigned] (SPARK-23209) HiveDelegationTokenProvider throws an exception if Hive jars are not the classpath

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23209: Assignee: Marcelo Vanzin > HiveDelegationTokenProvider throws an exception if Hive jars ar

[jira] [Resolved] (SPARK-23209) HiveDelegationTokenProvider throws an exception if Hive jars are not the classpath

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23209. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20399 [https://git

[jira] [Assigned] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23157: Assignee: Apache Spark > withColumn fails for a column that is a result of mapped DataSet

[jira] [Assigned] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23157: Assignee: (was: Apache Spark) > withColumn fails for a column that is a result of mapp

[jira] [Commented] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344078#comment-16344078 ] Apache Spark commented on SPARK-23157: -- User 'henryr' has created a pull request for

[jira] [Assigned] (SPARK-23261) Rename Pandas UDFs

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23261: Assignee: Apache Spark (was: Xiao Li) > Rename Pandas UDFs > -- > >

[jira] [Commented] (SPARK-23261) Rename Pandas UDFs

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344076#comment-16344076 ] Apache Spark commented on SPARK-23261: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-23261) Rename Pandas UDFs

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23261: Assignee: Xiao Li (was: Apache Spark) > Rename Pandas UDFs > -- > >

[jira] [Comment Edited] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-29 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344068#comment-16344068 ] MBA Learns to Code edited comment on SPARK-23246 at 1/29/18 9:45 PM: --

[jira] [Commented] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-29 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344068#comment-16344068 ] MBA Learns to Code commented on SPARK-23246: [~srowen] the Java driver heap d

[jira] [Updated] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-29 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MBA Learns to Code updated SPARK-23246: --- Attachment: SparkProgramHeapDump.bin.tar.xz > (Py)Spark OOM because of iteratively ac

[jira] [Created] (SPARK-23261) Rename Pandas UDFs

2018-01-29 Thread Xiao Li (JIRA)
Xiao Li created SPARK-23261: --- Summary: Rename Pandas UDFs Key: SPARK-23261 URL: https://issues.apache.org/jira/browse/SPARK-23261 Project: Spark Issue Type: Sub-task Components: PySpark

[jira] [Commented] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Henry Robinson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343962#comment-16343962 ] Henry Robinson commented on SPARK-23157: [~kretes] - I can see an argument for th

[jira] [Commented] (SPARK-23260) remove V2 from the class name of data source reader/writer

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343878#comment-16343878 ] Apache Spark commented on SPARK-23260: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-23260) remove V2 from the class name of data source reader/writer

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23260: Assignee: Apache Spark (was: Wenchen Fan) > remove V2 from the class name of data source

[jira] [Assigned] (SPARK-23260) remove V2 from the class name of data source reader/writer

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23260: Assignee: Wenchen Fan (was: Apache Spark) > remove V2 from the class name of data source

[jira] [Created] (SPARK-23260) remove V2 from the class name of data source reader/writer

2018-01-29 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-23260: --- Summary: remove V2 from the class name of data source reader/writer Key: SPARK-23260 URL: https://issues.apache.org/jira/browse/SPARK-23260 Project: Spark Issu

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343840#comment-16343840 ] Apache Spark commented on SPARK-23207: -- User 'jiangxb1987' has created a pull reques

[jira] [Assigned] (SPARK-23259) Clean up legacy code around hive external catalog

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23259: Assignee: Apache Spark > Clean up legacy code around hive external catalog > -

[jira] [Assigned] (SPARK-23259) Clean up legacy code around hive external catalog

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23259: Assignee: (was: Apache Spark) > Clean up legacy code around hive external catalog > --

[jira] [Commented] (SPARK-23259) Clean up legacy code around hive external catalog

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343814#comment-16343814 ] Apache Spark commented on SPARK-23259: -- User 'liufengdb' has created a pull request

[jira] [Created] (SPARK-23259) Clean up legacy code around hive external catalog

2018-01-29 Thread Feng Liu (JIRA)
Feng Liu created SPARK-23259: Summary: Clean up legacy code around hive external catalog Key: SPARK-23259 URL: https://issues.apache.org/jira/browse/SPARK-23259 Project: Spark Issue Type: Improve

[jira] [Assigned] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23240: Assignee: (was: Apache Spark) > PythonWorkerFactory issues unhelpful message when pysp

[jira] [Assigned] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23240: Assignee: Apache Spark > PythonWorkerFactory issues unhelpful message when pyspark.daemon

[jira] [Commented] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343792#comment-16343792 ] Apache Spark commented on SPARK-23240: -- User 'bersprockets' has created a pull reque

[jira] [Commented] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343785#comment-16343785 ] Apache Spark commented on SPARK-1: -- User 'BryanCutler' has created a pull reques

[jira] [Resolved] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2018-01-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-1. - Resolution: Fixed Assignee: Bryan Cutler Fix Version/s: 2.3.0 > Add User Documentation fo

[jira] [Created] (SPARK-23258) Should not split Arrow record batches based on row count

2018-01-29 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-23258: Summary: Should not split Arrow record batches based on row count Key: SPARK-23258 URL: https://issues.apache.org/jira/browse/SPARK-23258 Project: Spark Issu

[jira] [Commented] (SPARK-23020) Re-enable Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343693#comment-16343693 ] Marcelo Vanzin commented on SPARK-23020: :-/ It's getting harder and harder to r

[jira] [Comment Edited] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332698#comment-16332698 ] Bryan Cutler edited comment on SPARK-23109 at 1/29/18 5:26 PM:

[jira] [Comment Edited] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332698#comment-16332698 ] Bryan Cutler edited comment on SPARK-23109 at 1/29/18 5:25 PM:

[jira] [Commented] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343665#comment-16343665 ] Bryan Cutler commented on SPARK-23109: -- Thanks [~mlnick], yes this is done. > ML 2.

[jira] [Resolved] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-23109. -- Resolution: Done > ML 2.3 QA: API: Python API coverage > --- >

[jira] [Resolved] (SPARK-17006) WithColumn Performance Degrades with Number of Invocations

2018-01-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-17006. --- Resolution: Fixed Assignee: Herman van Hovell Fix Version/s: 2.3.0 >

[jira] [Resolved] (SPARK-23223) Stacking dataset transforms performs poorly

2018-01-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-23223. --- Resolution: Fixed Fix Version/s: 2.3.0 > Stacking dataset transforms performs

[jira] [Resolved] (SPARK-23059) Correct some improper with view related method usage

2018-01-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23059. - Resolution: Fixed Fix Version/s: 2.4.0 > Correct some improper with view related method usage > --

[jira] [Assigned] (SPARK-23059) Correct some improper with view related method usage

2018-01-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-23059: --- Assignee: xubo245 > Correct some improper with view related method usage > -

[jira] [Resolved] (SPARK-23199) improved Removes repetition from group expressions in Aggregate

2018-01-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23199. - Resolution: Fixed Assignee: caoxuewen Fix Version/s: 2.3.0 > improved Removes repetition

[jira] [Resolved] (SPARK-23219) Rename ReadTask to DataReaderFactory

2018-01-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23219. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20397 [https://githu

[jira] [Assigned] (SPARK-23219) Rename ReadTask to DataReaderFactory

2018-01-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23219: --- Assignee: Gengliang Wang > Rename ReadTask to DataReaderFactory > --

[jira] [Resolved] (SPARK-20129) JavaSparkContext should use SparkContext.getOrCreate

2018-01-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20129. --- Resolution: Won't Fix Assignee: (was: Xiangrui Meng) Per PR discussion, I believe this shou

[jira] [Commented] (SPARK-23252) When NodeManager and CoarseGrainedExecutorBackend processes are killed, the job will be blocked

2018-01-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343358#comment-16343358 ] Sean Owen commented on SPARK-23252: --- That much looks normal if the executor is removed

[jira] [Created] (SPARK-23257) Implement Kerberos Support in Kubernetes resource manager

2018-01-29 Thread Rob Keevil (JIRA)
Rob Keevil created SPARK-23257: -- Summary: Implement Kerberos Support in Kubernetes resource manager Key: SPARK-23257 URL: https://issues.apache.org/jira/browse/SPARK-23257 Project: Spark Issue T

[jira] [Commented] (SPARK-23252) When NodeManager and CoarseGrainedExecutorBackend processes are killed, the job will be blocked

2018-01-29 Thread Bang Xiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343317#comment-16343317 ] Bang Xiao commented on SPARK-23252: --- [~srowen] it seems the job  waits for the results 

[jira] [Commented] (SPARK-23252) When NodeManager and CoarseGrainedExecutorBackend processes are killed, the job will be blocked

2018-01-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343288#comment-16343288 ] Sean Owen commented on SPARK-23252: --- Blocked how? Waiting for the NodeManager? YARN wou

[jira] [Assigned] (SPARK-23108) ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath reassigned SPARK-23108: -- Assignee: Nick Pentreath > ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, s

[jira] [Comment Edited] (SPARK-23108) ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343278#comment-16343278 ] Nick Pentreath edited comment on SPARK-23108 at 1/29/18 12:14 PM: -

[jira] [Resolved] (SPARK-23108) ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-23108. Resolution: Resolved Fix Version/s: 2.3.0 > ML, Graph 2.3 QA: API: Experimental, Dev

[jira] [Commented] (SPARK-23108) ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343290#comment-16343290 ] Nick Pentreath commented on SPARK-23108: Also checked ml {{DeveloperAPI}}, nothin

[jira] [Updated] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23238: - Fix Version/s: 2.3.0 > Externalize SQLConf spark.sql.execution.arrow.enabled > -

[jira] [Resolved] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23238. -- Resolution: Fixed Fixed in https://github.com/apache/spark/pull/20403 > Externalize SQLConf sp

[jira] [Assigned] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23238: Assignee: Hyukjin Kwon > Externalize SQLConf spark.sql.execution.arrow.enabled >

[jira] [Commented] (SPARK-23108) ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343278#comment-16343278 ] Nick Pentreath commented on SPARK-23108: I think at this late stage we should not

[jira] [Commented] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343279#comment-16343279 ] Sean Owen commented on SPARK-23157: --- Agree this should not work . You are selecting a c

[jira] [Commented] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343276#comment-16343276 ] Nick Pentreath commented on SPARK-23109: Created SPARK-23256 to track {{columnSch

[jira] [Created] (SPARK-23256) Add columnSchema method to PySpark image reader

2018-01-29 Thread Nick Pentreath (JIRA)
Nick Pentreath created SPARK-23256: -- Summary: Add columnSchema method to PySpark image reader Key: SPARK-23256 URL: https://issues.apache.org/jira/browse/SPARK-23256 Project: Spark Issue Typ

[jira] [Commented] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343269#comment-16343269 ] Nick Pentreath commented on SPARK-23109: So [~bryanc] I think this is done then?

[jira] [Commented] (SPARK-21866) SPIP: Image support in Spark

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343266#comment-16343266 ] Nick Pentreath commented on SPARK-21866: Ok, added SPARK-23255 to track user guid

[jira] [Created] (SPARK-23255) Add user guide and examples for DataFrame image reading functions

2018-01-29 Thread Nick Pentreath (JIRA)
Nick Pentreath created SPARK-23255: -- Summary: Add user guide and examples for DataFrame image reading functions Key: SPARK-23255 URL: https://issues.apache.org/jira/browse/SPARK-23255 Project: Spark

[jira] [Updated] (SPARK-23107) ML, Graph 2.3 QA: API: New Scala APIs, docs

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23107: --- Description: Audit new public Scala APIs added to MLlib & GraphX. Take note of: * Protected/

[jira] [Updated] (SPARK-23227) Add user guide entry for collecting sub models for cross-validation classes

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23227: --- Priority: Minor (was: Major) > Add user guide entry for collecting sub models for cross-vali

[jira] [Updated] (SPARK-23127) Update FeatureHasher user guide for catCols parameter

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23127: --- Priority: Minor (was: Major) > Update FeatureHasher user guide for catCols parameter > -

  1   2   >