[jira] [Commented] (SPARK-24444) Improve pandas_udf GROUPED_MAP docs to explain column assignment

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497587#comment-16497587 ] Apache Spark commented on SPARK-2: -- User 'BryanCutler' has created a pull request for this

[jira] [Resolved] (SPARK-23920) High-order function: array_remove(x, element) → array

2018-05-31 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23920. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21069

[jira] [Assigned] (SPARK-23920) High-order function: array_remove(x, element) → array

2018-05-31 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23920: - Assignee: Huaxin Gao > High-order function: array_remove(x, element) → array >

[jira] [Commented] (SPARK-24448) File not found on the address SparkFiles.get returns on standalone cluster

2018-05-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497548#comment-16497548 ] Saisai Shao commented on SPARK-24448: - Does it only happen in standalone cluster mode, have you

[jira] [Resolved] (SPARK-24326) Add local:// scheme support for the app jar in mesos cluster mode

2018-05-31 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-24326. -- Resolution: Fixed Assignee: Stavros Kontopoulos Fix Version/s: 2.4.0 > Add

[jira] [Resolved] (SPARK-24444) Improve pandas_udf GROUPED_MAP docs to explain column assignment

2018-05-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-2. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21471

[jira] [Created] (SPARK-24448) File not found on the address SparkFiles.get returns on standalone cluster

2018-05-31 Thread Pritpal Singh (JIRA)
Pritpal Singh created SPARK-24448: - Summary: File not found on the address SparkFiles.get returns on standalone cluster Key: SPARK-24448 URL: https://issues.apache.org/jira/browse/SPARK-24448

[jira] [Created] (SPARK-24447) Pyspark RowMatrix.columnSimilarities() loses spark context

2018-05-31 Thread Perry Chu (JIRA)
Perry Chu created SPARK-24447: - Summary: Pyspark RowMatrix.columnSimilarities() loses spark context Key: SPARK-24447 URL: https://issues.apache.org/jira/browse/SPARK-24447 Project: Spark Issue

[jira] [Assigned] (SPARK-24330) Refactor ExecuteWriteTask in FileFormatWriter with DataWriter(V2)

2018-05-31 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24330: --- Assignee: Gengliang Wang > Refactor ExecuteWriteTask in FileFormatWriter with

[jira] [Resolved] (SPARK-24330) Refactor ExecuteWriteTask in FileFormatWriter with DataWriter(V2)

2018-05-31 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24330. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21381

[jira] [Commented] (SPARK-23754) StopIterator exception in Python UDF results in partial result

2018-05-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497461#comment-16497461 ] Hyukjin Kwon commented on SPARK-23754: -- For clarification, this is _fixed_. We are just trying to

[jira] [Commented] (SPARK-21918) HiveClient shouldn't share Hive object between different thread

2018-05-31 Thread fengchaoge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497456#comment-16497456 ] fengchaoge commented on SPARK-21918: Hu Liu gone? > HiveClient shouldn't share Hive object

[jira] [Updated] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show()

2018-05-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24442: - Fix Version/s: (was: 2.4.0) > Add configuration parameter to adjust the numbers of records

[jira] [Updated] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show()

2018-05-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24442: - Component/s: (was: Input/Output) SQL > Add configuration parameter to

[jira] [Commented] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show()

2018-05-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497458#comment-16497458 ] Hyukjin Kwon commented on SPARK-24442: -- Please avoid setting Fix Version which is usually set when

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2018-05-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497448#comment-16497448 ] Hyukjin Kwon commented on SPARK-21187: -- (y) > Complete support for remaining Spark data types in

[jira] [Updated] (SPARK-24444) Improve pandas_udf GROUPED_MAP docs to explain column assignment

2018-05-31 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-2: - Target Version/s: 2.3.1, 2.4.0 (was: 2.3.1) > Improve pandas_udf GROUPED_MAP docs to explain

[jira] [Commented] (SPARK-24396) Add Structured Streaming ForeachWriter for python

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497283#comment-16497283 ] Apache Spark commented on SPARK-24396: -- User 'tdas' has created a pull request for this issue:

[jira] [Commented] (SPARK-24446) Library path with special characters breaks Spark on YARN

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497275#comment-16497275 ] Apache Spark commented on SPARK-24446: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24446) Library path with special characters breaks Spark on YARN

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24446: Assignee: (was: Apache Spark) > Library path with special characters breaks Spark on

[jira] [Assigned] (SPARK-24446) Library path with special characters breaks Spark on YARN

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24446: Assignee: Apache Spark > Library path with special characters breaks Spark on YARN >

[jira] [Assigned] (SPARK-24416) Update configuration definition for spark.blacklist.killBlacklistedExecutors

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24416: Assignee: (was: Apache Spark) > Update configuration definition for

[jira] [Assigned] (SPARK-24416) Update configuration definition for spark.blacklist.killBlacklistedExecutors

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24416: Assignee: Apache Spark > Update configuration definition for

[jira] [Commented] (SPARK-24416) Update configuration definition for spark.blacklist.killBlacklistedExecutors

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497259#comment-16497259 ] Apache Spark commented on SPARK-24416: -- User 'redsanket' has created a pull request for this issue:

[jira] [Commented] (SPARK-21063) Spark return an empty result from remote hadoop cluster

2018-05-31 Thread Federico Lasa (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497250#comment-16497250 ] Federico Lasa commented on SPARK-21063: --- Affected as well on 2.0.0 (HDP 2.5.3) > Spark return an

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2018-05-31 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497244#comment-16497244 ] Bryan Cutler commented on SPARK-21187: -- Hi [~teddy.choi], MapType still needs some work to be done

[jira] [Assigned] (SPARK-24297) Change default value for spark.maxRemoteBlockSizeFetchToMem to be < 2GB

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24297: Assignee: (was: Apache Spark) > Change default value for

[jira] [Assigned] (SPARK-24297) Change default value for spark.maxRemoteBlockSizeFetchToMem to be < 2GB

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24297: Assignee: Apache Spark > Change default value for spark.maxRemoteBlockSizeFetchToMem to

[jira] [Commented] (SPARK-24297) Change default value for spark.maxRemoteBlockSizeFetchToMem to be < 2GB

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497219#comment-16497219 ] Apache Spark commented on SPARK-24297: -- User 'squito' has created a pull request for this issue:

[jira] [Resolved] (SPARK-24232) Allow referring to kubernetes secrets as env variable

2018-05-31 Thread Anirudh Ramanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-24232. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request

[jira] [Assigned] (SPARK-24232) Allow referring to kubernetes secrets as env variable

2018-05-31 Thread Anirudh Ramanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-24232: -- Assignee: Stavros Kontopoulos > Allow referring to kubernetes secrets as env

[jira] [Commented] (SPARK-24359) SPIP: ML Pipelines in R

2018-05-31 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497203#comment-16497203 ] Joseph K. Bradley commented on SPARK-24359: --- Clarification question: [~falaki] did you mean to

[jira] [Updated] (SPARK-24446) Library path with special characters breaks Spark on YARN

2018-05-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24446: --- Description: When YARN runs the application's main command, it does it like this: {code}

[jira] [Created] (SPARK-24446) Library path with special characters breaks Spark on YARN

2018-05-31 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-24446: -- Summary: Library path with special characters breaks Spark on YARN Key: SPARK-24446 URL: https://issues.apache.org/jira/browse/SPARK-24446 Project: Spark

[jira] [Commented] (SPARK-21896) Stack Overflow when window function nested inside aggregate function

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497160#comment-16497160 ] Apache Spark commented on SPARK-21896: -- User 'aokolnychyi' has created a pull request for this

[jira] [Commented] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show()

2018-05-31 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497087#comment-16497087 ] Reynold Xin commented on SPARK-24442: - Actually a pretty good idea. I've often wished there's a way

[jira] [Assigned] (SPARK-24445) Schema in json format for from_json in SQL

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24445: Assignee: Apache Spark > Schema in json format for from_json in SQL >

[jira] [Commented] (SPARK-24445) Schema in json format for from_json in SQL

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497084#comment-16497084 ] Apache Spark commented on SPARK-24445: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24445) Schema in json format for from_json in SQL

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24445: Assignee: (was: Apache Spark) > Schema in json format for from_json in SQL >

[jira] [Commented] (SPARK-24445) Schema in json format for from_json in SQL

2018-05-31 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497075#comment-16497075 ] Maxim Gekk commented on SPARK-24445: I am working on the ticket at the moment. > Schema in json

[jira] [Created] (SPARK-24445) Schema in json format for from_json in SQL

2018-05-31 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-24445: -- Summary: Schema in json format for from_json in SQL Key: SPARK-24445 URL: https://issues.apache.org/jira/browse/SPARK-24445 Project: Spark Issue Type:

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-05-31 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497070#comment-16497070 ] Ted Yu commented on SPARK-18057: I tend to agree with Cody. Just wondering if other people would accept

[jira] [Assigned] (SPARK-24444) Improve pandas_udf GROUPED_MAP docs to explain column assignment

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-2: Assignee: Apache Spark (was: Bryan Cutler) > Improve pandas_udf GROUPED_MAP docs to

[jira] [Assigned] (SPARK-24444) Improve pandas_udf GROUPED_MAP docs to explain column assignment

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-2: Assignee: Bryan Cutler (was: Apache Spark) > Improve pandas_udf GROUPED_MAP docs to

[jira] [Commented] (SPARK-24444) Improve pandas_udf GROUPED_MAP docs to explain column assignment

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497032#comment-16497032 ] Apache Spark commented on SPARK-2: -- User 'BryanCutler' has created a pull request for this

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-31 Thread Anirudh Ramanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497014#comment-16497014 ] Anirudh Ramanathan commented on SPARK-24434: Open to suggestions on what could be intuitive

[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-31 Thread Anirudh Ramanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497009#comment-16497009 ] Anirudh Ramanathan edited comment on SPARK-24434 at 5/31/18 6:54 PM: -

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-31 Thread Anirudh Ramanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497009#comment-16497009 ] Anirudh Ramanathan commented on SPARK-24434: Good point. I was basing my suggestion of JSON

[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-31 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497000#comment-16497000 ] Stavros Kontopoulos edited comment on SPARK-24434 at 5/31/18 6:49 PM:

[jira] [Created] (SPARK-24444) Improve pandas_udf GROUPED_MAP docs to explain column assignment

2018-05-31 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-2: Summary: Improve pandas_udf GROUPED_MAP docs to explain column assignment Key: SPARK-2 URL: https://issues.apache.org/jira/browse/SPARK-2 Project: Spark

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-31 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497000#comment-16497000 ] Stavros Kontopoulos commented on SPARK-24434: - [~foxish] JSON will be exposed to the user?

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-31 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496991#comment-16496991 ] Erik Erlandson commented on SPARK-24434: [~foxish] is there a technical (or ux) argument for

[jira] [Resolved] (SPARK-23900) format_number udf should take user specifed format as argument

2018-05-31 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23900. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21010

[jira] [Commented] (SPARK-18981) The last job hung when speculation is on

2018-05-31 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496990#comment-16496990 ] John Zhuge commented on SPARK-18981: Fixed by SPARK-11334. > The last job hung when speculation is

[jira] [Assigned] (SPARK-23900) format_number udf should take user specifed format as argument

2018-05-31 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23900: - Assignee: Yuming Wang > format_number udf should take user specifed format as argument

[jira] [Resolved] (SPARK-24397) Add TaskContext.getLocalProperties in Python

2018-05-31 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24397. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21437

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-31 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496967#comment-16496967 ] Yinan Li commented on SPARK-24434: -- [~foxish] that sounds like the approach to go.  > Support

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-05-31 Thread Anirudh Ramanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496962#comment-16496962 ] Anirudh Ramanathan commented on SPARK-24434: The way several custom APIs have done this

[jira] [Commented] (SPARK-24443) comparison should accept structurally-equal types

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496956#comment-16496956 ] Apache Spark commented on SPARK-24443: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24443) comparison should accept structurally-equal types

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24443: Assignee: Apache Spark (was: Wenchen Fan) > comparison should accept structurally-equal

[jira] [Assigned] (SPARK-24443) comparison should accept structurally-equal types

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24443: Assignee: Wenchen Fan (was: Apache Spark) > comparison should accept structurally-equal

[jira] [Updated] (SPARK-23874) Upgrade apache/arrow to 0.10.0

2018-05-31 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-23874: - Description: Version 0.10.0 will allow for the following improvements and bug fixes: * Allow

[jira] [Updated] (SPARK-24356) Duplicate strings in File.path managed by FileSegmentManagedBuffer

2018-05-31 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misha Dmitriev updated SPARK-24356: --- Attachment: dup-file-strings-details.png > Duplicate strings in File.path managed by

[jira] [Created] (SPARK-24443) comparison should accept structurally-equal types

2018-05-31 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-24443: --- Summary: comparison should accept structurally-equal types Key: SPARK-24443 URL: https://issues.apache.org/jira/browse/SPARK-24443 Project: Spark Issue Type:

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-05-31 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496924#comment-16496924 ] Ismael Juma commented on SPARK-18057: - Apache Kafka 2.0.0 will include KIP-266 and KAFKA-4879 has

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-05-31 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496910#comment-16496910 ] Marco Gaido commented on SPARK-24437: - Reproducing the issue is quite easy: you just need to run

[jira] [Commented] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show()

2018-05-31 Thread Andrew K Long (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496868#comment-16496868 ] Andrew K Long commented on SPARK-24442: --- Hey Sean,   Thanks for commenting!   "There are

[jira] [Comment Edited] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show()

2018-05-31 Thread Andrew K Long (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496868#comment-16496868 ] Andrew K Long edited comment on SPARK-24442 at 5/31/18 5:09 PM: Hey

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-05-31 Thread gagan taneja (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496866#comment-16496866 ] gagan taneja commented on SPARK-24437: -- No Dynamic allocation. Also this is an issue with Driver

[jira] [Updated] (SPARK-24381) Improve Unit Test Coverage of NOT IN subqueries

2018-05-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24381: --- Fix Version/s: (was: 2.3.1) 2.4.0 > Improve Unit Test Coverage of

[jira] [Updated] (SPARK-24381) Improve Unit Test Coverage of NOT IN subqueries

2018-05-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24381: --- Fix Version/s: (was: 2.3.2) 2.3.1 > Improve Unit Test Coverage of

[jira] [Assigned] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-24414: -- Assignee: Marcelo Vanzin > Stages page doesn't show all task attempts when failures

[jira] [Resolved] (SPARK-24414) Stages page doesn't show all task attempts when failures

2018-05-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24414. Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by

[jira] [Commented] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show()

2018-05-31 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496850#comment-16496850 ] Sean Owen commented on SPARK-24442: --- There are already method arguments for truncation and max rows,

[jira] [Commented] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show()

2018-05-31 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496840#comment-16496840 ] Sean Owen commented on SPARK-24442: --- Have a look at [https://spark.apache.org/contributing.html] first

[jira] [Updated] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show()

2018-05-31 Thread Andrew K Long (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew K Long updated SPARK-24442: -- Attachment: spark-adjustable-display-size.diff > Add configuration parameter to adjust the

[jira] [Created] (SPARK-24442) Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show()

2018-05-31 Thread Andrew K Long (JIRA)
Andrew K Long created SPARK-24442: - Summary: Add configuration parameter to adjust the numbers of records and the charters per row before truncation when a user runs.show() Key: SPARK-24442 URL:

[jira] [Assigned] (SPARK-24441) Expose total size of states in HDFSBackedStateStoreProvider

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24441: Assignee: Apache Spark > Expose total size of states in HDFSBackedStateStoreProvider >

[jira] [Assigned] (SPARK-24441) Expose total size of states in HDFSBackedStateStoreProvider

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24441: Assignee: (was: Apache Spark) > Expose total size of states in

[jira] [Commented] (SPARK-24441) Expose total size of states in HDFSBackedStateStoreProvider

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496665#comment-16496665 ] Apache Spark commented on SPARK-24441: -- User 'HeartSaVioR' has created a pull request for this

[jira] [Assigned] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22151: Assignee: Apache Spark > PYTHONPATH not picked up from the spark.yarn.appMasterEnv

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496648#comment-16496648 ] Apache Spark commented on SPARK-22151: -- User 'pgandhi999' has created a pull request for this

[jira] [Assigned] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22151: Assignee: (was: Apache Spark) > PYTHONPATH not picked up from the

[jira] [Created] (SPARK-24441) Expose total size of states in HDFSBackedStateStoreProvider

2018-05-31 Thread Jungtaek Lim (JIRA)
Jungtaek Lim created SPARK-24441: Summary: Expose total size of states in HDFSBackedStateStoreProvider Key: SPARK-24441 URL: https://issues.apache.org/jira/browse/SPARK-24441 Project: Spark

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0

2018-05-31 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496633#comment-16496633 ] Cody Koeninger commented on SPARK-18057: I'd just modify KafkaTestUtils to match the way things

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-05-31 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496598#comment-16496598 ] Marco Gaido commented on SPARK-24437: - Do you have dynamic allocation enabled? > Memory leak in

[jira] [Commented] (SPARK-23936) High-order function: map_concat(map1, map2, ..., mapN) → map

2018-05-31 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496599#comment-16496599 ] Bruce Robbins commented on SPARK-23936: --- tl;dr version: Spark's Map type allows duplicates.

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-05-31 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496585#comment-16496585 ] Marco Gaido commented on SPARK-24437: - I just remembered that I started working on this some time

[jira] [Resolved] (SPARK-24146) spark.ml parity for sequential pattern mining - PrefixSpan: Python API

2018-05-31 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24146. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21265

[jira] [Commented] (SPARK-23754) StopIterator exception in Python UDF results in partial result

2018-05-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496470#comment-16496470 ] Apache Spark commented on SPARK-23754: -- User 'e-dorigatti' has created a pull request for this

[jira] [Commented] (SPARK-24427) Spark 2.2 - Exception occurred while saving table in spark. Multiple sources found for parquet

2018-05-31 Thread Ashok Rai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496367#comment-16496367 ] Ashok Rai commented on SPARK-24427: --- Ok. Please let me know the mailing list. I will send my error

[jira] [Resolved] (SPARK-24427) Spark 2.2 - Exception occurred while saving table in spark. Multiple sources found for parquet

2018-05-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24427. -- Resolution: Invalid Here, https://spark.apache.org/community.html Let's leave this closed

[jira] [Created] (SPARK-24440) When use constant as column we may get wrong answer versus impala

2018-05-31 Thread zhoukang (JIRA)
zhoukang created SPARK-24440: Summary: When use constant as column we may get wrong answer versus impala Key: SPARK-24440 URL: https://issues.apache.org/jira/browse/SPARK-24440 Project: Spark

[jira] [Commented] (SPARK-24427) Spark 2.2 - Exception occurred while saving table in spark. Multiple sources found for parquet

2018-05-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496359#comment-16496359 ] Hyukjin Kwon commented on SPARK-24427: -- The log messages basically mean it detected multiple

[jira] [Commented] (SPARK-24427) Spark 2.2 - Exception occurred while saving table in spark. Multiple sources found for parquet

2018-05-31 Thread Ashok Rai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496358#comment-16496358 ] Ashok Rai commented on SPARK-24427: --- I have not specified any version in spark-submit. I am using

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

2018-05-31 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496331#comment-16496331 ] Marco Gaido commented on SPARK-24437: - I remember another JIRA about this. Anyway, this is indeed a

[jira] [Commented] (SPARK-23266) Matrix Inversion on BlockMatrix

2018-05-31 Thread Chandan Misra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496306#comment-16496306 ] Chandan Misra commented on SPARK-23266: --- I want to add this feature in any of the coming versions.

[jira] [Commented] (SPARK-23904) Big execution plan cause OOM

2018-05-31 Thread Ruben Berenguel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496290#comment-16496290 ] Ruben Berenguel commented on SPARK-23904: - Thanks [~igreenfi], still at it then :) > Big

[jira] [Commented] (SPARK-23257) Implement Kerberos Support in Kubernetes resource manager

2018-05-31 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496284#comment-16496284 ] Rob Vesse commented on SPARK-23257: --- [~ifilonenko] Any updates on this? We're currently using the

[jira] [Comment Edited] (SPARK-23904) Big execution plan cause OOM

2018-05-31 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496272#comment-16496272 ] Izek Greenfield edited comment on SPARK-23904 at 5/31/18 8:43 AM: --

  1   2   >