[jira] [Commented] (SPARK-19442) Unable to add column to the dataset using Dataset.WithColumn() api

2017-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865256#comment-15865256 ] Hyukjin Kwon commented on SPARK-19442: -- How about something like this? {code} import

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2017-02-13 Thread Deenbandhu Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865229#comment-15865229 ] Deenbandhu Agarwal commented on SPARK-15716: What happened to this issue ? > Memory usage

[jira] [Resolved] (SPARK-19585) Fix the cacheTable and uncacheTable API call in the SQL Programming Guide

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19585. - Resolution: Fixed Assignee: Sunitha Kambhampati Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-19442) Unable to add column to the dataset using Dataset.WithColumn() api

2017-02-13 Thread Navya Krishnappa (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865135#comment-15865135 ] Navya Krishnappa commented on SPARK-19442: -- If the source file has 3 columns NameAge

[jira] [Commented] (SPARK-19583) CTAS for data source tables with an created location does not work

2017-02-13 Thread Song Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865111#comment-15865111 ] Song Jun commented on SPARK-19583: -- ok, I'd like to take this one, thanks a lot! > CTAS for data source

[jira] [Assigned] (SPARK-19591) Add sample weights to decision trees

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19591: Assignee: Apache Spark > Add sample weights to decision trees >

[jira] [Commented] (SPARK-9478) Add sample weights to Random Forest

2017-02-13 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865079#comment-15865079 ] Seth Hendrickson commented on SPARK-9478: - [~josephkb] Done. Thanks for your feedback on sampling!

[jira] [Commented] (SPARK-19591) Add sample weights to decision trees

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865078#comment-15865078 ] Apache Spark commented on SPARK-19591: -- User 'sethah' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19591) Add sample weights to decision trees

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19591: Assignee: (was: Apache Spark) > Add sample weights to decision trees >

[jira] [Created] (SPARK-19591) Add sample weights to decision trees

2017-02-13 Thread Seth Hendrickson (JIRA)
Seth Hendrickson created SPARK-19591: Summary: Add sample weights to decision trees Key: SPARK-19591 URL: https://issues.apache.org/jira/browse/SPARK-19591 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-19590) Update the document for QuantileDiscretizer in pyspark

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19590: Assignee: (was: Apache Spark) > Update the document for QuantileDiscretizer in

[jira] [Commented] (SPARK-19590) Update the document for QuantileDiscretizer in pyspark

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865054#comment-15865054 ] Apache Spark commented on SPARK-19590: -- User 'VinceShieh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19590) Update the document for QuantileDiscretizer in pyspark

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19590: Assignee: Apache Spark > Update the document for QuantileDiscretizer in pyspark >

[jira] [Commented] (SPARK-19568) Must include class/method documentation for CRAN check

2017-02-13 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865022#comment-15865022 ] Shivaram Venkataraman commented on SPARK-19568: --- Is it possible to add this as a part of

[jira] [Comment Edited] (SPARK-19038) Can't find keytab file when using Hive catalog

2017-02-13 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865013#comment-15865013 ] Ruslan Dautkhanov edited comment on SPARK-19038 at 2/14/17 4:12 AM:

[jira] [Commented] (SPARK-19588) Allow putting keytab file to HDFS location specified in spark.yarn.keytab

2017-02-13 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865020#comment-15865020 ] Ruslan Dautkhanov commented on SPARK-19588: --- Our corporate sshd's are integrated with Active

[jira] [Created] (SPARK-19590) Update the document for QuantileDiscretizer in pyspark

2017-02-13 Thread Vincent (JIRA)
Vincent created SPARK-19590: --- Summary: Update the document for QuantileDiscretizer in pyspark Key: SPARK-19590 URL: https://issues.apache.org/jira/browse/SPARK-19590 Project: Spark Issue Type:

[jira] [Commented] (SPARK-19038) Can't find keytab file when using Hive catalog

2017-02-13 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865013#comment-15865013 ] Ruslan Dautkhanov commented on SPARK-19038: --- Another possible workaround is to pass principal

[jira] [Commented] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

2017-02-13 Thread KaiXu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865010#comment-15865010 ] KaiXu commented on SPARK-19528: --- nodemanager log see below exception, is that helpful? 2017-02-09

[jira] [Commented] (SPARK-19588) Allow putting keytab file to HDFS location specified in spark.yarn.keytab

2017-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865005#comment-15865005 ] Marcelo Vanzin commented on SPARK-19588: >From a feature perspective this is probably ok, but I'm

[jira] [Assigned] (SPARK-19589) Removal of SQLGEN files

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19589: Assignee: Apache Spark (was: Xiao Li) > Removal of SQLGEN files >

[jira] [Commented] (SPARK-19589) Removal of SQLGEN files

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864999#comment-15864999 ] Apache Spark commented on SPARK-19589: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19589) Removal of SQLGEN files

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19589: Assignee: Xiao Li (was: Apache Spark) > Removal of SQLGEN files >

[jira] [Commented] (SPARK-19038) Can't find keytab file when using Hive catalog

2017-02-13 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864995#comment-15864995 ] Ruslan Dautkhanov commented on SPARK-19038: --- Thank you [~jerryshao] > Can't find keytab file

[jira] [Updated] (SPARK-19589) Removal of SQLGEN files

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19589: Description: SQLGen is removed. Thus, the generated files should be removed too. > Removal of SQLGEN

[jira] [Created] (SPARK-19589) Removal of SQLGEN files

2017-02-13 Thread Xiao Li (JIRA)
Xiao Li created SPARK-19589: --- Summary: Removal of SQLGEN files Key: SPARK-19589 URL: https://issues.apache.org/jira/browse/SPARK-19589 Project: Spark Issue Type: Improvement Components:

[jira] [Commented] (SPARK-16026) Cost-based Optimizer framework

2017-02-13 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864992#comment-15864992 ] Ruslan Dautkhanov commented on SPARK-16026: --- [~ioana-delaney], (y) > Cost-based Optimizer

[jira] [Resolved] (SPARK-19539) CREATE TEMPORARY TABLE needs to avoid existing temp view

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19539. - Resolution: Fixed Assignee: Xin Wu Fix Version/s: 2.2.0 > CREATE TEMPORARY TABLE needs

[jira] [Assigned] (SPARK-19115) SparkSQL unsupports the command " create external table if not exist new_tbl like old_tbl location '/warehouse/new_tbl' "

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-19115: --- Assignee: Xiaochen Ouyang (was: Xiao Li) > SparkSQL unsupports the command " create external table

[jira] [Resolved] (SPARK-19115) SparkSQL unsupports the command " create external table if not exist new_tbl like old_tbl location '/warehouse/new_tbl' "

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19115. - Resolution: Fixed Fix Version/s: 2.2.0 > SparkSQL unsupports the command " create external table

[jira] [Commented] (SPARK-19038) Can't find keytab file when using Hive catalog

2017-02-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864983#comment-15864983 ] Saisai Shao commented on SPARK-19038: - I think the issue you met is the same as this JIRA mentioned,

[jira] [Resolved] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-13219. - Resolution: Duplicate > Pushdown predicate propagation in SparkSQL with join >

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864969#comment-15864969 ] Xiao Li commented on SPARK-13219: - Actually, this PR has already been resolved. We can close it now.

[jira] [Commented] (SPARK-19038) Can't find keytab file when using Hive catalog

2017-02-13 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864968#comment-15864968 ] Ruslan Dautkhanov commented on SPARK-19038: --- [~jerryshao] PR 16482 is for a different issue

[jira] [Updated] (SPARK-19588) Allow putting keytab file to HDFS location specified in spark.yarn.keytab

2017-02-13 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SPARK-19588: -- Summary: Allow putting keytab file to HDFS location specified in spark.yarn.keytab

[jira] [Created] (SPARK-19588) Allow putting keytab files specified by

2017-02-13 Thread Ruslan Dautkhanov (JIRA)
Ruslan Dautkhanov created SPARK-19588: - Summary: Allow putting keytab files specified by Key: SPARK-19588 URL: https://issues.apache.org/jira/browse/SPARK-19588 Project: Spark Issue

[jira] [Commented] (SPARK-19532) [Core]`DataStreamer for file` threads of DFSOutputStream leak if set `spark.speculation` to true

2017-02-13 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864953#comment-15864953 ] StanZhai commented on SPARK-19532: -- I can reproduce this by split our online data to the production test

[jira] [Updated] (SPARK-19587) Disallow when sort columns are part of partitioning columns

2017-02-13 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tejas Patil updated SPARK-19587: Description: This came up in discussion at

[jira] [Updated] (SPARK-19586) Incorrect push down filter for double negative in SQL

2017-02-13 Thread Everett Anderson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Everett Anderson updated SPARK-19586: - Description: Opening this as it's a somewhat serious issue in the 2.0.x tree in case

[jira] [Created] (SPARK-19586) Incorrect push down filter for double negative in SQL

2017-02-13 Thread Everett Anderson (JIRA)
Everett Anderson created SPARK-19586: Summary: Incorrect push down filter for double negative in SQL Key: SPARK-19586 URL: https://issues.apache.org/jira/browse/SPARK-19586 Project: Spark

[jira] [Created] (SPARK-19587) Disallow when sort columns are part of partitioning columns

2017-02-13 Thread Tejas Patil (JIRA)
Tejas Patil created SPARK-19587: --- Summary: Disallow when sort columns are part of partitioning columns Key: SPARK-19587 URL: https://issues.apache.org/jira/browse/SPARK-19587 Project: Spark

[jira] [Commented] (SPARK-14503) spark.ml Scala API for FPGrowth

2017-02-13 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864872#comment-15864872 ] Nick Pentreath commented on SPARK-14503: Seems {{PrefixSpan}} even takes different input:

[jira] [Commented] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

2017-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864850#comment-15864850 ] Shixiong Zhu commented on SPARK-19528: -- The external shuffle service runs inside the node manager.

[jira] [Commented] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

2017-02-13 Thread KaiXu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864844#comment-15864844 ] KaiXu commented on SPARK-19528: --- Thanks [~zsxwing] for the comment, this issue can be occurred frequently,

[jira] [Commented] (SPARK-12661) Drop Python 2.6 support in PySpark

2017-02-13 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864834#comment-15864834 ] Josh Rosen commented on SPARK-12661: IIRC the Jenkins work was to make sure that we have the new

[jira] [Commented] (SPARK-12661) Drop Python 2.6 support in PySpark

2017-02-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864824#comment-15864824 ] holdenk commented on SPARK-12661: - Coming back to this since Sean's thread reminded me - who should we

[jira] [Commented] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

2017-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864814#comment-15864814 ] Shixiong Zhu commented on SPARK-19528: -- This error is because the executor cannot connect to the

[jira] [Commented] (SPARK-19579) spark-submit fails to run Kafka Stream python script

2017-02-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864811#comment-15864811 ] Saisai Shao commented on SPARK-19579: - Spark Streaming Kafka Python API doesn't support Kafka 0.10

[jira] [Commented] (SPARK-12957) Derive and propagate data constrains in logical plan

2017-02-13 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864808#comment-15864808 ] Sameer Agarwal commented on SPARK-12957: [~ndimiduk] yes, I believe you should be able to observe

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2017-02-13 Thread Nick Dimiduk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864798#comment-15864798 ] Nick Dimiduk commented on SPARK-13219: -- Now that all the subtasks on SPARK-12957 are resolved, where

[jira] [Commented] (SPARK-12957) Derive and propagate data constrains in logical plan

2017-02-13 Thread Nick Dimiduk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864796#comment-15864796 ] Nick Dimiduk commented on SPARK-12957: -- I'm trying to understand the current state of

[jira] [Updated] (SPARK-19517) KafkaSource fails to initialize partition offsets

2017-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19517: - Attachment: SPARK-19517ProposalforfixingKafkaOffsetMetadata.pdf > KafkaSource fails to

[jira] [Updated] (SPARK-19517) KafkaSource fails to initialize partition offsets

2017-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19517: - Attachment: (was: SPARK-19517ProposalforfixingKafkaOffsetMetadata.pdf) > KafkaSource fails

[jira] [Updated] (SPARK-19517) KafkaSource fails to initialize partition offsets

2017-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19517: - Attachment: SPARK-19517ProposalforfixingKafkaOffsetMetadata.pdf > KafkaSource fails to

[jira] [Assigned] (SPARK-19585) Fix the cacheTable and uncacheTable API call in the SQL Programming Guide

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19585: Assignee: Apache Spark > Fix the cacheTable and uncacheTable API call in the SQL

[jira] [Commented] (SPARK-19585) Fix the cacheTable and uncacheTable API call in the SQL Programming Guide

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864759#comment-15864759 ] Apache Spark commented on SPARK-19585: -- User 'skambha' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19585) Fix the cacheTable and uncacheTable API call in the SQL Programming Guide

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19585: Assignee: (was: Apache Spark) > Fix the cacheTable and uncacheTable API call in the

[jira] [Created] (SPARK-19585) Fix the cacheTable and uncacheTable API call in the SQL Programming Guide

2017-02-13 Thread Sunitha Kambhampati (JIRA)
Sunitha Kambhampati created SPARK-19585: --- Summary: Fix the cacheTable and uncacheTable API call in the SQL Programming Guide Key: SPARK-19585 URL: https://issues.apache.org/jira/browse/SPARK-19585

[jira] [Assigned] (SPARK-19584) Update Structured Streaming documentation to include Batch query description

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19584: Assignee: (was: Apache Spark) > Update Structured Streaming documentation to include

[jira] [Assigned] (SPARK-19584) Update Structured Streaming documentation to include Batch query description

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19584: Assignee: Apache Spark > Update Structured Streaming documentation to include Batch query

[jira] [Commented] (SPARK-19584) Update Structured Streaming documentation to include Batch query description

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864734#comment-15864734 ] Apache Spark commented on SPARK-19584: -- User 'tcondie' has created a pull request for this issue:

[jira] [Created] (SPARK-19584) Update Structured Streaming documentation to include Batch query description

2017-02-13 Thread Tyson Condie (JIRA)
Tyson Condie created SPARK-19584: Summary: Update Structured Streaming documentation to include Batch query description Key: SPARK-19584 URL: https://issues.apache.org/jira/browse/SPARK-19584

[jira] [Commented] (SPARK-16026) Cost-based Optimizer framework

2017-02-13 Thread Ioana Delaney (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864708#comment-15864708 ] Ioana Delaney commented on SPARK-16026: --- [~Tagar], Our team is currently working on the support for

[jira] [Resolved] (SPARK-19429) Column.__getitem__ should support slice arguments

2017-02-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-19429. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16771

[jira] [Assigned] (SPARK-19429) Column.__getitem__ should support slice arguments

2017-02-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-19429: --- Assignee: Maciej Szymkiewicz > Column.__getitem__ should support slice arguments >

[jira] [Resolved] (SPARK-19520) WAL should not be encrypted

2017-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19520. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-9279) Spark Master Refuses to Bind WebUI to a Privileged Port

2017-02-13 Thread Julian Gamble (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864595#comment-15864595 ] Julian Gamble commented on SPARK-9279: -- This is a serious issue in Cloud/Docker environments when

[jira] [Updated] (SPARK-19529) TransportClientFactory.createClient() shouldn't call awaitUninterruptibly()

2017-02-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-19529: --- Target Version/s: 1.6.3, 2.0.3, 2.1.1, 2.2.0 (was: 2.0.3, 2.1.1, 2.2.0) >

[jira] [Commented] (SPARK-19529) TransportClientFactory.createClient() shouldn't call awaitUninterruptibly()

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864574#comment-15864574 ] Apache Spark commented on SPARK-19529: -- User 'liancheng' has created a pull request for this issue:

[jira] [Commented] (SPARK-15857) Add Caller Context in Spark

2017-02-13 Thread Weiqing Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864491#comment-15864491 ] Weiqing Yang commented on SPARK-15857: -- Thanks. [~zsxwing] > Add Caller Context in Spark >

[jira] [Resolved] (SPARK-15857) Add Caller Context in Spark

2017-02-13 Thread Weiqing Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang resolved SPARK-15857. -- Resolution: Fixed > Add Caller Context in Spark > --- > >

[jira] [Resolved] (SPARK-19435) Type coercion between ArrayTypes

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19435. - Resolution: Fixed Fix Version/s: 2.2.0 > Type coercion between ArrayTypes >

[jira] [Assigned] (SPARK-19435) Type coercion between ArrayTypes

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-19435: --- Assignee: Hyukjin Kwon > Type coercion between ArrayTypes > > >

[jira] [Commented] (SPARK-19583) CTAS for data source tables with an created location does not work

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864367#comment-15864367 ] Xiao Li commented on SPARK-19583: - This works well for hive tables. We should make it work. cc

[jira] [Comment Edited] (SPARK-19583) CTAS for data source tables with an created location does not work

2017-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864367#comment-15864367 ] Xiao Li edited comment on SPARK-19583 at 2/13/17 8:28 PM: -- This works well for

[jira] [Created] (SPARK-19583) CTAS for data source tables with an created location does not work

2017-02-13 Thread Xiao Li (JIRA)
Xiao Li created SPARK-19583: --- Summary: CTAS for data source tables with an created location does not work Key: SPARK-19583 URL: https://issues.apache.org/jira/browse/SPARK-19583 Project: Spark

[jira] [Commented] (SPARK-15857) Add Caller Context in Spark

2017-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864333#comment-15864333 ] Shixiong Zhu commented on SPARK-15857: -- Can we close this one now? > Add Caller Context in Spark >

[jira] [Updated] (SPARK-17714) ClassCircularityError is thrown when using org.apache.spark.util.Utils.classForName 

2017-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17714: - Component/s: Spark Core > ClassCircularityError is thrown when using >

[jira] [Resolved] (SPARK-17714) ClassCircularityError is thrown when using org.apache.spark.util.Utils.classForName 

2017-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17714. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-19553) Add GroupedData.countApprox()

2017-02-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864326#comment-15864326 ] Michael Armbrust commented on SPARK-19553: -- It seems like there are a couple of distinct feature

[jira] [Resolved] (SPARK-19542) Delete the temp checkpoint if a query is stopped without errors

2017-02-13 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-19542. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Delete the temp

[jira] [Commented] (SPARK-6883) Fork pyspark's cloudpickle as a separate dependency

2017-02-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864309#comment-15864309 ] holdenk commented on SPARK-6883: Let's consider re-opening this for discussion - do we maybe want to just

[jira] [Reopened] (SPARK-6883) Fork pyspark's cloudpickle as a separate dependency

2017-02-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reopened SPARK-6883: > Fork pyspark's cloudpickle as a separate dependency > --- > >

[jira] [Commented] (SPARK-19501) Slow checking if there are many spark.yarn.jars, which are already on HDFS

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864286#comment-15864286 ] Apache Spark commented on SPARK-19501: -- User 'jongwook' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19501) Slow checking if there are many spark.yarn.jars, which are already on HDFS

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19501: Assignee: Apache Spark > Slow checking if there are many spark.yarn.jars, which are

[jira] [Assigned] (SPARK-19501) Slow checking if there are many spark.yarn.jars, which are already on HDFS

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19501: Assignee: (was: Apache Spark) > Slow checking if there are many spark.yarn.jars,

[jira] [Commented] (SPARK-18871) New test cases for IN/NOT IN subquery

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864229#comment-15864229 ] Apache Spark commented on SPARK-18871: -- User 'kevinyu98' has created a pull request for this issue:

[jira] [Updated] (SPARK-19529) TransportClientFactory.createClient() shouldn't call awaitUninterruptibly()

2017-02-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-19529: --- Target Version/s: 2.0.3, 2.1.1, 2.2.0 (was: 2.0.3, 2.1.1) > TransportClientFactory.createClient()

[jira] [Resolved] (SPARK-19427) UserDefinedFunction should support data types strings

2017-02-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-19427. - Resolution: Fixed Assignee: Maciej Szymkiewicz Fix Version/s: 2.2.0 Thanks for doing all

[jira] [Created] (SPARK-19582) DataFrameReader conceptually inadequate

2017-02-13 Thread James Q. Arnold (JIRA)
James Q. Arnold created SPARK-19582: --- Summary: DataFrameReader conceptually inadequate Key: SPARK-19582 URL: https://issues.apache.org/jira/browse/SPARK-19582 Project: Spark Issue Type:

[jira] [Commented] (SPARK-19163) Lazy creation of the _judf

2017-02-13 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864110#comment-15864110 ] Maciej Szymkiewicz commented on SPARK-19163: [~holdenk] I see you've sorted out Jira

[jira] [Resolved] (SPARK-19506) Missing warnings import in pyspark.ml.util

2017-02-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-19506. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 Thanks for reporting and fixing

[jira] [Assigned] (SPARK-19506) Missing warnings import in pyspark.ml.util

2017-02-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-19506: --- Assignee: Maciej Szymkiewicz > Missing warnings import in pyspark.ml.util >

[jira] [Commented] (SPARK-19571) appveyor windows tests are failing

2017-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864096#comment-15864096 ] Hyukjin Kwon commented on SPARK-19571: -- Oh, I overlooked and I thought it is just because of

[jira] [Commented] (SPARK-19578) Poor pyspark performance + incorrect UI input-size metrics

2017-02-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864097#comment-15864097 ] Nicholas Chammas commented on SPARK-19578: -- I'm seeing the same thing too. You can get a much

[jira] [Resolved] (SPARK-19342) Datatype tImestamp is converted to numeric in collect method

2017-02-13 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-19342. -- Resolution: Fixed > Datatype tImestamp is converted to numeric in collect method >

[jira] [Commented] (SPARK-19571) appveyor windows tests are failing

2017-02-13 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864033#comment-15864033 ] Shivaram Venkataraman commented on SPARK-19571: --- cc [~hyukjin.kwon] > appveyor windows

[jira] [Commented] (SPARK-19553) Add GroupedData.countApprox()

2017-02-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864000#comment-15864000 ] Nicholas Chammas commented on SPARK-19553: -- Quick API question for you [~marmbrus]: Is this

[jira] [Commented] (SPARK-19514) Range is not interruptible

2017-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863946#comment-15863946 ] Apache Spark commented on SPARK-19514: -- User 'ala' has created a pull request for this issue:

[jira] [Commented] (SPARK-19578) Poor pyspark performance + incorrect UI input-size metrics

2017-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863829#comment-15863829 ] Sean Owen commented on SPARK-19578: --- I see something similar. It takes a couple seconds in Scala but

  1   2   >