[jira] [Updated] (SPARK-12913) Reimplement stat functions as declarative function

2016-01-28 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-12913: --- Summary: Reimplement stat functions as declarative function (was: Reimplement stddev/variance as

[jira] [Resolved] (SPARK-13050) Scalatest tags fail builds with the addition of the sketch module

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-13050. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 10954

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2016-01-28 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121792#comment-15121792 ] Michel Lemay commented on SPARK-10528: -- Your issue is about deleting temp files at time of shutdown.

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2016-01-28 Thread Andres Ivaldi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121822#comment-15121822 ] Andres Ivaldi commented on SPARK-10528: --- You are right, I didn't pay attention to the beginning of

[jira] [Commented] (SPARK-13074) Add getPersistentRDDs() API to JavaSparkContext

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123111#comment-15123111 ] Apache Spark commented on SPARK-13074: -- User 'flyjy' has created a pull request for this issue:

[jira] [Commented] (SPARK-13031) Improve test coverage for whole stage codegen

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123057#comment-15123057 ] Apache Spark commented on SPARK-13031: -- User 'davies' has created a pull request for this issue:

[jira] [Assigned] (SPARK-13074) Add getPersistentRDDs() API to JavaSparkContext

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13074: Assignee: Apache Spark > Add getPersistentRDDs() API to JavaSparkContext >

[jira] [Assigned] (SPARK-13074) Add getPersistentRDDs() API to JavaSparkContext

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13074: Assignee: (was: Apache Spark) > Add getPersistentRDDs() API to JavaSparkContext >

[jira] [Commented] (SPARK-5891) Add Binarizer

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123017#comment-15123017 ] Apache Spark commented on SPARK-5891: - User 'seddonm1' has created a pull request for this issue:

[jira] [Commented] (SPARK-3374) Spark on Yarn remove deprecated configs for 2.0

2016-01-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123043#comment-15123043 ] Saisai Shao commented on SPARK-3374: +1 to address this in the upcoming 2.0 release. 1. Currently in

[jira] [Commented] (SPARK-12982) SQLContext: temporary table registration does not accept valid identifier

2016-01-28 Thread Jayadevan M (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123023#comment-15123023 ] Jayadevan M commented on SPARK-12982: - I tried your scenario in spark 1.6.0. But I can register table

[jira] [Resolved] (SPARK-12968) Implement command to set current database

2016-01-28 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12968. - Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.0.0 > Implement

[jira] [Resolved] (SPARK-13067) DataFrameSuite.simple explode fail locally

2016-01-28 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-13067. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 10970

[jira] [Created] (SPARK-13074) Add getPersistentRDDs() API to JavaSparkContext

2016-01-28 Thread Junyang Shen (JIRA)
Junyang Shen created SPARK-13074: Summary: Add getPersistentRDDs() API to JavaSparkContext Key: SPARK-13074 URL: https://issues.apache.org/jira/browse/SPARK-13074 Project: Spark Issue Type:

[jira] [Commented] (SPARK-12725) SQL generation suffers from name conficts introduced by some analysis rules

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122013#comment-15122013 ] Cheng Lian commented on SPARK-12725: One possible solution I was thinking about is that we can add a

[jira] [Commented] (SPARK-12725) SQL generation suffers from name conficts introduced by some analysis rules

2016-01-28 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122016#comment-15122016 ] Michael Armbrust commented on SPARK-12725: -- Why don't we just add a flag to AttributeReference

[jira] [Updated] (SPARK-13064) api/v1/application/jobs/attempt lacks "attempId" field for spark-shell

2016-01-28 Thread Zhuo Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhuo Liu updated SPARK-13064: - Description: For any application launches with spark-shell will not have attemptId field in their rest

[jira] [Updated] (SPARK-13064) api/v1/application/jobs/attempt lacks "attempId" field for spark-shell

2016-01-28 Thread Zhuo Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhuo Liu updated SPARK-13064: - Description: For any application launches with spark-shell will not have attemptId field in their rest

[jira] [Commented] (SPARK-13053) Rectify ignored tests in InternalAccumulatorSuite

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122081#comment-15122081 ] Apache Spark commented on SPARK-13053: -- User 'andrewor14' has created a pull request for this issue:

[jira] [Commented] (SPARK-13064) api/v1/application/jobs/attempt lacks "attempId" field for spark-shell

2016-01-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122086#comment-15122086 ] Marcelo Vanzin commented on SPARK-13064: The real reason it's not set is because

[jira] [Assigned] (SPARK-13053) Rectify ignored tests in InternalAccumulatorSuite

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13053: Assignee: Andrew Or (was: Apache Spark) > Rectify ignored tests in

[jira] [Assigned] (SPARK-13053) Rectify ignored tests in InternalAccumulatorSuite

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13053: Assignee: Apache Spark (was: Andrew Or) > Rectify ignored tests in

[jira] [Commented] (SPARK-12723) Comprehensive SQL generation support for expressions

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122102#comment-15122102 ] Cheng Lian commented on SPARK-12723: Most of this issue is probably fixed in SPARK-12799 and PR

[jira] [Commented] (SPARK-12727) SQL generation support for distinct aggregation patterns that fit DistinctAggregationRewriter analysis rule

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122124#comment-15122124 ] Cheng Lian commented on SPARK-12727: It would be nice if we can recover the original distinct

[jira] [Commented] (SPARK-12739) Details of batch in Streaming tab uses two Duration columns

2016-01-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122187#comment-15122187 ] Shixiong Zhu commented on SPARK-12739: -- [~jlaskowski] could you submit a PR soon? Should be pretty

[jira] [Updated] (SPARK-11012) Canonicalize view definitions

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-11012: --- Description: In SPARK-10337, we added the first step of supporting view natively, which is

[jira] [Updated] (SPARK-12719) SQL generation support for generators (including UDTF)

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-12719: --- Description: {{HiveCompatibilitySuite}} can be useful for bootstrapping test coverage. Please refer

[jira] [Updated] (SPARK-12721) SQL generation support for script transformation

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-12721: --- Description: {{HiveCompatibilitySuite}} can be useful for bootstrapping test coverage. Please refer

[jira] [Updated] (SPARK-12720) SQL generation support for cube, rollup, and grouping set

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-12720: --- Description: {{HiveCompatibilitySuite}} can be useful for bootstrapping test coverage. Please refer

[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input

2016-01-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122169#comment-15122169 ] Shixiong Zhu commented on SPARK-13027: -- Could you submit a PR? > Add API for updateStateByKey to

[jira] [Commented] (SPARK-13039) Spark Streaming with Mesos shutdown without any reason on logs

2016-01-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122166#comment-15122166 ] Shixiong Zhu commented on SPARK-13039: -- May be killed by Mesos because exceeding the memory

[jira] [Commented] (SPARK-13064) api/v1/application/jobs/attempt lacks "attempId" field for spark-shell

2016-01-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122077#comment-15122077 ] Thomas Graves commented on SPARK-13064: --- thanks [~vanzin] We can just have the rest api add it as

[jira] [Updated] (SPARK-12718) SQL generation support for window functions

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-12718: --- Description: {{HiveWindowFunctionQuerySuite}} and {{HiveWindowFunctionQueryFileSuite}} can be useful

[jira] [Resolved] (SPARK-12926) SQLContext to display warning message when non-sql configs are being set

2016-01-28 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-12926. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 10849

[jira] [Issue Comment Deleted] (SPARK-6328) Python API for StreamingListener

2016-01-28 Thread Daniel Jalova (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Jalova updated SPARK-6328: - Comment: was deleted (was: @tdas Could you change the assignee to me please?) > Python API for

[jira] [Resolved] (SPARK-13031) Improve test coverage for whole stage codegen

2016-01-28 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-13031. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 10944

[jira] [Commented] (SPARK-13067) DataFrameSuite.simple explode fail locally

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122501#comment-15122501 ] Apache Spark commented on SPARK-13067: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-13067) DataFrameSuite.simple explode fail locally

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13067: Assignee: Apache Spark > DataFrameSuite.simple explode fail locally >

[jira] [Assigned] (SPARK-13067) DataFrameSuite.simple explode fail locally

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13067: Assignee: (was: Apache Spark) > DataFrameSuite.simple explode fail locally >

[jira] [Commented] (SPARK-13066) Specify types for per-model/estimator params in ML to allow automatic type conversion

2016-01-28 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122503#comment-15122503 ] Seth Hendrickson commented on SPARK-13066: -- Posted this on a [previous

[jira] [Resolved] (SPARK-12401) Add support for enums in postgres

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-12401. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 10596

[jira] [Updated] (SPARK-12401) Add support for enums in postgres

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-12401: --- Assignee: Takeshi Yamamuro > Add support for enums in postgres > - >

[jira] [Commented] (SPARK-6328) Python API for StreamingListener

2016-01-28 Thread Daniel Jalova (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122473#comment-15122473 ] Daniel Jalova commented on SPARK-6328: -- [~apachespark] Hi, can you change the assignee to me please?

[jira] [Created] (SPARK-13067) DataFrameSuite.simple explode fail locally

2016-01-28 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-13067: --- Summary: DataFrameSuite.simple explode fail locally Key: SPARK-13067 URL: https://issues.apache.org/jira/browse/SPARK-13067 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-12749) Spark SQL JSON schema infernce should allow floating-point numbers as BigDecimal

2016-01-28 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12749. - Resolution: Fixed Assignee: Brandon Bradley Fix Version/s: 2.0.0 > Spark SQL

[jira] [Resolved] (SPARK-9835) Iteratively reweighted least squares solver for GLMs

2016-01-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-9835. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 10639

[jira] [Updated] (SPARK-13050) Scalatest tags fail builds with the addition of the sketch module

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-13050: --- Description: Builds fail at the new sketch module when a scalatest tag is used. Found when using

[jira] [Commented] (SPARK-12944) CrossValidator doesn't accept a Pipeline as an estimator

2016-01-28 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122518#comment-15122518 ] Seth Hendrickson commented on SPARK-12944: -- Wanted to update - There are two parts to this

[jira] [Commented] (SPARK-10009) PySpark Param of Vector type can be set with Python array or numpy.array

2016-01-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122358#comment-15122358 ] holdenk commented on SPARK-10009: - We could handle this the param type conversion code that went in

[jira] [Commented] (SPARK-9301) collect_set and collect_list aggregate functions

2016-01-28 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122291#comment-15122291 ] Herman van Hovell commented on SPARK-9301: -- You could implement this as an

[jira] [Created] (SPARK-13065) streaming-twitter pass twitter4j.FilterQuery argument to TwitterUtils.createStream()

2016-01-28 Thread Andrew Davidson (JIRA)
Andrew Davidson created SPARK-13065: --- Summary: streaming-twitter pass twitter4j.FilterQuery argument to TwitterUtils.createStream() Key: SPARK-13065 URL: https://issues.apache.org/jira/browse/SPARK-13065

[jira] [Created] (SPARK-13066) Specify types for per-model/estimator params in ML to allow automatic type conversion

2016-01-28 Thread holdenk (JIRA)
holdenk created SPARK-13066: --- Summary: Specify types for per-model/estimator params in ML to allow automatic type conversion Key: SPARK-13066 URL: https://issues.apache.org/jira/browse/SPARK-13066 Project:

[jira] [Assigned] (SPARK-13055) SQLHistoryListener throws ClassCastException

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13055: Assignee: Andrew Or (was: Apache Spark) > SQLHistoryListener throws ClassCastException >

[jira] [Assigned] (SPARK-13055) SQLHistoryListener throws ClassCastException

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13055: Assignee: Apache Spark (was: Andrew Or) > SQLHistoryListener throws ClassCastException >

[jira] [Commented] (SPARK-13055) SQLHistoryListener throws ClassCastException

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122545#comment-15122545 ] Apache Spark commented on SPARK-13055: -- User 'andrewor14' has created a pull request for this issue:

[jira] [Commented] (SPARK-13066) Specify types for per-model/estimator params in ML to allow automatic type conversion

2016-01-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122567#comment-15122567 ] holdenk commented on SPARK-13066: - See https://issues.apache.org/jira/browse/SPARK-13068 > Specify types

[jira] [Updated] (SPARK-11955) Mark one side fields in merging schema for safely pushdowning filters in parquet

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-11955: --- Assignee: Liang-Chi Hsieh > Mark one side fields in merging schema for safely pushdowning filters in

[jira] [Resolved] (SPARK-11955) Mark one side fields in merging schema for safely pushdowning filters in parquet

2016-01-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-11955. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 9940

[jira] [Assigned] (SPARK-12913) Reimplement stddev/variance as declarative function

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12913: Assignee: Apache Spark > Reimplement stddev/variance as declarative function >

[jira] [Updated] (SPARK-13055) SQLHistoryListener throws ClassCastException

2016-01-28 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13055: -- Component/s: SQL > SQLHistoryListener throws ClassCastException >

[jira] [Created] (SPARK-13069) ActorHelper is not throttled by rate limiter

2016-01-28 Thread Lin Zhao (JIRA)
Lin Zhao created SPARK-13069: Summary: ActorHelper is not throttled by rate limiter Key: SPARK-13069 URL: https://issues.apache.org/jira/browse/SPARK-13069 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-12196) Store/retrieve blocks in different speed storage devices by hierarchy way

2016-01-28 Thread yucai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122684#comment-15122684 ] yucai commented on SPARK-12196: --- Thanks Holden. Usually, customers have 7-11 HDDs installed in server

[jira] [Created] (SPARK-13068) Extend pyspark ml paramtype conversion to support lists

2016-01-28 Thread holdenk (JIRA)
holdenk created SPARK-13068: --- Summary: Extend pyspark ml paramtype conversion to support lists Key: SPARK-13068 URL: https://issues.apache.org/jira/browse/SPARK-13068 Project: Spark Issue Type:

[jira] [Commented] (SPARK-13068) Extend pyspark ml paramtype conversion to support lists

2016-01-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122536#comment-15122536 ] holdenk commented on SPARK-13068: - cc [~sethah] > Extend pyspark ml paramtype conversion to support

[jira] [Commented] (SPARK-12913) Reimplement stddev/variance as declarative function

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122645#comment-15122645 ] Apache Spark commented on SPARK-12913: -- User 'davies' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12913) Reimplement stddev/variance as declarative function

2016-01-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12913: Assignee: (was: Apache Spark) > Reimplement stddev/variance as declarative function >

[jira] [Commented] (SPARK-12526) `ifelse`, `when`, `otherwise` unable to take Column as value

2016-01-28 Thread Deborah Siegel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122611#comment-15122611 ] Deborah Siegel commented on SPARK-12526: I'm trying to use ifelse (or when and otherwise) to

[jira] [Commented] (SPARK-12526) `ifelse`, `when`, `otherwise` unable to take Column as value

2016-01-28 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122631#comment-15122631 ] Shivaram Venkataraman commented on SPARK-12526: --- Yeah this fix was checked in after 1.6.0

[jira] [Updated] (SPARK-13060) CoarsedExecutorBackend register to driver should wait Executor was ready?

2016-01-28 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-13060: -- Description: Hi Josh I am spark user, currently I feel confused about executor registration. {code} Why

[jira] [Created] (SPARK-13060) CoarsedExecutorBackend register to driver should wait Executor was ready?

2016-01-28 Thread SuYan (JIRA)
SuYan created SPARK-13060: - Summary: CoarsedExecutorBackend register to driver should wait Executor was ready? Key: SPARK-13060 URL: https://issues.apache.org/jira/browse/SPARK-13060 Project: Spark

[jira] [Updated] (SPARK-13061) Error in spark rest api application info for job names contains spaces

2016-01-28 Thread Avihoo Mamka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avihoo Mamka updated SPARK-13061: - Description: When accessing spark rest api with application id to get job specific id status, a

[jira] [Updated] (SPARK-13061) Error in spark rest api application info for job names contains spaces

2016-01-28 Thread Avihoo Mamka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avihoo Mamka updated SPARK-13061: - Description: When accessing spark rest api with application id to get job specific id status, a

[jira] [Resolved] (SPARK-13058) spark-shell start fail

2016-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-13058. --- Resolution: Not A Problem I can't reproduce this and I expect it's a problem with your bash, like

[jira] [Commented] (SPARK-12963) In cluster mode,spark_local_ip will cause driver exception:Service 'Driver' failed after 16 retries!

2016-01-28 Thread lichenglin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121065#comment-15121065 ] lichenglin commented on SPARK-12963: I think the driver's ip should be stationary like 'localhost'

[jira] [Created] (SPARK-13061) Error in spark rest api application info for job names contains spaces

2016-01-28 Thread Avihoo Mamka (JIRA)
Avihoo Mamka created SPARK-13061: Summary: Error in spark rest api application info for job names contains spaces Key: SPARK-13061 URL: https://issues.apache.org/jira/browse/SPARK-13061 Project:

[jira] [Updated] (SPARK-13061) Error in spark rest api application info for job names contains spaces

2016-01-28 Thread Avihoo Mamka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avihoo Mamka updated SPARK-13061: - Description: When accessing spark rest api with application id to get job specific id status, a

[jira] [Created] (SPARK-13059) Sort inputsplits by size in HadoopRDD to avoid long tails

2016-01-28 Thread Rajesh Balamohan (JIRA)
Rajesh Balamohan created SPARK-13059: Summary: Sort inputsplits by size in HadoopRDD to avoid long tails Key: SPARK-13059 URL: https://issues.apache.org/jira/browse/SPARK-13059 Project: Spark

[jira] [Commented] (SPARK-12403) "Simba Spark ODBC Driver 1.0" not working with 1.5.2 anymore

2016-01-28 Thread Lunen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121080#comment-15121080 ] Lunen commented on SPARK-12403: --- Microsoft just updated their driver and it's working. (Just search for

[jira] [Closed] (SPARK-12403) "Simba Spark ODBC Driver 1.0" not working with 1.5.2 anymore

2016-01-28 Thread Lunen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lunen closed SPARK-12403. - Microsoft released an updated driver. Tested and working. > "Simba Spark ODBC Driver 1.0" not working with 1.5.2

[jira] [Commented] (SPARK-13059) Sort inputsplits by size in HadoopRDD to avoid long tails

2016-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121044#comment-15121044 ] Sean Owen commented on SPARK-13059: --- I like the idea. Although I'm not 100% sure I think that wouldn't

[jira] [Commented] (SPARK-12963) In cluster mode,spark_local_ip will cause driver exception:Service 'Driver' failed after 16 retries!

2016-01-28 Thread lichenglin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121053#comment-15121053 ] lichenglin commented on SPARK-12963: I think the driver's ip should be stationary like 'localhost'

[jira] [Commented] (SPARK-12261) pyspark crash for large dataset

2016-01-28 Thread Christopher Bourez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121114#comment-15121114 ] Christopher Bourez commented on SPARK-12261: I recompiled Spark on Windows but the problem

[jira] [Commented] (SPARK-12861) Changes to support KMeans with large feature space

2016-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121412#comment-15121412 ] Sean Owen commented on SPARK-12861: --- Yes, I'm inclined to close this as a duplicate of SPARK-4039 >

[jira] [Updated] (SPARK-13062) Overwriting same file with new schema destroys original file.

2016-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-13062: -- Target Version/s: (was: 1.6.0) (Don't set target version; 1.6.0 can't make sense anyway) Is that

[jira] [Updated] (SPARK-13062) Overwriting same file with new schema destroys original file.

2016-01-28 Thread Vincent Warmerdam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Warmerdam updated SPARK-13062: -- Description: I am using Hadoop with Spark 1.5.2. Using pyspark, let's create two

[jira] [Commented] (SPARK-8402) Add DP means clustering to MLlib

2016-01-28 Thread Meethu Mathew (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121332#comment-15121332 ] Meethu Mathew commented on SPARK-8402: -- [~mengxr] [~josephkb] This ticket is in idle state for a long

[jira] [Commented] (SPARK-13058) spark-shell start fail

2016-01-28 Thread Gabor Nemeth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121357#comment-15121357 ] Gabor Nemeth commented on SPARK-13058: -- Bash supports += array assignment only from version 3.1. >

[jira] [Resolved] (SPARK-6579) save as parquet with overwrite failed when linking with Hadoop 1.0.4

2016-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-6579. -- Resolution: Won't Fix If this is still an issue, I can't see it being resolved, as we're already

[jira] [Resolved] (SPARK-6581) Metadata is missing when saving parquet file using hadoop 1.0.4

2016-01-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-6581. -- Resolution: Won't Fix If this is still an issue, I can't see it being resolved, as we're already

[jira] [Commented] (SPARK-12861) Changes to support KMeans with large feature space

2016-01-28 Thread Roy Levin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121428#comment-15121428 ] Roy Levin commented on SPARK-12861: --- Please note that I also explain the difference and why the

[jira] [Updated] (SPARK-12861) Changes to support KMeans with large feature space

2016-01-28 Thread Roy Levin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roy Levin updated SPARK-12861: -- Description: The problem: - In Spark's KMeans code the center vectors are

[jira] [Created] (SPARK-13062) Using same file name with new schema destroys original file.

2016-01-28 Thread Vincent Warmerdam (JIRA)
Vincent Warmerdam created SPARK-13062: - Summary: Using same file name with new schema destroys original file. Key: SPARK-13062 URL: https://issues.apache.org/jira/browse/SPARK-13062 Project:

[jira] [Commented] (SPARK-12261) pyspark crash for large dataset

2016-01-28 Thread Christopher Bourez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121182#comment-15121182 ] Christopher Bourez commented on SPARK-12261: There is a strange "remove broadcast variable"

[jira] [Updated] (SPARK-13062) Using same file name with new schema destroys original file.

2016-01-28 Thread Vincent Warmerdam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Warmerdam updated SPARK-13062: -- Description: Let's create two dataframes. {code} ddf1 =

[jira] [Updated] (SPARK-12926) SQLContext to display warning message when non-sql configs are being set

2016-01-28 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tejas Patil updated SPARK-12926: Summary: SQLContext to display warning message when non-sql configs are being set (was:

[jira] [Commented] (SPARK-13063) Make the SPARK YARN STAGING DIR as configurable

2016-01-28 Thread Devaraj K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121427#comment-15121427 ] Devaraj K commented on SPARK-13063: --- Yes, the sub-directories under the staging dir are specific to

[jira] [Commented] (SPARK-12811) Estimator interface for generalized linear models (GLMs)

2016-01-28 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121176#comment-15121176 ] Yanbo Liang commented on SPARK-12811: - Should we put it under a new folder named "ml/glm"? >

[jira] [Updated] (SPARK-13062) Overwriting same file with new schema destroys original file.

2016-01-28 Thread Vincent Warmerdam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Warmerdam updated SPARK-13062: -- Description: I am using Hadoop with Spark 5.2. Using pyspark, let's create two

[jira] [Updated] (SPARK-13062) Overwriting same file with new schema destroys original file.

2016-01-28 Thread Vincent Warmerdam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Warmerdam updated SPARK-13062: -- Description: I am using Hadoop with Spark 5.2. Using pyspark, let's create two

[jira] [Updated] (SPARK-13062) Overwriting same file with new schema destroys original file.

2016-01-28 Thread Vincent Warmerdam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Warmerdam updated SPARK-13062: -- Summary: Overwriting same file with new schema destroys original file. (was: Using

  1   2   >