[jira] [Updated] (SPARK-15171) Deprecate registerTempTable and add dataset.createTempView

2016-05-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-15171: - Target Version/s: 2.0.0 > Deprecate registerTempTable and add dataset.createTempView >

[jira] [Updated] (SPARK-15171) Deprecate registerTempTable and add dataset.createTempView

2016-05-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-15171: - Priority: Critical (was: Minor) > Deprecate registerTempTable and add dataset.createTempView >

[jira] [Updated] (SPARK-15171) Deprecate registerTempTable and add dataset.createTempView

2016-05-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-15171: - Labels: release_notes releasenotes (was: ) > Deprecate registerTempTable and add dataset.createTempView

[jira] [Resolved] (SPARK-15195) Improve PyDoc for ml.tuning

2016-05-10 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-15195. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12967

[jira] [Updated] (SPARK-15119) DecisionTreeParams.minInfoGain does not have a validator

2016-05-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-15119: -- Issue Type: Improvement (was: Bug) > DecisionTreeParams.minInfoGain does not have a

[jira] [Commented] (SPARK-14813) ML 2.0 QA: API: Python API coverage

2016-05-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278700#comment-15278700 ] Joseph K. Bradley commented on SPARK-14813: --- [~holdenk] Can you please use the "requires" link

[jira] [Updated] (SPARK-12854) Vectorize Parquet reader

2016-05-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-12854: - Fix Version/s: 2.0.0 > Vectorize Parquet reader > > > Key:

[jira] [Created] (SPARK-15255) RDD name from DataFrame op should not include full local relation data

2016-05-10 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-15255: - Summary: RDD name from DataFrame op should not include full local relation data Key: SPARK-15255 URL: https://issues.apache.org/jira/browse/SPARK-15255

[jira] [Commented] (SPARK-15037) Use SparkSession instead of SQLContext in testsuites

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278677#comment-15278677 ] Apache Spark commented on SPARK-15037: -- User 'andrewor14' has created a pull request for this issue:

[jira] [Updated] (SPARK-14857) Table/Database Name Validation in SessionCatalog

2016-05-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-14857: -- Assignee: Xiao Li > Table/Database Name Validation in SessionCatalog >

[jira] [Resolved] (SPARK-14603) SessionCatalog needs to check if a metadata operation is valid

2016-05-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-14603. --- Resolution: Fixed Assignee: Xiao Li Fix Version/s: 2.0.0 > SessionCatalog needs to

[jira] [Created] (SPARK-15254) Improve ML pipeline Cross Validation Scaladoc & PyDoc

2016-05-10 Thread holdenk (JIRA)
holdenk created SPARK-15254: --- Summary: Improve ML pipeline Cross Validation Scaladoc & PyDoc Key: SPARK-15254 URL: https://issues.apache.org/jira/browse/SPARK-15254 Project: Spark Issue Type:

[jira] [Updated] (SPARK-15254) Improve ML pipeline Cross Validation Scaladoc & PyDoc

2016-05-10 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-15254: Component/s: ML > Improve ML pipeline Cross Validation Scaladoc & PyDoc >

[jira] [Updated] (SPARK-14684) Verification of partition specs in SessionCatalog

2016-05-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-14684: -- Assignee: Xiao Li > Verification of partition specs in SessionCatalog >

[jira] [Updated] (SPARK-15037) Use SparkSession instead of SQLContext in testsuites

2016-05-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15037: -- Component/s: SQL > Use SparkSession instead of SQLContext in testsuites >

[jira] [Updated] (SPARK-12661) Drop Python 2.6 support in PySpark

2016-05-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-12661: --- Target Version/s: 2.1.0 (was: 2.0.0) > Drop Python 2.6 support in PySpark >

[jira] [Commented] (SPARK-12661) Drop Python 2.6 support in PySpark

2016-05-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278630#comment-15278630 ] Davies Liu commented on SPARK-12661: I think the goal is clear we did not enough to do that, so I

[jira] [Updated] (SPARK-15037) Use SparkSession instead of SQLContext in testsuites

2016-05-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-15037: -- Component/s: Tests > Use SparkSession instead of SQLContext in testsuites >

[jira] [Resolved] (SPARK-15037) Use SparkSession instead of SQLContext in testsuites

2016-05-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-15037. --- Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: 2.0.0 > Use SparkSession

[jira] [Updated] (SPARK-12177) Update KafkaDStreams to new Kafka 0.9 Consumer API

2016-05-10 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated SPARK-12177: Target Version/s: (was: 2.0.0) Removing the target version of 2.0.0. Holler if you disagree. >

[jira] [Updated] (SPARK-15165) Codegen can break because toCommentSafeString is not actually safe

2016-05-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-15165: - Target Version/s: 1.5.3, 1.6.2, 2.0.0 (was: 2.0.0) > Codegen can break because toCommentSafeString is

[jira] [Commented] (SPARK-14737) Kafka Brokers are down - spark stream should retry

2016-05-10 Thread Faisal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278560#comment-15278560 ] Faisal commented on SPARK-14737: I am not sure if i am following you correctly. You mean to resubmit the

[jira] [Commented] (SPARK-13382) Update PySpark testing notes

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278544#comment-15278544 ] Sean Owen commented on SPARK-13382: --- I can change the wiki. I don't know if we have a clear theory

[jira] [Resolved] (SPARK-14560) Cooperative Memory Management for Spillables

2016-05-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-14560. Resolution: Fixed Assignee: Lianhui Wang (was: Imran Rashid) Fix Version/s: 2.0.0

[jira] [Resolved] (SPARK-11249) [Launcher] Launcher library fails is app resource is not added

2016-05-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11249. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.0.0 >

[jira] [Resolved] (SPARK-13670) spark-class doesn't bubble up error from launcher command

2016-05-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-13670. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.0.0 >

[jira] [Resolved] (SPARK-13382) Update PySpark testing notes

2016-05-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-13382. Resolution: Fixed Assignee: holdenk Fix Version/s: 2.0.0 Maybe [~srowen]

[jira] [Updated] (SPARK-14986) Spark SQL returns incorrect results for LATERAL VIEW OUTER queries if all inner columns are projected out

2016-05-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-14986: - Target Version/s: 2.0.0 > Spark SQL returns incorrect results for LATERAL VIEW OUTER queries if all >

[jira] [Updated] (SPARK-15179) Enable SQL generation for subqueries

2016-05-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-15179: --- Assignee: Herman van Hovell > Enable SQL generation for subqueries >

[jira] [Resolved] (SPARK-14773) Enable the tests in HiveCompatibilitySuite for subquery

2016-05-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-14773. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12988

[jira] [Resolved] (SPARK-15179) Enable SQL generation for subqueries

2016-05-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-15179. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12988

[jira] [Updated] (SPARK-15154) LongHashedRelation test fails on Big Endian platform

2016-05-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-15154: --- Assignee: Pete Robbins > LongHashedRelation test fails on Big Endian platform >

[jira] [Resolved] (SPARK-15154) LongHashedRelation test fails on Big Endian platform

2016-05-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-15154. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13009

[jira] [Commented] (SPARK-15250) Remove deprecated json API in DataFrameReader

2016-05-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278432#comment-15278432 ] Reynold Xin commented on SPARK-15250: - You'd need to change some Python methods too to get it

[jira] [Comment Edited] (SPARK-15193) samplingRatio should default to 1.0 across the board

2016-05-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278393#comment-15278393 ] Nicholas Chammas edited comment on SPARK-15193 at 5/10/16 4:27 PM: ---

[jira] [Commented] (SPARK-15193) samplingRatio should default to 1.0 across the board

2016-05-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278393#comment-15278393 ] Nicholas Chammas commented on SPARK-15193: -- Nope, a sampling ratio of 1.0 and None mean

[jira] [Commented] (SPARK-14737) Kafka Brokers are down - spark stream should retry

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278387#comment-15278387 ] Sean Owen commented on SPARK-14737: --- You could retry the entire app if it fails, in general. That's

[jira] [Commented] (SPARK-14737) Kafka Brokers are down - spark stream should retry

2016-05-10 Thread Faisal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278375#comment-15278375 ] Faisal commented on SPARK-14737: {code} import java.io.Serializable; import java.util.Arrays; import

[jira] [Commented] (SPARK-14737) Kafka Brokers are down - spark stream should retry

2016-05-10 Thread Faisal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278374#comment-15278374 ] Faisal commented on SPARK-14737: Hi Sean, Here are few points to note as per my original reported

[jira] [Updated] (SPARK-15253) For a data source table, Describe table needs to handle spark.sql.sources.schema

2016-05-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-15253: - Assignee: (was: Yin Huai) > For a data source table, Describe table needs to handle >

[jira] [Created] (SPARK-15253) For a data source table, Describe table needs to handle spark.sql.sources.schema

2016-05-10 Thread Yin Huai (JIRA)
Yin Huai created SPARK-15253: Summary: For a data source table, Describe table needs to handle spark.sql.sources.schema Key: SPARK-15253 URL: https://issues.apache.org/jira/browse/SPARK-15253 Project:

[jira] [Assigned] (SPARK-15253) For a data source table, Describe table needs to handle spark.sql.sources.schema

2016-05-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai reassigned SPARK-15253: Assignee: Yin Huai > For a data source table, Describe table needs to handle >

[jira] [Commented] (SPARK-15247) sqlCtx.read.parquet yields at least n_executors * n_cores tasks

2016-05-10 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278288#comment-15278288 ] Takeshi Yamamuro commented on SPARK-15247: -- You tried this in master? Seems this issue has been

[jira] [Resolved] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

2016-05-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-14963. --- Resolution: Fixed Fix Version/s: 2.1.0 > YarnShuffleService should use YARN

[jira] [Updated] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

2016-05-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-14963: -- Assignee: Saisai Shao > YarnShuffleService should use YARN getRecoveryPath() for leveldb

[jira] [Updated] (SPARK-15206) Add testcases for Distinct Aggregation in Having clause

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15206: -- Target Version/s: (was: 2.0.0) > Add testcases for Distinct Aggregation in Having clause >

[jira] [Commented] (SPARK-15193) samplingRatio should default to 1.0 across the board

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278274#comment-15278274 ] Sean Owen commented on SPARK-15193: --- Pardon my ignorance but are those not the same thing semantically?

[jira] [Updated] (SPARK-15224) Can not delete jar and list jar in spark Thrift server

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15224: -- Target Version/s: (was: 1.6.1) Priority: Minor (was: Major) (I'm not sure that's

[jira] [Updated] (SPARK-13605) Bean encoder cannot handle nonbean properties - no way to Encode nonbean Java objects with columns

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-13605: -- Target Version/s: (was: 2.0.0) > Bean encoder cannot handle nonbean properties - no way to Encode

[jira] [Updated] (SPARK-15220) Add hyperlink to "running application" and "completed application"

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15220: -- Assignee: Mao, Wei Resolved by https://github.com/apache/spark/pull/12997 > Add hyperlink to "running

[jira] [Commented] (SPARK-15252) add accumulator wrapper to have more control of it

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278232#comment-15278232 ] Apache Spark commented on SPARK-15252: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15252) add accumulator wrapper to have more control of it

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15252: Assignee: Apache Spark (was: Wenchen Fan) > add accumulator wrapper to have more control

[jira] [Assigned] (SPARK-15252) add accumulator wrapper to have more control of it

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15252: Assignee: Wenchen Fan (was: Apache Spark) > add accumulator wrapper to have more control

[jira] [Created] (SPARK-15252) add accumulator wrapper to have more control of it

2016-05-10 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-15252: --- Summary: add accumulator wrapper to have more control of it Key: SPARK-15252 URL: https://issues.apache.org/jira/browse/SPARK-15252 Project: Spark Issue Type:

[jira] [Updated] (SPARK-15189) ml.Evaluation pydoc issues

2016-05-10 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-15189: --- Assignee: holdenk > ml.Evaluation pydoc issues > -- > >

[jira] [Updated] (SPARK-15195) Improve PyDoc for ml.tuning

2016-05-10 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-15195: --- Assignee: holdenk > Improve PyDoc for ml.tuning > --- > >

[jira] [Commented] (SPARK-14815) ML, Graph, R 2.0 QA: Update user guide for new features & APIs

2016-05-10 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278193#comment-15278193 ] Nick Pentreath commented on SPARK-14815: Ok, I don't feel very strongly about removing them.

[jira] [Updated] (SPARK-14542) PipeRDD should allow configurable buffer size for the stdin writer

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-14542: -- Assignee: Sital Kedia > PipeRDD should allow configurable buffer size for the stdin writer >

[jira] [Resolved] (SPARK-14542) PipeRDD should allow configurable buffer size for the stdin writer

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-14542. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12309

[jira] [Assigned] (SPARK-9860) Join: Determine the join strategy (broadcast join or shuffle join) at runtime

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9860: --- Assignee: (was: Apache Spark) > Join: Determine the join strategy (broadcast join or

[jira] [Assigned] (SPARK-11293) Spillable collections leak shuffle memory

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11293: Assignee: Apache Spark (was: Josh Rosen) > Spillable collections leak shuffle memory >

[jira] [Commented] (SPARK-11293) Spillable collections leak shuffle memory

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278148#comment-15278148 ] Apache Spark commented on SPARK-11293: -- User 'lianhuiwang' has created a pull request for this

[jira] [Commented] (SPARK-4452) Shuffle data structures can starve others on the same thread for memory

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278147#comment-15278147 ] Apache Spark commented on SPARK-4452: - User 'lianhuiwang' has created a pull request for this issue:

[jira] [Assigned] (SPARK-11293) Spillable collections leak shuffle memory

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11293: Assignee: Josh Rosen (was: Apache Spark) > Spillable collections leak shuffle memory >

[jira] [Commented] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2016-05-10 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278124#comment-15278124 ] SuYan commented on SPARK-10796: --- main changes: 1. make DAGScheuler only receive Task Resubmit events from

[jira] [Resolved] (SPARK-14065) serialize MapStatuses in serial model

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-14065. --- Resolution: Won't Fix > serialize MapStatuses in serial model >

[jira] [Comment Edited] (SPARK-15159) Remove usage of HiveContext in SparkR.

2016-05-10 Thread Kai Jiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278010#comment-15278010 ] Kai Jiang edited comment on SPARK-15159 at 5/10/16 12:21 PM: - Also I think we

[jira] [Updated] (SPARK-15159) Remove usage of HiveContext in SparkR.

2016-05-10 Thread Kai Jiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang updated SPARK-15159: -- Description: HiveContext is to be deprecated in 2.0. Replace them with

[jira] [Commented] (SPARK-15159) Remove usage of HiveContext in SparkR.

2016-05-10 Thread Kai Jiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278010#comment-15278010 ] Kai Jiang commented on SPARK-15159: --- So I think we should implement SparkSession first. > Remove usage

[jira] [Updated] (SPARK-15159) Remove usage of HiveContext in SparkR.

2016-05-10 Thread Kai Jiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang updated SPARK-15159: -- Description: HiveContext is to be deprecated in 2.0. Replace them with SparkSession.enableHiveSupport

[jira] [Commented] (SPARK-15245) stream API throws an exception with an incorrect message when the path is not a direcotry

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277984#comment-15277984 ] Sean Owen commented on SPARK-15245: --- Ultimately, the lower levels raise the correct error in this case.

[jira] [Commented] (SPARK-15245) stream API throws an exception with an incorrect message when the path is not a direcotry

2016-05-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277979#comment-15277979 ] Hyukjin Kwon commented on SPARK-15245: -- Sorry for leaving comments again and again but I think this

[jira] [Comment Edited] (SPARK-15218) Error: Could not find or load main class org.apache.spark.launcher.Main when run from a directory containing colon ':'

2016-05-10 Thread Viacheslav Saevskiy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277961#comment-15277961 ] Viacheslav Saevskiy edited comment on SPARK-15218 at 5/10/16 11:34 AM:

[jira] [Commented] (SPARK-15218) Error: Could not find or load main class org.apache.spark.launcher.Main when run from a directory containing colon ':'

2016-05-10 Thread Viacheslav Saevskiy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277961#comment-15277961 ] Viacheslav Saevskiy commented on SPARK-15218: - It's a bash script as states at 1-st line. ```

[jira] [Commented] (SPARK-14162) java.lang.IllegalStateException: Did not find registered driver with class oracle.jdbc.OracleDriver

2016-05-10 Thread Kevin McHale (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277955#comment-15277955 ] Kevin McHale commented on SPARK-14162: -- [~sunrui] you are incorrect. You should take a look at

[jira] [Commented] (SPARK-15218) Error: Could not find or load main class org.apache.spark.launcher.Main when run from a directory containing colon ':'

2016-05-10 Thread Adam Cecile (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277952#comment-15277952 ] Adam Cecile commented on SPARK-15218: - Hehe, that's dirty :D But well, if it's able to workaround the

[jira] [Assigned] (SPARK-15218) Error: Could not find or load main class org.apache.spark.launcher.Main when run from a directory containing colon ':'

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15218: Assignee: (was: Apache Spark) > Error: Could not find or load main class

[jira] [Commented] (SPARK-15218) Error: Could not find or load main class org.apache.spark.launcher.Main when run from a directory containing colon ':'

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277943#comment-15277943 ] Apache Spark commented on SPARK-15218: -- User 'sayevsky' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15218) Error: Could not find or load main class org.apache.spark.launcher.Main when run from a directory containing colon ':'

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15218: Assignee: Apache Spark > Error: Could not find or load main class

[jira] [Created] (SPARK-15251) Cannot apply PythonUDF to aggregated column

2016-05-10 Thread Matthew Livesey (JIRA)
Matthew Livesey created SPARK-15251: --- Summary: Cannot apply PythonUDF to aggregated column Key: SPARK-15251 URL: https://issues.apache.org/jira/browse/SPARK-15251 Project: Spark Issue

[jira] [Commented] (SPARK-14127) [Table related commands] Describe table

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277936#comment-15277936 ] Apache Spark commented on SPARK-14127: -- User 'liancheng' has created a pull request for this issue:

[jira] [Commented] (SPARK-15250) Remove deprecated json API in DataFrameReader

2016-05-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277907#comment-15277907 ] Hyukjin Kwon commented on SPARK-15250: -- [~rxin] I searched and track down the related PRs but could

[jira] [Resolved] (SPARK-15245) stream API throws an exception with an incorrect message when the path is not a direcotry

2016-05-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15245. --- Resolution: Won't Fix Oh you're referring just to the arg name. Yeah it's because it comes from

[jira] [Created] (SPARK-15250) Remove deprecated json API in DataFrameReader

2016-05-10 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-15250: Summary: Remove deprecated json API in DataFrameReader Key: SPARK-15250 URL: https://issues.apache.org/jira/browse/SPARK-15250 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-15215) Fix Explain Parsing and Output

2016-05-10 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-15215. --- Resolution: Resolved Assignee: Xiao Li Target Version/s: 2.0.0

[jira] [Commented] (SPARK-15218) Error: Could not find or load main class org.apache.spark.launcher.Main when run from a directory containing colon ':'

2016-05-10 Thread Viacheslav Saevskiy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277852#comment-15277852 ] Viacheslav Saevskiy commented on SPARK-15218: - Unfortunately escaping ':' in classpath not

[jira] [Assigned] (SPARK-15249) Use FunctionResource instead of (String, String) in CreateFunction and CatalogFunction for resource

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15249: Assignee: Apache Spark > Use FunctionResource instead of (String, String) in

[jira] [Commented] (SPARK-15249) Use FunctionResource instead of (String, String) in CreateFunction and CatalogFunction for resource

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277848#comment-15277848 ] Apache Spark commented on SPARK-15249: -- User 'techaddict' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15249) Use FunctionResource instead of (String, String) in CreateFunction and CatalogFunction for resource

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15249: Assignee: (was: Apache Spark) > Use FunctionResource instead of (String, String) in

[jira] [Created] (SPARK-15249) Use FunctionResource instead of (String, String) in CreateFunction and CatalogFunction for resource

2016-05-10 Thread Sandeep Singh (JIRA)
Sandeep Singh created SPARK-15249: - Summary: Use FunctionResource instead of (String, String) in CreateFunction and CatalogFunction for resource Key: SPARK-15249 URL:

[jira] [Assigned] (SPARK-15177) SparkR 2.0 QA: New R APIs and API docs for mllib.R

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15177: Assignee: Apache Spark > SparkR 2.0 QA: New R APIs and API docs for mllib.R >

[jira] [Commented] (SPARK-15177) SparkR 2.0 QA: New R APIs and API docs for mllib.R

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277837#comment-15277837 ] Apache Spark commented on SPARK-15177: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15177) SparkR 2.0 QA: New R APIs and API docs for mllib.R

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15177: Assignee: (was: Apache Spark) > SparkR 2.0 QA: New R APIs and API docs for mllib.R >

[jira] [Assigned] (SPARK-15248) Partition added with ALTER TABLE to a hive partitioned table is not read while querying

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15248: Assignee: Tathagata Das (was: Apache Spark) > Partition added with ALTER TABLE to a hive

[jira] [Assigned] (SPARK-15248) Partition added with ALTER TABLE to a hive partitioned table is not read while querying

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15248: Assignee: Apache Spark (was: Tathagata Das) > Partition added with ALTER TABLE to a hive

[jira] [Commented] (SPARK-15248) Partition added with ALTER TABLE to a hive partitioned table is not read while querying

2016-05-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277826#comment-15277826 ] Apache Spark commented on SPARK-15248: -- User 'tdas' has created a pull request for this issue:

[jira] [Updated] (SPARK-15248) Partition added with ALTER TABLE to a hive partitioned table is not read while querying

2016-05-10 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-15248: -- Description: Table partitions can be added with locations different from default warehouse

[jira] [Created] (SPARK-15248) Partition added with ALTER TABLE to a hive partitioned table is not read while querying

2016-05-10 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-15248: - Summary: Partition added with ALTER TABLE to a hive partitioned table is not read while querying Key: SPARK-15248 URL: https://issues.apache.org/jira/browse/SPARK-15248

[jira] [Updated] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2016-05-10 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-10796: -- Description: {code} test("Resubmit stage while lost partition in ZombieTasksets or RemovedTaskSets") {

[jira] [Updated] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2016-05-10 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-10796: -- Description: desc: 1. We know a running ShuffleMapStage will have multiple TaskSet: one Active TaskSet,

<    1   2   3   >