[jira] [Commented] (SPARK-18332) SparkR 2.1 QA: Programming guide, migration guide, vignettes updates

2016-12-08 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733608#comment-15733608 ] Felix Cheung commented on SPARK-18332: -- Agreed. This piece is in need of updates, urgently ;) >

[jira] [Updated] (SPARK-14914) Test Cases fail on Windows

2016-12-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-14914: -- Component/s: Tests > Test Cases fail on Windows > -- > >

[jira] [Assigned] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18796: Assignee: Apache Spark > StreamingQueryManager should not hold a lock when starting a

[jira] [Assigned] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18796: Assignee: (was: Apache Spark) > StreamingQueryManager should not hold a lock when

[jira] [Commented] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733606#comment-15733606 ] Apache Spark commented on SPARK-18796: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Updated] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

2016-12-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18796: - Component/s: Structured Streaming > StreamingQueryManager should not hold a lock when starting a

[jira] [Updated] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

2016-12-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18796: - Description: Otherwise, the user cannot start any queries when a query is starting. If a query

[jira] [Updated] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

2016-12-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18796: - Affects Version/s: 2.1.0 2.0.2 > StreamingQueryManager should not hold a

[jira] [Created] (SPARK-18796) StreamingQueryManager should not hold a lock when starting a query

2016-12-08 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18796: Summary: StreamingQueryManager should not hold a lock when starting a query Key: SPARK-18796 URL: https://issues.apache.org/jira/browse/SPARK-18796 Project: Spark

[jira] [Commented] (SPARK-18745) java.lang.IndexOutOfBoundsException running query 68 Spark SQL on (100TB)

2016-12-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733586#comment-15733586 ] Dongjoon Hyun commented on SPARK-18745: --- Hi, I removed the FIX VERSION because this issue is

[jira] [Updated] (SPARK-18745) java.lang.IndexOutOfBoundsException running query 68 Spark SQL on (100TB)

2016-12-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-18745: -- Fix Version/s: (was: 2.1.0) > java.lang.IndexOutOfBoundsException running query 68 Spark

[jira] [Commented] (SPARK-18676) Spark 2.x query plan data size estimation can crash join queries versus 1.x

2016-12-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733569#comment-15733569 ] Davies Liu commented on SPARK-18676: Yes, it can, see WholeStageCodegen.doExecute() as an example.

[jira] [Updated] (SPARK-18795) SparkR vignette update: ksTest

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18795: -- Description: Update vignettes to cover ksTest (was: Update vignettes to cover

[jira] [Created] (SPARK-18795) SparkR vignette update: ksTest

2016-12-08 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-18795: - Summary: SparkR vignette update: ksTest Key: SPARK-18795 URL: https://issues.apache.org/jira/browse/SPARK-18795 Project: Spark Issue Type:

[jira] [Updated] (SPARK-18794) SparkR vignette update: gbt

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18794: -- Description: Update vignettes to cover gradient boosted trees (was: Update vignettes

[jira] [Created] (SPARK-18794) SparkR vignette update: gbt

2016-12-08 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-18794: - Summary: SparkR vignette update: gbt Key: SPARK-18794 URL: https://issues.apache.org/jira/browse/SPARK-18794 Project: Spark Issue Type:

[jira] [Commented] (SPARK-18332) SparkR 2.1 QA: Programming guide, migration guide, vignettes updates

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733552#comment-15733552 ] Joseph K. Bradley commented on SPARK-18332: --- Btw, it will also be good to get rid of the

[jira] [Created] (SPARK-18793) SparkR vignette update: random forest

2016-12-08 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-18793: - Summary: SparkR vignette update: random forest Key: SPARK-18793 URL: https://issues.apache.org/jira/browse/SPARK-18793 Project: Spark Issue Type:

[jira] [Updated] (SPARK-18792) SparkR vignette update: logit

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18792: -- Summary: SparkR vignette update: logit (was: SparkR vignette update: multiclass

[jira] [Updated] (SPARK-18793) SparkR vignette update: random forest

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18793: -- Description: Update vignettes to cover randomForest was:Update vignettes to cover

[jira] [Created] (SPARK-18792) SparkR vignette update: multiclass logistic regression

2016-12-08 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-18792: - Summary: SparkR vignette update: multiclass logistic regression Key: SPARK-18792 URL: https://issues.apache.org/jira/browse/SPARK-18792 Project: Spark

[jira] [Commented] (SPARK-18332) SparkR 2.1 QA: Programming guide, migration guide, vignettes updates

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733539#comment-15733539 ] Joseph K. Bradley commented on SPARK-18332: --- I'll make some subtasks > SparkR 2.1 QA:

[jira] [Created] (SPARK-18791) Stream-Stream Joins

2016-12-08 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-18791: Summary: Stream-Stream Joins Key: SPARK-18791 URL: https://issues.apache.org/jira/browse/SPARK-18791 Project: Spark Issue Type: New Feature

[jira] [Resolved] (SPARK-17823) Make JVMObjectTracker.objMap thread-safe

2016-12-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-17823. --- Resolution: Duplicate This is contained by SPARK-17822. > Make JVMObjectTracker.objMap

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-12-08 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733439#comment-15733439 ] Erik Erlandson commented on SPARK-18278: As I understand it (and as I've built them) an "MVP"

[jira] [Commented] (SPARK-18590) R - Include package vignettes and help pages, build source package in Spark distribution

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733395#comment-15733395 ] Apache Spark commented on SPARK-18590: -- User 'shivaram' has created a pull request for this issue:

[jira] [Commented] (SPARK-18752) "isSrcLocal" parameter to Hive loadTable / loadPartition should come from user

2016-12-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733397#comment-15733397 ] Marcelo Vanzin commented on SPARK-18752: For posterity, here's the exception we hit in our tests:

[jira] [Resolved] (SPARK-15218) Error: Could not find or load main class org.apache.spark.launcher.Main when run from a directory containing colon ':'

2016-12-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-15218. Resolution: Won't Fix I'm gonna close this since it's an issue in Mesos. If anyone has a

[jira] [Assigned] (SPARK-18790) Keep a general offset history of stream batches

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18790: Assignee: (was: Apache Spark) > Keep a general offset history of stream batches >

[jira] [Commented] (SPARK-18790) Keep a general offset history of stream batches

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733383#comment-15733383 ] Apache Spark commented on SPARK-18790: -- User 'tcondie' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18790) Keep a general offset history of stream batches

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18790: Assignee: Apache Spark > Keep a general offset history of stream batches >

[jira] [Commented] (SPARK-18331) Update SparkR website for 2.1

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733376#comment-15733376 ] Joseph K. Bradley commented on SPARK-18331: --- Hm, I forgot SparkR does not have a website.

[jira] [Updated] (SPARK-18331) Update SparkR website for 2.1

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18331: -- Target Version/s: (was: 2.1.0) > Update SparkR website for 2.1 >

[jira] [Closed] (SPARK-18331) Update SparkR website for 2.1

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley closed SPARK-18331. - Resolution: Not A Problem > Update SparkR website for 2.1 >

[jira] [Commented] (SPARK-16599) java.util.NoSuchElementException: None.get at at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:343)

2016-12-08 Thread Madhumita Nagle (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733363#comment-15733363 ] Madhumita Nagle commented on SPARK-16599: - Hi, I get a similar error. In my code i also try to

[jira] [Resolved] (SPARK-18760) Provide consistent format output for all file formats

2016-12-08 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18760. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Provide consistent

[jira] [Updated] (SPARK-18790) Keep a general offset history of stream batches

2016-12-08 Thread Tyson Condie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyson Condie updated SPARK-18790: - Component/s: Structured Streaming > Keep a general offset history of stream batches >

[jira] [Created] (SPARK-18790) Keep a general offset history of stream batches

2016-12-08 Thread Tyson Condie (JIRA)
Tyson Condie created SPARK-18790: Summary: Keep a general offset history of stream batches Key: SPARK-18790 URL: https://issues.apache.org/jira/browse/SPARK-18790 Project: Spark Issue Type:

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

2016-12-08 Thread Saikat Kanjilal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733342#comment-15733342 ] Saikat Kanjilal commented on SPARK-9487: Given the latest thread on the devlist thoughts

[jira] [Closed] (SPARK-18330) SparkR 2.1 QA: Update user guide for new features & APIs

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley closed SPARK-18330. - Resolution: Duplicate > SparkR 2.1 QA: Update user guide for new features & APIs >

[jira] [Updated] (SPARK-18330) SparkR 2.1 QA: Update user guide for new features & APIs

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18330: -- Target Version/s: (was: 2.1.0) > SparkR 2.1 QA: Update user guide for new features &

[jira] [Commented] (SPARK-18330) SparkR 2.1 QA: Update user guide for new features & APIs

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733270#comment-15733270 ] Joseph K. Bradley commented on SPARK-18330: --- I'll close this task. Its elements can be done

[jira] [Commented] (SPARK-18330) SparkR 2.1 QA: Update user guide for new features & APIs

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733264#comment-15733264 ] Joseph K. Bradley commented on SPARK-18330: --- Yeah, maybe we can consolidate some of these tasks

[jira] [Created] (SPARK-18789) Save Data frame with Null column exception

2016-12-08 Thread Harish (JIRA)
Harish created SPARK-18789: -- Summary: Save Data frame with Null column exception Key: SPARK-18789 URL: https://issues.apache.org/jira/browse/SPARK-18789 Project: Spark Issue Type: Bug Affects

[jira] [Commented] (SPARK-17859) persist should not impede with spark's ability to perform a broadcast join.

2016-12-08 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733247#comment-15733247 ] Andrew Ray commented on SPARK-17859: this appears to be fixed in 2.0.2 {code} scala>

[jira] [Updated] (SPARK-18751) Deadlock when SparkContext.stop is called in Utils.tryOrStopSparkContext

2016-12-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18751: - Affects Version/s: 2.1.0 > Deadlock when SparkContext.stop is called in

[jira] [Resolved] (SPARK-18751) Deadlock when SparkContext.stop is called in Utils.tryOrStopSparkContext

2016-12-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18751. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Deadlock when

[jira] [Updated] (SPARK-16589) Chained cartesian produces incorrect number of records

2016-12-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-16589: --- Fix Version/s: (was: 2.1.0) 2.1.1 > Chained cartesian produces incorrect

[jira] [Assigned] (SPARK-18323) Update MLlib, GraphX websites for 2.1

2016-12-08 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-18323: - Assignee: Joseph K. Bradley > Update MLlib, GraphX websites for 2.1 >

[jira] [Resolved] (SPARK-9384) Easier setting of executor and driver classpath

2016-12-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-9384. --- Resolution: Won't Fix It seems there hasn't really been much interest in pushing this forward

[jira] [Commented] (SPARK-18591) Replace hash-based aggregates with sort-based ones if inputs already sorted

2016-12-08 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733218#comment-15733218 ] Nattavut Sutyanyong commented on SPARK-18591: - [~maropu]: Why do you choose to implement it

[jira] [Resolved] (SPARK-18590) R - Include package vignettes and help pages, build source package in Spark distribution

2016-12-08 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-18590. --- Resolution: Fixed Fix Version/s: 2.1.1 Issue resolved by pull request

[jira] [Commented] (SPARK-18774) Ignore non-existing files when ignoreCorruptFiles is enabled

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733163#comment-15733163 ] Apache Spark commented on SPARK-18774: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Resolved] (SPARK-16589) Chained cartesian produces incorrect number of records

2016-12-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-16589. Resolution: Fixed Fix Version/s: 2.1.0 2.0.3 > Chained cartesian

[jira] [Updated] (SPARK-16589) Chained cartesian produces incorrect number of records

2016-12-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-16589: --- Assignee: Andrew Ray > Chained cartesian produces incorrect number of records >

[jira] [Commented] (SPARK-18332) SparkR 2.1 QA: Programming guide, migration guide, vignettes updates

2016-12-08 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732975#comment-15732975 ] Miao Wang commented on SPARK-18332: --- [~felixcheung]Let me add the spark.logit by today. Then, we can

[jira] [Resolved] (SPARK-8617) Handle history files better

2016-12-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-8617. --- Resolution: Fixed Assignee: Ergin Seyfe Fix Version/s: 2.2.0 > Handle history

[jira] [Commented] (SPARK-18688) Interpolated time series join

2016-12-08 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732969#comment-15732969 ] Nattavut Sutyanyong commented on SPARK-18688: - It looks like this "interpolation" join is

[jira] [Created] (SPARK-18788) Add getNumPartitions() to SparkR

2016-12-08 Thread Raela Wang (JIRA)
Raela Wang created SPARK-18788: -- Summary: Add getNumPartitions() to SparkR Key: SPARK-18788 URL: https://issues.apache.org/jira/browse/SPARK-18788 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-18330) SparkR 2.1 QA: Update user guide for new features & APIs

2016-12-08 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732827#comment-15732827 ] Felix Cheung commented on SPARK-18330: -- Yanbo has already updated the programming guide with for

[jira] [Commented] (SPARK-14932) Allow DataFrame.replace() to replace values with None

2016-12-08 Thread Jiayue Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732794#comment-15732794 ] Jiayue Zhang commented on SPARK-14932: -- I can work on this. Thanks. > Allow DataFrame.replace() to

[jira] [Commented] (SPARK-18332) SparkR 2.1 QA: Programming guide, migration guide, vignettes updates

2016-12-08 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732796#comment-15732796 ] Felix Cheung commented on SPARK-18332: -- Would you have time to do this [~wangmiao1981]? Particularly

[jira] [Created] (SPARK-18787) spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer usage by Netty

2016-12-08 Thread Aniket Bhatnagar (JIRA)
Aniket Bhatnagar created SPARK-18787: Summary: spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer usage by Netty Key: SPARK-18787 URL:

[jira] [Created] (SPARK-18786) pySpark SQLContext.getOrCreate(sc) take stopped sparkContext

2016-12-08 Thread Alex Liu (JIRA)
Alex Liu created SPARK-18786: Summary: pySpark SQLContext.getOrCreate(sc) take stopped sparkContext Key: SPARK-18786 URL: https://issues.apache.org/jira/browse/SPARK-18786 Project: Spark Issue

[jira] [Commented] (SPARK-18785) Fix/Investigate the test failures in Java/Scala on Windows

2016-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732594#comment-15732594 ] Hyukjin Kwon commented on SPARK-18785: -- I will work on this. > Fix/Investigate the test failures in

[jira] [Updated] (SPARK-18785) Fix/Investigate the test failures in Java/Scala on Windows

2016-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18785: - Description: - {{FileSuite}} {code} [info] - binary file input as byte array *** FAILED ***

[jira] [Created] (SPARK-18785) Fix/Investigate the test failures in Java/Scala on Windows

2016-12-08 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18785: Summary: Fix/Investigate the test failures in Java/Scala on Windows Key: SPARK-18785 URL: https://issues.apache.org/jira/browse/SPARK-18785 Project: Spark

[jira] [Resolved] (SPARK-18667) input_file_name function does not work with UDF

2016-12-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-18667. - Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 2.1.0 >

[jira] [Resolved] (SPARK-17591) Fix/investigate the failure of tests in Scala On Windows

2016-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-17591. -- Resolution: Done I am resolving this because I can proceed the tests further and the sub-tasks

[jira] [Updated] (SPARK-18781) Allow MatrixFactorizationModel.predict to skip user/product approximation count

2016-12-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18781: -- Priority: Minor (was: Major) It makes sense, though I'm also not sure of a great way to get around

[jira] [Resolved] (SPARK-18718) Skip some test failures due to path length limitation and fix tests to pass on Windows

2016-12-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18718. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16147

[jira] [Resolved] (SPARK-18784) Managed memory leak - spark-2.0.2

2016-12-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18784. --- Resolution: Duplicate It's actually ignorable > Managed memory leak - spark-2.0.2 >

[jira] [Commented] (SPARK-18779) Messages being received only from one partition when using Spark Streaming integration for Kafka 0.10 with kafka client library at 0.10.1

2016-12-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732371#comment-15732371 ] Sean Owen commented on SPARK-18779: --- It sounds like you're saying that you need a particular client API

[jira] [Updated] (SPARK-18779) Messages being received only from one partition when using Spark Streaming integration for Kafka 0.10 with kafka client library at 0.10.1

2016-12-08 Thread Pranav Nakhe (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pranav Nakhe updated SPARK-18779: - Description: I apologize for the earlier descripion which wasnt very clear about the issue. I

[jira] [Commented] (SPARK-18325) SparkR 2.1 QA: Check for new R APIs requiring example code

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732182#comment-15732182 ] Apache Spark commented on SPARK-18325: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Updated] (SPARK-18783) ML StringIndexer does not work with nested fields

2016-12-08 Thread manuel garrido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] manuel garrido updated SPARK-18783: --- Description: Using StringIndexer.transform with a nested field (from parsing json data)

[jira] [Created] (SPARK-18784) Managed memory leak - spark-2.0.2

2016-12-08 Thread Appu K (JIRA)
Appu K created SPARK-18784: -- Summary: Managed memory leak - spark-2.0.2 Key: SPARK-18784 URL: https://issues.apache.org/jira/browse/SPARK-18784 Project: Spark Issue Type: Bug Affects Versions:

[jira] [Updated] (SPARK-18783) ML StringIndexer does not work with nested fields

2016-12-08 Thread manuel garrido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] manuel garrido updated SPARK-18783: --- Description: Using StringIndexer.transform with a nested field (from parsing json data)

[jira] [Updated] (SPARK-18783) ML StringIndexer does not work with nested fields

2016-12-08 Thread manuel garrido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] manuel garrido updated SPARK-18783: --- Description: Using StringIndexer.transform with a nested field (from parsing json data)

[jira] [Created] (SPARK-18783) ML StringIndexer does not work with nested fields

2016-12-08 Thread manuel garrido (JIRA)
manuel garrido created SPARK-18783: -- Summary: ML StringIndexer does not work with nested fields Key: SPARK-18783 URL: https://issues.apache.org/jira/browse/SPARK-18783 Project: Spark Issue

[jira] [Assigned] (SPARK-18020) Kinesis receiver does not snapshot when shard completes

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18020: Assignee: (was: Apache Spark) > Kinesis receiver does not snapshot when shard

[jira] [Commented] (SPARK-18020) Kinesis receiver does not snapshot when shard completes

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731884#comment-15731884 ] Apache Spark commented on SPARK-18020: -- User 'maropu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18020) Kinesis receiver does not snapshot when shard completes

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18020: Assignee: Apache Spark > Kinesis receiver does not snapshot when shard completes >

[jira] [Commented] (SPARK-18782) Bump Hadoop 2.6 version to use Hadoop 2.6.5

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731806#comment-15731806 ] Apache Spark commented on SPARK-18782: -- User 'a-roberts' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18782) Bump Hadoop 2.6 version to use Hadoop 2.6.5

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18782: Assignee: (was: Apache Spark) > Bump Hadoop 2.6 version to use Hadoop 2.6.5 >

[jira] [Assigned] (SPARK-18782) Bump Hadoop 2.6 version to use Hadoop 2.6.5

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18782: Assignee: Apache Spark > Bump Hadoop 2.6 version to use Hadoop 2.6.5 >

[jira] [Created] (SPARK-18782) Bump Hadoop 2.6 version to use Hadoop 2.6.5

2016-12-08 Thread Adam Roberts (JIRA)
Adam Roberts created SPARK-18782: Summary: Bump Hadoop 2.6 version to use Hadoop 2.6.5 Key: SPARK-18782 URL: https://issues.apache.org/jira/browse/SPARK-18782 Project: Spark Issue Type:

[jira] [Created] (SPARK-18781) Allow MatrixFactorizationModel.predict to skip user/product approximation count

2016-12-08 Thread Eyal Allweil (JIRA)
Eyal Allweil created SPARK-18781: Summary: Allow MatrixFactorizationModel.predict to skip user/product approximation count Key: SPARK-18781 URL: https://issues.apache.org/jira/browse/SPARK-18781

[jira] [Assigned] (SPARK-18576) Expose basic TaskContext info in PySpark

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18576: Assignee: (was: Apache Spark) > Expose basic TaskContext info in PySpark >

[jira] [Assigned] (SPARK-18576) Expose basic TaskContext info in PySpark

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18576: Assignee: Apache Spark > Expose basic TaskContext info in PySpark >

[jira] [Commented] (SPARK-18576) Expose basic TaskContext info in PySpark

2016-12-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731722#comment-15731722 ] Apache Spark commented on SPARK-18576: -- User 'holdenk' has created a pull request for this issue:

[jira] [Commented] (SPARK-18779) Upgrade Spark Streaming integration for Kafka 0.10 to use Kafka client library at 0.10.1

2016-12-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731643#comment-15731643 ] Sean Owen commented on SPARK-18779: --- So you need a 0.10.1 feature on the client side for your app to

[jira] [Updated] (SPARK-18780) "org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree fromunixtime(cast(…))"

2016-12-08 Thread SunYonggang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SunYonggang updated SPARK-18780: Description: In Spark-Shell, I want to generate RDD from hivecontext.sql(hql_content), the syntax

[jira] [Commented] (SPARK-18780) "org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree fromunixtime(cast(…))"

2016-12-08 Thread SunYonggang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731634#comment-15731634 ] SunYonggang commented on SPARK-18780: - But if you use first_value(), it can avoid this error: "select

[jira] [Commented] (SPARK-18780) "org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree fromunixtime(cast(…))"

2016-12-08 Thread SunYonggang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731630#comment-15731630 ] SunYonggang commented on SPARK-18780: - Here when i remove collect_set(b)[0] from the SQL, or when I

[jira] [Commented] (SPARK-18278) Support native submission of spark jobs to a kubernetes cluster

2016-12-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731626#comment-15731626 ] Sean Owen commented on SPARK-18278: --- Generally, ASF projects produce source not binaries, but,

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-12-08 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731628#comment-15731628 ] Michael Schmeißer commented on SPARK-650: - No, it's not just about propagating information - some

[jira] [Updated] (SPARK-18780) "org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree fromunixtime(cast(…))"

2016-12-08 Thread SunYonggang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SunYonggang updated SPARK-18780: Environment: hdp 2.4.0.0-169 with 10 servers in CentOS 6.5; spark 1.6.0 hive 1.2.1hadoop

[jira] [Created] (SPARK-18780) "org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree fromunixtime(cast(…))"

2016-12-08 Thread SunYonggang (JIRA)
SunYonggang created SPARK-18780: --- Summary: "org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree fromunixtime(cast(…))" Key: SPARK-18780 URL:

[jira] [Comment Edited] (SPARK-13955) Spark in yarn mode fails

2016-12-08 Thread liyunzhang_intel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731574#comment-15731574 ] liyunzhang_intel edited comment on SPARK-13955 at 12/8/16 8:54 AM: ---

<    1   2   3   >