[jira] [Commented] (SPARK-11672) Flaky test: ml.JavaDefaultReadWriteSuite

2015-11-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005782#comment-15005782 ] Apache Spark commented on SPARK-11672: -- User 'mengxr' has created a pull request for

[jira] [Resolved] (SPARK-11736) Add MonotonicallyIncreasingID to function registry

2015-11-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-11736. - Resolution: Fixed Fix Version/s: 1.6.0 > Add MonotonicallyIncreasingID to function registr

[jira] [Assigned] (SPARK-11738) Make ArrayType orderable

2015-11-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11738: Assignee: (was: Apache Spark) > Make ArrayType orderable > >

[jira] [Commented] (SPARK-11738) Make ArrayType orderable

2015-11-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005689#comment-15005689 ] Apache Spark commented on SPARK-11738: -- User 'yhuai' has created a pull request for

[jira] [Assigned] (SPARK-11738) Make ArrayType orderable

2015-11-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11738: Assignee: Apache Spark > Make ArrayType orderable > > >

[jira] [Commented] (SPARK-11704) Optimize the Cartesian Join

2015-11-14 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005682#comment-15005682 ] Zhan Zhang commented on SPARK-11704: [~maropu] You are right. I mean fetching from ne

[jira] [Updated] (SPARK-11738) Make ArrayType orderable

2015-11-14 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-11738: - Summary: Make ArrayType orderable (was: Make array orderable) > Make ArrayType orderable > -

[jira] [Commented] (SPARK-10673) spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

2015-11-14 Thread Xin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005577#comment-15005577 ] Xin Wu commented on SPARK-10673: The fix is being tested.. will submit PR shortly. > spa

[jira] [Updated] (SPARK-11744) bin/pyspark --version doesn't return version and exit

2015-11-14 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-11744: - Description: {{bin/pyspark \-\-help}} offers a {{\-\-version}} option: {code} $ ./spark/

[jira] [Commented] (SPARK-11744) bin/pyspark --version doesn't return version and exit

2015-11-14 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005572#comment-15005572 ] Nicholas Chammas commented on SPARK-11744: -- Not sure who would be the best perso

[jira] [Updated] (SPARK-11744) bin/pyspark --version doesn't return version and exit

2015-11-14 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-11744: - Description: {{bin/pyspark \-\-help}} offers a {{\-\-version}} option: {code} $ ./spark/

[jira] [Created] (SPARK-11744) bin/pyspark --version doesn't return version and exit

2015-11-14 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-11744: Summary: bin/pyspark --version doesn't return version and exit Key: SPARK-11744 URL: https://issues.apache.org/jira/browse/SPARK-11744 Project: Spark

[jira] [Commented] (SPARK-11153) Turns off Parquet filter push-down for string and binary columns

2015-11-14 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005542#comment-15005542 ] Mark Hamstra commented on SPARK-11153: -- Thanks. > Turns off Parquet filter push-dow

[jira] [Assigned] (SPARK-9928) LogicalLocalTable in ExistingRDD.scala is not referenced by any code else

2015-11-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9928: --- Assignee: Apache Spark > LogicalLocalTable in ExistingRDD.scala is not referenced by any code

[jira] [Assigned] (SPARK-9928) LogicalLocalTable in ExistingRDD.scala is not referenced by any code else

2015-11-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9928: --- Assignee: (was: Apache Spark) > LogicalLocalTable in ExistingRDD.scala is not referenced

[jira] [Commented] (SPARK-9928) LogicalLocalTable in ExistingRDD.scala is not referenced by any code else

2015-11-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005525#comment-15005525 ] Apache Spark commented on SPARK-9928: - User 'gatorsmile' has created a pull request fo

[jira] [Commented] (SPARK-10759) Missing Python code example in ML Programming guide

2015-11-14 Thread Nathan Davis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005522#comment-15005522 ] Nathan Davis commented on SPARK-10759: -- [~lmoos], is this in progress? I can take it

[jira] [Commented] (SPARK-11725) Let UDF to handle null value

2015-11-14 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005509#comment-15005509 ] Reynold Xin commented on SPARK-11725: - This is the problem of default value in codege

[jira] [Comment Edited] (SPARK-9844) File appender race condition during SparkWorker shutdown

2015-11-14 Thread Jason Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005504#comment-15005504 ] Jason Huang edited comment on SPARK-9844 at 11/14/15 5:38 PM: --

[jira] [Commented] (SPARK-9844) File appender race condition during SparkWorker shutdown

2015-11-14 Thread Jason Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005504#comment-15005504 ] Jason Huang commented on SPARK-9844: Got the same error log in workers and my workers

[jira] [Commented] (SPARK-11725) Let UDF to handle null value

2015-11-14 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005433#comment-15005433 ] Herman van Hovell commented on SPARK-11725: --- I can reproduce the {{-1}} default

[jira] [Commented] (SPARK-11725) Let UDF to handle null value

2015-11-14 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005399#comment-15005399 ] Jeff Zhang commented on SPARK-11725: I am on master > Let UDF to handle null value

[jira] [Updated] (SPARK-7799) Move "StreamingContext.actorStream" to a separate project and deprecate it in StreamingContext

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-7799: - Target Version/s: (was: 1.6.0) > Move "StreamingContext.actorStream" to a separate project and deprecate

[jira] [Updated] (SPARK-10530) Kill other task attempts when one taskattempt belonging the same task is succeeded in speculation

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-10530: -- Target Version/s: (was: 1.6.0) Priority: Minor (was: Major) > Kill other task attempts w

[jira] [Resolved] (SPARK-10081) Skip re-computing getMissingParentStages in DAGScheduler

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-10081. --- Resolution: Won't Fix Target Version/s: (was: 1.6.0) > Skip re-computing getMissingParent

[jira] [Resolved] (SPARK-10526) Display cores/memory on ExecutorsTab

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-10526. --- Resolution: Won't Fix Target Version/s: (was: 1.6.0) > Display cores/memory on ExecutorsT

[jira] [Updated] (SPARK-9516) Improve Thread Dump page

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-9516: - Target Version/s: (was: 1.6.0) > Improve Thread Dump page > > >

[jira] [Updated] (SPARK-9516) Improve Thread Dump page

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-9516: - Priority: Minor (was: Major) > Improve Thread Dump page > > > Ke

[jira] [Updated] (SPARK-10062) Use tut for typechecking and running code in user guides

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-10062: -- Target Version/s: (was: 1.6.0) > Use tut for typechecking and running code in user guides > -

[jira] [Updated] (SPARK-10250) Scala PairRDDFuncitons.groupByKey() should be fault-tolerant of single large groups

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-10250: -- Target Version/s: (was: 1.6.0) > Scala PairRDDFuncitons.groupByKey() should be fault-tolerant of sing

[jira] [Updated] (SPARK-6227) PCA and SVD for PySpark

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-6227: - Target Version/s: (was: 1.6.0) > PCA and SVD for PySpark > --- > > K

[jira] [Updated] (SPARK-7441) Implement microbatch functionality so that Spark Streaming can process a large backlog of existing files discovered in batch in smaller batches

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-7441: - Target Version/s: (was: 1.6.0) > Implement microbatch functionality so that Spark Streaming can process

[jira] [Commented] (SPARK-6280) Remove Akka systemName from Spark

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005396#comment-15005396 ] Sean Owen commented on SPARK-6280: -- Are this and the other Akka-related items targeted fo

[jira] [Updated] (SPARK-11720) Return Double.NaN instead of null for Mean and Average when count = 0

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11720: -- Component/s: SQL > Return Double.NaN instead of null for Mean and Average when count = 0 >

[jira] [Resolved] (SPARK-11669) Python interface to SparkR GLM module

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-11669. --- Resolution: Not A Problem > Python interface to SparkR GLM module > -

[jira] [Updated] (SPARK-11702) Guava ClassLoading Issue When Using Different Hive Metastore Version

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11702: -- Component/s: Spark Core Got it, makes more sense now. > Guava ClassLoading Issue When Using Different

[jira] [Updated] (SPARK-11669) Python interface to SparkR GLM module

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11669: -- Target Version/s: (was: 1.5.0, 1.5.1) [~shubhanshumis...@gmail.com] it doesn't make sense to target r

[jira] [Updated] (SPARK-11732) MiMa excludes miss private classes

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11732: -- Labels: (was: newbie) Fix Version/s: (was: 1.6.0) [~thunterdb] don't set Fix version u

[jira] [Updated] (SPARK-11727) split ExpressionEncoder into FlatEncoder and ProductEncoder

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11727: -- Assignee: Wenchen Fan > split ExpressionEncoder into FlatEncoder and ProductEncoder > -

[jira] [Commented] (SPARK-11725) Let UDF to handle null value

2015-11-14 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005391#comment-15005391 ] Herman van Hovell commented on SPARK-11725: --- I'd rather add a warning than prev

[jira] [Commented] (SPARK-5226) Add DBSCAN Clustering Algorithm to MLlib

2015-11-14 Thread mustafa elbehery (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005355#comment-15005355 ] mustafa elbehery commented on SPARK-5226: - Hello, I would like to use DBSCAN on

[jira] [Updated] (SPARK-11573) correct 'reflective access of structural type member method should be enabled' Scala warnings

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11573: -- Assignee: Gabor Liptak Priority: Trivial (was: Minor) Description: was: > corr

[jira] [Resolved] (SPARK-11573) correct 'reflective access of structural type member method should be enabled' Scala warnings

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-11573. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9550 [https://github.com

[jira] [Commented] (SPARK-11337) Make example code in user guide testable

2015-11-14 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005338#comment-15005338 ] Xusen Yin commented on SPARK-11337: --- [~mengxr] Until now, all docs of ML and MLlib pack

[jira] [Commented] (SPARK-11728) Replace example code in ml-ensembles.md using include_example

2015-11-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005334#comment-15005334 ] Apache Spark commented on SPARK-11728: -- User 'yinxusen' has created a pull request f

[jira] [Assigned] (SPARK-11728) Replace example code in ml-ensembles.md using include_example

2015-11-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11728: Assignee: (was: Apache Spark) > Replace example code in ml-ensembles.md using include_

[jira] [Assigned] (SPARK-11728) Replace example code in ml-ensembles.md using include_example

2015-11-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11728: Assignee: Apache Spark > Replace example code in ml-ensembles.md using include_example > -

[jira] [Updated] (SPARK-11694) Parquet logical types are not being tested properly

2015-11-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-11694: --- Assignee: Hyukjin Kwon > Parquet logical types are not being tested properly > --

[jira] [Resolved] (SPARK-11694) Parquet logical types are not being tested properly

2015-11-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-11694. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9660 [https://github.c

[jira] [Commented] (SPARK-11553) row.getInt(i) if row[i]=null returns 0

2015-11-14 Thread Bartlomiej Alberski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005292#comment-15005292 ] Bartlomiej Alberski commented on SPARK-11553: - Thanks - good to know > row.g

[jira] [Commented] (SPARK-11553) row.getInt(i) if row[i]=null returns 0

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005284#comment-15005284 ] Sean Owen commented on SPARK-11553: --- That's clear already. We normally assign after it'

[jira] [Commented] (SPARK-11553) row.getInt(i) if row[i]=null returns 0

2015-11-14 Thread Bartlomiej Alberski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005270#comment-15005270 ] Bartlomiej Alberski commented on SPARK-11553: - Please assign me to this issue

[jira] [Resolved] (SPARK-11721) The programming guide for Spark SQL in Spark 1.3.0 needs additional imports to work

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-11721. --- Resolution: Not A Problem > The programming guide for Spark SQL in Spark 1.3.0 needs additional impor

[jira] [Reopened] (SPARK-11721) The programming guide for Spark SQL in Spark 1.3.0 needs additional imports to work

2015-11-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-11721: --- > The programming guide for Spark SQL in Spark 1.3.0 needs additional imports > to work > --

[jira] [Comment Edited] (SPARK-10673) spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

2015-11-14 Thread Xin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005249#comment-15005249 ] Xin Wu edited comment on SPARK-10673 at 11/14/15 8:19 AM: -- if th

[jira] [Commented] (SPARK-10673) spark.sql.hive.verifyPartitionPath Attempts to Verify Unregistered Partitions

2015-11-14 Thread Xin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005249#comment-15005249 ] Xin Wu commented on SPARK-10673: if the default is false, {code} if (!sc.conf.verifyParti