[jira] [Updated] (SPARK-11176) Umbrella ticket for wholeTextFiles bugs

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-11176: --- Summary: Umbrella ticket for wholeTextFiles bugs (was: Umbrella ticket for wholeTextFiles + S3

[jira] [Updated] (SPARK-10994) Clustering coefficient computation in GraphX

2015-10-19 Thread Yang Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yang updated SPARK-10994: -- Description: The Clustering Coefficient (CC) is a fundamental measure in social (or other type of)

[jira] [Commented] (SPARK-10994) Clustering coefficient computation in GraphX

2015-10-19 Thread Yang Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962929#comment-14962929 ] Yang Yang commented on SPARK-10994: --- update with describing our motivation in more details >

[jira] [Created] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

2015-10-19 Thread prakhar jauhari (JIRA)
prakhar jauhari created SPARK-11181: --- Summary: Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled. Key: SPARK-11181 URL: https://issues.apache.org/jira/browse/SPARK-11181

[jira] [Created] (SPARK-11179) Push filters through aggregate if filters are subset of 'group by' expressions

2015-10-19 Thread Nitin Goyal (JIRA)
Nitin Goyal created SPARK-11179: --- Summary: Push filters through aggregate if filters are subset of 'group by' expressions Key: SPARK-11179 URL: https://issues.apache.org/jira/browse/SPARK-11179

[jira] [Commented] (SPARK-11144) Add SparkLauncher for Spark Streaming, Spark SQL, etc

2015-10-19 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962903#comment-14962903 ] Jean-Baptiste Onofré commented on SPARK-11144: -- Hi Yuhang, just to confirm: an utility like

[jira] [Commented] (SPARK-11157) Allow Spark to be built without assemblies

2015-10-19 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962897#comment-14962897 ] Jean-Baptiste Onofré commented on SPARK-11157: -- Agree with Marcelo. It's something that I

[jira] [Updated] (SPARK-11176) Umbrella ticket for wholeTextFiles bugs

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-11176: --- Description: This umbrella ticket gathers together several distinct bug reports related to problems

[jira] [Assigned] (SPARK-11180) DataFrame.na.fill does not support Boolean Type:

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11180: Assignee: Apache Spark > DataFrame.na.fill does not support Boolean Type: >

[jira] [Commented] (SPARK-11180) DataFrame.na.fill does not support Boolean Type:

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962923#comment-14962923 ] Apache Spark commented on SPARK-11180: -- User 'rishabhbhardwaj' has created a pull request for this

[jira] [Assigned] (SPARK-11180) DataFrame.na.fill does not support Boolean Type:

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11180: Assignee: (was: Apache Spark) > DataFrame.na.fill does not support Boolean Type: >

[jira] [Commented] (SPARK-11179) Push filters through aggregate if filters are subset of 'group by' expressions

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962931#comment-14962931 ] Apache Spark commented on SPARK-11179: -- User 'nitin2goyal' has created a pull request for this

[jira] [Assigned] (SPARK-11179) Push filters through aggregate if filters are subset of 'group by' expressions

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11179: Assignee: Apache Spark > Push filters through aggregate if filters are subset of 'group

[jira] [Commented] (SPARK-11132) Mean Shift algorithm integration

2015-10-19 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962960#comment-14962960 ] Beck Gaël commented on SPARK-11132: --- Thank you, It's not the case for mean shift, i hope it will be.

[jira] [Commented] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

2015-10-19 Thread prakhar jauhari (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962961#comment-14962961 ] prakhar jauhari commented on SPARK-11181: - On analysing the code (Spark 1.3.1): When my DN goes

[jira] [Updated] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

2015-10-19 Thread prakhar jauhari (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] prakhar jauhari updated SPARK-11181: Fix Version/s: (was: 1.5.2) > Spark Yarn : Spark reducing total executors count even

[jira] [Created] (SPARK-11180) DataFrameNaFunctions fills does not support Boolean Type:

2015-10-19 Thread Satya Narayan (JIRA)
Satya Narayan created SPARK-11180: - Summary: DataFrameNaFunctions fills does not support Boolean Type: Key: SPARK-11180 URL: https://issues.apache.org/jira/browse/SPARK-11180 Project: Spark

[jira] [Commented] (SPARK-11177) sc.wholeTextFiles throws ArrayIndexOutOfBoundsException when S3 file has zero bytes

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962913#comment-14962913 ] Josh Rosen commented on SPARK-11177: It looks like this is caused by MAPREDUCE-4470, which is not

[jira] [Updated] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

2015-10-19 Thread prakhar jauhari (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] prakhar jauhari updated SPARK-11181: Target Version/s: 1.3.2 (was: 1.3.2, 1.5.2) > Spark Yarn : Spark reducing total executors

[jira] [Assigned] (SPARK-6541) Executor table on Stage page should sort by Executor ID numerically, not lexically

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6541: --- Assignee: (was: Apache Spark) > Executor table on Stage page should sort by Executor ID

[jira] [Updated] (SPARK-11180) DataFrame.na.fill does not support Boolean Type:

2015-10-19 Thread Satya Narayan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satya Narayan updated SPARK-11180: -- Summary: DataFrame.na.fill does not support Boolean Type: (was: DataFrameNaFunctions fills

[jira] [Updated] (SPARK-11180) DataFrame.na.fill does not support Boolean Type:

2015-10-19 Thread Satya Narayan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satya Narayan updated SPARK-11180: -- Description: Currently DataFrame.na.fill does not support Boolean primitive type. We have

[jira] [Commented] (SPARK-6541) Executor table on Stage page should sort by Executor ID numerically, not lexically

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962895#comment-14962895 ] Apache Spark commented on SPARK-6541: - User 'jbonofre' has created a pull request for this issue:

[jira] [Assigned] (SPARK-6541) Executor table on Stage page should sort by Executor ID numerically, not lexically

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6541: --- Assignee: Apache Spark > Executor table on Stage page should sort by Executor ID

[jira] [Commented] (SPARK-11167) Incorrect type resolution on heterogeneous data structures

2015-10-19 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962900#comment-14962900 ] Sun Rui commented on SPARK-11167: - For a DataFrame, each column is a collection of values of same type.

[jira] [Assigned] (SPARK-11179) Push filters through aggregate if filters are subset of 'group by' expressions

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11179: Assignee: (was: Apache Spark) > Push filters through aggregate if filters are subset

[jira] [Resolved] (SPARK-11128) strange NPE when writing in non-existing S3 bucket

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-11128. --- Resolution: Not A Problem Not a problem with Spark, that is. > strange NPE when writing in

[jira] [Commented] (SPARK-10352) Replace SQLTestData internal usages of String with UTF8String

2015-10-19 Thread Harsh Rathi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963194#comment-14963194 ] Harsh Rathi commented on SPARK-10352: - Why this is not a problem ? I am writing custom explode

[jira] [Created] (SPARK-11184) Declare most of .mllib code not-Experimental

2015-10-19 Thread Sean Owen (JIRA)
Sean Owen created SPARK-11184: - Summary: Declare most of .mllib code not-Experimental Key: SPARK-11184 URL: https://issues.apache.org/jira/browse/SPARK-11184 Project: Spark Issue Type:

[jira] [Created] (SPARK-11182) HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode

2015-10-19 Thread Liangliang Gu (JIRA)
Liangliang Gu created SPARK-11182: - Summary: HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode Key: SPARK-11182 URL:

[jira] [Updated] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11181: -- Flags: (was: Patch,Important) Target Version/s: (was: 1.3.2) Fix Version/s:

[jira] [Updated] (SPARK-10921) Completely remove the use of SparkContext.preferredNodeLocationData

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-10921: -- Assignee: Jacek Laskowski > Completely remove the use of SparkContext.preferredNodeLocationData >

[jira] [Resolved] (SPARK-10633) Persisting Spark stream to MySQL - Spark tries to create the table for every stream even if it exist already.

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-10633. --- Resolution: Not A Problem > Persisting Spark stream to MySQL - Spark tries to create the table for

[jira] [Assigned] (SPARK-11182) HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11182: Assignee: (was: Apache Spark) > HDFS Delegation Token will be expired when calling >

[jira] [Commented] (SPARK-11182) HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963092#comment-14963092 ] Apache Spark commented on SPARK-11182: -- User 'marsishandsome' has created a pull request for this

[jira] [Commented] (SPARK-11182) HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode

2015-10-19 Thread Liangliang Gu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963091#comment-14963091 ] Liangliang Gu commented on SPARK-11182: --- https://github.com/apache/spark/pull/9168 > HDFS

[jira] [Assigned] (SPARK-11182) HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11182: Assignee: Apache Spark > HDFS Delegation Token will be expired when calling >

[jira] [Created] (SPARK-11183) enable support for mesos 0.24+

2015-10-19 Thread Ioannis Polyzos (JIRA)
Ioannis Polyzos created SPARK-11183: --- Summary: enable support for mesos 0.24+ Key: SPARK-11183 URL: https://issues.apache.org/jira/browse/SPARK-11183 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-5250) EOFException in when reading gzipped files from S3 with wholeTextFiles

2015-10-19 Thread Mojmir Vinkler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963110#comment-14963110 ] Mojmir Vinkler commented on SPARK-5250: --- Yes, it's caused by reading a corrupt file (we only

[jira] [Resolved] (SPARK-10921) Completely remove the use of SparkContext.preferredNodeLocationData

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-10921. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8976

[jira] [Commented] (SPARK-10861) Univariate Statistics: Adding range support as UDAF

2015-10-19 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963029#comment-14963029 ] Jeff Zhang commented on SPARK-10861: [~JihongMA] what's your progress on this ? > Univariate

[jira] [Commented] (SPARK-6645) StructField/StructType and related classes are not in the Scaladoc

2015-10-19 Thread Rishabh Bhardwaj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963345#comment-14963345 ] Rishabh Bhardwaj commented on SPARK-6645: - I can see StructField/StructType classes in ScalaDoc in

[jira] [Created] (SPARK-11185) Add more task metrics to the "all Stages Page"

2015-10-19 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-11185: - Summary: Add more task metrics to the "all Stages Page" Key: SPARK-11185 URL: https://issues.apache.org/jira/browse/SPARK-11185 Project: Spark Issue Type:

[jira] [Updated] (SPARK-11186) Caseness inconsistency between SQLContext and HiveContext

2015-10-19 Thread Santiago M. Mola (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Santiago M. Mola updated SPARK-11186: - Description: Default catalog behaviour for caseness is different in {{SQLContext}} and

[jira] [Comment Edited] (SPARK-11162) Allow enabling debug logging from the command line

2015-10-19 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963509#comment-14963509 ] Ryan Williams edited comment on SPARK-11162 at 10/19/15 4:01 PM: - In the

[jira] [Commented] (SPARK-11162) Allow enabling debug logging from the command line

2015-10-19 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963509#comment-14963509 ] Ryan Williams commented on SPARK-11162: --- In the second message (of 2 total, afaict) on the thread,

[jira] [Comment Edited] (SPARK-11162) Allow enabling debug logging from the command line

2015-10-19 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963509#comment-14963509 ] Ryan Williams edited comment on SPARK-11162 at 10/19/15 3:59 PM: - In the

[jira] [Commented] (SPARK-11162) Allow enabling debug logging from the command line

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963511#comment-14963511 ] Sean Owen commented on SPARK-11162: --- Related: https://issues.apache.org/jira/browse/SPARK-11105 In

[jira] [Commented] (SPARK-11161) Viewing the web UI for the first time unpersists a cached RDD

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963522#comment-14963522 ] Sean Owen commented on SPARK-11161: --- Why would it be useful to continue to cache an RDD that can't be

[jira] [Assigned] (SPARK-11176) Umbrella ticket for wholeTextFiles bugs

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-11176: -- Assignee: Josh Rosen > Umbrella ticket for wholeTextFiles bugs >

[jira] [Commented] (SPARK-10780) Set initialModel in KMeans in Pipelines API

2015-10-19 Thread Jayant Shekhar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963827#comment-14963827 ] Jayant Shekhar commented on SPARK-10780: Sounds good [~xusen] and [~josephkb] In the process of

[jira] [Commented] (SPARK-11176) Umbrella ticket for wholeTextFiles bugs

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963825#comment-14963825 ] Josh Rosen commented on SPARK-11176: Going to close this as now, since all child tickets have been

[jira] [Resolved] (SPARK-11176) Umbrella ticket for wholeTextFiles bugs

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-11176. Resolution: Incomplete > Umbrella ticket for wholeTextFiles bugs >

[jira] [Resolved] (SPARK-11027) Better group distinct columns in query compilation

2015-10-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-11027. -- Resolution: Won't Fix > Better group distinct columns in query compilation >

[jira] [Created] (SPARK-11193) Spark 1.5+ Kinesis Streaming - ClassCastException when starting KinesisReceiver

2015-10-19 Thread Phil Kallos (JIRA)
Phil Kallos created SPARK-11193: --- Summary: Spark 1.5+ Kinesis Streaming - ClassCastException when starting KinesisReceiver Key: SPARK-11193 URL: https://issues.apache.org/jira/browse/SPARK-11193

[jira] [Created] (SPARK-11192) When graphite metric sink is enabled, spark sql leaks org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time

2015-10-19 Thread Blake Livingston (JIRA)
Blake Livingston created SPARK-11192: Summary: When graphite metric sink is enabled, spark sql leaks org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time Key: SPARK-11192 URL:

[jira] [Assigned] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11194: Assignee: Yin Huai (was: Apache Spark) > Use a single URLClassLoader for jars added

[jira] [Assigned] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11194: Assignee: Apache Spark (was: Yin Huai) > Use a single URLClassLoader for jars added

[jira] [Updated] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-11194: - Description: Right now, we stack a new URLClassLoader when a user add a jar through SQL's add jar

[jira] [Created] (SPARK-11190) SparkR support for cassandra collection types.

2015-10-19 Thread Bilind Hajer (JIRA)
Bilind Hajer created SPARK-11190: Summary: SparkR support for cassandra collection types. Key: SPARK-11190 URL: https://issues.apache.org/jira/browse/SPARK-11190 Project: Spark Issue Type:

[jira] [Updated] (SPARK-10955) Warn if dynamic allocation is enabled for Streaming jobs

2015-10-19 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-10955: -- Summary: Warn if dynamic allocation is enabled for Streaming jobs (was: Disable dynamic

[jira] [Commented] (SPARK-11027) Better group distinct columns in query compilation

2015-10-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963990#comment-14963990 ] Yin Huai commented on SPARK-11027: -- As pointed out by [~joshrosen] (see

[jira] [Created] (SPARK-11191) [1.5] Can't create UDF's using hive thrift service

2015-10-19 Thread David Ross (JIRA)
David Ross created SPARK-11191: -- Summary: [1.5] Can't create UDF's using hive thrift service Key: SPARK-11191 URL: https://issues.apache.org/jira/browse/SPARK-11191 Project: Spark Issue Type:

[jira] [Commented] (SPARK-11191) [1.5] Can't create UDF's using hive thrift service

2015-10-19 Thread David Ross (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964048#comment-14964048 ] David Ross commented on SPARK-11191: I will add that the exact same thing happens when you don't use

[jira] [Updated] (SPARK-11180) Support BooleanType in DataFrame.na.fill

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-11180: Summary: Support BooleanType in DataFrame.na.fill (was: DataFrame.na.fill does not support

[jira] [Commented] (SPARK-10754) table and column name are case sensitive when json Dataframe was registered as tempTable using JavaSparkContext.

2015-10-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964041#comment-14964041 ] Yin Huai commented on SPARK-10754: -- Can you use {{HiveContext}}, which set {{spark.sql.caseSensitive}}

[jira] [Updated] (SPARK-11192) When graphite metric sink is enabled, spark sql leaks org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time

2015-10-19 Thread Blake Livingston (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Livingston updated SPARK-11192: - Description: Noticed that slowly, over the course of a day or two, heap memory usage on

[jira] [Commented] (SPARK-11184) Declare most of .mllib code not-Experimental

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964077#comment-14964077 ] Apache Spark commented on SPARK-11184: -- User 'srowen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-11184) Declare most of .mllib code not-Experimental

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11184: Assignee: Apache Spark > Declare most of .mllib code not-Experimental >

[jira] [Assigned] (SPARK-11184) Declare most of .mllib code not-Experimental

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11184: Assignee: (was: Apache Spark) > Declare most of .mllib code not-Experimental >

[jira] [Commented] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964113#comment-14964113 ] Apache Spark commented on SPARK-11194: -- User 'yhuai' has created a pull request for this issue:

[jira] [Updated] (SPARK-11180) DataFrame.na.fill does not support Boolean Type:

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-11180: Description: Currently DataFrame.na.fill does not support Boolean primitive type. We have use

[jira] [Commented] (SPARK-10645) Bivariate Statistics: Spearman's Correlation support as UDAF

2015-10-19 Thread Arvind Surve (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964068#comment-14964068 ] Arvind Surve commented on SPARK-10645: -- Spearman's correlation coefficient (SpCoeff) does not fit

[jira] [Resolved] (SPARK-11180) DataFrame.na.fill does not support Boolean Type:

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-11180. - Resolution: Fixed Fix Version/s: 1.6.0 > DataFrame.na.fill does not support Boolean

[jira] [Updated] (SPARK-10955) Warn if dynamic allocation is enabled for Streaming jobs

2015-10-19 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-10955: -- Description: Spark streaming can be tricky with dynamic allocation and can lose data if not

[jira] [Commented] (SPARK-11190) SparkR support for cassandra collection types.

2015-10-19 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964032#comment-14964032 ] Shivaram Venkataraman commented on SPARK-11190: --- cc [~sunrui] Could you try this on the

[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps

2015-10-19 Thread Charles Allen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964043#comment-14964043 ] Charles Allen commented on SPARK-11016: --- [~srowen] I confirmed locally that

[jira] [Created] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Yin Huai (JIRA)
Yin Huai created SPARK-11194: Summary: Use a single URLClassLoader for jars added through SQL's "ADD JAR" command Key: SPARK-11194 URL: https://issues.apache.org/jira/browse/SPARK-11194 Project: Spark

[jira] [Commented] (SPARK-5929) Pyspark: Register a pip requirements file with spark_context

2015-10-19 Thread buckhx (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963655#comment-14963655 ] buckhx commented on SPARK-5929: --- I also included an add module that will bundle and ship a module that has

[jira] [Updated] (SPARK-11179) Push filters through aggregate if filters are subset of 'group by' expressions

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11179: -- Fix Version/s: (was: 1.6.0) [~nitin2goyal] this can't have a Fix version. > Push filters through

[jira] [Resolved] (SPARK-11119) cleanup unsafe array and map

2015-10-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9131

[jira] [Resolved] (SPARK-5250) EOFException in when reading gzipped files from S3 with wholeTextFiles

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-5250. --- Resolution: Cannot Reproduce > EOFException in when reading gzipped files from S3 with wholeTextFiles

[jira] [Created] (SPARK-11188) Elide stacktraces in bin/spark-sql for AnalysisExceptions

2015-10-19 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-11188: Summary: Elide stacktraces in bin/spark-sql for AnalysisExceptions Key: SPARK-11188 URL: https://issues.apache.org/jira/browse/SPARK-11188 Project: Spark

[jira] [Commented] (SPARK-11184) Declare most of .mllib code not-Experimental

2015-10-19 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963692#comment-14963692 ] Joseph K. Bradley commented on SPARK-11184: --- I agree we need to remove more of those tags;

[jira] [Resolved] (SPARK-11177) sc.wholeTextFiles throws ArrayIndexOutOfBoundsException when S3 file has zero bytes

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-11177. Resolution: Won't Fix I'm going to resolve this as "Won't Fix", since I think that the difficultly

[jira] [Updated] (SPARK-11187) Add Newton-Raphson Step per Tree to GBDT Implementation

2015-10-19 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-11187: Shepherd: DB Tsai Affects Version/s: (was: 1.5.1) 1.6.0

[jira] [Commented] (SPARK-5250) EOFException in when reading gzipped files from S3 with wholeTextFiles

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963817#comment-14963817 ] Josh Rosen commented on SPARK-5250: --- Ah, gotcha. I'm going to resolve this as "Cannot Reproduce" for the

[jira] [Commented] (SPARK-11150) Dynamic partition pruning

2015-10-19 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963849#comment-14963849 ] Ruslan Dautkhanov commented on SPARK-11150: --- Will partition-wise join will also be handled by

[jira] [Created] (SPARK-11189) History server is not able to parse some application report

2015-10-19 Thread JIRA
Jean-Baptiste Onofré created SPARK-11189: Summary: History server is not able to parse some application report Key: SPARK-11189 URL: https://issues.apache.org/jira/browse/SPARK-11189 Project:

[jira] [Commented] (SPARK-11189) History server is not able to parse some application report

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963936#comment-14963936 ] Sean Owen commented on SPARK-11189: --- It looks like you have a truncated input file. Are there any other

[jira] [Commented] (SPARK-4240) Refine Tree Predictions in Gradient Boosting to Improve Prediction Accuracy.

2015-10-19 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963648#comment-14963648 ] Joseph K. Bradley commented on SPARK-4240: -- This conversation slipped under my radar somehow; my

[jira] [Updated] (SPARK-11177) sc.wholeTextFiles throws ArrayIndexOutOfBoundsException when S3 file has zero bytes

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11177: -- Component/s: Input/Output > sc.wholeTextFiles throws ArrayIndexOutOfBoundsException when S3 file has

[jira] [Resolved] (SPARK-10668) Use WeightedLeastSquares in LinearRegression with L2 regularization if the number of features is small

2015-10-19 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-10668. - Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8884

[jira] [Resolved] (SPARK-4414) SparkContext.wholeTextFiles Doesn't work with S3 Buckets

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4414. --- Resolution: Won't Fix I'm going to resolve this as "Won't Fix", since I think that the difficultly /

[jira] [Created] (SPARK-11187) Add Newton-Raphson Step per Tree to GBDT Implementation

2015-10-19 Thread Joseph Babcock (JIRA)
Joseph Babcock created SPARK-11187: -- Summary: Add Newton-Raphson Step per Tree to GBDT Implementation Key: SPARK-11187 URL: https://issues.apache.org/jira/browse/SPARK-11187 Project: Spark

[jira] [Updated] (SPARK-9643) Error serializing datetimes with timezones using Dataframes and Parquet

2015-10-19 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-9643: - Assignee: Alex Angelini > Error serializing datetimes with timezones using Dataframes and Parquet >

[jira] [Commented] (SPARK-10994) Clustering coefficient computation in GraphX

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963806#comment-14963806 ] Reynold Xin commented on SPARK-10994: - [~sherlockbourne] I am sure this is a pretty good algorithm,

[jira] [Commented] (SPARK-11186) Caseness inconsistency between SQLContext and HiveContext

2015-10-19 Thread kevin yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963835#comment-14963835 ] kevin yu commented on SPARK-11186: -- Hello Santiago: How did you run the above code? did you get any

[jira] [Updated] (SPARK-11188) Elide stacktraces in bin/spark-sql for AnalysisExceptions

2015-10-19 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-11188: - Target Version/s: 1.4.2, 1.5.2, 1.6.0 (was: 1.6.0) > Elide stacktraces in bin/spark-sql

[jira] [Created] (SPARK-11196) Support for equality and pushdown of filters on some UDTs

2015-10-19 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-11196: Summary: Support for equality and pushdown of filters on some UDTs Key: SPARK-11196 URL: https://issues.apache.org/jira/browse/SPARK-11196 Project: Spark

  1   2   >