[GitHub] spark pull request: [SPARK-2024] Add saveAsSequenceFile to PySpark

2014-07-29 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1338#issuecomment-50442216 Looks good to me as well. @JoshRosen any comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508491 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) { ne

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508543 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) { ne

[GitHub] spark pull request: [SPARK-1630] Turn Null of Java/Scala into None...

2014-07-29 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1551#issuecomment-50442362 We aren't passing completely arbitrary iterators of Java objects to writeIteratorToStream; instead, we only handle iterators of strings and byte arrays. Nulls in data r

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508571 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala --- @@ -104,13 +105,11 @@ class RowMatrix( val nt: Int = n * (

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508559 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) { ne

[GitHub] spark pull request: SPARK-2686 Add Length support to Spark SQL and...

2014-07-29 Thread ueshin
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/1586#issuecomment-50442618 First, I would like to confirm, but which do you want to add to HQL, `Length` or `Strlen`? The title of this PR says to add `Length` to HQL, but the implementation add

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508694 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) {

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508695 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) {

[GitHub] spark pull request: [WIP][SPARK-2677]BasicBlockFetchIterator#next ...

2014-07-29 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/1619#issuecomment-50442727 @witgo @pwendell I have already noticed there is not a configuration for timeout for ConnectionManager, but the timeout for ConnectionManager does not resolve this issue

[GitHub] spark pull request: [SPARK-2024] Add saveAsSequenceFile to PySpark

2014-07-29 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/1338#discussion_r15508664 --- Diff: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala --- @@ -65,20 +66,49 @@ private[python] object SerDeUtil extends Logging {

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508718 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) {

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508725 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) {

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508785 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) { ne

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508790 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) {

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508808 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) {

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1110#discussion_r15508803 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/rdd/RDDFunctions.scala --- @@ -44,6 +47,65 @@ class RDDFunctions[T: ClassTag](self: RDD[T]) {

[GitHub] spark pull request: [SPARK-1667] Jobs never finish successfully on...

2014-07-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1383#issuecomment-50443081 Sorry to come back to this after a while. Disk faults can be transient as well right? I'm not sure if we'd want to exit the executor simply because of one disk fault. ---

[GitHub] spark pull request: [SPARK-2580] [PySpark] keep silent in worker i...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1625#issuecomment-50443121 QA tests have started for PR 1625. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17343/consoleFull --- If

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1346#issuecomment-50443467 QA tests have started for PR 1346. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17344/consoleFull --- If

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1110#issuecomment-50443472 QA tests have started for PR 1110. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17345/consoleFull --- If

[GitHub] spark pull request: [SPARK-2580] [PySpark] keep silent in worker i...

2014-07-29 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1625#issuecomment-50443446 I've merged this into `master` and `branch-1.0`. Thanks Davies! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-2726] and [SPARK-2727] Remove SortOrder...

2014-07-29 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1631#issuecomment-50443804 Looks good to me too, though it might be better to use Java's Arrays.sort instead of Scala's quickSort since Java has fancier algorithms in new versions. --- If your proj

[GitHub] spark pull request: [SPARK-2580] [PySpark] keep silent in worker i...

2014-07-29 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/1625#issuecomment-50443830 @JoshRosen did it include the last commit? I didn't find them in master nor branch-1.0. It's delayed? --- If your project is set up for it, you can reply to this email an

[GitHub] spark pull request: SPARK-2380: Support displaying accumulator val...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1309#issuecomment-50443836 QA results for PR 1309:- This patch FAILED unit tests.For more information see test ouptut:https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17340/consol

[GitHub] spark pull request: SPARK-2686 Add Length support to Spark SQL and...

2014-07-29 Thread javadba
Github user javadba commented on the pull request: https://github.com/apache/spark/pull/1586#issuecomment-50443929 @ueshin That is not what the title reads. Here is the title: Add Length support to Spark SQL and HQL and Strlen support to SQ --- If your project is set u

[GitHub] spark pull request: [SPARK-2580] [PySpark] keep silent in worker i...

2014-07-29 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1625#issuecomment-50443990 @davies Yeah, it included both commits. If you check the [Apache repo](https://git-wip-us.apache.org/repos/asf/spark/repo?p=spark.git;a=summary), you should see the co

[GitHub] spark pull request: [SPARK-1981] Add AWS Kinesis streaming support

2014-07-29 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1434#issuecomment-50443974 @pdeyhim can you take a look over this too when you have a chance? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub a

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1110#issuecomment-50444114 QA tests have started for PR 1110. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17346/consoleFull --- If

[GitHub] spark pull request: [SPARK-2710] [SQL] Build SchemaRDD from a Jdbc...

2014-07-29 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1612#discussion_r15509316 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcResultSetRDD.scala --- @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2710] [SQL] Build SchemaRDD from a Jdbc...

2014-07-29 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1612#discussion_r15509342 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcTypes.scala --- @@ -0,0 +1,56 @@ +/* +* Licensed to the Apache Software Foundat

[GitHub] spark pull request: [SPARK-2710] [SQL] Build SchemaRDD from a Jdbc...

2014-07-29 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1612#discussion_r15509361 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcTypes.scala --- @@ -0,0 +1,56 @@ +/* +* Licensed to the Apache Software Foundat

[GitHub] spark pull request: fix a mistaken type of "if" in description of ...

2014-07-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1633#issuecomment-50444578 Do you mind closing the pull request? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec

[GitHub] spark pull request: [SPARK-2726] and [SPARK-2727] Remove SortOrder...

2014-07-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1631#issuecomment-50444740 I tried that - had some issues with types between Scala and Java and resorted to the current implementation. In any case because this code will likely be replaced soon by th

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1110#issuecomment-50445699 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request: [SPARK-2580] [PySpark] keep silent in worker i...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1625#issuecomment-50446807 QA results for PR 1625:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1110#issuecomment-50446850 QA results for PR 1110:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2726] and [SPARK-2727] Remove SortOrder...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1631#issuecomment-50447222 QA results for PR 1631:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds the following public classes (experimental):class SparkSQLOperationMana

[GitHub] spark pull request: [SPARK-2726] and [SPARK-2727] Remove SortOrder...

2014-07-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1631#issuecomment-50447480 Ok I'm merging this. Thanks for reviewing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project d

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1110#issuecomment-50447573 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1110#issuecomment-50447677 QA results for PR 1110:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1110#issuecomment-50447729 Merging this in master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [SPARK-791] [PySpark] fix pickle itemgetter wi...

2014-07-29 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1627#issuecomment-50447804 Merged into master, branch-1.0, and branch-0.9. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1110#issuecomment-50447939 QA tests have started for PR 1110. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17347/consoleFull --- If

[GitHub] spark pull request: [SPARK-2726] and [SPARK-2727] Remove SortOrder...

2014-07-29 Thread rxin
Github user rxin closed the pull request at: https://github.com/apache/spark/pull/1631 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15510908 --- Diff: python/pyspark/sql.py --- @@ -20,8 +20,457 @@ from py4j.protocol import Py4JError -__all__ = ["SQLContext", "HiveContext", "LocalHi

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15510935 --- Diff: python/pyspark/sql.py --- @@ -20,8 +20,457 @@ from py4j.protocol import Py4JError -__all__ = ["SQLContext", "HiveContext", "LocalHi

[GitHub] spark pull request: SPARK-2646. log4j initialization not quite com...

2014-07-29 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1547#discussion_r15510944 --- Diff: core/src/main/scala/org/apache/spark/Logging.scala --- @@ -110,23 +110,26 @@ trait Logging { } private def initializeLogging() {

[GitHub] spark pull request: SPARK-2646. log4j initialization not quite com...

2014-07-29 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1547#discussion_r15510992 --- Diff: core/src/main/scala/org/apache/spark/Logging.scala --- @@ -110,23 +110,26 @@ trait Logging { } private def initializeLogging() {

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15510995 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/WrapDynamic.scala --- @@ -21,7 +21,9 @@ import scala.language.dynamics

[GitHub] spark pull request: SPARK-2646. log4j initialization not quite com...

2014-07-29 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1547#issuecomment-50448651 Thanks for catching this and digging into the fix. Some small questions in the PR, but generally looks good! --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15511092 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala --- @@ -201,47 +231,139 @@ object FractionalType { } }

[GitHub] spark pull request: Minor indentation and comment typo fixes.

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1630#issuecomment-50448836 QA results for PR 1630:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15511210 --- Diff: sql/core/src/main/java/org/apache/spark/sql/api/java/types/DataType.java --- @@ -0,0 +1,212 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request: SPARK-2646. log4j initialization not quite com...

2014-07-29 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/1547#discussion_r15511218 --- Diff: core/src/main/scala/org/apache/spark/Logging.scala --- @@ -110,23 +110,26 @@ trait Logging { } private def initializeLogging() {

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15511199 --- Diff: sql/core/src/main/java/org/apache/spark/sql/api/java/types/DataType.java --- @@ -0,0 +1,212 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15511259 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala --- @@ -89,6 +90,45 @@ class SQLContext(@transient val sparkContext: SparkContext)

[GitHub] spark pull request: SPARK-2646. log4j initialization not quite com...

2014-07-29 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/1547#discussion_r15511267 --- Diff: core/src/main/scala/org/apache/spark/Logging.scala --- @@ -110,23 +110,26 @@ trait Logging { } private def initializeLogging() {

[GitHub] spark pull request: [SPARK-1687] [PySpark] pickable namedtuple

2014-07-29 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1623#issuecomment-50449220 Is there a way to do this that doesn't require PySpark to be imported before the namedtuples are created? Can you directly replace the `__reduce__` method on the `name

[GitHub] spark pull request: Minor indentation and comment typo fixes.

2014-07-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1630#issuecomment-50449299 Merging this in master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15511405 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala --- @@ -259,8 +268,12 @@ private[sql] object JsonRDD extends Logging { // th

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15511457 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTypes.scala --- @@ -140,10 +142,12 @@ private[parquet] object ParquetTypesConverter extends

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15511453 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTypes.scala --- @@ -116,7 +116,7 @@ private[parquet] object ParquetTypesConverter extends

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r15511498 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/types/util/DataTypeConversions.scala --- @@ -0,0 +1,124 @@ +/* + * Licensed to the Apache Softwar

[GitHub] spark pull request: [SPARK-2521] Broadcast RDD object (instead of ...

2014-07-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1498#issuecomment-50450013 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-2521] Broadcast RDD object (instead of ...

2014-07-29 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1498#issuecomment-50450041 For the sake of it, run the test one more time to make sure it is passing deterministically. --- If your project is set up for it, you can reply to this email and hav

[GitHub] spark pull request: SPARK-2686 Add Length and Strlen support to Sp...

2014-07-29 Thread ueshin
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/1586#issuecomment-50450124 @javadba Ah, I see. Thank you for your detail. I'll continue to review :) --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: SPARK-2686 Add Length and Strlen support to Sp...

2014-07-29 Thread javadba
Github user javadba commented on the pull request: https://github.com/apache/spark/pull/1586#issuecomment-50450103 The latest changes (also pushed to the PR on github) were re-run an verified on my local machine. The hive tests take over 4 hours which is most of the 4:45 for all combi

[GitHub] spark pull request: [SQL]change some test lists

2014-07-29 Thread adrian-wang
GitHub user adrian-wang opened a pull request: https://github.com/apache/spark/pull/1634 [SQL]change some test lists 1. there's no `hook_context.q` but a `hook_context_cs.q` in query folder 2. there's no `compute_stats_table.q` in query folder 3. there's no `having1.q` in que

[GitHub] spark pull request: [SQL]change some test lists

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1634#issuecomment-50450576 QA tests have started for PR 1634. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17348/consoleFull --- If

[GitHub] spark pull request: [SPARK-2521] Broadcast RDD object (instead of ...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1498#issuecomment-50450582 QA tests have started for PR 1498. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17349/consoleFull --- If

[GitHub] spark pull request: SPARK-2078 Use ISO8601 date format in logging

2014-07-29 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1018#issuecomment-50450560 Jumping in as I tour emails this morning -- the current log format is like `14/07/29 09:43:53`. Full ISO format would be like `2014-07-29T09:43:53.1234-05:00`. Verbose ind

[GitHub] spark pull request: fix a mistaken type of "if" in description of ...

2014-07-29 Thread hzw19900416
Github user hzw19900416 closed the pull request at: https://github.com/apache/spark/pull/1633 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2174][MLLIB] treeReduce and treeAggrega...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1110#issuecomment-50451405 QA results for PR 1110:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/1346#issuecomment-50451426 @yhuai can you describe a little more about the `containsNull` for `ArrayType` and `MapType`, in my understanding, `Map` and `Array` contains null in most of cases

[GitHub] spark pull request: SPARK-1416: PySpark support for SequenceFile a...

2014-07-29 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/455#issuecomment-50451633 @rjurney This means you have two conflicting versions of Netty in the build. It may or may not be to do with the JIRA you cite, just because there are lots of Netties float

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1346#issuecomment-50452464 QA results for PR 1346:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [WIP][SPARK-2677]BasicBlockFetchIterator#next ...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1619#issuecomment-50454065 QA tests have started for PR 1619. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17350/consoleFull --- If

[GitHub] spark pull request: SPARK-2632, SPARK-2576. Fixed by only importin...

2014-07-29 Thread ScrapCodes
GitHub user ScrapCodes opened a pull request: https://github.com/apache/spark/pull/1635 SPARK-2632, SPARK-2576. Fixed by only importing what is necessary during class definition. You can merge this pull request into a Git repository by running: $ git pull https://github.com/S

[GitHub] spark pull request: [SPARK-2521] Broadcast RDD object (instead of ...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1498#issuecomment-50455156 QA results for PR 1498:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds the following public classes (experimental):abstract class Dependency[T

[GitHub] spark pull request: SPARK-2632, SPARK-2576. Fixed by only importin...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1635#issuecomment-50455375 QA tests have started for PR 1635. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17351/consoleFull --- If

[GitHub] spark pull request: [SPARK-2532] Consolidated shuffle fixes

2014-07-29 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/1609#issuecomment-50455483 All pending fixes work be done. I dont think there are any pieces missing in the merge from internal branch to master. Open for review, thanks ! --- If your proj

[GitHub] spark pull request: [SPARK-2532] Consolidated shuffle fixes

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1609#issuecomment-50455818 QA tests have started for PR 1609. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17352/consoleFull --- If

[GitHub] spark pull request: [WIP][SPARK-2677]BasicBlockFetchIterator#next ...

2014-07-29 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1619#issuecomment-50456598 @sarutak I think add a heartbeat detection mechanism is a good solution --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] spark pull request: [WIP][SPARK-2677]BasicBlockFetchIterator#next ...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1619#issuecomment-50458592 QA results for PR 1619:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: SPARK-2000:cannot connect to cluster in Standa...

2014-07-29 Thread CrazyJvm
Github user CrazyJvm commented on the pull request: https://github.com/apache/spark/pull/952#issuecomment-50458866 @mateiz YES, i agree. I was motivated by the "http://spark.apache.org/docs/latest/spark-standalone.html"; , which says "Note that if you are running spark-shell from one

[GitHub] spark pull request: [SPARK-2583] ConnectionManager cannot distingu...

2014-07-29 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/1490#issuecomment-50459001 Thank you for you comment @JoshRosen , @rxin . Of course I'll try it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitH

[GitHub] spark pull request: [WIP][SPARK-2677]BasicBlockFetchIterator#next ...

2014-07-29 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1619#issuecomment-50458991 @sarutak [ConnectionManager.scala#L259](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/network/ConnectionManager.scala#L259) to deal wit

[GitHub] spark pull request: SPARK-2632, SPARK-2576. Fixed by only importin...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1635#issuecomment-50459369 QA results for PR 1635:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2532] Consolidated shuffle fixes

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1609#issuecomment-50460011 QA results for PR 1609:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SQL]change some test lists

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1634#issuecomment-50460326 QA results for PR 1634:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: SPARK-2686 Add Length and Strlen support to Sp...

2014-07-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/1586#discussion_r15516740 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -209,6 +212,82 @@ case class EndsWith(left: Expr

[GitHub] spark pull request: SPARK-2686 Add Length and Strlen support to Sp...

2014-07-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/1586#discussion_r15517240 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -209,6 +212,82 @@ case class EndsWith(left: Expr

[GitHub] spark pull request: SPARK-2686 Add Length and Strlen support to Sp...

2014-07-29 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/1586#discussion_r15517292 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -209,6 +212,82 @@ case class EndsWith(left: Expr

[GitHub] spark pull request: SPARK-2686 Add Length and Strlen support to Sp...

2014-07-29 Thread ueshin
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/1586#issuecomment-50465289 @javadba I couldn't understand what you want `Strlen` to return. Could you clarify the semantics of `Strlen` again, please? --- If your project is set up for it, you c

[GitHub] spark pull request: SPARK-2294: fix locality inversion bug in Task...

2014-07-29 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/1313#discussion_r15518169 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -751,20 +787,7 @@ private[spark] class TaskSetManager( levels.

[GitHub] spark pull request: SPARK-2294: fix locality inversion bug in Task...

2014-07-29 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/1313#discussion_r15518594 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -246,28 +246,36 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: SPARK-2294: fix locality inversion bug in Task...

2014-07-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1313#issuecomment-50468872 QA tests have started for PR 1313. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17353/consoleFull --- If

[GitHub] spark pull request: SPARK-2294: fix locality inversion bug in Task...

2014-07-29 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/1313#issuecomment-50469155 Hi, @mateiz , I just updated the patch, The reason I kept a per-node tracking of NODE_LOCAL-only tasks is that, we don't want to delay the scheduling of NoPref

[GitHub] spark pull request: [SPARK-2580] [PySpark] keep silent in worker i...

2014-07-29 Thread mattf
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/1625#issuecomment-50470361 +1 lgtm, validated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

  1   2   3   4   5   6   7   >