git commit: HOTFIX: Bump spark-ec2 version to 1.1.0
Repository: spark Updated Branches: refs/heads/branch-1.1 a4a7a2414 - c71b5c6db HOTFIX: Bump spark-ec2 version to 1.1.0 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c71b5c6d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c71b5c6d Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c71b5c6d Branch: refs/heads/branch-1.1 Commit: c71b5c6db151cfc63bfeabdc88034c3dd9dc9e60 Parents: a4a7a24 Author: Patrick Wendell pwend...@gmail.com Authored: Fri Aug 29 11:20:45 2014 -0700 Committer: Patrick Wendell pwend...@gmail.com Committed: Fri Aug 29 11:20:45 2014 -0700 -- ec2/spark_ec2.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/c71b5c6d/ec2/spark_ec2.py -- diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py index afef4ef..59356c5 100755 --- a/ec2/spark_ec2.py +++ b/ec2/spark_ec2.py @@ -82,7 +82,7 @@ def parse_args(): between zones applies)) parser.add_option(-a, --ami, help=Amazon Machine Image ID to use) parser.add_option( --v, --spark-version, default=1.0.0, +-v, --spark-version, default=1.1.0, help=Version of Spark to use: 'X.Y.Z' or a specific git hash) parser.add_option( --spark-git-repo, - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [SPARK-3307] [PySpark] Fix doc string of SparkContext.broadcast()
Repository: spark
Updated Branches:
refs/heads/master 27df6ce6a - e248328b3
[SPARK-3307] [PySpark] Fix doc string of SparkContext.broadcast()
remove invalid docs
Author: Davies Liu davies@gmail.com
Closes #2202 from davies/keep and squashes the following commits:
aa3b44f [Davies Liu] remove invalid docs
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e248328b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e248328b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e248328b
Branch: refs/heads/master
Commit: e248328b39f52073422a12fd0388208de41be1c7
Parents: 27df6ce
Author: Davies Liu davies@gmail.com
Authored: Fri Aug 29 11:47:49 2014 -0700
Committer: Josh Rosen joshro...@apache.org
Committed: Fri Aug 29 11:47:49 2014 -0700
--
python/pyspark/context.py | 2 --
1 file changed, 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/e248328b/python/pyspark/context.py
--
diff --git a/python/pyspark/context.py b/python/pyspark/context.py
index a90870e..82f76de 100644
--- a/python/pyspark/context.py
+++ b/python/pyspark/context.py
@@ -568,8 +568,6 @@ class SparkContext(object):
L{Broadcastpyspark.broadcast.Broadcast}
object for reading it in distributed functions. The variable will
be sent to each cluster only once.
-
-:keep: Keep the `value` in driver or not.
ser = CompressedSerializer(PickleSerializer())
# pass large object by py4j is very slow and need much memory
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [SPARK-3307] [PySpark] Fix doc string of SparkContext.broadcast()
Repository: spark
Updated Branches:
refs/heads/branch-1.1 c71b5c6db - 98d0716a1
[SPARK-3307] [PySpark] Fix doc string of SparkContext.broadcast()
remove invalid docs
Author: Davies Liu davies@gmail.com
Closes #2202 from davies/keep and squashes the following commits:
aa3b44f [Davies Liu] remove invalid docs
(cherry picked from commit e248328b39f52073422a12fd0388208de41be1c7)
Signed-off-by: Josh Rosen joshro...@apache.org
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/98d0716a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/98d0716a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/98d0716a
Branch: refs/heads/branch-1.1
Commit: 98d0716a11e8e5731adfa6031a5d35916f42b6f5
Parents: c71b5c6
Author: Davies Liu davies@gmail.com
Authored: Fri Aug 29 11:47:49 2014 -0700
Committer: Josh Rosen joshro...@apache.org
Committed: Fri Aug 29 11:48:00 2014 -0700
--
python/pyspark/context.py | 2 --
1 file changed, 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/98d0716a/python/pyspark/context.py
--
diff --git a/python/pyspark/context.py b/python/pyspark/context.py
index a90870e..82f76de 100644
--- a/python/pyspark/context.py
+++ b/python/pyspark/context.py
@@ -568,8 +568,6 @@ class SparkContext(object):
L{Broadcastpyspark.broadcast.Broadcast}
object for reading it in distributed functions. The variable will
be sent to each cluster only once.
-
-:keep: Keep the `value` in driver or not.
ser = CompressedSerializer(PickleSerializer())
# pass large object by py4j is very slow and need much memory
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [Docs] SQL doc formatting and typo fixes
Repository: spark Updated Branches: refs/heads/branch-1.1 98d0716a1 - bfa2dc99a [Docs] SQL doc formatting and typo fixes As [reported on the dev list](http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-1-0-RC2-tp8107p8131.html): * Code fencing with triple-backticks doesnât seem to work like it does on GitHub. Newlines are lost. Instead, use 4-space indent to format small code blocks. * Nested bullets need 2 leading spaces, not 1. * Spellcheck! Author: Nicholas Chammas nicholas.cham...@gmail.com Author: nchammas nicholas.cham...@gmail.com Closes #2201 from nchammas/sql-doc-fixes and squashes the following commits: 873f889 [Nicholas Chammas] [Docs] fix skip-api flag 5195e0c [Nicholas Chammas] [Docs] SQL doc formatting and typo fixes 3b26c8d [nchammas] [Spark QA] Link to console output on test time out (cherry picked from commit 53aa8316e88980c6f46d3b9fc90d935a4738a370) Signed-off-by: Michael Armbrust mich...@databricks.com Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bfa2dc99 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bfa2dc99 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bfa2dc99 Branch: refs/heads/branch-1.1 Commit: bfa2dc99a22c23dc4b10d1f9e5dd9681f6f48537 Parents: 98d0716 Author: Nicholas Chammas nicholas.cham...@gmail.com Authored: Fri Aug 29 15:23:32 2014 -0700 Committer: Michael Armbrust mich...@databricks.com Committed: Fri Aug 29 15:23:41 2014 -0700 -- docs/README.md| 2 +- docs/sql-programming-guide.md | 109 + 2 files changed, 52 insertions(+), 59 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/bfa2dc99/docs/README.md -- diff --git a/docs/README.md b/docs/README.md index fd7ba4e..0a0126c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -30,7 +30,7 @@ called `_site` containing index.html as well as the rest of the compiled files. You can modify the default Jekyll build as follows: # Skip generating API docs (which takes a while) -$ SKIP_SCALADOC=1 jekyll build +$ SKIP_API=1 jekyll build # Serve content locally on port 4000 $ jekyll serve --watch # Build the site with extra features used on the live page http://git-wip-us.apache.org/repos/asf/spark/blob/bfa2dc99/docs/sql-programming-guide.md -- diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index c41f280..8f7fb54 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -474,10 +474,10 @@ anotherPeople = sqlContext.jsonRDD(anotherPeopleRDD) Spark SQL also supports reading and writing data stored in [Apache Hive](http://hive.apache.org/). However, since Hive has a large number of dependencies, it is not included in the default Spark assembly. -In order to use Hive you must first run '`sbt/sbt -Phive assembly/assembly`' (or use `-Phive` for maven). +In order to use Hive you must first run `sbt/sbt -Phive assembly/assembly` (or use `-Phive` for maven). This command builds a new assembly jar that includes Hive. Note that this Hive assembly jar must also be present on all of the worker nodes, as they will need access to the Hive serialization and deserialization libraries -(SerDes) in order to acccess data stored in Hive. +(SerDes) in order to access data stored in Hive. Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`. @@ -576,9 +576,8 @@ evaluated by the SQL execution engine. A full list of the functions supported c ## Running the Thrift JDBC server -The Thrift JDBC server implemented here corresponds to the [`HiveServer2`] -(https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2) in Hive 0.12. You can test -the JDBC server with the beeline script comes with either Spark or Hive 0.12. +The Thrift JDBC server implemented here corresponds to the [`HiveServer2`](https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2) +in Hive 0.12. You can test the JDBC server with the beeline script comes with either Spark or Hive 0.12. To start the JDBC server, run the following in the Spark directory: @@ -597,7 +596,7 @@ Connect to the JDBC server in beeline with: Beeline will ask you for a username and password. In non-secure mode, simply enter the username on your machine and a blank password. For secure mode, please follow the instructions given in the -[beeline documentation](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients) +[beeline documentation](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients). Configuration of Hive is done by placing your
git commit: SPARK-2813: [SQL] Implement SQRT() directly in Spark SQL
Repository: spark
Updated Branches:
refs/heads/master 53aa8316e - 2f1519def
SPARK-2813: [SQL] Implement SQRT() directly in Spark SQL
This PR adds a native implementation for SQL SQRT() and thus avoids delegating
this function to Hive.
Author: William Benton wi...@redhat.com
Closes #1750 from willb/spark-2813 and squashes the following commits:
22c8a79 [William Benton] Fixed missed newline from rebase
d673861 [William Benton] Added string coercions for SQRT and associated test
case
e125df4 [William Benton] Added ExpressionEvaluationSuite test cases for SQRT
7b84bcd [William Benton] SQL SQRT now properly returns NULL for NULL inputs
8256971 [William Benton] added SQRT test to SqlQuerySuite
504d2e5 [William Benton] Added native SQRT implementation
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2f1519de
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2f1519de
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2f1519de
Branch: refs/heads/master
Commit: 2f1519defaba4f3c7d536669f909bfd9e13e4069
Parents: 53aa831
Author: William Benton wi...@redhat.com
Authored: Fri Aug 29 15:26:59 2014 -0700
Committer: Michael Armbrust mich...@databricks.com
Committed: Fri Aug 29 15:26:59 2014 -0700
--
.../org/apache/spark/sql/catalyst/SqlParser.scala | 2 ++
.../sql/catalyst/analysis/HiveTypeCoercion.scala | 2 ++
.../spark/sql/catalyst/expressions/arithmetic.scala | 13 +
.../expressions/ExpressionEvaluationSuite.scala | 13 +
.../scala/org/apache/spark/sql/SQLQuerySuite.scala| 14 ++
.../main/scala/org/apache/spark/sql/hive/HiveQl.scala | 2 ++
6 files changed, 46 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/2f1519de/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
index 2c73a80..4f166c0 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
@@ -122,6 +122,7 @@ class SqlParser extends StandardTokenParsers with
PackratParsers {
protected val EXCEPT = Keyword(EXCEPT)
protected val SUBSTR = Keyword(SUBSTR)
protected val SUBSTRING = Keyword(SUBSTRING)
+ protected val SQRT = Keyword(SQRT)
// Use reflection to find the reserved words defined in this class.
protected val reservedWords =
@@ -323,6 +324,7 @@ class SqlParser extends StandardTokenParsers with
PackratParsers {
(SUBSTR | SUBSTRING) ~ ( ~ expression ~ , ~ expression ~ , ~
expression ~ ) ^^ {
case s ~ , ~ p ~ , ~ l = Substring(s,p,l)
} |
+SQRT ~ ( ~ expression ~ ) ^^ { case exp = Sqrt(exp) } |
ident ~ ( ~ repsep(expression, ,) ~ ) ^^ {
case udfName ~ _ ~ exprs = UnresolvedFunction(udfName, exprs)
}
http://git-wip-us.apache.org/repos/asf/spark/blob/2f1519de/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
index 15eb598..ecfcd62 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
@@ -227,6 +227,8 @@ trait HiveTypeCoercion {
Sum(Cast(e, DoubleType))
case Average(e) if e.dataType == StringType =
Average(Cast(e, DoubleType))
+ case Sqrt(e) if e.dataType == StringType =
+Sqrt(Cast(e, DoubleType))
}
}
http://git-wip-us.apache.org/repos/asf/spark/blob/2f1519de/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
index aae86a3..56f0428 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
@@ -33,6 +33,19 @@ case class UnaryMinus(child: Expression) extends
UnaryExpression {
}
}
+case class Sqrt(child: Expression) extends UnaryExpression {
+ type EvaluatedType = Any
+
+ def dataType = child.dataType
+
git commit: [SPARK-3234][Build] Fixed environment variables that rely on deprecated command line options in make-distribution.sh
Repository: spark Updated Branches: refs/heads/master 2f1519def - 287c0ac77 [SPARK-3234][Build] Fixed environment variables that rely on deprecated command line options in make-distribution.sh Please refer to [SPARK-3234](https://issues.apache.org/jira/browse/SPARK-3234) for details. Author: Cheng Lian lian.cs@gmail.com Closes #2208 from liancheng/spark-3234 and squashes the following commits: fb26de8 [Cheng Lian] Fixed SPARK-3234 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/287c0ac7 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/287c0ac7 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/287c0ac7 Branch: refs/heads/master Commit: 287c0ac7722dd4bc51b921ccc6f0e3c1625b5ff4 Parents: 2f1519d Author: Cheng Lian lian.cs@gmail.com Authored: Fri Aug 29 15:29:43 2014 -0700 Committer: Patrick Wendell pwend...@gmail.com Committed: Fri Aug 29 15:29:43 2014 -0700 -- make-distribution.sh | 12 +++- 1 file changed, 11 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/287c0ac7/make-distribution.sh -- diff --git a/make-distribution.sh b/make-distribution.sh index f7a6a9d..ee13990 100755 --- a/make-distribution.sh +++ b/make-distribution.sh @@ -113,7 +113,17 @@ if ! which mvn /dev/null; then echo -e Download Maven from https://maven.apache.org/; exit -1; fi + VERSION=$(mvn help:evaluate -Dexpression=project.version 2/dev/null | grep -v INFO | tail -n 1) +SPARK_HADOOP_VERSION=$(mvn help:evaluate -Dexpression=hadoop.version $@ 2/dev/null\ +| grep -v INFO\ +| tail -n 1) +SPARK_HIVE=$(mvn help:evaluate -Dexpression=project.activeProfiles $@ 2/dev/null\ +| grep -v INFO\ +| fgrep --count idhive/id;\ +# Reset exit status to 0, otherwise the script stops here if the last grep finds nothing\ +# because we use set -o pipefail +echo -n) JAVA_CMD=$JAVA_HOME/bin/java JAVA_VERSION=$($JAVA_CMD -version 21) @@ -175,7 +185,7 @@ cp $FWDIR/examples/target/scala*/spark-examples*.jar $DISTDIR/lib/ mkdir -p $DISTDIR/examples/src/main cp -r $FWDIR/examples/src/main $DISTDIR/examples/src/ -if [ $SPARK_HIVE == true ]; then +if [ $SPARK_HIVE == 1 ]; then cp $FWDIR/lib_managed/jars/datanucleus*.jar $DISTDIR/lib/ fi - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [SPARK-3198] [SQL] Remove the TreeNode.id
Repository: spark
Updated Branches:
refs/heads/master 287c0ac77 - dc4d577c6
[SPARK-3198] [SQL] Remove the TreeNode.id
Thus id property of the TreeNode API does save time in a faster way to compare
2 TreeNodes, it is kind of performance bottleneck during the expression object
creation in a multi-threading env (because of the memory barrier).
Fortunately, the tree node comparison only happen once in master, so even we
remove it, the entire performance will not be affected.
Author: Cheng Hao hao.ch...@intel.com
Closes #2155 from chenghao-intel/treenode and squashes the following commits:
7cf2cd2 [Cheng Hao] Remove the implicit keyword for TreeNodeRef and some other
small issues
5873415 [Cheng Hao] Remove the TreeNode.id
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/dc4d577c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/dc4d577c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/dc4d577c
Branch: refs/heads/master
Commit: dc4d577c6549df58f42c0e22cac354554d169896
Parents: 287c0ac
Author: Cheng Hao hao.ch...@intel.com
Authored: Fri Aug 29 15:32:26 2014 -0700
Committer: Michael Armbrust mich...@databricks.com
Committed: Fri Aug 29 15:32:26 2014 -0700
--
.../spark/sql/catalyst/planning/patterns.scala | 11 +
.../spark/sql/catalyst/plans/QueryPlan.scala| 12 +-
.../spark/sql/catalyst/trees/TreeNode.scala | 24 ++--
.../spark/sql/catalyst/trees/package.scala | 11 +
.../sql/catalyst/trees/TreeNodeSuite.scala | 5 +++-
.../sql/execution/GeneratedAggregate.scala | 10
.../spark/sql/execution/debug/package.scala | 7 +++---
.../apache/spark/sql/execution/pythonUdfs.scala | 2 +-
8 files changed, 40 insertions(+), 42 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/dc4d577c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
index 90923fe..f0fd9a8 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
@@ -19,8 +19,9 @@ package org.apache.spark.sql.catalyst.planning
import scala.annotation.tailrec
-import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.Logging
+import org.apache.spark.sql.catalyst.trees.TreeNodeRef
+import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical._
@@ -134,8 +135,8 @@ object PartialAggregation {
// Only do partial aggregation if supported by all aggregate expressions.
if (allAggregates.size == partialAggregates.size) {
// Create a map of expressions to their partial evaluations for all
aggregate expressions.
-val partialEvaluations: Map[Long, SplitEvaluation] =
- partialAggregates.map(a = (a.id, a.asPartial)).toMap
+val partialEvaluations: Map[TreeNodeRef, SplitEvaluation] =
+ partialAggregates.map(a = (new TreeNodeRef(a), a.asPartial)).toMap
// We need to pass all grouping expressions though so the grouping can
happen a second
// time. However some of them might be unnamed so we alias them
allowing them to be
@@ -148,8 +149,8 @@ object PartialAggregation {
// Replace aggregations with a new expression that computes the result
from the already
// computed partial evaluations and grouping values.
val rewrittenAggregateExpressions =
aggregateExpressions.map(_.transformUp {
- case e: Expression if partialEvaluations.contains(e.id) =
-partialEvaluations(e.id).finalEvaluation
+ case e: Expression if partialEvaluations.contains(new
TreeNodeRef(e)) =
+partialEvaluations(new TreeNodeRef(e)).finalEvaluation
case e: Expression if namedGroupingExpressions.contains(e) =
namedGroupingExpressions(e).toAttribute
}).asInstanceOf[Seq[NamedExpression]]
http://git-wip-us.apache.org/repos/asf/spark/blob/dc4d577c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
index 1e177e2..af9e4d8 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
+++
git commit: [SPARK-3269][SQL] Decreases initial buffer size for row set to prevent OOM
Repository: spark
Updated Branches:
refs/heads/master b1eccfc88 - d94a44d7c
[SPARK-3269][SQL] Decreases initial buffer size for row set to prevent OOM
When a large batch size is specified, `SparkSQLOperationManager` OOMs even if
the whole result set is much smaller than the batch size.
Author: Cheng Lian lian.cs@gmail.com
Closes #2171 from liancheng/jdbc-fetch-size and squashes the following commits:
5e1623b [Cheng Lian] Decreases initial buffer size for row set to prevent OOM
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d94a44d7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d94a44d7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d94a44d7
Branch: refs/heads/master
Commit: d94a44d7caaf3fe7559d9ad7b10872fa16cf81ca
Parents: b1eccfc
Author: Cheng Lian lian.cs@gmail.com
Authored: Fri Aug 29 15:36:04 2014 -0700
Committer: Michael Armbrust mich...@databricks.com
Committed: Fri Aug 29 15:36:04 2014 -0700
--
.../sql/hive/thriftserver/server/SparkSQLOperationManager.scala | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/d94a44d7/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala
--
diff --git
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala
index 6eccb1b..f12b5a6 100644
---
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala
+++
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala
@@ -66,9 +66,10 @@ class SparkSQLOperationManager(hiveContext: HiveContext)
extends OperationManage
if (!iter.hasNext) {
new RowSet()
} else {
- val maxRows = maxRowsL.toInt // Do you really want a row batch
larger than Int Max? No.
+ // maxRowsL here typically maps to java.sql.Statement.getFetchSize,
which is an int
+ val maxRows = maxRowsL.toInt
var curRow = 0
- var rowSet = new ArrayBuffer[Row](maxRows)
+ var rowSet = new ArrayBuffer[Row](maxRows.min(1024))
while (curRow maxRows iter.hasNext) {
val sparkRow = iter.next()
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [SPARK-3269][SQL] Decreases initial buffer size for row set to prevent OOM
Repository: spark
Updated Branches:
refs/heads/branch-1.1 cf049efdc - 9bae345ab
[SPARK-3269][SQL] Decreases initial buffer size for row set to prevent OOM
When a large batch size is specified, `SparkSQLOperationManager` OOMs even if
the whole result set is much smaller than the batch size.
Author: Cheng Lian lian.cs@gmail.com
Closes #2171 from liancheng/jdbc-fetch-size and squashes the following commits:
5e1623b [Cheng Lian] Decreases initial buffer size for row set to prevent OOM
(cherry picked from commit d94a44d7caaf3fe7559d9ad7b10872fa16cf81ca)
Signed-off-by: Michael Armbrust mich...@databricks.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9bae345a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9bae345a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9bae345a
Branch: refs/heads/branch-1.1
Commit: 9bae345ab543b686e8b32e504711140950d9ea68
Parents: cf049ef
Author: Cheng Lian lian.cs@gmail.com
Authored: Fri Aug 29 15:36:04 2014 -0700
Committer: Michael Armbrust mich...@databricks.com
Committed: Fri Aug 29 15:36:19 2014 -0700
--
.../sql/hive/thriftserver/server/SparkSQLOperationManager.scala | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/9bae345a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala
--
diff --git
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala
index 6eccb1b..f12b5a6 100644
---
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala
+++
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala
@@ -66,9 +66,10 @@ class SparkSQLOperationManager(hiveContext: HiveContext)
extends OperationManage
if (!iter.hasNext) {
new RowSet()
} else {
- val maxRows = maxRowsL.toInt // Do you really want a row batch
larger than Int Max? No.
+ // maxRowsL here typically maps to java.sql.Statement.getFetchSize,
which is an int
+ val maxRows = maxRowsL.toInt
var curRow = 0
- var rowSet = new ArrayBuffer[Row](maxRows)
+ var rowSet = new ArrayBuffer[Row](maxRows.min(1024))
while (curRow maxRows iter.hasNext) {
val sparkRow = iter.next()
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [SPARK-3291][SQL]TestcaseName in createQueryTest should not contain :
Repository: spark
Updated Branches:
refs/heads/branch-1.1 9bae345ab - c1333b852
[SPARK-3291][SQL]TestcaseName in createQueryTest should not contain :
: is not allowed to appear in a file name of Windows system. If file name
contains :, this file can't be checked out in a Windows system and developers
using Windows must be careful to not commit the deletion of such files, Which
is very inconvenient.
Author: qiping.lqp qiping@alibaba-inc.com
Closes #2191 from chouqin/querytest and squashes the following commits:
0e943a1 [qiping.lqp] rename golden file
60a863f [qiping.lqp] TestcaseName in createQueryTest should not contain :
(cherry picked from commit 634d04b87c2744d645e9c26e746ba2006371d9b5)
Signed-off-by: Michael Armbrust mich...@databricks.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c1333b85
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c1333b85
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c1333b85
Branch: refs/heads/branch-1.1
Commit: c1333b85292636c8db24ca08401bd559ed2c71f9
Parents: 9bae345
Author: qiping.lqp qiping@alibaba-inc.com
Authored: Fri Aug 29 15:37:43 2014 -0700
Committer: Michael Armbrust mich...@databricks.com
Committed: Fri Aug 29 15:38:00 2014 -0700
--
...uery Hive table-0-5d14d21a239daa42b086cc895215009a | 14 ++
...ity: Hive table-0-5d14d21a239daa42b086cc895215009a | 14 --
.../spark/sql/hive/execution/HiveComparisonTest.scala | 3 +++
.../spark/sql/hive/execution/HiveQuerySuite.scala | 2 +-
4 files changed, 18 insertions(+), 15 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/c1333b85/sql/hive/src/test/resources/golden/case
sensitivity when query Hive table-0-5d14d21a239daa42b086cc895215009a
--
diff --git a/sql/hive/src/test/resources/golden/case sensitivity when query
Hive table-0-5d14d21a239daa42b086cc895215009a
b/sql/hive/src/test/resources/golden/case sensitivity when query Hive
table-0-5d14d21a239daa42b086cc895215009a
new file mode 100644
index 000..4d7127c
--- /dev/null
+++ b/sql/hive/src/test/resources/golden/case sensitivity when query Hive
table-0-5d14d21a239daa42b086cc895215009a
@@ -0,0 +1,14 @@
+0 val_0
+4 val_4
+12 val_12
+8 val_8
+0 val_0
+0 val_0
+10 val_10
+5 val_5
+11 val_11
+5 val_5
+2 val_2
+12 val_12
+5 val_5
+9 val_9
http://git-wip-us.apache.org/repos/asf/spark/blob/c1333b85/sql/hive/src/test/resources/golden/case
sensitivity: Hive table-0-5d14d21a239daa42b086cc895215009a
--
diff --git a/sql/hive/src/test/resources/golden/case sensitivity: Hive
table-0-5d14d21a239daa42b086cc895215009a
b/sql/hive/src/test/resources/golden/case sensitivity: Hive
table-0-5d14d21a239daa42b086cc895215009a
deleted file mode 100644
index 4d7127c..000
--- a/sql/hive/src/test/resources/golden/case sensitivity: Hive
table-0-5d14d21a239daa42b086cc895215009a
+++ /dev/null
@@ -1,14 +0,0 @@
-0 val_0
-4 val_4
-12 val_12
-8 val_8
-0 val_0
-0 val_0
-10 val_10
-5 val_5
-11 val_11
-5 val_5
-2 val_2
-12 val_12
-5 val_5
-9 val_9
http://git-wip-us.apache.org/repos/asf/spark/blob/c1333b85/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
--
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
index 502ce8f..671c3b1 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
@@ -195,6 +195,9 @@ abstract class HiveComparisonTest
val installHooksCommand = (?i)SET.*hooks.r
def createQueryTest(testCaseName: String, sql: String, reset: Boolean =
true) {
+// testCaseName must not contain ':', which is not allowed to appear in a
filename of Windows
+assert(!testCaseName.contains(:))
+
// If test sharding is enable, skip tests that are not in the correct
shard.
shardInfo.foreach {
case (shardId, numShards) if testCaseName.hashCode % numShards !=
shardId = return
http://git-wip-us.apache.org/repos/asf/spark/blob/c1333b85/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
--
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
git commit: [SPARK-3173][SQL] Timestamp support in the parser
Repository: spark
Updated Branches:
refs/heads/master 634d04b87 - 98ddbe6cd
[SPARK-3173][SQL] Timestamp support in the parser
If you have a table with TIMESTAMP column, that column can't be used in WHERE
clause properly - it is not evaluated properly.
[More](https://issues.apache.org/jira/browse/SPARK-3173)
Motivation: http://www.aproint.com/aggregation-with-spark-sql/
- [x] modify SqlParser so it supports casting to TIMESTAMP (workaround for item
2)
- [x] the string literal should be converted into Timestamp if the column is
Timestamp.
Author: Zdenek Farana zdenek.far...@gmail.com
Author: Zdenek Farana zdenek.far...@aproint.com
Closes #2084 from byF/SPARK-3173 and squashes the following commits:
442b59d [Zdenek Farana] Fixed test merge conflict
2dbf4f6 [Zdenek Farana] Merge remote-tracking branch 'origin/SPARK-3173' into
SPARK-3173
65b6215 [Zdenek Farana] Fixed timezone sensitivity in the test
47b27b4 [Zdenek Farana] Now works in the case of StringLiteral=TimestampColumn
96a661b [Zdenek Farana] Code style change
491dfcf [Zdenek Farana] Added test cases for SPARK-3173
4446b1e [Zdenek Farana] A string literal is casted into Timestamp when the
column is Timestamp.
59af397 [Zdenek Farana] Added a new TIMESTAMP keyword; CAST to TIMESTAMP now
can be used in SQL expression.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/98ddbe6c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/98ddbe6c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/98ddbe6c
Branch: refs/heads/master
Commit: 98ddbe6cdbe4141df3d008dcb675ecd682c97492
Parents: 634d04b
Author: Zdenek Farana zdenek.far...@gmail.com
Authored: Fri Aug 29 15:39:15 2014 -0700
Committer: Michael Armbrust mich...@databricks.com
Committed: Fri Aug 29 15:39:15 2014 -0700
--
.../apache/spark/sql/catalyst/SqlParser.scala | 3 +-
.../catalyst/analysis/HiveTypeCoercion.scala| 10 +
.../org/apache/spark/sql/SQLQuerySuite.scala| 43 +++-
3 files changed, 54 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/98ddbe6c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
index 4f166c0..a88bd85 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
@@ -114,6 +114,7 @@ class SqlParser extends StandardTokenParsers with
PackratParsers {
protected val STRING = Keyword(STRING)
protected val SUM = Keyword(SUM)
protected val TABLE = Keyword(TABLE)
+ protected val TIMESTAMP = Keyword(TIMESTAMP)
protected val TRUE = Keyword(TRUE)
protected val UNCACHE = Keyword(UNCACHE)
protected val UNION = Keyword(UNION)
@@ -359,7 +360,7 @@ class SqlParser extends StandardTokenParsers with
PackratParsers {
literal
protected lazy val dataType: Parser[DataType] =
-STRING ^^^ StringType
+STRING ^^^ StringType | TIMESTAMP ^^^ TimestampType
}
class SqlLexical(val keywords: Seq[String]) extends StdLexical {
http://git-wip-us.apache.org/repos/asf/spark/blob/98ddbe6c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
index ecfcd62..d6758eb 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
@@ -218,11 +218,21 @@ trait HiveTypeCoercion {
case a: BinaryArithmetic if a.right.dataType == StringType =
a.makeCopy(Array(a.left, Cast(a.right, DoubleType)))
+ case p: BinaryPredicate if p.left.dataType == StringType
+ p.right.dataType == TimestampType =
+p.makeCopy(Array(Cast(p.left, TimestampType), p.right))
+ case p: BinaryPredicate if p.left.dataType == TimestampType
+ p.right.dataType == StringType =
+p.makeCopy(Array(p.left, Cast(p.right, TimestampType)))
+
case p: BinaryPredicate if p.left.dataType == StringType
p.right.dataType != StringType =
p.makeCopy(Array(Cast(p.left, DoubleType), p.right))
case p: BinaryPredicate if p.left.dataType != StringType
p.right.dataType == StringType =
p.makeCopy(Array(p.left, Cast(p.right, DoubleType)))
+ case i @ In(a,b) if
[1/2] git commit: Revert [maven-release-plugin] prepare for next development iteration
Repository: spark Updated Branches: refs/heads/branch-1.1 c1333b852 - c4b7ec8dd Revert [maven-release-plugin] prepare for next development iteration This reverts commit a4a7a241441489a0d31365e18476ae2e1c34464d. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/926f171c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/926f171c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/926f171c Branch: refs/heads/branch-1.1 Commit: 926f171c90d9c3cb83e99c31bccf79c21bd6f5e0 Parents: c1333b8 Author: Patrick Wendell pwend...@gmail.com Authored: Fri Aug 29 15:55:26 2014 -0700 Committer: Patrick Wendell pwend...@gmail.com Committed: Fri Aug 29 15:55:26 2014 -0700 -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 24 files changed, 25 insertions(+), 25 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/926f171c/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 4709b7d..799f8d9 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.1-SNAPSHOT/version +version1.1.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/926f171c/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index f29540b..8eec7e5 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.1-SNAPSHOT/version +version1.1.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/926f171c/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index debc4dd..83e6026 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.1-SNAPSHOT/version +version1.1.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/926f171c/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index f35d3d6..9bde90e 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.1-SNAPSHOT/version +version1.1.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/926f171c/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index 17d0fe2..daaae05 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.1-SNAPSHOT/version +version1.1.0/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/926f171c/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 402af35..830eb32 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.1-SNAPSHOT/version +version1.1.0/version relativePath../../pom.xml/relativePath /parent
[2/2] git commit: Revert [maven-release-plugin] prepare release v1.1.0-rc2
Revert [maven-release-plugin] prepare release v1.1.0-rc2 This reverts commit 711aebb329ca28046396af1e34395a0df92b5327. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c4b7ec8d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c4b7ec8d Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c4b7ec8d Branch: refs/heads/branch-1.1 Commit: c4b7ec8dd8d63255451acf7e62383217f1f63bdc Parents: 926f171 Author: Patrick Wendell pwend...@gmail.com Authored: Fri Aug 29 15:55:30 2014 -0700 Committer: Patrick Wendell pwend...@gmail.com Committed: Fri Aug 29 15:55:30 2014 -0700 -- assembly/pom.xml | 6 +++--- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 9 + repl/pom.xml | 2 +- sql/catalyst/pom.xml | 5 +++-- sql/core/pom.xml | 5 +++-- sql/hive-thriftserver/pom.xml | 5 +++-- sql/hive/pom.xml | 5 +++-- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 24 files changed, 38 insertions(+), 33 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/c4b7ec8d/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 799f8d9..9fbb037 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0/version +version1.1.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent @@ -124,8 +124,8 @@ transformer implementation=org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer resourcelog4j.properties/resource /transformer -transformer implementation=org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer / -transformer implementation=org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer / +transformer implementation=org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer/ +transformer implementation=org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer/ /transformers /configuration /execution http://git-wip-us.apache.org/repos/asf/spark/blob/c4b7ec8d/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 8eec7e5..bd51b11 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0/version +version1.1.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c4b7ec8d/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 83e6026..6d8be37 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0/version +version1.1.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c4b7ec8d/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 9bde90e..8c4c128 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0/version +version1.1.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c4b7ec8d/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index daaae05..b345276 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId
git commit: [SPARK-3296][mllib] spark-example should be run-example in head notation of DenseKMeans and SparseNaiveBayes
Repository: spark
Updated Branches:
refs/heads/master 98ddbe6cd - 13901764f
[SPARK-3296][mllib] spark-example should be run-example in head notation of
DenseKMeans and SparseNaiveBayes
`./bin/spark-example` should be `./bin/run-example` in DenseKMeans and
SparseNaiveBayes
Author: wangfei wangfei_he...@126.com
Closes #2193 from scwf/run-example and squashes the following commits:
207eb3a [wangfei] spark-example should be run-example
27a8999 [wangfei] ./bin/spark-example should be ./bin/run-example
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/13901764
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/13901764
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/13901764
Branch: refs/heads/master
Commit: 13901764f4e9ed3de03e420d88ab42bdce5d5140
Parents: 98ddbe6
Author: wangfei wangfei_he...@126.com
Authored: Fri Aug 29 17:37:15 2014 -0700
Committer: Xiangrui Meng m...@databricks.com
Committed: Fri Aug 29 17:37:15 2014 -0700
--
.../main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala | 2 +-
.../scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/13901764/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala
--
diff --git
a/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala
b/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala
index f96bc1b..89dfa26 100644
--- a/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala
+++ b/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala
@@ -27,7 +27,7 @@ import org.apache.spark.mllib.linalg.Vectors
/**
* An example k-means app. Run with
* {{{
- * ./bin/spark-example org.apache.spark.examples.mllib.DenseKMeans [options]
input
+ * ./bin/run-example org.apache.spark.examples.mllib.DenseKMeans [options]
input
* }}}
* If you use it as a template to create your own app, please use
`spark-submit` to submit your app.
*/
http://git-wip-us.apache.org/repos/asf/spark/blob/13901764/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala
--
diff --git
a/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala
b/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala
index 88acd9d..952fa2a 100644
---
a/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala
+++
b/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala
@@ -27,7 +27,7 @@ import org.apache.spark.mllib.util.MLUtils
/**
* An example naive Bayes app. Run with
* {{{
- * ./bin/spark-example org.apache.spark.examples.mllib.SparseNaiveBayes
[options] input
+ * ./bin/run-example org.apache.spark.examples.mllib.SparseNaiveBayes
[options] input
* }}}
* If you use it as a template to create your own app, please use
`spark-submit` to submit your app.
*/
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [SPARK-3296][mllib] spark-example should be run-example in head notation of DenseKMeans and SparseNaiveBayes
Repository: spark
Updated Branches:
refs/heads/branch-1.1 c4b7ec8dd - b0facb590
[SPARK-3296][mllib] spark-example should be run-example in head notation of
DenseKMeans and SparseNaiveBayes
`./bin/spark-example` should be `./bin/run-example` in DenseKMeans and
SparseNaiveBayes
Author: wangfei wangfei_he...@126.com
Closes #2193 from scwf/run-example and squashes the following commits:
207eb3a [wangfei] spark-example should be run-example
27a8999 [wangfei] ./bin/spark-example should be ./bin/run-example
(cherry picked from commit 13901764f4e9ed3de03e420d88ab42bdce5d5140)
Signed-off-by: Xiangrui Meng m...@databricks.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b0facb59
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b0facb59
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b0facb59
Branch: refs/heads/branch-1.1
Commit: b0facb590eac032f82cea35982c3ed335f2ebbf4
Parents: c4b7ec8
Author: wangfei wangfei_he...@126.com
Authored: Fri Aug 29 17:37:15 2014 -0700
Committer: Xiangrui Meng m...@databricks.com
Committed: Fri Aug 29 17:37:36 2014 -0700
--
.../main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala | 2 +-
.../scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b0facb59/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala
--
diff --git
a/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala
b/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala
index f96bc1b..89dfa26 100644
--- a/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala
+++ b/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala
@@ -27,7 +27,7 @@ import org.apache.spark.mllib.linalg.Vectors
/**
* An example k-means app. Run with
* {{{
- * ./bin/spark-example org.apache.spark.examples.mllib.DenseKMeans [options]
input
+ * ./bin/run-example org.apache.spark.examples.mllib.DenseKMeans [options]
input
* }}}
* If you use it as a template to create your own app, please use
`spark-submit` to submit your app.
*/
http://git-wip-us.apache.org/repos/asf/spark/blob/b0facb59/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala
--
diff --git
a/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala
b/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala
index 88acd9d..952fa2a 100644
---
a/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala
+++
b/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala
@@ -27,7 +27,7 @@ import org.apache.spark.mllib.util.MLUtils
/**
* An example naive Bayes app. Run with
* {{{
- * ./bin/spark-example org.apache.spark.examples.mllib.SparseNaiveBayes
[options] input
+ * ./bin/run-example org.apache.spark.examples.mllib.SparseNaiveBayes
[options] input
* }}}
* If you use it as a template to create your own app, please use
`spark-submit` to submit your app.
*/
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [SPARK-3320][SQL] Made batched in-memory column buffer building work for SchemaRDDs with empty partitions
Repository: spark
Updated Branches:
refs/heads/master 13901764f - 32b18dd52
[SPARK-3320][SQL] Made batched in-memory column buffer building work for
SchemaRDDs with empty partitions
Author: Cheng Lian lian.cs@gmail.com
Closes #2213 from liancheng/spark-3320 and squashes the following commits:
45a0139 [Cheng Lian] Fixed typo in InMemoryColumnarQuerySuite
f67067d [Cheng Lian] Fixed SPARK-3320
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/32b18dd5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/32b18dd5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/32b18dd5
Branch: refs/heads/master
Commit: 32b18dd52cf8920903819f23e406271ecd8ac6bb
Parents: 1390176
Author: Cheng Lian lian.cs@gmail.com
Authored: Fri Aug 29 18:16:47 2014 -0700
Committer: Michael Armbrust mich...@databricks.com
Committed: Fri Aug 29 18:16:47 2014 -0700
--
.../columnar/InMemoryColumnarTableScan.scala| 49
.../scala/org/apache/spark/sql/TestData.scala | 5 ++
.../columnar/InMemoryColumnarQuerySuite.scala | 19 ++--
3 files changed, 39 insertions(+), 34 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/32b18dd5/sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
b/sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
index bc36bac..cb055cd 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
@@ -104,40 +104,29 @@ private[sql] case class InMemoryColumnarTableScan(
override def execute() = {
relation.cachedColumnBuffers.mapPartitions { iterator =
// Find the ordinals of the requested columns. If none are requested,
use the first.
- val requestedColumns =
-if (attributes.isEmpty) {
- Seq(0)
-} else {
- attributes.map(a = relation.output.indexWhere(_.exprId == a.exprId))
-}
-
- new Iterator[Row] {
-private[this] var columnBuffers: Array[ByteBuffer] = null
-private[this] var columnAccessors: Seq[ColumnAccessor] = null
-nextBatch()
-
-private[this] val nextRow = new
GenericMutableRow(columnAccessors.length)
-
-def nextBatch() = {
- columnBuffers = iterator.next()
- columnAccessors =
requestedColumns.map(columnBuffers(_)).map(ColumnAccessor(_))
-}
+ val requestedColumns = if (attributes.isEmpty) {
+Seq(0)
+ } else {
+attributes.map(a = relation.output.indexWhere(_.exprId == a.exprId))
+ }
-override def next() = {
- if (!columnAccessors.head.hasNext) {
-nextBatch()
- }
+ iterator
+.map(batch = requestedColumns.map(batch(_)).map(ColumnAccessor(_)))
+.flatMap { columnAccessors =
+ val nextRow = new GenericMutableRow(columnAccessors.length)
+ new Iterator[Row] {
+override def next() = {
+ var i = 0
+ while (i nextRow.length) {
+columnAccessors(i).extractTo(nextRow, i)
+i += 1
+ }
+ nextRow
+}
- var i = 0
- while (i nextRow.length) {
-columnAccessors(i).extractTo(nextRow, i)
-i += 1
+override def hasNext = columnAccessors.head.hasNext
}
- nextRow
}
-
-override def hasNext = columnAccessors.head.hasNext || iterator.hasNext
- }
}
}
}
http://git-wip-us.apache.org/repos/asf/spark/blob/32b18dd5/sql/core/src/test/scala/org/apache/spark/sql/TestData.scala
--
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/TestData.scala
b/sql/core/src/test/scala/org/apache/spark/sql/TestData.scala
index c3ec82f..eb33a61 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/TestData.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/TestData.scala
@@ -151,4 +151,9 @@ object TestData {
TimestampField(new Timestamp(i))
})
timestamps.registerTempTable(timestamps)
+
+ case class IntField(i: Int)
+ // An RDD with 4 elements and 8 partitions
+ val withEmptyParts = TestSQLContext.sparkContext.parallelize((1 to
4).map(IntField), 8)
+ withEmptyParts.registerTempTable(withEmptyParts)
}
http://git-wip-us.apache.org/repos/asf/spark/blob/32b18dd5/sql/core/src/test/scala/org/apache/spark/sql/columnar/InMemoryColumnarQuerySuite.scala
git commit: [SPARK-3320][SQL] Made batched in-memory column buffer building work for SchemaRDDs with empty partitions
Repository: spark
Updated Branches:
refs/heads/branch-1.1 b0facb590 - aa9364a03
[SPARK-3320][SQL] Made batched in-memory column buffer building work for
SchemaRDDs with empty partitions
Author: Cheng Lian lian.cs@gmail.com
Closes #2213 from liancheng/spark-3320 and squashes the following commits:
45a0139 [Cheng Lian] Fixed typo in InMemoryColumnarQuerySuite
f67067d [Cheng Lian] Fixed SPARK-3320
(cherry picked from commit 32b18dd52cf8920903819f23e406271ecd8ac6bb)
Signed-off-by: Michael Armbrust mich...@databricks.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/aa9364a0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/aa9364a0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/aa9364a0
Branch: refs/heads/branch-1.1
Commit: aa9364a03ddc793fd2c94981fb168ef8100a507c
Parents: b0facb5
Author: Cheng Lian lian.cs@gmail.com
Authored: Fri Aug 29 18:16:47 2014 -0700
Committer: Michael Armbrust mich...@databricks.com
Committed: Fri Aug 29 18:16:58 2014 -0700
--
.../columnar/InMemoryColumnarTableScan.scala| 49
.../scala/org/apache/spark/sql/TestData.scala | 5 ++
.../columnar/InMemoryColumnarQuerySuite.scala | 19 ++--
3 files changed, 39 insertions(+), 34 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/aa9364a0/sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
b/sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
index bc36bac..cb055cd 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/columnar/InMemoryColumnarTableScan.scala
@@ -104,40 +104,29 @@ private[sql] case class InMemoryColumnarTableScan(
override def execute() = {
relation.cachedColumnBuffers.mapPartitions { iterator =
// Find the ordinals of the requested columns. If none are requested,
use the first.
- val requestedColumns =
-if (attributes.isEmpty) {
- Seq(0)
-} else {
- attributes.map(a = relation.output.indexWhere(_.exprId == a.exprId))
-}
-
- new Iterator[Row] {
-private[this] var columnBuffers: Array[ByteBuffer] = null
-private[this] var columnAccessors: Seq[ColumnAccessor] = null
-nextBatch()
-
-private[this] val nextRow = new
GenericMutableRow(columnAccessors.length)
-
-def nextBatch() = {
- columnBuffers = iterator.next()
- columnAccessors =
requestedColumns.map(columnBuffers(_)).map(ColumnAccessor(_))
-}
+ val requestedColumns = if (attributes.isEmpty) {
+Seq(0)
+ } else {
+attributes.map(a = relation.output.indexWhere(_.exprId == a.exprId))
+ }
-override def next() = {
- if (!columnAccessors.head.hasNext) {
-nextBatch()
- }
+ iterator
+.map(batch = requestedColumns.map(batch(_)).map(ColumnAccessor(_)))
+.flatMap { columnAccessors =
+ val nextRow = new GenericMutableRow(columnAccessors.length)
+ new Iterator[Row] {
+override def next() = {
+ var i = 0
+ while (i nextRow.length) {
+columnAccessors(i).extractTo(nextRow, i)
+i += 1
+ }
+ nextRow
+}
- var i = 0
- while (i nextRow.length) {
-columnAccessors(i).extractTo(nextRow, i)
-i += 1
+override def hasNext = columnAccessors.head.hasNext
}
- nextRow
}
-
-override def hasNext = columnAccessors.head.hasNext || iterator.hasNext
- }
}
}
}
http://git-wip-us.apache.org/repos/asf/spark/blob/aa9364a0/sql/core/src/test/scala/org/apache/spark/sql/TestData.scala
--
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/TestData.scala
b/sql/core/src/test/scala/org/apache/spark/sql/TestData.scala
index c3ec82f..eb33a61 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/TestData.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/TestData.scala
@@ -151,4 +151,9 @@ object TestData {
TimestampField(new Timestamp(i))
})
timestamps.registerTempTable(timestamps)
+
+ case class IntField(i: Int)
+ // An RDD with 4 elements and 8 partitions
+ val withEmptyParts = TestSQLContext.sparkContext.parallelize((1 to
4).map(IntField), 8)
+ withEmptyParts.registerTempTable(withEmptyParts)
}
git commit: Adding new CHANGES.txt
Repository: spark Updated Branches: refs/heads/branch-1.1 aa9364a03 - 272b4a68d Adding new CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/272b4a68 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/272b4a68 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/272b4a68 Branch: refs/heads/branch-1.1 Commit: 272b4a68d9b4ae2d74df7615122c587bfac2dddf Parents: aa9364a Author: Patrick Wendell pwend...@gmail.com Authored: Fri Aug 29 18:49:51 2014 -0700 Committer: Patrick Wendell pwend...@gmail.com Committed: Fri Aug 29 18:49:51 2014 -0700 -- CHANGES.txt | 45 + 1 file changed, 45 insertions(+) -- http://git-wip-us.apache.org/repos/asf/spark/blob/272b4a68/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index b42721a..3693d59 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -3,6 +3,51 @@ Spark Change Log Release 1.1.0 + [SPARK-3320][SQL] Made batched in-memory column buffer building work for SchemaRDDs with empty partitions + Cheng Lian lian.cs@gmail.com + 2014-08-29 18:16:47 -0700 + Commit: aa9364a, github.com/apache/spark/pull/2213 + + [SPARK-3296][mllib] spark-example should be run-example in head notation of DenseKMeans and SparseNaiveBayes + wangfei wangfei_he...@126.com + 2014-08-29 17:37:15 -0700 + Commit: b0facb5, github.com/apache/spark/pull/2193 + + [SPARK-3291][SQL]TestcaseName in createQueryTest should not contain : + qiping.lqp qiping@alibaba-inc.com + 2014-08-29 15:37:43 -0700 + Commit: c1333b8, github.com/apache/spark/pull/2191 + + [SPARK-3269][SQL] Decreases initial buffer size for row set to prevent OOM + Cheng Lian lian.cs@gmail.com + 2014-08-29 15:36:04 -0700 + Commit: 9bae345, github.com/apache/spark/pull/2171 + + [SPARK-3234][Build] Fixed environment variables that rely on deprecated command line options in make-distribution.sh + Cheng Lian lian.cs@gmail.com + 2014-08-29 15:29:43 -0700 + Commit: cf049ef, github.com/apache/spark/pull/2208 + + [Docs] SQL doc formatting and typo fixes + Nicholas Chammas nicholas.cham...@gmail.com, nchammas nicholas.cham...@gmail.com + 2014-08-29 15:23:32 -0700 + Commit: bfa2dc9, github.com/apache/spark/pull/2201 + + [SPARK-3307] [PySpark] Fix doc string of SparkContext.broadcast() + Davies Liu davies@gmail.com + 2014-08-29 11:47:49 -0700 + Commit: 98d0716, github.com/apache/spark/pull/2202 + + HOTFIX: Bump spark-ec2 version to 1.1.0 + Patrick Wendell pwend...@gmail.com + 2014-08-29 11:20:45 -0700 + Commit: c71b5c6 + + Adding new CHANGES.txt + Patrick Wendell pwend...@gmail.com + 2014-08-28 17:17:30 -0700 + Commit: 7db87b3 + [SPARK-3277] Fix external spilling with LZ4 assertion error Andrew Or andrewo...@gmail.com, Patrick Wendell pwend...@gmail.com 2014-08-28 17:05:21 -0700 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [maven-release-plugin] prepare release v1.1.0-rc3
Repository: spark Updated Branches: refs/heads/branch-1.1 272b4a68d - 2b2e02265 [maven-release-plugin] prepare release v1.1.0-rc3 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2b2e0226 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2b2e0226 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2b2e0226 Branch: refs/heads/branch-1.1 Commit: 2b2e02265f80e4c5172c1e498aa9ba2c6b91c6c9 Parents: 272b4a6 Author: Patrick Wendell pwend...@gmail.com Authored: Sat Aug 30 02:26:03 2014 + Committer: Patrick Wendell pwend...@gmail.com Committed: Sat Aug 30 02:26:03 2014 + -- assembly/pom.xml | 6 +++--- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 9 - repl/pom.xml | 2 +- sql/catalyst/pom.xml | 5 ++--- sql/core/pom.xml | 5 ++--- sql/hive-thriftserver/pom.xml | 5 ++--- sql/hive/pom.xml | 5 ++--- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 24 files changed, 33 insertions(+), 38 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/2b2e0226/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 9fbb037..799f8d9 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0-SNAPSHOT/version +version1.1.0/version relativePath../pom.xml/relativePath /parent @@ -124,8 +124,8 @@ transformer implementation=org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer resourcelog4j.properties/resource /transformer -transformer implementation=org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer/ -transformer implementation=org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer/ +transformer implementation=org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer / +transformer implementation=org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer / /transformers /configuration /execution http://git-wip-us.apache.org/repos/asf/spark/blob/2b2e0226/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index bd51b11..8eec7e5 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0-SNAPSHOT/version +version1.1.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/2b2e0226/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 6d8be37..83e6026 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0-SNAPSHOT/version +version1.1.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/2b2e0226/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 8c4c128..9bde90e 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0-SNAPSHOT/version +version1.1.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/2b2e0226/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index b345276..daaae05 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId
git commit: [maven-release-plugin] prepare for next development iteration
Repository: spark Updated Branches: refs/heads/branch-1.1 2b2e02265 - 8b5f0dbd8 [maven-release-plugin] prepare for next development iteration Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8b5f0dbd Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8b5f0dbd Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8b5f0dbd Branch: refs/heads/branch-1.1 Commit: 8b5f0dbd8d32a25a4e7ba3ebe1a4c3c6310aeb85 Parents: 2b2e022 Author: Patrick Wendell pwend...@gmail.com Authored: Sat Aug 30 02:26:11 2014 + Committer: Patrick Wendell pwend...@gmail.com Committed: Sat Aug 30 02:26:11 2014 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- pom.xml | 4 ++-- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- yarn/stable/pom.xml | 2 +- 24 files changed, 25 insertions(+), 25 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/8b5f0dbd/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 799f8d9..4709b7d 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0/version +version1.1.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8b5f0dbd/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 8eec7e5..f29540b 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0/version +version1.1.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8b5f0dbd/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 83e6026..debc4dd 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0/version +version1.1.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8b5f0dbd/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 9bde90e..f35d3d6 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0/version +version1.1.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8b5f0dbd/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index daaae05..17d0fe2 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0/version +version1.1.1-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8b5f0dbd/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 830eb32..402af35 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent/artifactId -version1.1.0/version +version1.1.1-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/8b5f0dbd/external/kafka/pom.xml
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.1.0-rc3 [created] 25596d1da - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
git commit: BUILD: Adding back CDH4 as per user requests
Repository: spark
Updated Branches:
refs/heads/master 32b18dd52 - a004a8d87
BUILD: Adding back CDH4 as per user requests
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a004a8d8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a004a8d8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a004a8d8
Branch: refs/heads/master
Commit: a004a8d879a85af3be0aefa3f331116d4aabb1e4
Parents: 32b18dd
Author: Patrick Wendell pwend...@gmail.com
Authored: Fri Aug 29 22:24:35 2014 -0700
Committer: Patrick Wendell pwend...@gmail.com
Committed: Fri Aug 29 22:24:35 2014 -0700
--
dev/create-release/create-release.sh | 1 +
1 file changed, 1 insertion(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/a004a8d8/dev/create-release/create-release.sh
--
diff --git a/dev/create-release/create-release.sh
b/dev/create-release/create-release.sh
index 7549fbb..281e8d4 100755
--- a/dev/create-release/create-release.sh
+++ b/dev/create-release/create-release.sh
@@ -118,6 +118,7 @@ make_binary_release() {
}
make_binary_release hadoop1 -Phive -Dhadoop.version=1.0.4
+make_binary_release cdh4 -Phive -Dhadoop.version=2.0.0-mr1-cdh4.2.0
make_binary_release hadoop2.3 -Phadoop-2.3 -Phive -Pyarn
make_binary_release hadoop2.4 -Phadoop-2.4 -Phive -Pyarn
make_binary_release hadoop2.4-without-hive -Phadoop-2.4 -Pyarn
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
git commit: BUILD: Adding back CDH4 as per user requests
Repository: spark
Updated Branches:
refs/heads/branch-1.1 8b5f0dbd8 - d4ce26426
BUILD: Adding back CDH4 as per user requests
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d4ce2642
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d4ce2642
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d4ce2642
Branch: refs/heads/branch-1.1
Commit: d4ce26426832eb09a229c51cb5e64048272cf4b6
Parents: 8b5f0db
Author: Patrick Wendell pwend...@gmail.com
Authored: Fri Aug 29 22:24:35 2014 -0700
Committer: Patrick Wendell pwend...@gmail.com
Committed: Fri Aug 29 22:26:04 2014 -0700
--
dev/create-release/create-release.sh | 1 +
1 file changed, 1 insertion(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/d4ce2642/dev/create-release/create-release.sh
--
diff --git a/dev/create-release/create-release.sh
b/dev/create-release/create-release.sh
index 7549fbb..281e8d4 100755
--- a/dev/create-release/create-release.sh
+++ b/dev/create-release/create-release.sh
@@ -118,6 +118,7 @@ make_binary_release() {
}
make_binary_release hadoop1 -Phive -Dhadoop.version=1.0.4
+make_binary_release cdh4 -Phive -Dhadoop.version=2.0.0-mr1-cdh4.2.0
make_binary_release hadoop2.3 -Phadoop-2.3 -Phive -Pyarn
make_binary_release hadoop2.4 -Phadoop-2.4 -Phive -Pyarn
make_binary_release hadoop2.4-without-hive -Phadoop-2.4 -Pyarn
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
git commit: [SPARK-3305] Remove unused import from UI classes.
Repository: spark
Updated Branches:
refs/heads/master a004a8d87 - 7e662af33
[SPARK-3305] Remove unused import from UI classes.
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Closes #2200 from sarutak/SPARK-3305 and squashes the following commits:
3cbd6ee [Kousuke Saruta] Removed unused import from classes related to UI
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7e662af3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7e662af3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7e662af3
Branch: refs/heads/master
Commit: 7e662af332beb171dc89027a2562d0949d69cfa0
Parents: a004a8d
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Authored: Fri Aug 29 22:52:32 2014 -0700
Committer: Reynold Xin r...@apache.org
Committed: Fri Aug 29 22:52:32 2014 -0700
--
core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala | 1 -
.../main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala | 2 +-
2 files changed, 1 insertion(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/7e662af3/core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
--
diff --git a/core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
b/core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
index 02df4e8..b0e3bb3 100644
--- a/core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
@@ -21,7 +21,6 @@ import javax.servlet.http.HttpServletRequest
import scala.xml.Node
-import org.apache.spark.storage.StorageLevel
import org.apache.spark.ui.{ToolTips, UIUtils, WebUIPage}
import org.apache.spark.util.Utils
http://git-wip-us.apache.org/repos/asf/spark/blob/7e662af3/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
b/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
index f7f918f..eaeb861 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
@@ -17,7 +17,7 @@
package org.apache.spark.ui.jobs
-import scala.collection.mutable.{HashMap, ListBuffer, Map}
+import scala.collection.mutable.{HashMap, ListBuffer}
import org.apache.spark._
import org.apache.spark.annotation.DeveloperApi
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
