spark git commit: [SPARK-8406] [SQL] Adding UUID to output file name to avoid accidental overwriting
Repository: spark Updated Branches: refs/heads/master 47c1d5629 -> 0818fdec3 [SPARK-8406] [SQL] Adding UUID to output file name to avoid accidental overwriting This PR fixes a Parquet output file name collision bug which may cause data loss. Changes made: 1. Identify each write job issued by `InsertIntoHadoopFsRelation` with a UUID All concrete data sources which extend `HadoopFsRelation` (Parquet and ORC for now) must use this UUID to generate task output file path to avoid name collision. 2. Make `TestHive` use a local mode `SparkContext` with 32 threads to increase parallelism The major reason for this is that, the original parallelism of 2 is too low to reproduce the data loss issue. Also, higher concurrency may potentially caught more concurrency bugs during testing phase. (It did help us spotted SPARK-8501.) 3. `OrcSourceSuite` was updated to workaround SPARK-8501, which we detected along the way. NOTE: This PR is made a little bit more complicated than expected because we hit two other bugs on the way and have to work them around. See [SPARK-8501] [1] and [SPARK-8513] [2]. [1]: https://github.com/liancheng/spark/tree/spark-8501 [2]: https://github.com/liancheng/spark/tree/spark-8513 Some background and a summary of offline discussion with yhuai about this issue for better understanding: In 1.4.0, we added `HadoopFsRelation` to abstract partition support of all data sources that are based on Hadoop `FileSystem` interface. Specifically, this makes partition discovery, partition pruning, and writing dynamic partitions for data sources much easier. To support appending, the Parquet data source tries to find out the max part number of part-files in the destination directory (i.e., `` in output file name `part-r-.gz.parquet`) at the beginning of the write job. In 1.3.0, this step happens on driver side before any files are written. However, in 1.4.0, this is moved to task side. Unfortunately, for tasks scheduled later, they may see wrong max part number generated of files newly written by other finished tasks within the same job. This actually causes a race condition. In most cases, this only causes nonconsecutive part numbers in output file names. But when the DataFrame contains thousands of RDD partitions, it's likely that two tasks may choose the same part number, then one of them gets overwritten by the other. Before `HadoopFsRelation`, Spark SQL already supports appending data to Hive tables. From a user's perspective, these two look similar. However, they differ a lot internally. When data are inserted into Hive tables via Spark SQL, `InsertIntoHiveTable` simulates Hive's behaviors: 1. Write data to a temporary location 2. Move data in the temporary location to the final destination location using - `Hive.loadTable()` for non-partitioned table - `Hive.loadPartition()` for static partitions - `Hive.loadDynamicPartitions()` for dynamic partitions The important part is that, `Hive.copyFiles()` is invoked in step 2 to move the data to the destination directory (I found the name is kinda confusing since no "copying" occurs here, we are just moving and renaming stuff). If a file in the source directory and another file in the destination directory happen to have the same name, say `part-r-1.parquet`, the former is moved to the destination directory and renamed with a `_copy_N` postfix (`part-r-1_copy_1.parquet`). That's how Hive handles appending and avoids name collision between different write jobs. Some alternatives fixes considered for this issue: 1. Use a similar approach as Hive This approach is not preferred in Spark 1.4.0 mainly because file metadata operations in S3 tend to be slow, especially for tables with lots of file and/or partitions. That's why `InsertIntoHadoopFsRelation` just inserts to destination directory directly, and is often used together with `DirectParquetOutputCommitter` to reduce latency when working with S3. This means, we don't have the chance to do renaming, and must avoid name collision from the very beginning. 2. Same as 1.3, just move max part number detection back to driver side This isn't doable because unlike 1.3, 1.4 also takes dynamic partitioning into account. When inserting into dynamic partitions, we don't know which partition directories will be touched on driver side before issuing the write job. Checking all partition directories is simply too expensive for tables with thousands of partitions. 3. Add extra component to output file names to avoid name collision This seems to be the only reasonable solution for now. To be more specific, we need a JOB level unique identifier to identify all write jobs issued by `InsertIntoHadoopFile`. Notice that TASK level unique identifiers can NOT be used. Because in this way a speculative task will write to a different output file from the original task. If both tasks succ
spark git commit: [SPARK-8406] [SQL] Backports SPARK-8406 and PR #6864 to branch-1.4
Repository: spark
Updated Branches:
refs/heads/branch-1.4 b836bac3f -> 451c8722a
[SPARK-8406] [SQL] Backports SPARK-8406 and PR #6864 to branch-1.4
Author: Cheng Lian
Closes #6932 from liancheng/spark-8406-for-1.4 and squashes the following
commits:
a0168fe [Cheng Lian] Backports SPARK-8406 and PR #6864 to branch-1.4
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/451c8722
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/451c8722
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/451c8722
Branch: refs/heads/branch-1.4
Commit: 451c8722afea83e8e8f11c438469eea10e5acf4c
Parents: b836bac
Author: Cheng Lian
Authored: Mon Jun 22 10:04:29 2015 -0700
Committer: Yin Huai
Committed: Mon Jun 22 10:04:29 2015 -0700
--
.../apache/spark/sql/parquet/newParquet.scala | 41 ++
.../org/apache/spark/sql/sources/commands.scala | 59
.../spark/sql/hive/orc/OrcFileOperator.scala| 7 ++-
.../apache/spark/sql/hive/orc/OrcRelation.scala | 3 +-
.../apache/spark/sql/hive/test/TestHive.scala | 2 +-
.../spark/sql/hive/orc/OrcSourceSuite.scala | 22
.../spark/sql/sources/SimpleTextRelation.scala | 4 +-
.../sql/sources/hadoopFsRelationSuites.scala| 39 +++--
8 files changed, 110 insertions(+), 67 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/451c8722/sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala
b/sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala
index 3328e6f..abf9614 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/parquet/newParquet.scala
@@ -61,50 +61,21 @@ private[sql] class ParquetOutputWriter(path: String,
context: TaskAttemptContext
extends OutputWriter {
private val recordWriter: RecordWriter[Void, Row] = {
-val conf = context.getConfiguration
val outputFormat = {
- // When appending new Parquet files to an existing Parquet file
directory, to avoid
- // overwriting existing data files, we need to find out the max task ID
encoded in these data
- // file names.
- // TODO Make this snippet a utility function for other data source
developers
- val maxExistingTaskId = {
-// Note that `path` may point to a temporary location. Here we
retrieve the real
-// destination path from the configuration
-val outputPath = new Path(conf.get("spark.sql.sources.output.path"))
-val fs = outputPath.getFileSystem(conf)
-
-if (fs.exists(outputPath)) {
- // Pattern used to match task ID in part file names, e.g.:
- //
- // part-r-1.gz.parquet
- // ^
- val partFilePattern = """part-.-(\d{1,}).*""".r
-
- fs.listStatus(outputPath).map(_.getPath.getName).map {
-case partFilePattern(id) => id.toInt
-case name if name.startsWith("_") => 0
-case name if name.startsWith(".") => 0
-case name => throw new AnalysisException(
- s"Trying to write Parquet files to directory $outputPath, " +
-s"but found items with illegal name '$name'.")
- }.reduceOption(_ max _).getOrElse(0)
-} else {
- 0
-}
- }
-
new ParquetOutputFormat[Row]() {
// Here we override `getDefaultWorkFile` for two reasons:
//
-// 1. To allow appending. We need to generate output file name based
on the max available
-// task ID computed above.
+// 1. To allow appending. We need to generate unique output file
names to avoid
+// overwriting existing files (either exist before the write job,
or are just written
+// by other tasks within the same write job).
//
// 2. To allow dynamic partitioning. Default `getDefaultWorkFile`
uses
// `FileOutputCommitter.getWorkPath()`, which points to the base
directory of all
// partitions in the case of dynamic partitioning.
override def getDefaultWorkFile(context: TaskAttemptContext,
extension: String): Path = {
- val split = context.getTaskAttemptID.getTaskID.getId +
maxExistingTaskId + 1
- new Path(path, f"part-r-$split%05d$extension")
+ val uniqueWriteJobId =
context.getConfiguration.get("spark.sql.sources.writeJobUUID")
+ val split = context.getTaskAttemptID.getTaskID.getId
+ new Path(path, f"part-r-$split%05d-$uniqueWriteJobId$extension")
}
}
}
http://git-wip-us.apache.org/repos/asf
spark git commit: [SPARK-8420] [SQL] Fix comparision of timestamps/dates with strings (branch-1.4)
Repository: spark
Updated Branches:
refs/heads/branch-1.4 451c8722a -> 65981619b
[SPARK-8420] [SQL] Fix comparision of timestamps/dates with strings (branch-1.4)
This is branch 1.4 backport of https://github.com/apache/spark/pull/6888.
Below is the original description.
In earlier versions of Spark SQL we casted `TimestampType` and `DataType` to
`StringType` when it was involved in a binary comparison with a `StringType`.
This allowed comparing a timestamp with a partial date as a user would expect.
- `time > "2014-06-10"`
- `time > "2014"`
In 1.4.0 we tried to cast the String instead into a Timestamp. However, since
partial dates are not a valid complete timestamp this results in `null` which
results in the tuple being filtered.
This PR restores the earlier behavior. Note that we still special case
equality so that these comparisons are not affected by not printing zeros for
subsecond precision.
Author: Michael Armbrust
Closes #6888 from marmbrus/timeCompareString and squashes the following commits:
bdef29c [Michael Armbrust] test partial date
1f09adf [Michael Armbrust] special handling of equality
1172c60 [Michael Armbrust] more test fixing
4dfc412 [Michael Armbrust] fix tests
aaa9508 [Michael Armbrust] newline
04d908f [Michael Armbrust] [SPARK-8420][SQL] Fix comparision of
timestamps/dates with strings
Conflicts:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
Author: Michael Armbrust
Closes #6914 from yhuai/timeCompareString-1.4 and squashes the following
commits:
9882915 [Michael Armbrust] [SPARK-8420] [SQL] Fix comparision of
timestamps/dates with strings
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/65981619
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/65981619
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/65981619
Branch: refs/heads/branch-1.4
Commit: 65981619b26da03f0c5133133e318a180235e96d
Parents: 451c872
Author: Michael Armbrust
Authored: Mon Jun 22 10:45:33 2015 -0700
Committer: Yin Huai
Committed: Mon Jun 22 10:45:33 2015 -0700
--
.../catalyst/analysis/HiveTypeCoercion.scala| 17 --
.../sql/catalyst/expressions/predicates.scala | 9
.../apache/spark/sql/DataFrameDateSuite.scala | 56
.../org/apache/spark/sql/SQLQuerySuite.scala| 4 ++
.../scala/org/apache/spark/sql/TestData.scala | 6 ---
.../columnar/InMemoryColumnarQuerySuite.scala | 7 ++-
6 files changed, 88 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/65981619/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
index fa7968e..6d0f4a0 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
@@ -242,7 +242,16 @@ trait HiveTypeCoercion {
case a: BinaryArithmetic if a.right.dataType == StringType =>
a.makeCopy(Array(a.left, Cast(a.right, DoubleType)))
- // we should cast all timestamp/date/string compare into string compare
+ // For equality between string and timestamp we cast the string to a
timestamp
+ // so that things like rounding of subsecond precision does not affect
the comparison.
+ case p @ Equality(left @ StringType(), right @ TimestampType()) =>
+p.makeCopy(Array(Cast(left, TimestampType), right))
+ case p @ Equality(left @ TimestampType(), right @ StringType()) =>
+p.makeCopy(Array(left, Cast(right, TimestampType)))
+
+ // We should cast all relative timestamp/date/string comparison into
string comparisions
+ // This behaves as a user would expect because timestamp strings sort
lexicographically.
+ // i.e. TimeStamp(2013-01-01 00:00 ...) < "2014" = true
case p: BinaryComparison if p.left.dataType == StringType &&
p.right.dataType == DateType =>
p.makeCopy(Array(p.left, Cast(p.right, StringType)))
@@ -251,10 +260,12 @@ trait HiveTypeCoercion {
p.makeCopy(Array(Cast(p.left, StringType), p.right))
case p: BinaryComparison if p.left.dataType == StringType &&
p.right.dataType == TimestampType =>
-p.makeCopy(Array(Cast(p.left, TimestampType), p.right))
+p.makeCopy(Array(p.left, Cast(p.right, StringType
spark git commit: [SPARK-8429] [EC2] Add ability to set additional tags
Repository: spark
Updated Branches:
refs/heads/master 0818fdec3 -> 42a1f716f
[SPARK-8429] [EC2] Add ability to set additional tags
Add the `--additional-tags` parameter that allows to set additional tags to all
the created instances (masters and slaves).
The user can specify multiple tags by separating them with a comma (`,`), while
each tag name and value should be separated by a colon (`:`); for example,
`Task:MySparkProject,Env:production` would add two tags, `Task` and `Env`, with
the given values.
Author: Stefano Parmesan
Closes #6857 from armisael/patch-1 and squashes the following commits:
c5ac92c [Stefano Parmesan] python style (pep8)
8e614f1 [Stefano Parmesan] Set multiple tags in a single request
bfc56af [Stefano Parmesan] Address SPARK-7900 by inceasing sleep time
daf8615 [Stefano Parmesan] Add ability to set additional tags
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/42a1f716
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/42a1f716
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/42a1f716
Branch: refs/heads/master
Commit: 42a1f716fa35533507784be5e9117a984a03e62d
Parents: 0818fde
Author: Stefano Parmesan
Authored: Mon Jun 22 11:43:10 2015 -0700
Committer: Shivaram Venkataraman
Committed: Mon Jun 22 11:43:10 2015 -0700
--
ec2/spark_ec2.py | 28
1 file changed, 20 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/42a1f716/ec2/spark_ec2.py
--
diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py
index 5608749..1037356 100755
--- a/ec2/spark_ec2.py
+++ b/ec2/spark_ec2.py
@@ -290,6 +290,10 @@ def parse_args():
"--additional-security-group", type="string", default="",
help="Additional security group to place the machines in")
parser.add_option(
+"--additional-tags", type="string", default="",
+help="Additional tags to set on the machines; tags are
comma-separated, while name and " +
+ "value are colon separated; ex:
\"Task:MySparkProject,Env:production\"")
+parser.add_option(
"--copy-aws-credentials", action="store_true", default=False,
help="Add AWS credentials to hadoop configuration to allow Spark to
access S3")
parser.add_option(
@@ -684,16 +688,24 @@ def launch_cluster(conn, opts, cluster_name):
# This wait time corresponds to SPARK-4983
print("Waiting for AWS to propagate instance metadata...")
-time.sleep(5)
-# Give the instances descriptive names
+time.sleep(15)
+
+# Give the instances descriptive names and set additional tags
+additional_tags = {}
+if opts.additional_tags.strip():
+additional_tags = dict(
+map(str.strip, tag.split(':', 1)) for tag in
opts.additional_tags.split(',')
+)
+
for master in master_nodes:
-master.add_tag(
-key='Name',
-value='{cn}-master-{iid}'.format(cn=cluster_name, iid=master.id))
+master.add_tags(
+dict(additional_tags,
Name='{cn}-master-{iid}'.format(cn=cluster_name, iid=master.id))
+)
+
for slave in slave_nodes:
-slave.add_tag(
-key='Name',
-value='{cn}-slave-{iid}'.format(cn=cluster_name, iid=slave.id))
+slave.add_tags(
+dict(additional_tags,
Name='{cn}-slave-{iid}'.format(cn=cluster_name, iid=slave.id))
+)
# Return all the instances
return (master_nodes, slave_nodes)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8482] Added M4 instances to the list.
Repository: spark
Updated Branches:
refs/heads/master 42a1f716f -> ba8a4537f
[SPARK-8482] Added M4 instances to the list.
AWS recently added M4 instances
(https://aws.amazon.com/blogs/aws/the-new-m4-instance-type-bonus-price-reduction-on-m3-c4/).
Author: Pradeep Chhetri
Closes #6899 from pradeepchhetri/master and squashes the following commits:
4f4ea79 [Pradeep Chhetri] Added t2.large instance
3d2bb6c [Pradeep Chhetri] Added M4 instances to the list
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ba8a4537
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ba8a4537
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ba8a4537
Branch: refs/heads/master
Commit: ba8a4537fee7d85f968cccf8d1c607731daae307
Parents: 42a1f71
Author: Pradeep Chhetri
Authored: Mon Jun 22 11:45:31 2015 -0700
Committer: Shivaram Venkataraman
Committed: Mon Jun 22 11:45:31 2015 -0700
--
ec2/spark_ec2.py | 16 ++--
1 file changed, 14 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/ba8a4537/ec2/spark_ec2.py
--
diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py
index 1037356..63e2c79 100755
--- a/ec2/spark_ec2.py
+++ b/ec2/spark_ec2.py
@@ -362,7 +362,7 @@ def get_validate_spark_version(version, repo):
# Source: http://aws.amazon.com/amazon-linux-ami/instance-type-matrix/
-# Last Updated: 2015-05-08
+# Last Updated: 2015-06-19
# For easy maintainability, please keep this manually-inputted dictionary
sorted by key.
EC2_INSTANCE_TYPES = {
"c1.medium": "pvm",
@@ -404,6 +404,11 @@ EC2_INSTANCE_TYPES = {
"m3.large":"hvm",
"m3.xlarge": "hvm",
"m3.2xlarge": "hvm",
+"m4.large":"hvm",
+"m4.xlarge": "hvm",
+"m4.2xlarge": "hvm",
+"m4.4xlarge": "hvm",
+"m4.10xlarge": "hvm",
"r3.large":"hvm",
"r3.xlarge": "hvm",
"r3.2xlarge": "hvm",
@@ -413,6 +418,7 @@ EC2_INSTANCE_TYPES = {
"t2.micro":"hvm",
"t2.small":"hvm",
"t2.medium": "hvm",
+"t2.large":"hvm",
}
@@ -923,7 +929,7 @@ def wait_for_cluster_state(conn, opts, cluster_instances,
cluster_state):
# Get number of local disks available for a given EC2 instance type.
def get_num_disks(instance_type):
# Source:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
-# Last Updated: 2015-05-08
+# Last Updated: 2015-06-19
# For easy maintainability, please keep this manually-inputted dictionary
sorted by key.
disks_by_instance = {
"c1.medium": 1,
@@ -965,6 +971,11 @@ def get_num_disks(instance_type):
"m3.large":1,
"m3.xlarge": 2,
"m3.2xlarge": 2,
+"m4.large":0,
+"m4.xlarge": 0,
+"m4.2xlarge": 0,
+"m4.4xlarge": 0,
+"m4.10xlarge": 0,
"r3.large":1,
"r3.xlarge": 1,
"r3.2xlarge": 1,
@@ -974,6 +985,7 @@ def get_num_disks(instance_type):
"t2.micro":0,
"t2.small":0,
"t2.medium": 0,
+"t2.large":0,
}
if instance_type in disks_by_instance:
return disks_by_instance[instance_type]
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8511] [PYSPARK] Modify a test to remove a saved model in `regression.py`
Repository: spark
Updated Branches:
refs/heads/master ba8a4537f -> 5d89d9f00
[SPARK-8511] [PYSPARK] Modify a test to remove a saved model in `regression.py`
[[SPARK-8511] Modify a test to remove a saved model in `regression.py` - ASF
JIRA](https://issues.apache.org/jira/browse/SPARK-8511)
Author: Yu ISHIKAWA
Closes #6926 from yu-iskw/SPARK-8511 and squashes the following commits:
7cd0948 [Yu ISHIKAWA] Use `shutil.rmtree()` to temporary directories for saving
model testings, instead of `os.removedirs()`
4a01c9e [Yu ISHIKAWA] [SPARK-8511][pyspark] Modify a test to remove a saved
model in `regression.py`
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5d89d9f0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5d89d9f0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5d89d9f0
Branch: refs/heads/master
Commit: 5d89d9f00ba4d6d0767a4c4964d3af324bf6f14b
Parents: ba8a453
Author: Yu ISHIKAWA
Authored: Mon Jun 22 11:53:11 2015 -0700
Committer: Joseph K. Bradley
Committed: Mon Jun 22 11:53:11 2015 -0700
--
python/pyspark/mllib/classification.py | 9 ++---
python/pyspark/mllib/clustering.py | 3 ++-
python/pyspark/mllib/recommendation.py | 3 ++-
python/pyspark/mllib/regression.py | 14 +-
python/pyspark/mllib/tests.py | 3 ++-
5 files changed, 21 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/5d89d9f0/python/pyspark/mllib/classification.py
--
diff --git a/python/pyspark/mllib/classification.py
b/python/pyspark/mllib/classification.py
index 42e4139..758accf 100644
--- a/python/pyspark/mllib/classification.py
+++ b/python/pyspark/mllib/classification.py
@@ -135,8 +135,9 @@ class LogisticRegressionModel(LinearClassificationModel):
1
>>> sameModel.predict(SparseVector(2, {0: 1.0}))
0
+>>> from shutil import rmtree
>>> try:
-...os.removedirs(path)
+...rmtree(path)
... except:
...pass
>>> multi_class_data = [
@@ -387,8 +388,9 @@ class SVMModel(LinearClassificationModel):
1
>>> sameModel.predict(SparseVector(2, {0: -1.0}))
0
+>>> from shutil import rmtree
>>> try:
-...os.removedirs(path)
+...rmtree(path)
... except:
...pass
"""
@@ -515,8 +517,9 @@ class NaiveBayesModel(Saveable, Loader):
>>> sameModel = NaiveBayesModel.load(sc, path)
>>> sameModel.predict(SparseVector(2, {0: 1.0})) ==
model.predict(SparseVector(2, {0: 1.0}))
True
+>>> from shutil import rmtree
>>> try:
-... os.removedirs(path)
+... rmtree(path)
... except OSError:
... pass
"""
http://git-wip-us.apache.org/repos/asf/spark/blob/5d89d9f0/python/pyspark/mllib/clustering.py
--
diff --git a/python/pyspark/mllib/clustering.py
b/python/pyspark/mllib/clustering.py
index c382298..e6ef729 100644
--- a/python/pyspark/mllib/clustering.py
+++ b/python/pyspark/mllib/clustering.py
@@ -79,8 +79,9 @@ class KMeansModel(Saveable, Loader):
>>> sameModel = KMeansModel.load(sc, path)
>>> sameModel.predict(sparse_data[0]) == model.predict(sparse_data[0])
True
+>>> from shutil import rmtree
>>> try:
-... os.removedirs(path)
+... rmtree(path)
... except OSError:
... pass
"""
http://git-wip-us.apache.org/repos/asf/spark/blob/5d89d9f0/python/pyspark/mllib/recommendation.py
--
diff --git a/python/pyspark/mllib/recommendation.py
b/python/pyspark/mllib/recommendation.py
index 9c4647d..506ca21 100644
--- a/python/pyspark/mllib/recommendation.py
+++ b/python/pyspark/mllib/recommendation.py
@@ -106,8 +106,9 @@ class MatrixFactorizationModel(JavaModelWrapper,
JavaSaveable, JavaLoader):
0.4...
>>> sameModel.predictAll(testset).collect()
[Rating(...
+>>> from shutil import rmtree
>>> try:
-... os.removedirs(path)
+... rmtree(path)
... except OSError:
... pass
"""
http://git-wip-us.apache.org/repos/asf/spark/blob/5d89d9f0/python/pyspark/mllib/regression.py
--
diff --git a/python/pyspark/mllib/regression.py
b/python/pyspark/mllib/regression.py
index 0c4d7d3..5ddbbee 100644
--- a/python/pyspark/mllib/regression.py
+++ b/python/pyspark/mllib/regression.py
@@ -133,10 +133,11 @@ class LinearRegressionModel(LinearRegressionModelBase):
True
>>> abs(sameModel.predict(SparseVector(1, {0: 1.0})) - 1) < 0.5
True
+>>> from shutil import rmtree
>>> try:
-...os.removedirs(path)
+... rmtree(path)
spark git commit: [SPARK-8511] [PYSPARK] Modify a test to remove a saved model in `regression.py`
Repository: spark
Updated Branches:
refs/heads/branch-1.4 65981619b -> 507381d39
[SPARK-8511] [PYSPARK] Modify a test to remove a saved model in `regression.py`
[[SPARK-8511] Modify a test to remove a saved model in `regression.py` - ASF
JIRA](https://issues.apache.org/jira/browse/SPARK-8511)
Author: Yu ISHIKAWA
Closes #6926 from yu-iskw/SPARK-8511 and squashes the following commits:
7cd0948 [Yu ISHIKAWA] Use `shutil.rmtree()` to temporary directories for saving
model testings, instead of `os.removedirs()`
4a01c9e [Yu ISHIKAWA] [SPARK-8511][pyspark] Modify a test to remove a saved
model in `regression.py`
(cherry picked from commit 5d89d9f00ba4d6d0767a4c4964d3af324bf6f14b)
Signed-off-by: Joseph K. Bradley
Conflicts:
python/pyspark/mllib/tests.py
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/507381d3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/507381d3
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/507381d3
Branch: refs/heads/branch-1.4
Commit: 507381d393cdfee697e8b67f9d1f8a048f383c05
Parents: 6598161
Author: Yu ISHIKAWA
Authored: Mon Jun 22 11:53:11 2015 -0700
Committer: Joseph K. Bradley
Committed: Mon Jun 22 11:59:53 2015 -0700
--
python/pyspark/mllib/classification.py | 9 ++---
python/pyspark/mllib/clustering.py | 3 ++-
python/pyspark/mllib/recommendation.py | 3 ++-
python/pyspark/mllib/regression.py | 14 +-
python/pyspark/mllib/tests.py | 3 ++-
5 files changed, 21 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/507381d3/python/pyspark/mllib/classification.py
--
diff --git a/python/pyspark/mllib/classification.py
b/python/pyspark/mllib/classification.py
index 42e4139..758accf 100644
--- a/python/pyspark/mllib/classification.py
+++ b/python/pyspark/mllib/classification.py
@@ -135,8 +135,9 @@ class LogisticRegressionModel(LinearClassificationModel):
1
>>> sameModel.predict(SparseVector(2, {0: 1.0}))
0
+>>> from shutil import rmtree
>>> try:
-...os.removedirs(path)
+...rmtree(path)
... except:
...pass
>>> multi_class_data = [
@@ -387,8 +388,9 @@ class SVMModel(LinearClassificationModel):
1
>>> sameModel.predict(SparseVector(2, {0: -1.0}))
0
+>>> from shutil import rmtree
>>> try:
-...os.removedirs(path)
+...rmtree(path)
... except:
...pass
"""
@@ -515,8 +517,9 @@ class NaiveBayesModel(Saveable, Loader):
>>> sameModel = NaiveBayesModel.load(sc, path)
>>> sameModel.predict(SparseVector(2, {0: 1.0})) ==
model.predict(SparseVector(2, {0: 1.0}))
True
+>>> from shutil import rmtree
>>> try:
-... os.removedirs(path)
+... rmtree(path)
... except OSError:
... pass
"""
http://git-wip-us.apache.org/repos/asf/spark/blob/507381d3/python/pyspark/mllib/clustering.py
--
diff --git a/python/pyspark/mllib/clustering.py
b/python/pyspark/mllib/clustering.py
index b55583f..15e4527 100644
--- a/python/pyspark/mllib/clustering.py
+++ b/python/pyspark/mllib/clustering.py
@@ -75,8 +75,9 @@ class KMeansModel(Saveable, Loader):
>>> sameModel = KMeansModel.load(sc, path)
>>> sameModel.predict(sparse_data[0]) == model.predict(sparse_data[0])
True
+>>> from shutil import rmtree
>>> try:
-... os.removedirs(path)
+... rmtree(path)
... except OSError:
... pass
"""
http://git-wip-us.apache.org/repos/asf/spark/blob/507381d3/python/pyspark/mllib/recommendation.py
--
diff --git a/python/pyspark/mllib/recommendation.py
b/python/pyspark/mllib/recommendation.py
index 9c4647d..506ca21 100644
--- a/python/pyspark/mllib/recommendation.py
+++ b/python/pyspark/mllib/recommendation.py
@@ -106,8 +106,9 @@ class MatrixFactorizationModel(JavaModelWrapper,
JavaSaveable, JavaLoader):
0.4...
>>> sameModel.predictAll(testset).collect()
[Rating(...
+>>> from shutil import rmtree
>>> try:
-... os.removedirs(path)
+... rmtree(path)
... except OSError:
... pass
"""
http://git-wip-us.apache.org/repos/asf/spark/blob/507381d3/python/pyspark/mllib/regression.py
--
diff --git a/python/pyspark/mllib/regression.py
b/python/pyspark/mllib/regression.py
index 0c4d7d3..5ddbbee 100644
--- a/python/pyspark/mllib/regression.py
+++ b/python/pyspark/mllib/regression.py
@@ -133,10 +133,11 @@ class LinearRegressionModel(LinearRegressionModelBase):
True
>>> abs(sameModel.p
spark git commit: [SPARK-8104] [SQL] auto alias expressions in analyzer
Repository: spark
Updated Branches:
refs/heads/master 5d89d9f00 -> da7bbb943
[SPARK-8104] [SQL] auto alias expressions in analyzer
Currently we auto alias expression in parser. However, during parser phase we
don't have enough information to do the right alias. For example, Generator
that has more than 1 kind of element need MultiAlias, ExtractValue don't need
Alias if it's in middle of a ExtractValue chain.
Author: Wenchen Fan
Closes #6647 from cloud-fan/alias and squashes the following commits:
552eba4 [Wenchen Fan] fix python
5b5786d [Wenchen Fan] fix agg
73a90cb [Wenchen Fan] fix case-preserve of ExtractValue
4cfd23c [Wenchen Fan] fix order by
d18f401 [Wenchen Fan] refine
9f07359 [Wenchen Fan] address comments
39c1aef [Wenchen Fan] small fix
33640ec [Wenchen Fan] auto alias expressions in analyzer
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/da7bbb94
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/da7bbb94
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/da7bbb94
Branch: refs/heads/master
Commit: da7bbb9435dae9a3bedad578599d96ea858f349e
Parents: 5d89d9f
Author: Wenchen Fan
Authored: Mon Jun 22 12:13:00 2015 -0700
Committer: Michael Armbrust
Committed: Mon Jun 22 12:13:00 2015 -0700
--
python/pyspark/sql/context.py | 9 ++-
.../apache/spark/sql/catalyst/SqlParser.scala | 11 +--
.../spark/sql/catalyst/analysis/Analyzer.scala | 77
.../sql/catalyst/analysis/CheckAnalysis.scala | 9 +--
.../sql/catalyst/analysis/unresolved.scala | 20 -
.../sql/catalyst/expressions/ExtractValue.scala | 36 +
.../spark/sql/catalyst/planning/patterns.scala | 6 +-
.../catalyst/plans/logical/LogicalPlan.scala| 11 ++-
.../catalyst/plans/logical/basicOperators.scala | 20 -
.../scala/org/apache/spark/sql/Column.scala | 1 -
.../scala/org/apache/spark/sql/DataFrame.scala | 6 +-
.../org/apache/spark/sql/GroupedData.scala | 43 +--
.../apache/spark/sql/execution/pythonUdfs.scala | 2 +-
.../org/apache/spark/sql/SQLQuerySuite.scala| 6 +-
.../scala/org/apache/spark/sql/TestData.scala | 1 -
.../org/apache/spark/sql/hive/HiveQl.scala | 9 +--
16 files changed, 150 insertions(+), 117 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/da7bbb94/python/pyspark/sql/context.py
--
diff --git a/python/pyspark/sql/context.py b/python/pyspark/sql/context.py
index 599c9ac..dc23922 100644
--- a/python/pyspark/sql/context.py
+++ b/python/pyspark/sql/context.py
@@ -86,7 +86,8 @@ class SQLContext(object):
>>> df.registerTempTable("allTypes")
>>> sqlContext.sql('select i+1, d+1, not b, list[1], dict["s"], time,
row.a '
...'from allTypes where b and i > 0').collect()
-[Row(c0=2, c1=2.0, c2=False, c3=2, c4=0, time=datetime.datetime(2014,
8, 1, 14, 1, 5), a=1)]
+[Row(_c0=2, _c1=2.0, _c2=False, _c3=2, _c4=0, \
+time=datetime.datetime(2014, 8, 1, 14, 1, 5), a=1)]
>>> df.map(lambda x: (x.i, x.s, x.d, x.l, x.b, x.time, x.row.a,
x.list)).collect()
[(1, u'string', 1.0, 1, True, datetime.datetime(2014, 8, 1, 14, 1, 5),
1, [1, 2, 3])]
"""
@@ -176,17 +177,17 @@ class SQLContext(object):
>>> sqlContext.registerFunction("stringLengthString", lambda x: len(x))
>>> sqlContext.sql("SELECT stringLengthString('test')").collect()
-[Row(c0=u'4')]
+[Row(_c0=u'4')]
>>> from pyspark.sql.types import IntegerType
>>> sqlContext.registerFunction("stringLengthInt", lambda x: len(x),
IntegerType())
>>> sqlContext.sql("SELECT stringLengthInt('test')").collect()
-[Row(c0=4)]
+[Row(_c0=4)]
>>> from pyspark.sql.types import IntegerType
>>> sqlContext.udf.register("stringLengthInt", lambda x: len(x),
IntegerType())
>>> sqlContext.sql("SELECT stringLengthInt('test')").collect()
-[Row(c0=4)]
+[Row(_c0=4)]
"""
func = lambda _, it: map(lambda x: f(*x), it)
ser = AutoBatchedSerializer(PickleSerializer())
http://git-wip-us.apache.org/repos/asf/spark/blob/da7bbb94/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
index da3a717..79f526e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
@@ -99,13 +99,6 @@ class SqlParser extends AbstractSpa
spark git commit: [SPARK-8532] [SQL] In Python's DataFrameWriter, save/saveAsTable/json/parquet/jdbc always override mode
Repository: spark
Updated Branches:
refs/heads/branch-1.4 507381d39 -> 994abbaeb
[SPARK-8532] [SQL] In Python's DataFrameWriter,
save/saveAsTable/json/parquet/jdbc always override mode
https://issues.apache.org/jira/browse/SPARK-8532
This PR has two changes. First, it fixes the bug that save actions (i.e.
`save/saveAsTable/json/parquet/jdbc`) always override mode. Second, it adds
input argument `partitionBy` to `save/saveAsTable/parquet`.
Author: Yin Huai
Closes #6937 from yhuai/SPARK-8532 and squashes the following commits:
f972d5d [Yin Huai] davies's comment.
d37abd2 [Yin Huai] style.
d21290a [Yin Huai] Python doc.
889eb25 [Yin Huai] Minor refactoring and add partitionBy to save, saveAsTable,
and parquet.
7fbc24b [Yin Huai] Use None instead of "error" as the default value of mode
since JVM-side already uses "error" as the default value.
d696dff [Yin Huai] Python style.
88eb6c4 [Yin Huai] If mode is "error", do not call mode method.
c40c461 [Yin Huai] Regression test.
(cherry picked from commit 5ab9fcfb01a0ad2f6c103f67c1a785d3b49e33f0)
Signed-off-by: Yin Huai
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/994abbae
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/994abbae
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/994abbae
Branch: refs/heads/branch-1.4
Commit: 994abbaeb3c5444d09548291f865373ba4f1909f
Parents: 507381d
Author: Yin Huai
Authored: Mon Jun 22 13:51:23 2015 -0700
Committer: Yin Huai
Committed: Mon Jun 22 13:51:34 2015 -0700
--
python/pyspark/sql/readwriter.py | 30 +++---
python/pyspark/sql/tests.py | 32
2 files changed, 51 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/994abbae/python/pyspark/sql/readwriter.py
--
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index f036644..1b7bc0f 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -218,7 +218,10 @@ class DataFrameWriter(object):
>>> df.write.mode('append').parquet(os.path.join(tempfile.mkdtemp(),
'data'))
"""
-self._jwrite = self._jwrite.mode(saveMode)
+# At the JVM side, the default value of mode is already set to "error".
+# So, if the given saveMode is None, we will not call JVM-side's mode
method.
+if saveMode is not None:
+self._jwrite = self._jwrite.mode(saveMode)
return self
@since(1.4)
@@ -253,11 +256,12 @@ class DataFrameWriter(object):
"""
if len(cols) == 1 and isinstance(cols[0], (list, tuple)):
cols = cols[0]
-self._jwrite = self._jwrite.partitionBy(_to_seq(self._sqlContext._sc,
cols))
+if len(cols) > 0:
+self._jwrite =
self._jwrite.partitionBy(_to_seq(self._sqlContext._sc, cols))
return self
@since(1.4)
-def save(self, path=None, format=None, mode="error", **options):
+def save(self, path=None, format=None, mode=None, partitionBy=(),
**options):
"""Saves the contents of the :class:`DataFrame` to a data source.
The data source is specified by the ``format`` and a set of
``options``.
@@ -272,11 +276,12 @@ class DataFrameWriter(object):
* ``overwrite``: Overwrite existing data.
* ``ignore``: Silently ignore this operation if data already
exists.
* ``error`` (default case): Throw an exception if data already
exists.
+:param partitionBy: names of partitioning columns
:param options: all other string options
>>> df.write.mode('append').parquet(os.path.join(tempfile.mkdtemp(),
'data'))
"""
-self.mode(mode).options(**options)
+self.partitionBy(partitionBy).mode(mode).options(**options)
if format is not None:
self.format(format)
if path is None:
@@ -296,7 +301,7 @@ class DataFrameWriter(object):
self._jwrite.mode("overwrite" if overwrite else
"append").insertInto(tableName)
@since(1.4)
-def saveAsTable(self, name, format=None, mode="error", **options):
+def saveAsTable(self, name, format=None, mode=None, partitionBy=(),
**options):
"""Saves the content of the :class:`DataFrame` as the specified table.
In the case the table already exists, behavior of this function
depends on the
@@ -312,15 +317,16 @@ class DataFrameWriter(object):
:param name: the table name
:param format: the format used to save
:param mode: one of `append`, `overwrite`, `error`, `ignore` (default:
error)
+:param partitionBy: names of partitioning columns
:param options: all other string o
spark git commit: [SPARK-8532] [SQL] In Python's DataFrameWriter, save/saveAsTable/json/parquet/jdbc always override mode
Repository: spark
Updated Branches:
refs/heads/master da7bbb943 -> 5ab9fcfb0
[SPARK-8532] [SQL] In Python's DataFrameWriter,
save/saveAsTable/json/parquet/jdbc always override mode
https://issues.apache.org/jira/browse/SPARK-8532
This PR has two changes. First, it fixes the bug that save actions (i.e.
`save/saveAsTable/json/parquet/jdbc`) always override mode. Second, it adds
input argument `partitionBy` to `save/saveAsTable/parquet`.
Author: Yin Huai
Closes #6937 from yhuai/SPARK-8532 and squashes the following commits:
f972d5d [Yin Huai] davies's comment.
d37abd2 [Yin Huai] style.
d21290a [Yin Huai] Python doc.
889eb25 [Yin Huai] Minor refactoring and add partitionBy to save, saveAsTable,
and parquet.
7fbc24b [Yin Huai] Use None instead of "error" as the default value of mode
since JVM-side already uses "error" as the default value.
d696dff [Yin Huai] Python style.
88eb6c4 [Yin Huai] If mode is "error", do not call mode method.
c40c461 [Yin Huai] Regression test.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5ab9fcfb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5ab9fcfb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5ab9fcfb
Branch: refs/heads/master
Commit: 5ab9fcfb01a0ad2f6c103f67c1a785d3b49e33f0
Parents: da7bbb9
Author: Yin Huai
Authored: Mon Jun 22 13:51:23 2015 -0700
Committer: Yin Huai
Committed: Mon Jun 22 13:51:23 2015 -0700
--
python/pyspark/sql/readwriter.py | 30 +++---
python/pyspark/sql/tests.py | 32
2 files changed, 51 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/5ab9fcfb/python/pyspark/sql/readwriter.py
--
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index f036644..1b7bc0f 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -218,7 +218,10 @@ class DataFrameWriter(object):
>>> df.write.mode('append').parquet(os.path.join(tempfile.mkdtemp(),
'data'))
"""
-self._jwrite = self._jwrite.mode(saveMode)
+# At the JVM side, the default value of mode is already set to "error".
+# So, if the given saveMode is None, we will not call JVM-side's mode
method.
+if saveMode is not None:
+self._jwrite = self._jwrite.mode(saveMode)
return self
@since(1.4)
@@ -253,11 +256,12 @@ class DataFrameWriter(object):
"""
if len(cols) == 1 and isinstance(cols[0], (list, tuple)):
cols = cols[0]
-self._jwrite = self._jwrite.partitionBy(_to_seq(self._sqlContext._sc,
cols))
+if len(cols) > 0:
+self._jwrite =
self._jwrite.partitionBy(_to_seq(self._sqlContext._sc, cols))
return self
@since(1.4)
-def save(self, path=None, format=None, mode="error", **options):
+def save(self, path=None, format=None, mode=None, partitionBy=(),
**options):
"""Saves the contents of the :class:`DataFrame` to a data source.
The data source is specified by the ``format`` and a set of
``options``.
@@ -272,11 +276,12 @@ class DataFrameWriter(object):
* ``overwrite``: Overwrite existing data.
* ``ignore``: Silently ignore this operation if data already
exists.
* ``error`` (default case): Throw an exception if data already
exists.
+:param partitionBy: names of partitioning columns
:param options: all other string options
>>> df.write.mode('append').parquet(os.path.join(tempfile.mkdtemp(),
'data'))
"""
-self.mode(mode).options(**options)
+self.partitionBy(partitionBy).mode(mode).options(**options)
if format is not None:
self.format(format)
if path is None:
@@ -296,7 +301,7 @@ class DataFrameWriter(object):
self._jwrite.mode("overwrite" if overwrite else
"append").insertInto(tableName)
@since(1.4)
-def saveAsTable(self, name, format=None, mode="error", **options):
+def saveAsTable(self, name, format=None, mode=None, partitionBy=(),
**options):
"""Saves the content of the :class:`DataFrame` as the specified table.
In the case the table already exists, behavior of this function
depends on the
@@ -312,15 +317,16 @@ class DataFrameWriter(object):
:param name: the table name
:param format: the format used to save
:param mode: one of `append`, `overwrite`, `error`, `ignore` (default:
error)
+:param partitionBy: names of partitioning columns
:param options: all other string options
"""
-self.mode(mode).options(**options)
+self.partitionBy(partitionBy).
spark git commit: [SPARK-8455] [ML] Implement n-gram feature transformer
Repository: spark
Updated Branches:
refs/heads/master 5ab9fcfb0 -> afe35f051
[SPARK-8455] [ML] Implement n-gram feature transformer
Implementation of n-gram feature transformer for ML.
Author: Feynman Liang
Closes #6887 from feynmanliang/ngram-featurizer and squashes the following
commits:
d2c839f [Feynman Liang] Make n > input length yield empty output
9fadd36 [Feynman Liang] Add empty and corner test cases, fix names and spaces
fe93873 [Feynman Liang] Implement n-gram feature transformer
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/afe35f05
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/afe35f05
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/afe35f05
Branch: refs/heads/master
Commit: afe35f0519bc7dcb85010a7eedcff854d4fc313a
Parents: 5ab9fcf
Author: Feynman Liang
Authored: Mon Jun 22 14:15:35 2015 -0700
Committer: Joseph K. Bradley
Committed: Mon Jun 22 14:15:35 2015 -0700
--
.../org/apache/spark/ml/feature/NGram.scala | 69 ++
.../apache/spark/ml/feature/NGramSuite.scala| 94
2 files changed, 163 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/afe35f05/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
--
diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
new file mode 100644
index 000..8de10eb
--- /dev/null
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/NGram.scala
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.ml.UnaryTransformer
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.util.Identifiable
+import org.apache.spark.sql.types.{ArrayType, DataType, StringType}
+
+/**
+ * :: Experimental ::
+ * A feature transformer that converts the input array of strings into an
array of n-grams. Null
+ * values in the input array are ignored.
+ * It returns an array of n-grams where each n-gram is represented by a
space-separated string of
+ * words.
+ *
+ * When the input is empty, an empty array is returned.
+ * When the input array length is less than n (number of elements per n-gram),
no n-grams are
+ * returned.
+ */
+@Experimental
+class NGram(override val uid: String)
+ extends UnaryTransformer[Seq[String], Seq[String], NGram] {
+
+ def this() = this(Identifiable.randomUID("ngram"))
+
+ /**
+ * Minimum n-gram length, >= 1.
+ * Default: 2, bigram features
+ * @group param
+ */
+ val n: IntParam = new IntParam(this, "n", "number elements per n-gram (>=1)",
+ParamValidators.gtEq(1))
+
+ /** @group setParam */
+ def setN(value: Int): this.type = set(n, value)
+
+ /** @group getParam */
+ def getN: Int = $(n)
+
+ setDefault(n -> 2)
+
+ override protected def createTransformFunc: Seq[String] => Seq[String] = {
+_.iterator.sliding($(n)).withPartial(false).map(_.mkString(" ")).toSeq
+ }
+
+ override protected def validateInputType(inputType: DataType): Unit = {
+require(inputType.sameType(ArrayType(StringType)),
+ s"Input type must be ArrayType(StringType) but got $inputType.")
+ }
+
+ override protected def outputDataType: DataType = new ArrayType(StringType,
false)
+}
http://git-wip-us.apache.org/repos/asf/spark/blob/afe35f05/mllib/src/test/scala/org/apache/spark/ml/feature/NGramSuite.scala
--
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/NGramSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/feature/NGramSuite.scala
new file mode 100644
index 000..ab97e3d
--- /dev/null
+++ b/mllib/src/test/scala/org/apache/spark/ml/feature/NGramSuite.scala
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
spark git commit: [SPARK-8537] [SPARKR] Add a validation rule about the curly braces in SparkR to `.lintr`
Repository: spark
Updated Branches:
refs/heads/master afe35f051 -> b1f3a489e
[SPARK-8537] [SPARKR] Add a validation rule about the curly braces in SparkR to
`.lintr`
[[SPARK-8537] Add a validation rule about the curly braces in SparkR to
`.lintr` - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-8537)
Author: Yu ISHIKAWA
Closes #6940 from yu-iskw/SPARK-8537 and squashes the following commits:
7eec1a0 [Yu ISHIKAWA] [SPARK-8537][SparkR] Add a validation rule about the
curly braces in SparkR to `.lintr`
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b1f3a489
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b1f3a489
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b1f3a489
Branch: refs/heads/master
Commit: b1f3a489efc6f4f9d172344c3345b9b38ae235e0
Parents: afe35f0
Author: Yu ISHIKAWA
Authored: Mon Jun 22 14:35:38 2015 -0700
Committer: Shivaram Venkataraman
Committed: Mon Jun 22 14:35:38 2015 -0700
--
R/pkg/.lintr | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b1f3a489/R/pkg/.lintr
--
diff --git a/R/pkg/.lintr b/R/pkg/.lintr
index b10ebd3..038236f 100644
--- a/R/pkg/.lintr
+++ b/R/pkg/.lintr
@@ -1,2 +1,2 @@
-linters: with_defaults(line_length_linter(100), camel_case_linter = NULL)
+linters: with_defaults(line_length_linter(100), camel_case_linter = NULL,
open_curly_linter(allow_single_line = TRUE),
closed_curly_linter(allow_single_line = TRUE))
exclusions: list("inst/profile/general.R" = 1, "inst/profile/shell.R")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8095][BACKPORT] Resolve dependencies of --packages in local ivy cache
Repository: spark
Updated Branches:
refs/heads/branch-1.3 0b8dce0c0 -> 45b4527e3
[SPARK-8095][BACKPORT] Resolve dependencies of --packages in local ivy cache
Backported PR #6788
cc andrewor14
Author: Burak Yavuz
Closes #6923 from brkyvz/backport-local-ivy and squashes the following commits:
eb17384 [Burak Yavuz] [SPARK-8095][BACKPORT] Resolve dependencies of --packages
in local ivy cache
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/45b4527e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/45b4527e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/45b4527e
Branch: refs/heads/branch-1.3
Commit: 45b4527e34d67a0143eeaa70660feee04ab8d85a
Parents: 0b8dce0
Author: Burak Yavuz
Authored: Mon Jun 22 14:45:52 2015 -0700
Committer: Andrew Or
Committed: Mon Jun 22 14:45:52 2015 -0700
--
.../org/apache/spark/deploy/SparkSubmit.scala | 24 ++--
.../org/apache/spark/deploy/IvyTestUtils.scala | 127 ---
.../spark/deploy/SparkSubmitUtilsSuite.scala| 22 ++--
3 files changed, 138 insertions(+), 35 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/45b4527e/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
--
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index 02fb360..831679e 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -35,7 +35,8 @@ import org.apache.ivy.core.resolve.ResolveOptions
import org.apache.ivy.core.retrieve.RetrieveOptions
import org.apache.ivy.core.settings.IvySettings
import org.apache.ivy.plugins.matcher.GlobPatternMatcher
-import org.apache.ivy.plugins.resolver.{ChainResolver, IBiblioResolver}
+import org.apache.ivy.plugins.repository.file.FileRepository
+import org.apache.ivy.plugins.resolver.{FileSystemResolver, ChainResolver,
IBiblioResolver}
import org.apache.spark.SPARK_VERSION
import org.apache.spark.deploy.rest._
@@ -677,8 +678,14 @@ private[spark] object SparkSubmitUtils {
}
/** Path of the local Maven cache. */
- private[spark] def m2Path: File = new File(System.getProperty("user.home"),
-".m2" + File.separator + "repository" + File.separator)
+ private[spark] def m2Path: File = {
+if (Utils.isTesting) {
+ // test builds delete the maven cache, and this can cause flakiness
+ new File("dummy", ".m2" + File.separator + "repository")
+} else {
+ new File(System.getProperty("user.home"), ".m2" + File.separator +
"repository")
+}
+ }
/**
* Extracts maven coordinates from a comma-delimited string
@@ -700,12 +707,13 @@ private[spark] object SparkSubmitUtils {
localM2.setName("local-m2-cache")
cr.add(localM2)
-val localIvy = new IBiblioResolver
-localIvy.setRoot(new File(ivySettings.getDefaultIvyUserDir,
- "local" + File.separator).toURI.toString)
+val localIvy = new FileSystemResolver
+val localIvyRoot = new File(ivySettings.getDefaultIvyUserDir, "local")
+localIvy.setLocal(true)
+localIvy.setRepository(new FileRepository(localIvyRoot))
val ivyPattern = Seq("[organisation]", "[module]", "[revision]", "[type]s",
"[artifact](-[classifier]).[ext]").mkString(File.separator)
-localIvy.setPattern(ivyPattern)
+localIvy.addIvyPattern(localIvyRoot.getAbsolutePath + File.separator +
ivyPattern)
localIvy.setName("local-ivy-cache")
cr.add(localIvy)
@@ -769,7 +777,7 @@ private[spark] object SparkSubmitUtils {
md.addDependency(dd)
}
}
-
+
/** Add exclusion rules for dependencies already included in the
spark-assembly */
private[spark] def addExclusionRules(
ivySettings: IvySettings,
http://git-wip-us.apache.org/repos/asf/spark/blob/45b4527e/core/src/test/scala/org/apache/spark/deploy/IvyTestUtils.scala
--
diff --git a/core/src/test/scala/org/apache/spark/deploy/IvyTestUtils.scala
b/core/src/test/scala/org/apache/spark/deploy/IvyTestUtils.scala
index 7d39984..8cb5228 100644
--- a/core/src/test/scala/org/apache/spark/deploy/IvyTestUtils.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/IvyTestUtils.scala
@@ -24,6 +24,8 @@ import com.google.common.io.{Files, ByteStreams}
import org.apache.commons.io.FileUtils
+import org.apache.ivy.core.settings.IvySettings
+
import org.apache.spark.TestUtils.{createCompiledClass, JavaSourceFromString}
import org.apache.spark.deploy.SparkSubmitUtils.MavenCoordinate
@@ -44,13 +46,30 @@ private[deploy] object IvyTestUtils {
if (!useIvyLayout) {
Seq(groupDirs, artifactDirs, artifact.ve
spark git commit: [SPARK-8356] [SQL] Reconcile callUDF and callUdf
Repository: spark
Updated Branches:
refs/heads/master b1f3a489e -> 50d3242d6
[SPARK-8356] [SQL] Reconcile callUDF and callUdf
Deprecates ```callUdf``` in favor of ```callUDF```.
Author: BenFradet
Closes #6902 from BenFradet/SPARK-8356 and squashes the following commits:
ef4e9d8 [BenFradet] deprecated callUDF, use udf instead
9b1de4d [BenFradet] reinstated unit test for the deprecated callUdf
cbd80a5 [BenFradet] deprecated callUdf in favor of callUDF
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/50d3242d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/50d3242d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/50d3242d
Branch: refs/heads/master
Commit: 50d3242d6a5530a51dacab249e3f3d49e2d50635
Parents: b1f3a48
Author: BenFradet
Authored: Mon Jun 22 15:06:47 2015 -0700
Committer: Michael Armbrust
Committed: Mon Jun 22 15:06:47 2015 -0700
--
.../scala/org/apache/spark/sql/functions.scala | 45
.../org/apache/spark/sql/DataFrameSuite.scala | 11 -
2 files changed, 55 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/50d3242d/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
--
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
index 7e7a099..8cea826 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
@@ -1448,7 +1448,9 @@ object functions {
*
* @group udf_funcs
* @since 1.3.0
+ * @deprecated As of 1.5.0, since it's redundant with udf()
*/
+@deprecated("Use udf", "1.5.0")
def callUDF(f: Function$x[$fTypes], returnType: DataType${if (args.length
> 0) ", " + args else ""}): Column = {
ScalaUdf(f, returnType, Seq($argsInUdf))
}""")
@@ -1584,7 +1586,9 @@ object functions {
*
* @group udf_funcs
* @since 1.3.0
+ * @deprecated As of 1.5.0, since it's redundant with udf()
*/
+ @deprecated("Use udf", "1.5.0")
def callUDF(f: Function0[_], returnType: DataType): Column = {
ScalaUdf(f, returnType, Seq())
}
@@ -1595,7 +1599,9 @@ object functions {
*
* @group udf_funcs
* @since 1.3.0
+ * @deprecated As of 1.5.0, since it's redundant with udf()
*/
+ @deprecated("Use udf", "1.5.0")
def callUDF(f: Function1[_, _], returnType: DataType, arg1: Column): Column
= {
ScalaUdf(f, returnType, Seq(arg1.expr))
}
@@ -1606,7 +1612,9 @@ object functions {
*
* @group udf_funcs
* @since 1.3.0
+ * @deprecated As of 1.5.0, since it's redundant with udf()
*/
+ @deprecated("Use udf", "1.5.0")
def callUDF(f: Function2[_, _, _], returnType: DataType, arg1: Column, arg2:
Column): Column = {
ScalaUdf(f, returnType, Seq(arg1.expr, arg2.expr))
}
@@ -1617,7 +1625,9 @@ object functions {
*
* @group udf_funcs
* @since 1.3.0
+ * @deprecated As of 1.5.0, since it's redundant with udf()
*/
+ @deprecated("Use udf", "1.5.0")
def callUDF(f: Function3[_, _, _, _], returnType: DataType, arg1: Column,
arg2: Column, arg3: Column): Column = {
ScalaUdf(f, returnType, Seq(arg1.expr, arg2.expr, arg3.expr))
}
@@ -1628,7 +1638,9 @@ object functions {
*
* @group udf_funcs
* @since 1.3.0
+ * @deprecated As of 1.5.0, since it's redundant with udf()
*/
+ @deprecated("Use udf", "1.5.0")
def callUDF(f: Function4[_, _, _, _, _], returnType: DataType, arg1: Column,
arg2: Column, arg3: Column, arg4: Column): Column = {
ScalaUdf(f, returnType, Seq(arg1.expr, arg2.expr, arg3.expr, arg4.expr))
}
@@ -1639,7 +1651,9 @@ object functions {
*
* @group udf_funcs
* @since 1.3.0
+ * @deprecated As of 1.5.0, since it's redundant with udf()
*/
+ @deprecated("Use udf", "1.5.0")
def callUDF(f: Function5[_, _, _, _, _, _], returnType: DataType, arg1:
Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column): Column = {
ScalaUdf(f, returnType, Seq(arg1.expr, arg2.expr, arg3.expr, arg4.expr,
arg5.expr))
}
@@ -1650,7 +1664,9 @@ object functions {
*
* @group udf_funcs
* @since 1.3.0
+ * @deprecated As of 1.5.0, since it's redundant with udf()
*/
+ @deprecated("Use udf", "1.5.0")
def callUDF(f: Function6[_, _, _, _, _, _, _], returnType: DataType, arg1:
Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column):
Column = {
ScalaUdf(f, returnType, Seq(arg1.expr, arg2.expr, arg3.expr, arg4.expr,
arg5.expr, arg6.expr))
}
@@ -1661,7 +1677,9 @@ object functions {
*
* @group udf_funcs
* @since 1.3.0
+ * @deprecated As of 1.5.0, since it's redundant with udf()
spark git commit: [SPARK-8492] [SQL] support binaryType in UnsafeRow
Repository: spark
Updated Branches:
refs/heads/master 50d3242d6 -> 96aa01378
[SPARK-8492] [SQL] support binaryType in UnsafeRow
Support BinaryType in UnsafeRow, just like StringType.
Also change the layout of StringType and BinaryType in UnsafeRow, by combining
offset and size together as Long, which will limit the size of Row to under 2G
(given that fact that any single buffer can not be bigger than 2G in JVM).
Author: Davies Liu
Closes #6911 from davies/unsafe_bin and squashes the following commits:
d68706f [Davies Liu] update comment
519f698 [Davies Liu] address comment
98a964b [Davies Liu] Merge branch 'master' of github.com:apache/spark into
unsafe_bin
180b49d [Davies Liu] fix zero-out
22e4c0a [Davies Liu] zero-out padding bytes
6abfe93 [Davies Liu] fix style
447dea0 [Davies Liu] support binaryType in UnsafeRow
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/96aa0137
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/96aa0137
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/96aa0137
Branch: refs/heads/master
Commit: 96aa01378e3b3dbb4601d31c7312a311cb65b22e
Parents: 50d3242
Author: Davies Liu
Authored: Mon Jun 22 15:22:17 2015 -0700
Committer: Davies Liu
Committed: Mon Jun 22 15:22:17 2015 -0700
--
.../UnsafeFixedWidthAggregationMap.java | 8 ---
.../sql/catalyst/expressions/UnsafeRow.java | 34 ++-
.../expressions/UnsafeRowConverter.scala| 60 +++-
.../expressions/UnsafeRowConverterSuite.scala | 16 +++---
4 files changed, 72 insertions(+), 46 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/96aa0137/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeFixedWidthAggregationMap.java
--
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeFixedWidthAggregationMap.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeFixedWidthAggregationMap.java
index f7849eb..83f2a31 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeFixedWidthAggregationMap.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeFixedWidthAggregationMap.java
@@ -17,7 +17,6 @@
package org.apache.spark.sql.catalyst.expressions;
-import java.util.Arrays;
import java.util.Iterator;
import org.apache.spark.sql.catalyst.InternalRow;
@@ -142,14 +141,7 @@ public final class UnsafeFixedWidthAggregationMap {
final int groupingKeySize =
groupingKeyToUnsafeRowConverter.getSizeRequirement(groupingKey);
// Make sure that the buffer is large enough to hold the key. If it's not,
grow it:
if (groupingKeySize > groupingKeyConversionScratchSpace.length) {
- // This new array will be initially zero, so there's no need to zero it
out here
groupingKeyConversionScratchSpace = new byte[groupingKeySize];
-} else {
- // Zero out the buffer that's used to hold the current row. This is
necessary in order
- // to ensure that rows hash properly, since garbage data from the
previous row could
- // otherwise end up as padding in this row. As a performance
optimization, we only zero out
- // the portion of the buffer that we'll actually write to.
- Arrays.fill(groupingKeyConversionScratchSpace, 0, groupingKeySize,
(byte) 0);
}
final int actualGroupingKeySize = groupingKeyToUnsafeRowConverter.writeRow(
groupingKey,
http://git-wip-us.apache.org/repos/asf/spark/blob/96aa0137/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
--
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
index ed04d2e..bb2f207 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
@@ -47,7 +47,8 @@ import static org.apache.spark.sql.types.DataTypes.*;
* In the `values` region, we store one 8-byte word per field. For fields that
hold fixed-length
* primitive types, such as long, double, or int, we store the value directly
in the word. For
* fields with non-primitive or variable-length values, we store a relative
offset (w.r.t. the
- * base address of the row) that points to the beginning of the
variable-length field.
+ * base address of the row) that points to the beginning of the
variable-length field, and length
+ * (they are combined into a long).
*
* Instances of `UnsafeRow` act as pointers to row data store
spark git commit: [HOTFIX] [TESTS] Typo mqqt -> mqtt
Repository: spark Updated Branches: refs/heads/master 96aa01378 -> 1dfb0f7b2 [HOTFIX] [TESTS] Typo mqqt -> mqtt This was introduced in #6866. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1dfb0f7b Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1dfb0f7b Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1dfb0f7b Branch: refs/heads/master Commit: 1dfb0f7b2aed5ee6d07543fdeac8ff7c777b63b9 Parents: 96aa013 Author: Andrew Or Authored: Mon Jun 22 16:16:26 2015 -0700 Committer: Andrew Or Committed: Mon Jun 22 16:16:26 2015 -0700 -- dev/run-tests.py | 8 1 file changed, 4 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/1dfb0f7b/dev/run-tests.py -- diff --git a/dev/run-tests.py b/dev/run-tests.py index 2cccfed..de1b453 100755 --- a/dev/run-tests.py +++ b/dev/run-tests.py @@ -179,14 +179,14 @@ streaming_twitter = Module( ) -streaming_mqqt = Module( -name="streaming-mqqt", +streaming_mqtt = Module( +name="streaming-mqtt", dependencies=[streaming], source_file_regexes=[ -"external/mqqt", +"external/mqtt", ], sbt_test_goals=[ -"streaming-mqqt/test", +"streaming-mqtt/test", ] ) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-7153] [SQL] support all integral type ordinal in GetArrayItem
Repository: spark
Updated Branches:
refs/heads/master 1dfb0f7b2 -> 860a49ef2
[SPARK-7153] [SQL] support all integral type ordinal in GetArrayItem
first convert `ordinal` to `Number`, then convert to int type.
Author: Wenchen Fan
Closes #5706 from cloud-fan/7153 and squashes the following commits:
915db79 [Wenchen Fan] fix 7153
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/860a49ef
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/860a49ef
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/860a49ef
Branch: refs/heads/master
Commit: 860a49ef20cea5711a7f54de0053ea33647e56a7
Parents: 1dfb0f7
Author: Wenchen Fan
Authored: Mon Jun 22 17:37:35 2015 -0700
Committer: Michael Armbrust
Committed: Mon Jun 22 17:37:35 2015 -0700
--
.../sql/catalyst/expressions/ExtractValue.scala | 2 +-
.../expressions/complexTypeCreator.scala| 80 +++
.../sql/catalyst/expressions/complexTypes.scala | 81
.../catalyst/expressions/ComplexTypeSuite.scala | 20 +
4 files changed, 101 insertions(+), 82 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/860a49ef/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExtractValue.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExtractValue.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExtractValue.scala
index 013027b..4d6c1c2 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExtractValue.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExtractValue.scala
@@ -186,7 +186,7 @@ case class GetArrayItem(child: Expression, ordinal:
Expression)
// TODO: consider using Array[_] for ArrayType child to avoid
// boxing of primitives
val baseValue = value.asInstanceOf[Seq[_]]
-val index = ordinal.asInstanceOf[Int]
+val index = ordinal.asInstanceOf[Number].intValue()
if (index >= baseValue.size || index < 0) {
null
} else {
http://git-wip-us.apache.org/repos/asf/spark/blob/860a49ef/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
new file mode 100644
index 000..e0bf07e
--- /dev/null
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.types._
+
+
+/**
+ * Returns an Array containing the evaluation of all children expressions.
+ */
+case class CreateArray(children: Seq[Expression]) extends Expression {
+
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ lazy val childTypes = children.map(_.dataType).distinct
+
+ override lazy val resolved =
+childrenResolved && childTypes.size <= 1
+
+ override def dataType: DataType = {
+assert(resolved, s"Invalid dataType of mixed ArrayType
${childTypes.mkString(",")}")
+ArrayType(
+ childTypes.headOption.getOrElse(NullType),
+ containsNull = children.exists(_.nullable))
+ }
+
+ override def nullable: Boolean = false
+
+ override def eval(input: InternalRow): Any = {
+children.map(_.eval(input))
+ }
+
+ override def toString: String = s"Array(${children.mkString(",")})"
+}
+
+/**
+ * Returns a Row containing the evaluation of all children expressions.
+ * TODO: [[CreateStruct]] does not support codegen.
+ */
+case class CreateStruct(children: Seq[Expression]) extends Expression {
+
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ override lazy val resolved: Boolean
spark git commit: [SPARK-8307] [SQL] improve timestamp from parquet
Repository: spark
Updated Branches:
refs/heads/master 860a49ef2 -> 6b7f2ceaf
[SPARK-8307] [SQL] improve timestamp from parquet
This PR change to convert julian day to unix timestamp directly (without
Calendar and Timestamp).
cc adrian-wang rxin
Author: Davies Liu
Closes #6759 from davies/improve_ts and squashes the following commits:
849e301 [Davies Liu] Merge branch 'master' of github.com:apache/spark into
improve_ts
b0e4cad [Davies Liu] Merge branch 'master' of github.com:apache/spark into
improve_ts
8e2d56f [Davies Liu] address comments
634b9f5 [Davies Liu] fix mima
4891efb [Davies Liu] address comment
bfc437c [Davies Liu] fix build
ae5979c [Davies Liu] Merge branch 'master' of github.com:apache/spark into
improve_ts
602b969 [Davies Liu] remove jodd
2f2e48c [Davies Liu] fix test
8ace611 [Davies Liu] fix mima
212143b [Davies Liu] fix mina
c834108 [Davies Liu] Merge branch 'master' of github.com:apache/spark into
improve_ts
a3171b8 [Davies Liu] Merge branch 'master' of github.com:apache/spark into
improve_ts
5233974 [Davies Liu] fix scala style
361fd62 [Davies Liu] address comments
ea196d4 [Davies Liu] improve timestamp from parquet
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6b7f2cea
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6b7f2cea
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6b7f2cea
Branch: refs/heads/master
Commit: 6b7f2ceafdcbb014791909747c2210b527305df9
Parents: 860a49e
Author: Davies Liu
Authored: Mon Jun 22 18:03:59 2015 -0700
Committer: Davies Liu
Committed: Mon Jun 22 18:03:59 2015 -0700
--
pom.xml | 1 -
project/MimaExcludes.scala | 12 +-
.../sql/catalyst/CatalystTypeConverters.scala | 14 +-
.../spark/sql/catalyst/expressions/Cast.scala | 16 +-
.../sql/catalyst/expressions/literals.scala | 6 +-
.../spark/sql/catalyst/util/DateTimeUtils.scala | 151 +++
.../spark/sql/catalyst/util/DateUtils.scala | 120 ---
.../sql/catalyst/expressions/CastSuite.scala| 11 +-
.../catalyst/expressions/PredicateSuite.scala | 6 +-
.../expressions/UnsafeRowConverterSuite.scala | 10 +-
.../sql/catalyst/util/DateTimeUtilsSuite.scala | 51 +++
.../sql/catalyst/util/DateUtilsSuite.scala | 39 -
sql/core/pom.xml| 5 -
.../apache/spark/sql/execution/pythonUdfs.scala | 10 +-
.../org/apache/spark/sql/jdbc/JDBCRDD.scala | 12 +-
.../apache/spark/sql/json/JacksonParser.scala | 6 +-
.../org/apache/spark/sql/json/JsonRDD.scala | 8 +-
.../spark/sql/parquet/ParquetConverter.scala| 86 ++-
.../spark/sql/parquet/ParquetTableSupport.scala | 19 ++-
.../spark/sql/parquet/timestamp/NanoTime.scala | 69 -
.../org/apache/spark/sql/json/JsonSuite.scala | 20 +--
.../spark/sql/parquet/ParquetIOSuite.scala | 4 +-
.../spark/sql/sources/TableScanSuite.scala | 11 +-
.../apache/spark/sql/hive/HiveInspectors.scala | 20 +--
.../org/apache/spark/sql/hive/TableReader.scala | 8 +-
.../spark/sql/hive/hiveWriterContainers.scala | 4 +-
26 files changed, 321 insertions(+), 398 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6b7f2cea/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 6d4f717..80cacb5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -156,7 +156,6 @@
2.10
${scala.version}
org.scala-lang
-3.6.3
1.9.13
2.4.4
1.1.1.7
http://git-wip-us.apache.org/repos/asf/spark/blob/6b7f2cea/project/MimaExcludes.scala
--
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index 015d029..7a748fb 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -54,7 +54,17 @@ object MimaExcludes {
ProblemFilters.exclude[MissingMethodProblem](
"org.apache.spark.streaming.kafka.KafkaTestUtils.waitUntilLeaderOffset"),
// SQL execution is considered private.
-excludePackage("org.apache.spark.sql.execution")
+excludePackage("org.apache.spark.sql.execution"),
+// NanoTime and CatalystTimestampConverter is only used inside
catalyst,
+// not needed anymore
+ProblemFilters.exclude[MissingClassProblem](
+ "org.apache.spark.sql.parquet.timestamp.NanoTime"),
+ ProblemFilters.exclude[MissingClassProblem](
+ "org.apache.spark.sql.parquet.timestamp.NanoTime$"),
+ProblemFilters.exclude[MissingClassProblem](
+ "org.apache.spark.sql.parquet.CatalystTimestampConverter"),
+ProblemFilters.exclude[MissingC
spark git commit: [SPARK-7859] [SQL] Collect_set() behavior differences which fails the unit test under jdk8
Repository: spark
Updated Branches:
refs/heads/master 6b7f2ceaf -> 13321e655
[SPARK-7859] [SQL] Collect_set() behavior differences which fails the unit test
under jdk8
To reproduce that:
```
JAVA_HOME=/home/hcheng/Java/jdk1.8.0_45 | build/sbt -Phadoop-2.3 -Phive
'test-only
org.apache.spark.sql.hive.execution.HiveWindowFunctionQueryWithoutCodeGenSuite'
```
A simple workaround to fix that is update the original query, for getting the
output size instead of the exact elements of the array (output by collect_set())
Author: Cheng Hao
Closes #6402 from chenghao-intel/windowing and squashes the following commits:
99312ad [Cheng Hao] add order by for the select clause
edf8ce3 [Cheng Hao] update the code as suggested
7062da7 [Cheng Hao] fix the collect_set() behaviour differences under different
versions of JDK
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/13321e65
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/13321e65
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/13321e65
Branch: refs/heads/master
Commit: 13321e65559f6354ec1287a690580fd6f498ef89
Parents: 6b7f2ce
Author: Cheng Hao
Authored: Mon Jun 22 20:04:49 2015 -0700
Committer: Yin Huai
Committed: Mon Jun 22 20:04:49 2015 -0700
--
.../HiveWindowFunctionQuerySuite.scala | 8 ++
... testSTATs-0-6dfcd7925fb267699c4bf82737d4609 | 97
...testSTATs-0-da0e0cca69e42118a96b8609b8fa5838 | 26 --
3 files changed, 105 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/13321e65/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala
--
diff --git
a/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala
b/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala
index 934452f..31a49a3 100644
---
a/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala
+++
b/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala
@@ -526,8 +526,14 @@ abstract class HiveWindowFunctionQueryBaseSuite extends
HiveComparisonTest with
| rows between 2 preceding and 2 following);
""".stripMargin, reset = false)
+ // collect_set() output array in an arbitrary order, hence causes different
result
+ // when running this test suite under Java 7 and 8.
+ // We change the original sql query a little bit for making the test suite
passed
+ // under different JDK
createQueryTest("windowing.q -- 20. testSTATs",
"""
+ |select p_mfgr,p_name, p_size, sdev, sdev_pop, uniq_data, var, cor,
covarp
+ |from (
|select p_mfgr,p_name, p_size,
|stddev(p_retailprice) over w1 as sdev,
|stddev_pop(p_retailprice) over w1 as sdev_pop,
@@ -538,6 +544,8 @@ abstract class HiveWindowFunctionQueryBaseSuite extends
HiveComparisonTest with
|from part
|window w1 as (distribute by p_mfgr sort by p_mfgr, p_name
| rows between 2 preceding and 2 following)
+ |) t lateral view explode(uniq_size) d as uniq_data
+ |order by p_mfgr,p_name, p_size, sdev, sdev_pop, uniq_data, var, cor,
covarp
""".stripMargin, reset = false)
createQueryTest("windowing.q -- 21. testDISTs",
http://git-wip-us.apache.org/repos/asf/spark/blob/13321e65/sql/hive/src/test/resources/golden/windowing.q
-- 20. testSTATs-0-6dfcd7925fb267699c4bf82737d4609
--
diff --git a/sql/hive/src/test/resources/golden/windowing.q -- 20.
testSTATs-0-6dfcd7925fb267699c4bf82737d4609
b/sql/hive/src/test/resources/golden/windowing.q -- 20.
testSTATs-0-6dfcd7925fb267699c4bf82737d4609
new file mode 100644
index 000..7e5fcee
--- /dev/null
+++ b/sql/hive/src/test/resources/golden/windowing.q -- 20.
testSTATs-0-6dfcd7925fb267699c4bf82737d4609
@@ -0,0 +1,97 @@
+Manufacturer#1 almond antique burnished rose metallic 2
258.10677784349235 258.10677784349235 2 66619.10876874991
0.811328754177887 2801.70745
+Manufacturer#1 almond antique burnished rose metallic 2
258.10677784349235 258.10677784349235 6 66619.10876874991
0.811328754177887 2801.70745
+Manufacturer#1 almond antique burnished rose metallic 2
258.10677784349235 258.10677784349235 34 66619.10876874991
0.811328754177887 2801.70745
+Manufacturer#1 almond antique burnished rose metallic 2
273.70217881648074 273.70217881648074 2 74912.882688
spark git commit: [SPARK-7859] [SQL] Collect_set() behavior differences which fails the unit test under jdk8
Repository: spark
Updated Branches:
refs/heads/branch-1.4 994abbaeb -> d73900a90
[SPARK-7859] [SQL] Collect_set() behavior differences which fails the unit test
under jdk8
To reproduce that:
```
JAVA_HOME=/home/hcheng/Java/jdk1.8.0_45 | build/sbt -Phadoop-2.3 -Phive
'test-only
org.apache.spark.sql.hive.execution.HiveWindowFunctionQueryWithoutCodeGenSuite'
```
A simple workaround to fix that is update the original query, for getting the
output size instead of the exact elements of the array (output by collect_set())
Author: Cheng Hao
Closes #6402 from chenghao-intel/windowing and squashes the following commits:
99312ad [Cheng Hao] add order by for the select clause
edf8ce3 [Cheng Hao] update the code as suggested
7062da7 [Cheng Hao] fix the collect_set() behaviour differences under different
versions of JDK
(cherry picked from commit 13321e65559f6354ec1287a690580fd6f498ef89)
Signed-off-by: Yin Huai
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d73900a9
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d73900a9
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d73900a9
Branch: refs/heads/branch-1.4
Commit: d73900a9034b22191e1440b18ee63b1835f09582
Parents: 994abba
Author: Cheng Hao
Authored: Mon Jun 22 20:04:49 2015 -0700
Committer: Yin Huai
Committed: Mon Jun 22 20:05:00 2015 -0700
--
.../HiveWindowFunctionQuerySuite.scala | 8 ++
... testSTATs-0-6dfcd7925fb267699c4bf82737d4609 | 97
...testSTATs-0-da0e0cca69e42118a96b8609b8fa5838 | 26 --
3 files changed, 105 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/d73900a9/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala
--
diff --git
a/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala
b/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala
index 934452f..31a49a3 100644
---
a/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala
+++
b/sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveWindowFunctionQuerySuite.scala
@@ -526,8 +526,14 @@ abstract class HiveWindowFunctionQueryBaseSuite extends
HiveComparisonTest with
| rows between 2 preceding and 2 following);
""".stripMargin, reset = false)
+ // collect_set() output array in an arbitrary order, hence causes different
result
+ // when running this test suite under Java 7 and 8.
+ // We change the original sql query a little bit for making the test suite
passed
+ // under different JDK
createQueryTest("windowing.q -- 20. testSTATs",
"""
+ |select p_mfgr,p_name, p_size, sdev, sdev_pop, uniq_data, var, cor,
covarp
+ |from (
|select p_mfgr,p_name, p_size,
|stddev(p_retailprice) over w1 as sdev,
|stddev_pop(p_retailprice) over w1 as sdev_pop,
@@ -538,6 +544,8 @@ abstract class HiveWindowFunctionQueryBaseSuite extends
HiveComparisonTest with
|from part
|window w1 as (distribute by p_mfgr sort by p_mfgr, p_name
| rows between 2 preceding and 2 following)
+ |) t lateral view explode(uniq_size) d as uniq_data
+ |order by p_mfgr,p_name, p_size, sdev, sdev_pop, uniq_data, var, cor,
covarp
""".stripMargin, reset = false)
createQueryTest("windowing.q -- 21. testDISTs",
http://git-wip-us.apache.org/repos/asf/spark/blob/d73900a9/sql/hive/src/test/resources/golden/windowing.q
-- 20. testSTATs-0-6dfcd7925fb267699c4bf82737d4609
--
diff --git a/sql/hive/src/test/resources/golden/windowing.q -- 20.
testSTATs-0-6dfcd7925fb267699c4bf82737d4609
b/sql/hive/src/test/resources/golden/windowing.q -- 20.
testSTATs-0-6dfcd7925fb267699c4bf82737d4609
new file mode 100644
index 000..7e5fcee
--- /dev/null
+++ b/sql/hive/src/test/resources/golden/windowing.q -- 20.
testSTATs-0-6dfcd7925fb267699c4bf82737d4609
@@ -0,0 +1,97 @@
+Manufacturer#1 almond antique burnished rose metallic 2
258.10677784349235 258.10677784349235 2 66619.10876874991
0.811328754177887 2801.70745
+Manufacturer#1 almond antique burnished rose metallic 2
258.10677784349235 258.10677784349235 6 66619.10876874991
0.811328754177887 2801.70745
+Manufacturer#1 almond antique burnished rose metallic 2
258.10677784349235 258.10677784349235 34 66619.10876874991
0.811328754177887 2801.70745
+Manufacturer#1 almond antique
spark git commit: MAINTENANCE: Automated closing of pull requests.
Repository: spark Updated Branches: refs/heads/master 13321e655 -> c4d234396 MAINTENANCE: Automated closing of pull requests. This commit exists to close the following pull requests on Github: Closes #2849 (close requested by 'srowen') Closes #2786 (close requested by 'andrewor14') Closes #4678 (close requested by 'JoshRosen') Closes #5457 (close requested by 'andrewor14') Closes #3346 (close requested by 'andrewor14') Closes #6518 (close requested by 'andrewor14') Closes #5403 (close requested by 'pwendell') Closes #2110 (close requested by 'srowen') Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c4d23439 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c4d23439 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c4d23439 Branch: refs/heads/master Commit: c4d2343966cbae40a8271a2e6cad66227d2f8249 Parents: 13321e6 Author: Patrick Wendell Authored: Mon Jun 22 20:25:32 2015 -0700 Committer: Patrick Wendell Committed: Mon Jun 22 20:25:32 2015 -0700 -- -- - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8548] [SPARKR] Remove the trailing whitespaces from the SparkR files
Repository: spark
Updated Branches:
refs/heads/branch-1.4 d73900a90 -> 250179485
[SPARK-8548] [SPARKR] Remove the trailing whitespaces from the SparkR files
[[SPARK-8548] Remove the trailing whitespaces from the SparkR files - ASF
JIRA](https://issues.apache.org/jira/browse/SPARK-8548)
- This is the result of `lint-r`
https://gist.github.com/yu-iskw/0019b37a2c1167f33986
Author: Yu ISHIKAWA
Closes #6945 from yu-iskw/SPARK-8548 and squashes the following commits:
0bd567a [Yu ISHIKAWA] [SPARK-8548][SparkR] Remove the trailing whitespaces from
the SparkR files
(cherry picked from commit 44fa7df64daa55bd6eb1f2c219a9701b34e1c2a3)
Signed-off-by: Shivaram Venkataraman
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/25017948
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/25017948
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/25017948
Branch: refs/heads/branch-1.4
Commit: 250179485b59f3015fd2f44934b6cb1d3669de80
Parents: d73900a
Author: Yu ISHIKAWA
Authored: Mon Jun 22 20:55:38 2015 -0700
Committer: Shivaram Venkataraman
Committed: Mon Jun 22 20:55:55 2015 -0700
--
R/pkg/R/DataFrame.R | 96 ++--
R/pkg/R/RDD.R | 48 +++---
R/pkg/R/SQLContext.R| 14 ++--
R/pkg/R/broadcast.R | 6 +-
R/pkg/R/deserialize.R | 2 +-
R/pkg/R/generics.R | 15 ++---
R/pkg/R/group.R | 1 -
R/pkg/R/jobj.R | 2 +-
R/pkg/R/pairRDD.R | 4 +-
R/pkg/R/schema.R| 2 +-
R/pkg/R/serialize.R | 2 +-
R/pkg/R/sparkR.R| 6 +-
R/pkg/R/utils.R | 48 +++---
R/pkg/R/zzz.R | 1 -
R/pkg/inst/tests/test_binaryFile.R | 7 +-
R/pkg/inst/tests/test_binary_function.R | 28
R/pkg/inst/tests/test_rdd.R | 12 ++--
R/pkg/inst/tests/test_shuffle.R | 28
R/pkg/inst/tests/test_sparkSQL.R| 28
R/pkg/inst/tests/test_take.R| 1 -
R/pkg/inst/tests/test_textFile.R| 7 +-
R/pkg/inst/tests/test_utils.R | 12 ++--
22 files changed, 182 insertions(+), 188 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/25017948/R/pkg/R/DataFrame.R
--
diff --git a/R/pkg/R/DataFrame.R b/R/pkg/R/DataFrame.R
index 0af5cb8..6feabf4 100644
--- a/R/pkg/R/DataFrame.R
+++ b/R/pkg/R/DataFrame.R
@@ -38,7 +38,7 @@ setClass("DataFrame",
setMethod("initialize", "DataFrame", function(.Object, sdf, isCached) {
.Object@env <- new.env()
.Object@env$isCached <- isCached
-
+
.Object@sdf <- sdf
.Object
})
@@ -55,11 +55,11 @@ dataFrame <- function(sdf, isCached = FALSE) {
DataFrame Methods
##
#' Print Schema of a DataFrame
-#'
+#'
#' Prints out the schema in tree format
-#'
+#'
#' @param x A SparkSQL DataFrame
-#'
+#'
#' @rdname printSchema
#' @export
#' @examples
@@ -78,11 +78,11 @@ setMethod("printSchema",
})
#' Get schema object
-#'
+#'
#' Returns the schema of this DataFrame as a structType object.
-#'
+#'
#' @param x A SparkSQL DataFrame
-#'
+#'
#' @rdname schema
#' @export
#' @examples
@@ -100,9 +100,9 @@ setMethod("schema",
})
#' Explain
-#'
+#'
#' Print the logical and physical Catalyst plans to the console for debugging.
-#'
+#'
#' @param x A SparkSQL DataFrame
#' @param extended Logical. If extended is False, explain() only prints the
physical plan.
#' @rdname explain
@@ -200,11 +200,11 @@ setMethod("show", "DataFrame",
})
#' DataTypes
-#'
+#'
#' Return all column names and their data types as a list
-#'
+#'
#' @param x A SparkSQL DataFrame
-#'
+#'
#' @rdname dtypes
#' @export
#' @examples
@@ -224,11 +224,11 @@ setMethod("dtypes",
})
#' Column names
-#'
+#'
#' Return all column names as a list
-#'
+#'
#' @param x A SparkSQL DataFrame
-#'
+#'
#' @rdname columns
#' @export
#' @examples
@@ -256,12 +256,12 @@ setMethod("names",
})
#' Register Temporary Table
-#'
+#'
#' Registers a DataFrame as a Temporary Table in the SQLContext
-#'
+#'
#' @param x A SparkSQL DataFrame
#' @param tableName A character vector containing the name of the table
-#'
+#'
#' @rdname registerTempTable
#' @export
#' @examples
@@ -306,11 +306,11 @@ setMethod("insertInto",
})
#' Cache
-#'
+#'
#' Persist with the default storage level (MEMORY_ONLY).
-#'
+#'
#' @param x A SparkSQL DataFrame
-#'
+#'
#' @rdname cache-methods
#' @export
#'
spark git commit: [SPARK-8548] [SPARKR] Remove the trailing whitespaces from the SparkR files
Repository: spark
Updated Branches:
refs/heads/master c4d234396 -> 44fa7df64
[SPARK-8548] [SPARKR] Remove the trailing whitespaces from the SparkR files
[[SPARK-8548] Remove the trailing whitespaces from the SparkR files - ASF
JIRA](https://issues.apache.org/jira/browse/SPARK-8548)
- This is the result of `lint-r`
https://gist.github.com/yu-iskw/0019b37a2c1167f33986
Author: Yu ISHIKAWA
Closes #6945 from yu-iskw/SPARK-8548 and squashes the following commits:
0bd567a [Yu ISHIKAWA] [SPARK-8548][SparkR] Remove the trailing whitespaces from
the SparkR files
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/44fa7df6
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/44fa7df6
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/44fa7df6
Branch: refs/heads/master
Commit: 44fa7df64daa55bd6eb1f2c219a9701b34e1c2a3
Parents: c4d2343
Author: Yu ISHIKAWA
Authored: Mon Jun 22 20:55:38 2015 -0700
Committer: Shivaram Venkataraman
Committed: Mon Jun 22 20:55:38 2015 -0700
--
R/pkg/R/DataFrame.R | 96 ++--
R/pkg/R/RDD.R | 48 +++---
R/pkg/R/SQLContext.R| 14 ++--
R/pkg/R/broadcast.R | 6 +-
R/pkg/R/deserialize.R | 2 +-
R/pkg/R/generics.R | 15 ++---
R/pkg/R/group.R | 1 -
R/pkg/R/jobj.R | 2 +-
R/pkg/R/pairRDD.R | 4 +-
R/pkg/R/schema.R| 2 +-
R/pkg/R/serialize.R | 2 +-
R/pkg/R/sparkR.R| 6 +-
R/pkg/R/utils.R | 48 +++---
R/pkg/R/zzz.R | 1 -
R/pkg/inst/tests/test_binaryFile.R | 7 +-
R/pkg/inst/tests/test_binary_function.R | 28
R/pkg/inst/tests/test_rdd.R | 12 ++--
R/pkg/inst/tests/test_shuffle.R | 28
R/pkg/inst/tests/test_sparkSQL.R| 28
R/pkg/inst/tests/test_take.R| 1 -
R/pkg/inst/tests/test_textFile.R| 7 +-
R/pkg/inst/tests/test_utils.R | 12 ++--
22 files changed, 182 insertions(+), 188 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/44fa7df6/R/pkg/R/DataFrame.R
--
diff --git a/R/pkg/R/DataFrame.R b/R/pkg/R/DataFrame.R
index 0af5cb8..6feabf4 100644
--- a/R/pkg/R/DataFrame.R
+++ b/R/pkg/R/DataFrame.R
@@ -38,7 +38,7 @@ setClass("DataFrame",
setMethod("initialize", "DataFrame", function(.Object, sdf, isCached) {
.Object@env <- new.env()
.Object@env$isCached <- isCached
-
+
.Object@sdf <- sdf
.Object
})
@@ -55,11 +55,11 @@ dataFrame <- function(sdf, isCached = FALSE) {
DataFrame Methods
##
#' Print Schema of a DataFrame
-#'
+#'
#' Prints out the schema in tree format
-#'
+#'
#' @param x A SparkSQL DataFrame
-#'
+#'
#' @rdname printSchema
#' @export
#' @examples
@@ -78,11 +78,11 @@ setMethod("printSchema",
})
#' Get schema object
-#'
+#'
#' Returns the schema of this DataFrame as a structType object.
-#'
+#'
#' @param x A SparkSQL DataFrame
-#'
+#'
#' @rdname schema
#' @export
#' @examples
@@ -100,9 +100,9 @@ setMethod("schema",
})
#' Explain
-#'
+#'
#' Print the logical and physical Catalyst plans to the console for debugging.
-#'
+#'
#' @param x A SparkSQL DataFrame
#' @param extended Logical. If extended is False, explain() only prints the
physical plan.
#' @rdname explain
@@ -200,11 +200,11 @@ setMethod("show", "DataFrame",
})
#' DataTypes
-#'
+#'
#' Return all column names and their data types as a list
-#'
+#'
#' @param x A SparkSQL DataFrame
-#'
+#'
#' @rdname dtypes
#' @export
#' @examples
@@ -224,11 +224,11 @@ setMethod("dtypes",
})
#' Column names
-#'
+#'
#' Return all column names as a list
-#'
+#'
#' @param x A SparkSQL DataFrame
-#'
+#'
#' @rdname columns
#' @export
#' @examples
@@ -256,12 +256,12 @@ setMethod("names",
})
#' Register Temporary Table
-#'
+#'
#' Registers a DataFrame as a Temporary Table in the SQLContext
-#'
+#'
#' @param x A SparkSQL DataFrame
#' @param tableName A character vector containing the name of the table
-#'
+#'
#' @rdname registerTempTable
#' @export
#' @examples
@@ -306,11 +306,11 @@ setMethod("insertInto",
})
#' Cache
-#'
+#'
#' Persist with the default storage level (MEMORY_ONLY).
-#'
+#'
#' @param x A SparkSQL DataFrame
-#'
+#'
#' @rdname cache-methods
#' @export
#' @examples
@@ -400,7 +400,7 @@ setMethod("repartition",
signature(x = "DataFrame", numPartitions = "numeri
spark git commit: [BUILD] Preparing Spark release 1.4.1
Repository: spark Updated Branches: refs/heads/branch-1.4 250179485 -> 48d683014 [BUILD] Preparing Spark release 1.4.1 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/48d68301 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/48d68301 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/48d68301 Branch: refs/heads/branch-1.4 Commit: 48d683014458d48453b59911b472224879746533 Parents: 2501794 Author: Patrick Wendell Authored: Mon Jun 22 22:18:52 2015 -0700 Committer: Patrick Wendell Committed: Mon Jun 22 22:18:52 2015 -0700 -- CHANGES.txt | 603 +++ R/pkg/DESCRIPTION | 2 +- assembly/pom.xml| 2 +- bagel/pom.xml | 2 +- core/pom.xml| 2 +- .../main/scala/org/apache/spark/package.scala | 2 +- dev/create-release/generate-changelist.py | 4 +- docs/_config.yml| 4 +- ec2/spark_ec2.py| 4 +- examples/pom.xml| 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml | 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml | 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml| 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml | 2 +- extras/kinesis-asl/pom.xml | 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml | 2 +- launcher/pom.xml| 2 +- mllib/pom.xml | 2 +- network/common/pom.xml | 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml| 2 +- pom.xml | 2 +- repl/pom.xml| 2 +- sql/catalyst/pom.xml| 2 +- sql/core/pom.xml| 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml| 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- unsafe/pom.xml | 2 +- yarn/pom.xml| 2 +- 36 files changed, 642 insertions(+), 37 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/48d68301/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 8c99404..d6fe305 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,6 +1,609 @@ Spark Change Log +Release 1.4.1 + + [SPARK-8548] [SPARKR] Remove the trailing whitespaces from the SparkR files + Yu ISHIKAWA + 2015-06-22 20:55:38 -0700 + Commit: 2501794, github.com/apache/spark/pull/6945 + + [SPARK-7859] [SQL] Collect_set() behavior differences which fails the unit test under jdk8 + Cheng Hao + 2015-06-22 20:04:49 -0700 + Commit: d73900a, github.com/apache/spark/pull/6402 + + [SPARK-8532] [SQL] In Python's DataFrameWriter, save/saveAsTable/json/parquet/jdbc always override mode + Yin Huai + 2015-06-22 13:51:23 -0700 + Commit: 994abba, github.com/apache/spark/pull/6937 + + [SPARK-8511] [PYSPARK] Modify a test to remove a saved model in `regression.py` + Yu ISHIKAWA + 2015-06-22 11:53:11 -0700 + Commit: 507381d, github.com/apache/spark/pull/6926 + + [SPARK-8420] [SQL] Fix comparision of timestamps/dates with strings (branch-1.4) + Michael Armbrust , Michael Armbrust + 2015-06-22 10:45:33 -0700 + Commit: 6598161, github.com/apache/spark/pull/6888 + + [SPARK-8406] [SQL] Backports SPARK-8406 and PR #6864 to branch-1.4 + Cheng Lian + 2015-06-22 10:04:29 -0700 + Commit: 451c872, github.com/apache/spark/pull/6932 + + [HOTFIX] Hotfix branch-1.4 building by removing avgMetrics in CrossValidatorSuite + Liang-Chi Hsieh + 2015-06-21 22:25:08 -0700 + Commit: b836bac, github.com/apache/spark/pull/6929 + + [SPARK-7715] [MLLIB] [ML] [DOC] Updated MLlib programming guide for release 1.4 + Joseph K. Bradley + 2015-06-21 16:25:25 -0700 + Commit: 2a7ea31, github.com/apache/spark/pull/6897 + + [SPARK-8379] [SQL] avoid speculative tasks write to the same file + jeanlyn + 2015-06-21 00:13:40 -0700 + Commit: f0e4040, github.com/apache/spark/pull/6833 + + [SPARK-8468] [ML] Take the negative of some metrics in Regress
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.4.1-rc1 [created] d0a5560ce - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[2/2] spark git commit: Preparing development version 1.4.2-SNAPSHOT
Preparing development version 1.4.2-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1cfa7302 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1cfa7302 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1cfa7302 Branch: refs/heads/branch-1.4 Commit: 1cfa7302ee9ac4ec31728f74c63714d53adf27b7 Parents: d0a5560 Author: Patrick Wendell Authored: Mon Jun 22 22:21:31 2015 -0700 Committer: Patrick Wendell Committed: Mon Jun 22 22:21:31 2015 -0700 -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml| 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- launcher/pom.xml | 2 +- mllib/pom.xml | 2 +- network/common/pom.xml| 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml | 2 +- pom.xml | 2 +- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- unsafe/pom.xml| 2 +- yarn/pom.xml | 2 +- 30 files changed, 30 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/1cfa7302/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index ba233e7..228db59 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1 +1.4.2-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/1cfa7302/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index c5e9183..ce791a6 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1 +1.4.2-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/1cfa7302/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index f0d236d..176ea9b 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1 +1.4.2-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/1cfa7302/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index e9a9cc2..877c2fb 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1 +1.4.2-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/1cfa7302/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index 9669507..37f2b1b 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1 +1.4.2-SNAPSHOT ../../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/1cfa7302/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index b3ad09a..9789435 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1 +1.4.2-SNAPSHOT ../../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/1cfa7302/external/kafka-assembly/pom.xml -- diff --git a/external/kafka-assembly/pom.xml b/external/kafka-assembly/pom.xml index c05bd1b..18b1d86 100644 --- a/external/kafka-assembly/pom.xml +++ b/external/kafka-assembly/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1 +1.4.2-SNAPSHOT ../../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/1cfa7302/external/kafka/pom.xml
[1/2] spark git commit: Preparing Spark release v1.4.1-rc1
Repository: spark Updated Branches: refs/heads/branch-1.4 48d683014 -> 1cfa7302e Preparing Spark release v1.4.1-rc1 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d0a5560c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d0a5560c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d0a5560c Branch: refs/heads/branch-1.4 Commit: d0a5560ce4b898ef931c37a315ad53e1d5fbe6c6 Parents: 48d6830 Author: Patrick Wendell Authored: Mon Jun 22 22:21:26 2015 -0700 Committer: Patrick Wendell Committed: Mon Jun 22 22:21:26 2015 -0700 -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml| 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- launcher/pom.xml | 2 +- mllib/pom.xml | 2 +- network/common/pom.xml| 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml | 2 +- pom.xml | 2 +- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- unsafe/pom.xml| 2 +- yarn/pom.xml | 2 +- 30 files changed, 30 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/d0a5560c/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index b53d7c3..ba233e7 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1-SNAPSHOT +1.4.1 ../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/d0a5560c/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index dce13c4..c5e9183 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1-SNAPSHOT +1.4.1 ../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/d0a5560c/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index 4f3375d..f0d236d 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1-SNAPSHOT +1.4.1 ../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/d0a5560c/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index eee557a..e9a9cc2 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1-SNAPSHOT +1.4.1 ../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/d0a5560c/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index fc0ef59..9669507 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1-SNAPSHOT +1.4.1 ../../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/d0a5560c/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index ee70d8e..b3ad09a 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1-SNAPSHOT +1.4.1 ../../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/d0a5560c/external/kafka-assembly/pom.xml -- diff --git a/external/kafka-assembly/pom.xml b/external/kafka-assembly/pom.xml index 72d3547..c05bd1b 100644 --- a/external/kafka-assembly/pom.xml +++ b/external/kafka-assembly/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.10 -1.4.1-SNAPSHOT +1.4.1 ../../pom.xml http://git-wip-us.apache.org/repos/asf/spar
spark git commit: [SPARK-7781] [MLLIB] gradient boosted trees.train regressor missing max bins
Repository: spark
Updated Branches:
refs/heads/master 44fa7df64 -> 164fe2aa4
[SPARK-7781] [MLLIB] gradient boosted trees.train regressor missing max bins
Author: Holden Karau
Closes #6331 from
holdenk/SPARK-7781-GradientBoostedTrees.trainRegressor-missing-max-bins and
squashes the following commits:
2894695 [Holden Karau] remove extra blank line
2573e8d [Holden Karau] Update the scala side of the pythonmllibapi and make the
test a bit nicer too
3a09170 [Holden Karau] add maxBins to to the train method as well
af7f274 [Holden Karau] Add maxBins to GradientBoostedTrees.trainRegressor and
correctly mention the default of 32 in other places where it mentioned 100
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/164fe2aa
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/164fe2aa
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/164fe2aa
Branch: refs/heads/master
Commit: 164fe2aa44993da6c77af6de5efdae47a8b3958c
Parents: 44fa7df
Author: Holden Karau
Authored: Mon Jun 22 22:40:19 2015 -0700
Committer: Joseph K. Bradley
Committed: Mon Jun 22 22:40:19 2015 -0700
--
.../spark/mllib/api/python/PythonMLLibAPI.scala | 4 +++-
python/pyspark/mllib/tests.py | 7 +++
python/pyspark/mllib/tree.py| 22 +---
3 files changed, 24 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/164fe2aa/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
index 634d56d..f9a271f 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
@@ -696,12 +696,14 @@ private[python] class PythonMLLibAPI extends Serializable
{
lossStr: String,
numIterations: Int,
learningRate: Double,
- maxDepth: Int): GradientBoostedTreesModel = {
+ maxDepth: Int,
+ maxBins: Int): GradientBoostedTreesModel = {
val boostingStrategy = BoostingStrategy.defaultParams(algoStr)
boostingStrategy.setLoss(Losses.fromString(lossStr))
boostingStrategy.setNumIterations(numIterations)
boostingStrategy.setLearningRate(learningRate)
boostingStrategy.treeStrategy.setMaxDepth(maxDepth)
+boostingStrategy.treeStrategy.setMaxBins(maxBins)
boostingStrategy.treeStrategy.categoricalFeaturesInfo =
categoricalFeaturesInfo.asScala.toMap
val cached = data.rdd.persist(StorageLevel.MEMORY_AND_DISK)
http://git-wip-us.apache.org/repos/asf/spark/blob/164fe2aa/python/pyspark/mllib/tests.py
--
diff --git a/python/pyspark/mllib/tests.py b/python/pyspark/mllib/tests.py
index b13159e..c8d61b9 100644
--- a/python/pyspark/mllib/tests.py
+++ b/python/pyspark/mllib/tests.py
@@ -463,6 +463,13 @@ class ListTests(MLlibTestCase):
except ValueError:
self.fail()
+# Verify that maxBins is being passed through
+GradientBoostedTrees.trainRegressor(
+rdd, categoricalFeaturesInfo=categoricalFeaturesInfo,
numIterations=4, maxBins=32)
+with self.assertRaises(Exception) as cm:
+GradientBoostedTrees.trainRegressor(
+rdd, categoricalFeaturesInfo=categoricalFeaturesInfo,
numIterations=4, maxBins=1)
+
class StatTests(MLlibTestCase):
# SPARK-4023
http://git-wip-us.apache.org/repos/asf/spark/blob/164fe2aa/python/pyspark/mllib/tree.py
--
diff --git a/python/pyspark/mllib/tree.py b/python/pyspark/mllib/tree.py
index cfcbea5..372b86a 100644
--- a/python/pyspark/mllib/tree.py
+++ b/python/pyspark/mllib/tree.py
@@ -299,7 +299,7 @@ class RandomForest(object):
1 internal node + 2 leaf nodes. (default: 4)
:param maxBins: maximum number of bins used for splitting
features
- (default: 100)
+ (default: 32)
:param seed: Random seed for bootstrapping and choosing feature
subsets.
:return: RandomForestModel that can be used for prediction
@@ -377,7 +377,7 @@ class RandomForest(object):
1 leaf node; depth 1 means 1 internal node + 2 leaf
nodes. (default: 4)
:param maxBins: maximum number of bins used for splitting
- features (default: 100)
+ features (default: 32)
:param seed: Random seed for bootstrapping and choosing feature
spark git commit: [SPARK-7781] [MLLIB] gradient boosted trees.train regressor missing max bins
Repository: spark
Updated Branches:
refs/heads/branch-1.4 1cfa7302e -> 22cc1ab66
[SPARK-7781] [MLLIB] gradient boosted trees.train regressor missing max bins
Author: Holden Karau
Closes #6331 from
holdenk/SPARK-7781-GradientBoostedTrees.trainRegressor-missing-max-bins and
squashes the following commits:
2894695 [Holden Karau] remove extra blank line
2573e8d [Holden Karau] Update the scala side of the pythonmllibapi and make the
test a bit nicer too
3a09170 [Holden Karau] add maxBins to to the train method as well
af7f274 [Holden Karau] Add maxBins to GradientBoostedTrees.trainRegressor and
correctly mention the default of 32 in other places where it mentioned 100
(cherry picked from commit 164fe2aa44993da6c77af6de5efdae47a8b3958c)
Signed-off-by: Joseph K. Bradley
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/22cc1ab6
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/22cc1ab6
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/22cc1ab6
Branch: refs/heads/branch-1.4
Commit: 22cc1ab66ec26aeac9dacaeb176f94d3bdaccbf4
Parents: 1cfa730
Author: Holden Karau
Authored: Mon Jun 22 22:40:19 2015 -0700
Committer: Joseph K. Bradley
Committed: Mon Jun 22 22:40:31 2015 -0700
--
.../spark/mllib/api/python/PythonMLLibAPI.scala | 4 +++-
python/pyspark/mllib/tests.py | 7 +++
python/pyspark/mllib/tree.py| 22 +---
3 files changed, 24 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/22cc1ab6/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
index 16f3131..d1b2c98 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
@@ -685,12 +685,14 @@ private[python] class PythonMLLibAPI extends Serializable
{
lossStr: String,
numIterations: Int,
learningRate: Double,
- maxDepth: Int): GradientBoostedTreesModel = {
+ maxDepth: Int,
+ maxBins: Int): GradientBoostedTreesModel = {
val boostingStrategy = BoostingStrategy.defaultParams(algoStr)
boostingStrategy.setLoss(Losses.fromString(lossStr))
boostingStrategy.setNumIterations(numIterations)
boostingStrategy.setLearningRate(learningRate)
boostingStrategy.treeStrategy.setMaxDepth(maxDepth)
+boostingStrategy.treeStrategy.setMaxBins(maxBins)
boostingStrategy.treeStrategy.categoricalFeaturesInfo =
categoricalFeaturesInfo.asScala.toMap
val cached = data.rdd.persist(StorageLevel.MEMORY_AND_DISK)
http://git-wip-us.apache.org/repos/asf/spark/blob/22cc1ab6/python/pyspark/mllib/tests.py
--
diff --git a/python/pyspark/mllib/tests.py b/python/pyspark/mllib/tests.py
index 7a113f8..4335143 100644
--- a/python/pyspark/mllib/tests.py
+++ b/python/pyspark/mllib/tests.py
@@ -444,6 +444,13 @@ class ListTests(MLlibTestCase):
except ValueError:
self.fail()
+# Verify that maxBins is being passed through
+GradientBoostedTrees.trainRegressor(
+rdd, categoricalFeaturesInfo=categoricalFeaturesInfo,
numIterations=4, maxBins=32)
+with self.assertRaises(Exception) as cm:
+GradientBoostedTrees.trainRegressor(
+rdd, categoricalFeaturesInfo=categoricalFeaturesInfo,
numIterations=4, maxBins=1)
+
class StatTests(MLlibTestCase):
# SPARK-4023
http://git-wip-us.apache.org/repos/asf/spark/blob/22cc1ab6/python/pyspark/mllib/tree.py
--
diff --git a/python/pyspark/mllib/tree.py b/python/pyspark/mllib/tree.py
index cfcbea5..372b86a 100644
--- a/python/pyspark/mllib/tree.py
+++ b/python/pyspark/mllib/tree.py
@@ -299,7 +299,7 @@ class RandomForest(object):
1 internal node + 2 leaf nodes. (default: 4)
:param maxBins: maximum number of bins used for splitting
features
- (default: 100)
+ (default: 32)
:param seed: Random seed for bootstrapping and choosing feature
subsets.
:return: RandomForestModel that can be used for prediction
@@ -377,7 +377,7 @@ class RandomForest(object):
1 leaf node; depth 1 means 1 internal node + 2 leaf
nodes. (default: 4)
:param maxBins: maximum number of bins used for splitting
- features (default: 100)
+ fe
spark git commit: [SPARK-8431] [SPARKR] Add in operator to DataFrame Column in SparkR
Repository: spark
Updated Branches:
refs/heads/master 164fe2aa4 -> d4f633514
[SPARK-8431] [SPARKR] Add in operator to DataFrame Column in SparkR
[[SPARK-8431] Add in operator to DataFrame Column in SparkR - ASF
JIRA](https://issues.apache.org/jira/browse/SPARK-8431)
Author: Yu ISHIKAWA
Closes #6941 from yu-iskw/SPARK-8431 and squashes the following commits:
1f64423 [Yu ISHIKAWA] Modify the comment
f4309a7 [Yu ISHIKAWA] Make a `setMethod` for `%in%` be independent
6e37936 [Yu ISHIKAWA] Modify a variable name
c196173 [Yu ISHIKAWA] [SPARK-8431][SparkR] Add in operator to DataFrame Column
in SparkR
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d4f63351
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d4f63351
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d4f63351
Branch: refs/heads/master
Commit: d4f633514a393320c9ae64c00a75f702e6f58c67
Parents: 164fe2a
Author: Yu ISHIKAWA
Authored: Mon Jun 22 23:04:36 2015 -0700
Committer: Davies Liu
Committed: Mon Jun 22 23:04:36 2015 -0700
--
R/pkg/R/column.R | 16
R/pkg/inst/tests/test_sparkSQL.R | 10 ++
2 files changed, 26 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/d4f63351/R/pkg/R/column.R
--
diff --git a/R/pkg/R/column.R b/R/pkg/R/column.R
index 80e92d3..8e4b0f5 100644
--- a/R/pkg/R/column.R
+++ b/R/pkg/R/column.R
@@ -210,6 +210,22 @@ setMethod("cast",
}
})
+#' Match a column with given values.
+#'
+#' @rdname column
+#' @return a matched values as a result of comparing with given values.
+#' \dontrun{
+#' filter(df, "age in (10, 30)")
+#' where(df, df$age %in% c(10, 30))
+#' }
+setMethod("%in%",
+ signature(x = "Column"),
+ function(x, table) {
+table <- listToSeq(as.list(table))
+jc <- callJMethod(x@jc, "in", table)
+return(column(jc))
+ })
+
#' Approx Count Distinct
#'
#' @rdname column
http://git-wip-us.apache.org/repos/asf/spark/blob/d4f63351/R/pkg/inst/tests/test_sparkSQL.R
--
diff --git a/R/pkg/inst/tests/test_sparkSQL.R b/R/pkg/inst/tests/test_sparkSQL.R
index fc7f3f0..417153d 100644
--- a/R/pkg/inst/tests/test_sparkSQL.R
+++ b/R/pkg/inst/tests/test_sparkSQL.R
@@ -693,6 +693,16 @@ test_that("filter() on a DataFrame", {
filtered2 <- where(df, df$name != "Michael")
expect_true(count(filtered2) == 2)
expect_true(collect(filtered2)$age[2] == 19)
+
+ # test suites for %in%
+ filtered3 <- filter(df, "age in (19)")
+ expect_equal(count(filtered3), 1)
+ filtered4 <- filter(df, "age in (19, 30)")
+ expect_equal(count(filtered4), 2)
+ filtered5 <- where(df, df$age %in% c(19))
+ expect_equal(count(filtered5), 1)
+ filtered6 <- where(df, df$age %in% c(19, 30))
+ expect_equal(count(filtered6), 2)
})
test_that("join() on a DataFrame", {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8359] [SQL] Fix incorrect decimal precision after multiplication
Repository: spark
Updated Branches:
refs/heads/master d4f633514 -> 31bd30687
[SPARK-8359] [SQL] Fix incorrect decimal precision after multiplication
JIRA: https://issues.apache.org/jira/browse/SPARK-8359
Author: Liang-Chi Hsieh
Closes #6814 from viirya/fix_decimal2 and squashes the following commits:
071a757 [Liang-Chi Hsieh] Remove maximum precision and use
MathContext.UNLIMITED.
df217d4 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into
fix_decimal2
a43bfc3 [Liang-Chi Hsieh] Add MathContext with maximum supported precision.
72eeb3f [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into
fix_decimal2
44c9348 [Liang-Chi Hsieh] Fix incorrect decimal precision after multiplication.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/31bd3068
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/31bd3068
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/31bd3068
Branch: refs/heads/master
Commit: 31bd30687bc29c0e457c37308d489ae2b6e5b72a
Parents: d4f6335
Author: Liang-Chi Hsieh
Authored: Mon Jun 22 23:11:56 2015 -0700
Committer: Davies Liu
Committed: Mon Jun 22 23:11:56 2015 -0700
--
.../src/main/scala/org/apache/spark/sql/types/Decimal.scala| 6 --
.../org/apache/spark/sql/types/decimal/DecimalSuite.scala | 5 +
2 files changed, 9 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/31bd3068/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
index a85af9e..bd9823b 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
@@ -17,6 +17,8 @@
package org.apache.spark.sql.types
+import java.math.{MathContext, RoundingMode}
+
import org.apache.spark.annotation.DeveloperApi
/**
@@ -137,9 +139,9 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def toBigDecimal: BigDecimal = {
if (decimalVal.ne(null)) {
- decimalVal
+ decimalVal(MathContext.UNLIMITED)
} else {
- BigDecimal(longVal, _scale)
+ BigDecimal(longVal, _scale)(MathContext.UNLIMITED)
}
}
http://git-wip-us.apache.org/repos/asf/spark/blob/31bd3068/sql/catalyst/src/test/scala/org/apache/spark/sql/types/decimal/DecimalSuite.scala
--
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/decimal/DecimalSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/decimal/DecimalSuite.scala
index 4c0365c..ccc29c0 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/decimal/DecimalSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/decimal/DecimalSuite.scala
@@ -162,4 +162,9 @@ class DecimalSuite extends SparkFunSuite with
PrivateMethodTester {
assert(new Decimal().set(100L, 10, 0).toUnscaledLong === 100L)
assert(Decimal(Long.MaxValue, 100, 0).toUnscaledLong === Long.MaxValue)
}
+
+ test("accurate precision after multiplication") {
+val decimal = (Decimal(Long.MaxValue, 38, 0) * Decimal(Long.MaxValue, 38,
0)).toJavaBigDecimal
+assert(decimal.unscaledValue.toString ===
"85070591730234615847396907784232501249")
+ }
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8483] [STREAMING] Remove commons-lang3 dependency from Flume Si…
Repository: spark
Updated Branches:
refs/heads/master 31bd30687 -> 9b618fb0d
[SPARK-8483] [STREAMING] Remove commons-lang3 dependency from Flume Siâ¦
â¦nk. Also bump Flume version to 1.6.0
Author: Hari Shreedharan
Closes #6910 from harishreedharan/remove-commons-lang3 and squashes the
following commits:
9875f7d [Hari Shreedharan] Revert back to Flume 1.4.0
ca35eb0 [Hari Shreedharan] [SPARK-8483][Streaming] Remove commons-lang3
dependency from Flume Sink. Also bump Flume version to 1.6.0
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9b618fb0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9b618fb0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9b618fb0
Branch: refs/heads/master
Commit: 9b618fb0d2536121d2784ff5341d74723e810fc5
Parents: 31bd306
Author: Hari Shreedharan
Authored: Mon Jun 22 23:34:17 2015 -0700
Committer: Tathagata Das
Committed: Mon Jun 22 23:34:17 2015 -0700
--
external/flume-sink/pom.xml | 4
.../spark/streaming/flume/sink/SparkAvroCallbackHandler.scala| 4 ++--
2 files changed, 2 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/9b618fb0/external/flume-sink/pom.xml
--
diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml
index 7a7dccc..0664cfb 100644
--- a/external/flume-sink/pom.xml
+++ b/external/flume-sink/pom.xml
@@ -36,10 +36,6 @@
- org.apache.commons
- commons-lang3
-
-
org.apache.flume
flume-ng-sdk
http://git-wip-us.apache.org/repos/asf/spark/blob/9b618fb0/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
--
diff --git
a/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
b/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
index dc2a4ab..719fca0 100644
---
a/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
+++
b/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
@@ -16,13 +16,13 @@
*/
package org.apache.spark.streaming.flume.sink
+import java.util.UUID
import java.util.concurrent.{CountDownLatch, Executors}
import java.util.concurrent.atomic.AtomicLong
import scala.collection.mutable
import org.apache.flume.Channel
-import org.apache.commons.lang3.RandomStringUtils
/**
* Class that implements the SparkFlumeProtocol, that is used by the Avro
Netty Server to process
@@ -53,7 +53,7 @@ private[flume] class SparkAvroCallbackHandler(val threads:
Int, val channel: Cha
// Since the new txn may not have the same sequence number we must guard
against accidentally
// committing a new transaction. To reduce the probability of that happening
a random string is
// prepended to the sequence number. Does not change for life of sink
- private val seqBase = RandomStringUtils.randomAlphanumeric(8)
+ private val seqBase = UUID.randomUUID().toString.substring(0, 8)
private val seqCounter = new AtomicLong(0)
// Protected by `sequenceNumberToProcessor`
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8541] [PYSPARK] test the absolute error in approx doctests
Repository: spark Updated Branches: refs/heads/master 9b618fb0d -> f0dcbe8a7 [SPARK-8541] [PYSPARK] test the absolute error in approx doctests A minor change but one which is (presumably) visible on the public api docs webpage. Author: Scott Taylor Closes #6942 from megatron-me-uk/patch-3 and squashes the following commits: fbed000 [Scott Taylor] test the absolute error in approx doctests Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f0dcbe8a Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f0dcbe8a Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f0dcbe8a Branch: refs/heads/master Commit: f0dcbe8a7c2de510b47a21eb45cde34777638758 Parents: 9b618fb Author: Scott Taylor Authored: Mon Jun 22 23:37:56 2015 -0700 Committer: Josh Rosen Committed: Mon Jun 22 23:37:56 2015 -0700 -- python/pyspark/rdd.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/f0dcbe8a/python/pyspark/rdd.py -- diff --git a/python/pyspark/rdd.py b/python/pyspark/rdd.py index 20c0bc9..1b64be2 100644 --- a/python/pyspark/rdd.py +++ b/python/pyspark/rdd.py @@ -2198,7 +2198,7 @@ class RDD(object): >>> rdd = sc.parallelize(range(1000), 10) >>> r = sum(range(1000)) ->>> (rdd.sumApprox(1000) - r) / r < 0.05 +>>> abs(rdd.sumApprox(1000) - r) / r < 0.05 True """ jrdd = self.mapPartitions(lambda it: [float(sum(it))])._to_java_object_rdd() @@ -2215,7 +2215,7 @@ class RDD(object): >>> rdd = sc.parallelize(range(1000), 10) >>> r = sum(range(1000)) / 1000.0 ->>> (rdd.meanApprox(1000) - r) / r < 0.05 +>>> abs(rdd.meanApprox(1000) - r) / r < 0.05 True """ jrdd = self.map(float)._to_java_object_rdd() - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8541] [PYSPARK] test the absolute error in approx doctests
Repository: spark Updated Branches: refs/heads/branch-1.4 22cc1ab66 -> d0943afbc [SPARK-8541] [PYSPARK] test the absolute error in approx doctests A minor change but one which is (presumably) visible on the public api docs webpage. Author: Scott Taylor Closes #6942 from megatron-me-uk/patch-3 and squashes the following commits: fbed000 [Scott Taylor] test the absolute error in approx doctests (cherry picked from commit f0dcbe8a7c2de510b47a21eb45cde34777638758) Signed-off-by: Josh Rosen Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d0943afb Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d0943afb Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d0943afb Branch: refs/heads/branch-1.4 Commit: d0943afbcffec5d8b668794dedc8d85fb10b0596 Parents: 22cc1ab Author: Scott Taylor Authored: Mon Jun 22 23:37:56 2015 -0700 Committer: Josh Rosen Committed: Mon Jun 22 23:38:21 2015 -0700 -- python/pyspark/rdd.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/d0943afb/python/pyspark/rdd.py -- diff --git a/python/pyspark/rdd.py b/python/pyspark/rdd.py index 20c0bc9..1b64be2 100644 --- a/python/pyspark/rdd.py +++ b/python/pyspark/rdd.py @@ -2198,7 +2198,7 @@ class RDD(object): >>> rdd = sc.parallelize(range(1000), 10) >>> r = sum(range(1000)) ->>> (rdd.sumApprox(1000) - r) / r < 0.05 +>>> abs(rdd.sumApprox(1000) - r) / r < 0.05 True """ jrdd = self.mapPartitions(lambda it: [float(sum(it))])._to_java_object_rdd() @@ -2215,7 +2215,7 @@ class RDD(object): >>> rdd = sc.parallelize(range(1000), 10) >>> r = sum(range(1000)) / 1000.0 ->>> (rdd.meanApprox(1000) - r) / r < 0.05 +>>> abs(rdd.meanApprox(1000) - r) / r < 0.05 True """ jrdd = self.map(float)._to_java_object_rdd() - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8541] [PYSPARK] test the absolute error in approx doctests
Repository: spark Updated Branches: refs/heads/branch-1.3 45b4527e3 -> 716dcf631 [SPARK-8541] [PYSPARK] test the absolute error in approx doctests A minor change but one which is (presumably) visible on the public api docs webpage. Author: Scott Taylor Closes #6942 from megatron-me-uk/patch-3 and squashes the following commits: fbed000 [Scott Taylor] test the absolute error in approx doctests (cherry picked from commit f0dcbe8a7c2de510b47a21eb45cde34777638758) Signed-off-by: Josh Rosen Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/716dcf63 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/716dcf63 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/716dcf63 Branch: refs/heads/branch-1.3 Commit: 716dcf631558920c080cb824dcd617789b9f96d5 Parents: 45b4527 Author: Scott Taylor Authored: Mon Jun 22 23:37:56 2015 -0700 Committer: Josh Rosen Committed: Mon Jun 22 23:39:39 2015 -0700 -- python/pyspark/rdd.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/716dcf63/python/pyspark/rdd.py -- diff --git a/python/pyspark/rdd.py b/python/pyspark/rdd.py index d80366a..bd18cb3 100644 --- a/python/pyspark/rdd.py +++ b/python/pyspark/rdd.py @@ -2130,7 +2130,7 @@ class RDD(object): >>> rdd = sc.parallelize(range(1000), 10) >>> r = sum(xrange(1000)) ->>> (rdd.sumApprox(1000) - r) / r < 0.05 +>>> abs(rdd.sumApprox(1000) - r) / r < 0.05 True """ jrdd = self.mapPartitions(lambda it: [float(sum(it))])._to_java_object_rdd() @@ -2147,7 +2147,7 @@ class RDD(object): >>> rdd = sc.parallelize(range(1000), 10) >>> r = sum(xrange(1000)) / 1000.0 ->>> (rdd.meanApprox(1000) - r) / r < 0.05 +>>> abs(rdd.meanApprox(1000) - r) / r < 0.05 True """ jrdd = self.map(float)._to_java_object_rdd() - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-8483] [STREAMING] Remove commons-lang3 dependency from Flume Sink
Repository: spark
Updated Branches:
refs/heads/branch-1.4 d0943afbc -> 929479675
[SPARK-8483] [STREAMING] Remove commons-lang3 dependency from Flume Sink
Author: Hari Shreedharan
Closes #6910 from harishreedharan/remove-commons-lang3 and squashes the
following commits:
9875f7d [Hari Shreedharan] Revert back to Flume 1.4.0
ca35eb0 [Hari Shreedharan] [SPARK-8483][Streaming] Remove commons-lang3
dependency from Flume Sink. Also bump Flume version to 1.6.0
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/92947967
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/92947967
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/92947967
Branch: refs/heads/branch-1.4
Commit: 9294796750f9c9330ab113f025763e68b624abc9
Parents: d0943af
Author: Hari Shreedharan
Authored: Mon Jun 22 23:34:17 2015 -0700
Committer: Tathagata Das
Committed: Mon Jun 22 23:41:35 2015 -0700
--
external/flume-sink/pom.xml | 4
.../spark/streaming/flume/sink/SparkAvroCallbackHandler.scala| 4 ++--
2 files changed, 2 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/92947967/external/flume-sink/pom.xml
--
diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml
index 37f2b1b..ad431fa 100644
--- a/external/flume-sink/pom.xml
+++ b/external/flume-sink/pom.xml
@@ -36,10 +36,6 @@
- org.apache.commons
- commons-lang3
-
-
org.apache.flume
flume-ng-sdk
http://git-wip-us.apache.org/repos/asf/spark/blob/92947967/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
--
diff --git
a/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
b/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
index dc2a4ab..719fca0 100644
---
a/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
+++
b/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala
@@ -16,13 +16,13 @@
*/
package org.apache.spark.streaming.flume.sink
+import java.util.UUID
import java.util.concurrent.{CountDownLatch, Executors}
import java.util.concurrent.atomic.AtomicLong
import scala.collection.mutable
import org.apache.flume.Channel
-import org.apache.commons.lang3.RandomStringUtils
/**
* Class that implements the SparkFlumeProtocol, that is used by the Avro
Netty Server to process
@@ -53,7 +53,7 @@ private[flume] class SparkAvroCallbackHandler(val threads:
Int, val channel: Cha
// Since the new txn may not have the same sequence number we must guard
against accidentally
// committing a new transaction. To reduce the probability of that happening
a random string is
// prepended to the sequence number. Does not change for life of sink
- private val seqBase = RandomStringUtils.randomAlphanumeric(8)
+ private val seqBase = UUID.randomUUID().toString.substring(0, 8)
private val seqCounter = new AtomicLong(0)
// Protected by `sequenceNumberToProcessor`
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
