[GitHub] spark issue #13758: [SPARK-16043][SQL] Prepare GenericArrayData implementati...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13758 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60768/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13746 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13746 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60760/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13758: [SPARK-16043][SQL] Prepare GenericArrayData implementati...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13758 **[Test build #60768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60768/consoleFull)** for PR 13758 at commit [`17bdfcf`](https://github.com/apache/spark/commit/17bdfcf7b9c155fe05eb75f73d661c2863cdf2d9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13758: [SPARK-16043][SQL] Prepare GenericArrayData imple...
GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/13758 [SPARK-16043][SQL] Prepare GenericArrayData implementation specialized for a primitive array ## What changes were proposed in this pull request? This PR addresses a ToDo of ```GenericArrayData``` class. Current implementation of ```GenericArrayData``` leads to boxing/unboxing if type of array elements are primitive. It would be good to eliminate boxing/unboxing from the view of runtime memory footprint and performance. This PR eliminattes boxing/unboxing by preparing sub classes of ```GenericArrayData``` to specialize operations in these classes. This PR prepare a new method ```GenericArrayData.allocate(...) ``` that can return generic ```GenericArrayData``` instance or a ```GenericArrayData``` instance specialized for `` type array. Here are major improvements: 1. Hold an array in a primitive array (previously ```Object[]``` is used and boxing happened in a constructor) 2. a method "get``()" gets a value from an primitive array (previously unboxing happened) 3. a method "to``Array" performs data copy using ```System.arraycopy``` (previously unboxing happened) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) add unit tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/kiszk/spark SPARK-16043 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13758.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13758 commit 17bdfcf7b9c155fe05eb75f73d661c2863cdf2d9 Author: Kazuaki Ishizaki Date: 2016-06-18T06:47:11Z Implementation of GenericArrayData specialized for primitive type array add unit tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13746 **[Test build #60760 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60760/consoleFull)** for PR 13746 at commit [`b1adc7a`](https://github.com/apache/spark/commit/b1adc7aa8a217d18ac0ab6eb89306e23d82c8f86). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13755: [SPARK-16040][MLlib][DOC]:spark.mllib PIC document extra...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/13755 OK --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13730: [SPARK-16006][SQL] Attemping to write empty DataFrame wi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13730 Hi, @rxin . Could you review this PR ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13724: [SPARK-15973] [PYSPARK] Fix GroupedData Documenta...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13724 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13744: [SPARK-16025] [CORE] Document OFF_HEAP storage level in ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/13744 Looks OK to me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13724: [SPARK-15973] [PYSPARK] Fix GroupedData Documentation
Github user rxin commented on the issue: https://github.com/apache/spark/pull/13724 @josh-howes what's your jira id? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13724: [SPARK-15973] [PYSPARK] Fix GroupedData Documentation
Github user rxin commented on the issue: https://github.com/apache/spark/pull/13724 Thanks - merging in master/2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13742: [SPARK-16023][SQL] Move InMemoryRelation to its o...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13742 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13742: [SPARK-16023][SQL] Move InMemoryRelation to its own file
Github user rxin commented on the issue: https://github.com/apache/spark/pull/13742 Merging in master/2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13746: [SPARK-16030] [SQL] Allow specifying static parti...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13746#discussion_r67596845
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -43,11 +44,116 @@ import org.apache.spark.unsafe.types.UTF8String
* Replaces generic operations with specific variants that are designed to
work with Spark
* SQL Data Sources.
*/
-private[sql] object DataSourceAnalysis extends Rule[LogicalPlan] {
+private[sql] case class DataSourceAnalysis(conf: CatalystConf) extends
Rule[LogicalPlan] {
+
+ def resolver: Resolver = {
+if (conf.caseSensitiveAnalysis) {
+ caseSensitiveResolution
+} else {
+ caseInsensitiveResolution
+}
+ }
+
override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+
+// If the InsertIntoTable command is for a partitioned
HadoopFsRelation and
+// the user has specified static partitions, we add a Project operator
on top of the query
+// to include those constant column values in the query result.
+//
+// Example:
+// Let's say that we have a table "t", which is created by
+// CREATE TABLE t (a INT, b INT, c INT) USING parquet PARTITIONED BY
(b, c)
+// The statement of "INSERT INTO TABLE t PARTITION (b=2, c) SELECT 1,
3"
+// will be converted to "INSERT INTO TABLE t PARTITION (b, c) SELECT
1, 2, 3".
+//
+// Basically, we will put those partition columns having a assigned
value back
+// to the SELECT clause. The output of the SELECT clause is organized
as
+// normal_columns static_partitioning_columns
dynamic_partitioning_columns.
+// static_partitioning_columns are partitioning columns having assigned
+// values in the PARTITION clause (e.g. b in the above example).
+// dynamic_partitioning_columns are partitioning columns that do not
assigned
+// values in the PARTITION clause (e.g. c in the above example).
+case i @ logical.InsertIntoTable(
+l @ LogicalRelation(t: HadoopFsRelation, _, _), parts, query,
overwrite, false)
--- End diff --
and don't use t or i here either ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13746: [SPARK-16030] [SQL] Allow specifying static parti...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13746#discussion_r67596839
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -43,11 +44,116 @@ import org.apache.spark.unsafe.types.UTF8String
* Replaces generic operations with specific variants that are designed to
work with Spark
* SQL Data Sources.
*/
-private[sql] object DataSourceAnalysis extends Rule[LogicalPlan] {
+private[sql] case class DataSourceAnalysis(conf: CatalystConf) extends
Rule[LogicalPlan] {
+
+ def resolver: Resolver = {
+if (conf.caseSensitiveAnalysis) {
+ caseSensitiveResolution
+} else {
+ caseInsensitiveResolution
+}
+ }
+
override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+
+// If the InsertIntoTable command is for a partitioned
HadoopFsRelation and
+// the user has specified static partitions, we add a Project operator
on top of the query
+// to include those constant column values in the query result.
+//
+// Example:
+// Let's say that we have a table "t", which is created by
+// CREATE TABLE t (a INT, b INT, c INT) USING parquet PARTITIONED BY
(b, c)
+// The statement of "INSERT INTO TABLE t PARTITION (b=2, c) SELECT 1,
3"
+// will be converted to "INSERT INTO TABLE t PARTITION (b, c) SELECT
1, 2, 3".
+//
+// Basically, we will put those partition columns having a assigned
value back
+// to the SELECT clause. The output of the SELECT clause is organized
as
+// normal_columns static_partitioning_columns
dynamic_partitioning_columns.
+// static_partitioning_columns are partitioning columns having assigned
+// values in the PARTITION clause (e.g. b in the above example).
+// dynamic_partitioning_columns are partitioning columns that do not
assigned
+// values in the PARTITION clause (e.g. c in the above example).
+case i @ logical.InsertIntoTable(
+l @ LogicalRelation(t: HadoopFsRelation, _, _), parts, query,
overwrite, false)
--- End diff --
l is a bad name to use
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/13746 Actually can you try do it in this? I worry similar to past pull requests we will never revisit and the code will just look like this 2 years from now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/13746 This is ok for 2.0, but for something like this we should write it in a way so the rewriting logic can be unit tested without end-to-end suites. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13749: [SPARK-16034][SQL] Checks the partition columns w...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13749#discussion_r67596783
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -435,26 +435,25 @@ case class DataSource(
// If we are appending to a table that already exists, make sure
the partitioning matches
// up. If we fail to load the table for whatever reason, ignore
the check.
if (mode == SaveMode.Append) {
- val existingPartitionColumnSet = try {
-Some(
- resolveRelation()
-.asInstanceOf[HadoopFsRelation]
-.location
-.partitionSpec()
-.partitionColumns
-.fieldNames
-.toSet)
- } catch {
-case e: Exception =>
- None
- }
-
- existingPartitionColumnSet.foreach { ex =>
-if (ex.map(_.toLowerCase) !=
partitionColumns.map(_.toLowerCase()).toSet) {
- throw new AnalysisException(
-s"Requested partitioning does not equal existing
partitioning: " +
-s"$ex != ${partitionColumns.toSet}.")
-}
+ val existingColumns = Try {
+resolveRelation()
+ .asInstanceOf[HadoopFsRelation]
+ .location
+ .partitionSpec()
+ .partitionColumns
+ .fieldNames
+ .toSeq
+ }.getOrElse(Seq.empty[String])
+ val sameColumns =
+existingColumns.map(_.toLowerCase) ==
partitionColumns.map(_.toLowerCase)
+ if (existingColumns.size > 0 && !sameColumns) {
+throw new AnalysisException(
+ s"""Requested partitioning does not match existing
partitioning.
--- End diff --
can you add "Requested partitioning does not match existing partitioning
for table $table" ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13749: [SPARK-16034][SQL] Checks the partition columns when cal...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13749 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60754/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13749: [SPARK-16034][SQL] Checks the partition columns when cal...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13749 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13749: [SPARK-16034][SQL] Checks the partition columns when cal...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13749 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60753/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13749: [SPARK-16034][SQL] Checks the partition columns when cal...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13749 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13749: [SPARK-16034][SQL] Checks the partition columns when cal...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13749 **[Test build #60754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60754/consoleFull)** for PR 13749 at commit [`7a4293b`](https://github.com/apache/spark/commit/7a4293bd68896701e4ca61a9df06133357bc0f8a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13757: [SPARK-16042][SQL] Eliminate nullcheck code at projectio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13757 **[Test build #60767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60767/consoleFull)** for PR 13757 at commit [`3875e1f`](https://github.com/apache/spark/commit/3875e1fe20f7032c3da007bcdccd9f2c627710db). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13749: [SPARK-16034][SQL] Checks the partition columns when cal...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13749 **[Test build #60753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60753/consoleFull)** for PR 13749 at commit [`5224802`](https://github.com/apache/spark/commit/5224802418a2917ae09f64d8bffce80a0c4d5201). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13757: [SPARK-16042][SQL] Eliminate nullcheck code at pr...
GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/13757
[SPARK-16042][SQL] Eliminate nullcheck code at projection for an array type
## What changes were proposed in this pull request?
This PR eliminates nullcheck code at projection for an array type. The
nullcheck a call to write each element of an array (lines 076-078 in "Before
applying this PR") is generated. If we know all of the elements do not have
``null`` at compilation time, we can eliminate code for nullcheck.
This PR checks whether ```ArrayType.containsNull``` is ```false``` to know
the all of array elements do not have ```null```.
An example program
```
val df = sparkContext.parallelize(Seq(1.0, 2.0), 1).toDF("v")
df.selectExpr("Array(v + 2.2, v + 3.3)").collect
```
Before applying this PR
```
/* 028 */ protected void processNext() throws java.io.IOException {
/* 029 */ while (inputadapter_input.hasNext()) {
/* 030 */ InternalRow inputadapter_row = (InternalRow)
inputadapter_input.next();
/* 031 */ double inputadapter_value = inputadapter_row.getDouble(0);
/* 032 */
/* 033 */ final boolean project_isNull = false;
/* 034 */ this.project_values = new Object[2];
/* 035 */ double project_value1 = -1.0;
/* 036 */ project_value1 = inputadapter_value + 2.2D;
/* 037 */ if (false) {
/* 038 */ project_values[0] = null;
/* 039 */ } else {
/* 040 */ project_values[0] = project_value1;
/* 041 */ }
/* 042 */
/* 043 */ double project_value4 = -1.0;
/* 044 */ project_value4 = inputadapter_value + 3.3D;
/* 045 */ if (false) {
/* 046 */ project_values[1] = null;
/* 047 */ } else {
/* 048 */ project_values[1] = project_value4;
/* 049 */ }
/* 050 */
/* 051 */ final ArrayData project_value = new
org.apache.spark.sql.catalyst.util.GenericArrayData(project_values);
/* 052 */ this.project_values = null;
/* 053 */ project_holder.reset();
/* 054 */
/* 055 */ project_rowWriter.zeroOutNullBytes();
/* 056 */
/* 057 */ if (project_isNull) {
/* 058 */ project_rowWriter.setNullAt(0);
/* 059 */ } else {
/* 060 */ // Remember the current cursor so that we can calculate
how many bytes are
/* 061 */ // written later.
/* 062 */ final int project_tmpCursor = project_holder.cursor;
/* 063 */
/* 064 */ if (project_value instanceof UnsafeArrayData) {
/* 065 */ final int project_sizeInBytes = ((UnsafeArrayData)
project_value).getSizeInBytes();
/* 066 */ // grow the global buffer before writing data.
/* 067 */ project_holder.grow(project_sizeInBytes);
/* 068 */ ((UnsafeArrayData)
project_value).writeToMemory(project_holder.buffer, project_holder.cursor);
/* 069 */ project_holder.cursor += project_sizeInBytes;
/* 070 */
/* 071 */ } else {
/* 072 */ final int project_numElements =
project_value.numElements();
/* 073 */ project_arrayWriter.initialize(project_holder,
project_numElements, 8);
/* 074 */
/* 075 */ for (int project_index = 0; project_index <
project_numElements; project_index++) {
/* 076 */ if (project_value.isNullAt(project_index)) {
/* 077 */ project_arrayWriter.setNullAt(project_index);
/* 078 */ } else {
/* 079 */ final double project_element =
project_value.getDouble(project_index);
/* 080 */ project_arrayWriter.write(project_index,
project_element);
/* 081 */ }
/* 082 */
/* 083 */ }
/* 084 */ }
/* 085 */
/* 086 */ project_rowWriter.setOffsetAndSize(0, project_tmpCursor,
project_holder.cursor - project_tmpCursor);
/* 087 */ project_rowWriter.alignToWords(project_holder.cursor -
project_tmpCursor);
/* 088 */ }
/* 089 */ project_result.setTotalSize(project_holder.totalSize());
/* 090 */ append(project_result);
/* 091 */ if (shouldStop()) return;
/* 092 */ }
```
After applying this PR
```
/* 028 */ protected void processNext() throws java.io.IOException {
/* 029 */ while (inputadapter_input.hasNext()) {
/* 030 */ InternalRow inputadapter_row = (InternalRow)
inputadapter_input.next();
/* 031 */ double inputadapter_value = inputadapter_row.getDouble(0);
/* 032 */
/* 033 */ final boolean project_isNull = false;
/* 034 */ this.project_values = new Object[2];
/* 035 */ double project_value1 = -1.0;
/* 036 */ project_value1 = inputadapter_value + 2.2D;
/* 037 */ if (f
[GitHub] spark issue #13734: [SPARK-14995][R] Add `since` tag in Roxygen documentatio...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13734 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60765/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13734: [SPARK-14995][R] Add `since` tag in Roxygen documentatio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13734 **[Test build #60765 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60765/consoleFull)** for PR 13734 at commit [`0092967`](https://github.com/apache/spark/commit/0092967df878bbaaf5ad2a007d56cc68146daea2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13734: [SPARK-14995][R] Add `since` tag in Roxygen documentatio...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13734 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13754: [SPARK-16036][SPARK-16037][SQL] fix various table insert...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13754 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13754: [SPARK-16036][SPARK-16037][SQL] fix various table insert...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13754 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60759/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13754: [SPARK-16036][SPARK-16037][SQL] fix various table insert...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13754 **[Test build #60759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60759/consoleFull)** for PR 13754 at commit [`52e67d4`](https://github.com/apache/spark/commit/52e67d48019c5b27cb5f59584a7de15e0a1869c9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13706: [SPARK-15988] [SQL] Implement DDL commands: Creat...
Github user lianhuiwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/13706#discussion_r67596616
--- Diff:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -97,6 +97,9 @@ statement
| CREATE TEMPORARY? FUNCTION qualifiedName AS className=STRING
(USING resource (',' resource)*)?
#createFunction
| DROP TEMPORARY? FUNCTION (IF EXISTS)? qualifiedName
#dropFunction
+| CREATE TEMPORARY MACRO macroName=identifier
--- End diff --
No, Now Hive only support temporary macro's.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12268: [SPARK-14480][SQL] Simplify CSV parsing process with a b...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12268 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12268: [SPARK-14480][SQL] Simplify CSV parsing process with a b...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12268 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60751/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #10896: [SPARK-12978][SQL] Skip unnecessary final group-by when ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/10896 **[Test build #60766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60766/consoleFull)** for PR 10896 at commit [`adad55d`](https://github.com/apache/spark/commit/adad55dbaf4adab313f970c82c2b7a45298c36eb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12268: [SPARK-14480][SQL] Simplify CSV parsing process with a b...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12268 **[Test build #60751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60751/consoleFull)** for PR 12268 at commit [`7abdfc1`](https://github.com/apache/spark/commit/7abdfc66f2bf275fc4318c0ffe8836dcbb70). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13541: [SPARK-15803][PYSPARK] Support with statement syn...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13541 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13541: [SPARK-15803][PYSPARK] Support with statement syntax for...
Github user davies commented on the issue: https://github.com/apache/spark/pull/13541 Merging this into master and 2.0, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13541: [SPARK-15803][PYSPARK] Support with statement syntax for...
Github user davies commented on the issue: https://github.com/apache/spark/pull/13541 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13734: [SPARK-14995][R] Add `since` tag in Roxygen documentatio...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13734 Hi, @shivaram and @felixcheung . Now the document is updated with master and shows merged notes correctly. I manually checked all the merged notes and used function signatures for overloaded functions. ``` Note sparkR.stop since 1.4.0 sparkR.session.stop since 2.0.0 ``` https://home.apache.org/~dongjoon/spark-2.0.0-docs/api/R/sparkR.session.stop.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13592: [SPARK-15863][SQL][DOC] Initial SQL programming guide up...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13592 @liancheng Is it worth adding two parameters `spark.sql.files.maxPartitionBytes` and `spark.sql.files.openCostInBytes` in `Other Configuration Options`? They are kinds of internal parameters though, it seems they are useful for the users that would like to control #partitions. https://issues.apache.org/jira/browse/SPARK-15894 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13752: [SPARK-16028][SPARKR] spark.lapply can work with active ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13752 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60763/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13752: [SPARK-16028][SPARKR] spark.lapply can work with active ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13752 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13752: [SPARK-16028][SPARKR] spark.lapply can work with active ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13752 **[Test build #60763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60763/consoleFull)** for PR 13752 at commit [`f5f401c`](https://github.com/apache/spark/commit/f5f401c42034396a45015cd558f62cbf83af1719). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13753: [SPARK-16029][SPARKR] SparkR add dropTempView and deprec...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13753 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60762/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13753: [SPARK-16029][SPARKR] SparkR add dropTempView and deprec...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13753 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13753: [SPARK-16029][SPARKR] SparkR add dropTempView and deprec...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13753 **[Test build #60762 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60762/consoleFull)** for PR 13753 at commit [`d70154c`](https://github.com/apache/spark/commit/d70154cee47ca9fe803e79d673396a833b057fa7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13734: [SPARK-14995][R] Add `since` tag in Roxygen documentatio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13734 **[Test build #60765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60765/consoleFull)** for PR 13734 at commit [`0092967`](https://github.com/apache/spark/commit/0092967df878bbaaf5ad2a007d56cc68146daea2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13750: [SPARK-16035][PYSPARK] Fix SparseVector parser as...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13750 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13750: [SPARK-16035][PYSPARK] Fix SparseVector parser assertion...
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/13750 Merged into master and branch-2.0. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13641: [SPARK-10258][DOC][ML] Add @Since annotations to ...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/13641#discussion_r67596152
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/ElementwiseProduct.scala ---
@@ -33,21 +33,26 @@ import org.apache.spark.sql.types.DataType
* multiplier.
*/
@Experimental
-class ElementwiseProduct(override val uid: String)
+@Since("1.4.0")
+class ElementwiseProduct @Since("1.4.0") (@Since("1.4.0") override val
uid: String)
extends UnaryTransformer[Vector, Vector, ElementwiseProduct] with
DefaultParamsWritable {
+ @Since("1.4.0")
def this() = this(Identifiable.randomUID("elemProd"))
/**
* the vector to multiply with input vectors
* @group param
*/
+ @Since("1.4.0")
val scalingVec: Param[Vector] = new Param(this, "scalingVec", "vector
for hadamard product")
/** @group setParam */
+ @Since("1.4.0")
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13641: [SPARK-10258][DOC][ML] Add @Since annotations to ...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/13641#discussion_r67596150
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/ElementwiseProduct.scala ---
@@ -33,21 +33,26 @@ import org.apache.spark.sql.types.DataType
* multiplier.
*/
@Experimental
-class ElementwiseProduct(override val uid: String)
+@Since("1.4.0")
+class ElementwiseProduct @Since("1.4.0") (@Since("1.4.0") override val
uid: String)
extends UnaryTransformer[Vector, Vector, ElementwiseProduct] with
DefaultParamsWritable {
+ @Since("1.4.0")
def this() = this(Identifiable.randomUID("elemProd"))
/**
* the vector to multiply with input vectors
* @group param
*/
+ @Since("1.4.0")
--- End diff --
This uses the new `Vector`. So ideally it is since `2.0.0`. I'm not sure
about whether we should change the class since version to 2.0 as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13755: [SPARK-16040][MLlib][DOC]:spark.mllib PIC document extra...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13755 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13755: [SPARK-16040][MLlib][DOC]:spark.mllib PIC document extra...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13755 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60761/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13755: [SPARK-16040][MLlib][DOC]:spark.mllib PIC document extra...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13755 **[Test build #60761 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60761/consoleFull)** for PR 13755 at commit [`4262ad3`](https://github.com/apache/spark/commit/4262ad30ea347f78d42990d0a39bb5db56648821). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13753: [SPARK-16029][SPARKR] SparkR add dropTempView and deprec...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13753 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13753: [SPARK-16029][SPARKR] SparkR add dropTempView and deprec...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13753 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60757/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13753: [SPARK-16029][SPARKR] SparkR add dropTempView and deprec...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13753 **[Test build #60757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60757/consoleFull)** for PR 13753 at commit [`0b2fe6f`](https://github.com/apache/spark/commit/0b2fe6fbb80599a00ab32b00d0ea256739682519). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13751: [SPARK-15159][SPARKR] SparkSession roxygen2 doc, program...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13751 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60756/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13751: [SPARK-15159][SPARKR] SparkSession roxygen2 doc, program...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13751 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13751: [SPARK-15159][SPARKR] SparkSession roxygen2 doc, program...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13751 **[Test build #60756 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60756/consoleFull)** for PR 13751 at commit [`c434614`](https://github.com/apache/spark/commit/c434614321a3650a18b228f6333ffe245b6ea7ee). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13752: [SPARK-16028][SPARKR] spark.lapply can work with active ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13752 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60755/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13752: [SPARK-16028][SPARKR] spark.lapply can work with active ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13752 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13752: [SPARK-16028][SPARKR] spark.lapply can work with active ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13752 **[Test build #60755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60755/consoleFull)** for PR 13752 at commit [`179fbdc`](https://github.com/apache/spark/commit/179fbdc7f93a929e00aaeeb575eb404038385252). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13745: [Spark 15997][DOC][ML] Update user guide for HashingTF, ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13745 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60758/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13745: [Spark 15997][DOC][ML] Update user guide for HashingTF, ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13745 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13756: [SPARK-16041][SQL] Disallow Duplicate Columns in partiti...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13756 **[Test build #60764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60764/consoleFull)** for PR 13756 at commit [`83082ff`](https://github.com/apache/spark/commit/83082ffb9dae922374b2d2f6051cce4bf24cb1de). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13745: [Spark 15997][DOC][ML] Update user guide for HashingTF, ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13745 **[Test build #60758 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60758/consoleFull)** for PR 13745 at commit [`3b01f11`](https://github.com/apache/spark/commit/3b01f111a417b398c659637b7b977df5c387011d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13756: [SPARK-16041][SQL] Disallow Duplicate Columns in ...
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/13756 [SPARK-16041][SQL] Disallow Duplicate Columns in partitionBy, blockBy and sortBy in DataFrameWriter What changes were proposed in this pull request? Duplicate columns are not allowed in `partitionBy`, `blockBy`, `sortBy` in `DataFrameWriter`. The duplicate columns could cause unpredictable results. For example, the resolution failure. This PR is to detect the duplicates and issue exceptions with appropriate messages. How was this patch tested? Added test cases in `DataFrameReaderWriterSuite` You can merge this pull request into a Git repository by running: $ git pull https://github.com/gatorsmile/spark dedup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13756.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13756 commit 83082ffb9dae922374b2d2f6051cce4bf24cb1de Author: gatorsmile Date: 2016-06-18T05:13:36Z fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13745: [Spark 15997][DOC][ML] Update user guide for HashingTF, ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13745 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13745: [Spark 15997][DOC][ML] Update user guide for HashingTF, ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13745 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60752/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13745: [Spark 15997][DOC][ML] Update user guide for HashingTF, ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13745 **[Test build #60752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60752/consoleFull)** for PR 13745 at commit [`01e4a08`](https://github.com/apache/spark/commit/01e4a08b3a6154a0e04391fa8299821894109bb3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13752: [SPARK-16028][SPARKR] spark.lapply can work with active ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13752 **[Test build #60763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60763/consoleFull)** for PR 13752 at commit [`f5f401c`](https://github.com/apache/spark/commit/f5f401c42034396a45015cd558f62cbf83af1719). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13753: [SPARK-16029][SPARKR] SparkR add dropTempView and deprec...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13753 **[Test build #60762 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60762/consoleFull)** for PR 13753 at commit [`d70154c`](https://github.com/apache/spark/commit/d70154cee47ca9fe803e79d673396a833b057fa7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13755: [SPARK-16040][MLlib][DOC]:spark.mllib PIC document extra...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13755 **[Test build #60761 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60761/consoleFull)** for PR 13755 at commit [`4262ad3`](https://github.com/apache/spark/commit/4262ad30ea347f78d42990d0a39bb5db56648821). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13755: [SPARK-16040][MLlib][DOC]:spark.mllib PIC documen...
GitHub user wangmiao1981 opened a pull request: https://github.com/apache/spark/pull/13755 [SPARK-16040][MLlib][DOC]:spark.mllib PIC document extra line of refernece ## What changes were proposed in this pull request? In the 2.0 document, Line "A full example that produces the experiment described in the PIC paper can be found under examples/." is redundant. There is already "Find full example code at "examples/src/main/scala/org/apache/spark/examples/mllib/PowerIterationClusteringExample.scala" in the Spark repo.". We should remove the first line, which is consistent with other documents. ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) Manual test You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangmiao1981/spark doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13755.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13755 commit 4262ad30ea347f78d42990d0a39bb5db56648821 Author: wm...@hotmail.com Date: 2016-06-18T05:04:20Z remove extra line --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13753: [SPARK-16029][SPARKR] SparkR add dropTempView and deprec...
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/13753 LGTM pending Jenkins. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13635: [SPARK-15159][SPARKR] SparkR SparkSession API
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/13635 Oh, great!! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13718: [SPARK-16002][SQL]Sleep when no new data arrives to avoi...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/13718 @tdas I fixed the test. Could you take another look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13746: [SPARK-16030] [SQL] Allow specifying static partitions w...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13746 **[Test build #60760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60760/consoleFull)** for PR 13746 at commit [`b1adc7a`](https://github.com/apache/spark/commit/b1adc7aa8a217d18ac0ab6eb89306e23d82c8f86). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13754: [SPARK-16036][SPARK-16037][SQL] fix various table insert...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13754 **[Test build #60759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60759/consoleFull)** for PR 13754 at commit [`52e67d4`](https://github.com/apache/spark/commit/52e67d48019c5b27cb5f59584a7de15e0a1869c9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13740: [SPARK-16020][SQL]Fix complete mode aggregation w...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13740 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13754: [SPARK-16036][SPARK-16037][SQL] fix various table...
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/13754 [SPARK-16036][SPARK-16037][SQL] fix various table insertion semantics ## What changes were proposed in this pull request? The current table insertion has some weird behaviours: 1. inserting into a partitioned table with mismatch columns has confusing error message for hive table, and wrong result for datasource table 2. inserting into a partitioned table without partition list has wrong result for hive table. This PR fixes these 2 problems. ## How was this patch tested? new test in hive `SQLQuerySuite` You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloud-fan/spark insert2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13754.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13754 commit 52e67d48019c5b27cb5f59584a7de15e0a1869c9 Author: Wenchen Fan Date: 2016-06-17T21:37:31Z fix table insertion semantics --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13754: [SPARK-16036][SPARK-16037][SQL] fix various table insert...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13754 cc @yhuai @marmbrus @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13740: [SPARK-16020][SQL]Fix complete mode aggregation with con...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/13740 The added test passed. I'm going to merge it into master and 2.0 to make sure it's in rc1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13753: [SPARK-16029][SPARKR] SparkR add dropTempView and deprec...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13753 **[Test build #60757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60757/consoleFull)** for PR 13753 at commit [`0b2fe6f`](https://github.com/apache/spark/commit/0b2fe6fbb80599a00ab32b00d0ea256739682519). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13745: [Spark 15997][DOC][ML] Update user guide for HashingTF, ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13745 **[Test build #60758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60758/consoleFull)** for PR 13745 at commit [`3b01f11`](https://github.com/apache/spark/commit/3b01f111a417b398c659637b7b977df5c387011d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13752: [SPARK-16028][SPARKR] spark.lapply can work with active ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13752 **[Test build #60755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60755/consoleFull)** for PR 13752 at commit [`179fbdc`](https://github.com/apache/spark/commit/179fbdc7f93a929e00aaeeb575eb404038385252). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13751: [SPARK-15159][SPARKR] SparkSession roxygen2 doc, program...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13751 **[Test build #60756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60756/consoleFull)** for PR 13751 at commit [`c434614`](https://github.com/apache/spark/commit/c434614321a3650a18b228f6333ffe245b6ea7ee). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13753: [SPARK-16029][SPARKR] SparkR add dropTempView and...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/13753 [SPARK-16029][SPARKR] SparkR add dropTempView and deprecate dropTempTable ## What changes were proposed in this pull request? Add dropTempView and deprecate dropTempTable ## How was this patch tested? unit tests @shivaram @liancheng You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rdroptempview Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13753.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13753 commit 0b2fe6fbb80599a00ab32b00d0ea256739682519 Author: Felix Cheung Date: 2016-06-18T04:50:53Z Add dropTempView and deprecate dropTempTable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13752: [SPARK-16028][SPARKR] spark.lapply can work with active ...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/13752 example update in the other PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13752: [SPARK-16028][SPARKR] spark.lapply can work with ...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/13752 [SPARK-16028][SPARKR] spark.lapply can work with active context ## What changes were proposed in this pull request? spark.lapply and setLogLevel ## How was this patch tested? unit test @shivaram @thunterdb You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rlapply Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13752.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13752 commit 179fbdc7f93a929e00aaeeb575eb404038385252 Author: Felix Cheung Date: 2016-06-18T04:48:05Z lapply should not need spark context --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13751: [SPARK-15159][SPARKR] SparkSession roxygen2 doc, ...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/13751 [SPARK-15159][SPARKR] SparkSession roxygen2 doc, programming guide, example updates ## What changes were proposed in this pull request? roxygen2 doc, programming guide, example updates ## How was this patch tested? manual checks @shivaram You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rsparksessiondoc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13751.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13751 commit c434614321a3650a18b228f6333ffe245b6ea7ee Author: Felix Cheung Date: 2016-06-18T04:39:44Z roxygen2 doc, programming guide, example updates --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13725: [SPARK-15892][ML] Backport correctly merging AFTAggregat...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/13725 I am closing this as this is merged. Thank you @mengxr again! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13725: [SPARK-15892][ML] Backport correctly merging AFTA...
Github user HyukjinKwon closed the pull request at: https://github.com/apache/spark/pull/13725 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13750: [SPARK-16035][PYSPARK] Fix SparseVector parser assertion...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13750 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60749/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13750: [SPARK-16035][PYSPARK] Fix SparseVector parser assertion...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13750 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13749: [SPARK-16034][SQL] Checks the partition columns when cal...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13749 **[Test build #60754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60754/consoleFull)** for PR 13749 at commit [`7a4293b`](https://github.com/apache/spark/commit/7a4293bd68896701e4ca61a9df06133357bc0f8a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
