[GitHub] spark issue #17348: [SPARK-20018][SQL] Pivot with timestamp and count should...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17348 **[Test build #74824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74824/testReport)** for PR 17348 at commit [`3c619df`](https://github.com/apache/spark/commit/3c619dfb94723bd7a7d6a0811ab6329bf107f81b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17348: [SPARK-20018][SQL] Pivot with timestamp and count should...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17348 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74824/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17348: [SPARK-20018][SQL] Pivot with timestamp and count should...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17348 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17350: [SPARK-20017][SQL] change the nullability of func...
GitHub user zhaorongsheng opened a pull request: https://github.com/apache/spark/pull/17350 [SPARK-20017][SQL] change the nullability of function 'StringToMap' from 'false' to 'true' ## What changes were proposed in this pull request? Change the nullability of function `StringToMap` from `false` to `true`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhaorongsheng/spark bug-fix_strToMap_NPE Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17350.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17350 commit d4bb1583650e44e556bf06d1503355d7529e1ac6 Author: zhaorongsheng <334362...@qq.com> Date: 2017-03-18T16:03:34Z Merge remote-tracking branch 'upstream/master' into master_git commit ee2d7e6aee4248cec124457b5b03da5aa790c984 Author: zhaorongsheng <334362...@qq.com> Date: 2017-03-19T15:48:15Z SPARK-20017 change the nullability of 'StringToMap' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17350: [SPARK-20017][SQL] change the nullability of function 'S...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17350 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16541: [SPARK-19088][SQL] Optimize sequence type deseria...
Github user michalsenkyr commented on a diff in the pull request:
https://github.com/apache/spark/pull/16541#discussion_r106810789
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -589,6 +590,170 @@ case class MapObjects private(
}
}
+object CollectObjects {
+ private val curId = new java.util.concurrent.atomic.AtomicInteger()
+
+ /**
+ * Construct an instance of CollectObjects case class.
+ *
+ * @param function The function applied on the collection elements.
+ * @param inputData An expression that when evaluated returns a
collection object.
+ * @param elementType The data type of elements in the collection.
+ * @param collClass The type of the resulting collection.
+ */
+ def apply(
+ function: Expression => Expression,
+ inputData: Expression,
+ elementType: DataType,
+ collClass: Class[_]): CollectObjects = {
+val loopValue = "CollectObjects_loopValue" + curId.getAndIncrement()
+val loopIsNull = "CollectObjects_loopIsNull" + curId.getAndIncrement()
+val loopVar = LambdaVariable(loopValue, loopIsNull, elementType)
+val builderValue = "CollectObjects_builderValue" +
curId.getAndIncrement()
+CollectObjects(loopValue, loopIsNull, elementType, function(loopVar),
inputData,
+ collClass, builderValue)
+ }
+}
+
+/**
+ * An equivalent to the [[MapObjects]] case class but returning an
ObjectType containing
+ * a Scala collection constructed using the associated builder, obtained
by calling `newBuilder`
+ * on the collection's companion object.
+ *
+ * @param loopValue the name of the loop variable that used when iterate
the collection, and used
+ * as input for the `lambdaFunction`
+ * @param loopIsNull the nullity of the loop variable that used when
iterate the collection, and
+ * used as input for the `lambdaFunction`
+ * @param loopVarDataType the data type of the loop variable that used
when iterate the collection,
+ *and used as input for the `lambdaFunction`
+ * @param lambdaFunction A function that take the `loopVar` as input, and
used as lambda function
+ * to handle collection elements.
+ * @param inputData An expression that when evaluated returns a collection
object.
+ * @param collClass The type of the resulting collection.
+ * @param builderValue The name of the builder variable used to construct
the resulting collection.
+ */
+case class CollectObjects private(
--- End diff --
Yes, we actually can. I merged `CollectObjects` into `MapObjects` in my
next commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFi...
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/17274#discussion_r106810830
--- Diff: R/pkg/inst/tests/testthat/test_context.R ---
@@ -177,6 +177,13 @@ test_that("add and get file to be downloaded with
Spark job on every node", {
spark.addFile(path)
download_path <- spark.getSparkFiles(filename)
expect_equal(readLines(download_path), words)
+
+ # Test spark.getSparkFiles works well on executors.
+ seq <- seq(from = 1, to = 10, length.out = 5)
+ f <- function(seq) { readLines(spark.getSparkFiles(filename)) }
+ results <- spark.lapply(seq, f)
+ for (i in 1:5) { expect_equal(results[[i]], words) }
+
--- End diff --
Reading files in the UDF is the main use of this fix, however, it can pass
test in SparkR console and jobs submitted by ```bin/spark-submit
test.R```(local mode) or ```bin/spark-submit --master yarn test.R```(yarn
mode). These two scenarios are the most common use cases for this function, and
passing the tests in real cluster is convincing enough.
I suspect the odd failure in the previous ```run-tests.sh``` was caused by
other issues(such as the test infrastructure), not the fix itself. So I think
we can get this in and leave todo if we can't figure out the root cause of that
odd test for the moment, since 2.2 code freeze is coming. What do you think of
it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17350: [SPARK-20017][SQL] change the nullability of function 'S...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/17350 Could you add tests? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16541: [SPARK-19088][SQL] Optimize sequence type deseria...
Github user michalsenkyr commented on a diff in the pull request:
https://github.com/apache/spark/pull/16541#discussion_r106810940
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/SequenceBenchmark.scala
---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.spark.util.Benchmark
+
+/**
+ * Benchmark [[Seq]], [[List]] and [[scala.collection.mutable.Queue]]
serialization
+ * performance.
+ * To run this:
+ * 1. replace ignore(...) with test(...)
+ * 2. build/sbt "sql/test-only *benchmark.SequenceBenchmark"
+ *
+ * Benchmarks in this file are skipped in normal builds.
+ */
+class SequenceBenchmark extends BenchmarkBase {
--- End diff --
Removed but it's still useful to know that the change didn't affect
performance in any negative way. It introduced a different approach to
collection construction after all. Should I also remove the results from PR
description?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16541: [SPARK-19088][SQL] Optimize sequence type deseria...
Github user michalsenkyr commented on a diff in the pull request:
https://github.com/apache/spark/pull/16541#discussion_r106810993
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -589,6 +590,170 @@ case class MapObjects private(
}
}
+object CollectObjects {
+ private val curId = new java.util.concurrent.atomic.AtomicInteger()
+
+ /**
+ * Construct an instance of CollectObjects case class.
+ *
+ * @param function The function applied on the collection elements.
+ * @param inputData An expression that when evaluated returns a
collection object.
+ * @param elementType The data type of elements in the collection.
+ * @param collClass The type of the resulting collection.
+ */
+ def apply(
+ function: Expression => Expression,
+ inputData: Expression,
+ elementType: DataType,
+ collClass: Class[_]): CollectObjects = {
+val loopValue = "CollectObjects_loopValue" + curId.getAndIncrement()
+val loopIsNull = "CollectObjects_loopIsNull" + curId.getAndIncrement()
+val loopVar = LambdaVariable(loopValue, loopIsNull, elementType)
+val builderValue = "CollectObjects_builderValue" +
curId.getAndIncrement()
--- End diff --
Altered vals in `MapObjects` to share the same `curId` instead
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFi...
Github user yanboliang closed the pull request at: https://github.com/apache/spark/pull/17274 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFi...
GitHub user yanboliang reopened a pull request: https://github.com/apache/spark/pull/17274 [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFiles fails when it was called on executors. ## What changes were proposed in this pull request? SparkR ```spark.getSparkFiles``` fails when it was called on executors, see details at [SPARK-19925](https://issues.apache.org/jira/browse/SPARK-19925). ## How was this patch tested? Add unit tests, and verify this fix at standalone and yarn cluster. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yanboliang/spark spark-19925 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17274.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17274 commit 6cb40ebf994e1263473ff1127c6ca7dc649c374d Author: Yanbo Liang Date: 2017-03-13T11:15:36Z Fix SparkR spark.getSparkFiles fails when it was called on executors. commit 2405eb6fe40aee1df20a35d225c4b9d9a875241e Author: Yanbo Liang Date: 2017-03-15T14:10:48Z Add prefix SPARKR_ and use file.path to construct file path. commit 44e6d3264f9460614792f94eceb41b1eb43bb239 Author: Yanbo Liang Date: 2017-03-16T10:06:00Z Update test cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17274: [SPARK-19925][SPARKR] Fix SparkR spark.getSparkFiles fai...
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/17274 I closed and reopened this PR to trigger the appveyor test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14615: [SPARK-17029] make toJSON not go through rdd form but op...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14615 **[Test build #74828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74828/testReport)** for PR 14615 at commit [`6250699`](https://github.com/apache/spark/commit/6250699b5a72890449417317014dd27b7c8048df). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17349: [MINOR][R] Reorder `Collate` fields in DESCRIPTION file
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17349 merged. thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17338 @srowen It sounds like we did not monitor the mvn testing results. Thus, we accumulate multiple failures. We need to resolve it one by one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16596 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17349: [MINOR][R] Reorder `Collate` fields in DESCRIPTIO...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17349 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST][test-maven][WIP] Use the database af...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17344 No idea. We need to monitor the mvn builds more closely. This flaky test case also impacts the sbt build. Let me merge this first. Then, we can continue to clean the others. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST] Use the database after Hive's curren...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17344 Let us run the SBT build too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST] Use the database after Hive's curren...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17344 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST] Use the database after Hive's curren...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17344 **[Test build #74829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74829/testReport)** for PR 17344 at commit [`90bd976`](https://github.com/apache/spark/commit/90bd9763399f2cbeed3c93b0d0c1adc024d6602e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output to te...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16330 merged to master and branch-2.1 thanks @shivaram @gatorsmile @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16330: [SPARK-18817][SPARKR][SQL] change derby log outpu...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16330 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17338: [SPARK-19990][SQL][test-maven]create a temp file ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17338#discussion_r106813669
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -700,20 +701,33 @@ abstract class DDLSuite extends QueryTest with
SQLTestUtils {
test("create temporary view using") {
val csvFile =
-
Thread.currentThread().getContextClassLoader.getResource("test-data/cars.csv").toString
-withView("testview") {
- sql(s"CREATE OR REPLACE TEMPORARY VIEW testview (c1 String, c2
String) USING " +
-"org.apache.spark.sql.execution.datasources.csv.CSVFileFormat " +
-s"OPTIONS (PATH '$csvFile')")
+
Thread.currentThread().getContextClassLoader.getResourceAsStream("test-data/cars.csv")
--- End diff --
Could you write the comment about this? When we read the code in the
future, we can know the history. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17302: [SPARK-19959][SQL] Fix to throw NullPointerException in ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17302 **[Test build #74830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74830/testReport)** for PR 17302 at commit [`fd4fc3d`](https://github.com/apache/spark/commit/fd4fc3d2058a1610f1429c89fe7c1146c42c7328). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17338 @windpiger This becomes a general issue for the test cases. Could you create a separate function `withXYZ` for it? We can reuse it later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17351: [SPARK-20020][SPARKR] DataFrame checkpoint API
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/17351 [SPARK-20020][SPARKR] DataFrame checkpoint API ## What changes were proposed in this pull request? Add checkpoint, setCheckpointDir API to R ## How was this patch tested? unit tests, manual tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rdfcheckpoint Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17351.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17351 commit a2b5e683532631c3366b778159054e978a960aa3 Author: Felix Cheung Date: 2017-03-19T17:48:26Z checkpoint and setCheckpointDir commit c6f114ef047e999ba56d8d4f6284336d27577501 Author: Felix Cheung Date: 2017-03-19T18:03:56Z doc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17351: [SPARK-20020][SPARKR] DataFrame checkpoint API
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17351 **[Test build #74831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74831/testReport)** for PR 17351 at commit [`87471cc`](https://github.com/apache/spark/commit/87471cc3a5dd9ed77a21bae50205e568421443f5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17191: [SPARK-14471][SQL] Aliases in SELECT could be used in GR...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17191
Could you add the related test case? Below is some examples
```Scala
withSQLConf(SQLConf.GROUP_BY_ALIASES_ENABLED.key -> "true",
SQLConf.GROUP_BY_ORDINAL.key -> "true") {
sql("SELECT k1 AS `2`, k2 AS `3`, SUM(v) FROM t GROUP BY 1, `3`")
}
withSQLConf(SQLConf.GROUP_BY_ALIASES_ENABLED.key -> "true",
SQLConf.GROUP_BY_ORDINAL.key -> "false") {
sql("SELECT k2 AS `3`, SUM(v) FROM t GROUP BY 5, `3`")
}
withSQLConf(SQLConf.GROUP_BY_ALIASES_ENABLED.key -> "false",
SQLConf.GROUP_BY_ORDINAL.key -> "true") {
// Interception
// sql("SELECT k2 AS `3`, SUM(v) FROM t GROUP BY 1, `3`")
}
withSQLConf(SQLConf.GROUP_BY_ALIASES_ENABLED.key -> "false",
SQLConf.GROUP_BY_ORDINAL.key -> "false") {
sql("SELECT k2 AS `3`, SUM(v) FROM t GROUP BY k2, 5")
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17191: [SPARK-14471][SQL] Aliases in SELECT could be use...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17191#discussion_r106814321
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -2598,4 +2598,27 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
assert(!jobStarted.get(), "Command should not trigger a Spark job.")
}
+
+ test("SPARK-14471 When groupByAliasesEnabled=true, aliases in SELECT
could exist in GROUP BY") {
+withSQLConf(SQLConf.GROUP_BY_ALIASES_ENABLED.key -> "true",
+SQLConf.GROUP_BY_ORDINAL.key -> "true") {
+ Seq((1, "a", 0), (2, "a", 1), (1, "a", 2)).toDF("k1", "k2", "v")
+.createOrReplaceTempView("t")
+ checkAnswer(
+sql("SELECT k1 AS key1, k2 AS key2, SUM(v) FROM t GROUP BY key1,
key2"),
+Row(1, "a", 2) :: Row(2, "a", 1) :: Nil)
+ // Check mixed cases: group-by ordinals and aliases
+ checkAnswer(
+sql("SELECT k1, k2 AS key2, SUM(v) FROM t GROUP BY key2, 1"),
+Row(1, "a", 2) :: Row(2, "a", 1) :: Nil)
+}
+withSQLConf(SQLConf.GROUP_BY_ALIASES_ENABLED.key -> "false") {
+ Seq((1, "a", 0), (2, "a", 1), (1, "a", 2)).toDF("k1", "k2", "v")
+.createOrReplaceTempView("t")
--- End diff --
Please remove it. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17191: [SPARK-14471][SQL] Aliases in SELECT could be use...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17191#discussion_r106814663 --- Diff: sql/core/src/test/resources/sql-tests/inputs/group-by.sql --- @@ -35,3 +35,6 @@ FROM testData; -- Aggregate with foldable input and multiple distinct groups. SELECT COUNT(DISTINCT b), COUNT(DISTINCT b, c) FROM (SELECT 1 AS a, 2 AS b, 3 AS c) GROUP BY a; + +-- Aliases in SELECT could be used in GROUP BY +SELECT a AS k, COUNT(b) FROM testData GROUP BY k; --- End diff -- Also add the extra EXPLAIN for us to know whether the plan is correct. ``` EXPLAIN SELECT a AS k, COUNT(b) FROM testData GROUP BY k ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17191: [SPARK-14471][SQL] Aliases in SELECT could be used in GR...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17191 We already have `group-by.sql` and `group-by-ordinal.sql`. Could you move the test cases there? You can use the following command to turn the conf off or on. ``` -- turn off group by ordinal set spark.sql.groupByOrdinal=false; ``` For some critical test cases, we should EXPLAIN the plan. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14615: [SPARK-17029] make toJSON not go through rdd form but op...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14615 **[Test build #74828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74828/testReport)** for PR 14615 at commit [`6250699`](https://github.com/apache/spark/commit/6250699b5a72890449417317014dd27b7c8048df). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14615: [SPARK-17029] make toJSON not go through rdd form but op...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14615 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14615: [SPARK-17029] make toJSON not go through rdd form but op...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14615 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74828/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17351: [SPARK-20020][SPARKR] DataFrame checkpoint API
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17351 **[Test build #74831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74831/testReport)** for PR 17351 at commit [`87471cc`](https://github.com/apache/spark/commit/87471cc3a5dd9ed77a21bae50205e568421443f5). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17351: [SPARK-20020][SPARKR] DataFrame checkpoint API
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17351 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74831/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17351: [SPARK-20020][SPARKR] DataFrame checkpoint API
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17351 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17351: [SPARK-20020][SPARKR] DataFrame checkpoint API
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17351 **[Test build #74832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74832/testReport)** for PR 17351 at commit [`207922f`](https://github.com/apache/spark/commit/207922f2ae2334456d50b984766541781acd7425). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16596 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16596 **[Test build #74833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74833/testReport)** for PR 16596 at commit [`e33b50a`](https://github.com/apache/spark/commit/e33b50aae78c79a425ab1e935498919eb0350c97). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17351: [SPARK-20020][SPARKR] DataFrame checkpoint API
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17351 **[Test build #74832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74832/testReport)** for PR 17351 at commit [`207922f`](https://github.com/apache/spark/commit/207922f2ae2334456d50b984766541781acd7425). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17351: [SPARK-20020][SPARKR] DataFrame checkpoint API
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17351 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74832/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17351: [SPARK-20020][SPARKR] DataFrame checkpoint API
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17351 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17302: [SPARK-19959][SQL] Fix to throw NullPointerException in ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17302 **[Test build #74830 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74830/testReport)** for PR 17302 at commit [`fd4fc3d`](https://github.com/apache/spark/commit/fd4fc3d2058a1610f1429c89fe7c1146c42c7328). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17302: [SPARK-19959][SQL] Fix to throw NullPointerException in ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17302 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17302: [SPARK-19959][SQL] Fix to throw NullPointerException in ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17302 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74830/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST] Use the database after Hive's curren...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17344 **[Test build #74829 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74829/testReport)** for PR 17344 at commit [`90bd976`](https://github.com/apache/spark/commit/90bd9763399f2cbeed3c93b0d0c1adc024d6602e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST] Use the database after Hive's curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17344 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST] Use the database after Hive's curren...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17344 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74829/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST] Use the database after Hive's curren...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17344 Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17344: [SPARK-19990][TEST] Use the database after Hive's...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17344 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17179: [SPARK-19067][SS] Processing-time-based timeout in MapGr...
Github user tdas commented on the issue: https://github.com/apache/spark/pull/17179 I am merging this to master in the interest of time. If there are comments, I will address them in a follow up PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17179: [SPARK-19067][SS] Processing-time-based timeout i...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17179 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17250: [SPARK-19911][STREAMING] Add builder interface for Kines...
Github user budde commented on the issue: https://github.com/apache/spark/pull/17250 @brkyvz A conference took up a lot of my time last week but I should have it updated later today --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16596 **[Test build #74833 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74833/testReport)** for PR 16596 at commit [`e33b50a`](https://github.com/apache/spark/commit/e33b50aae78c79a425ab1e935498919eb0350c97). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16596 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74833/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16596: [SPARK-19237][SPARKR][CORE] On Windows spark-submit shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16596 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST] Use the database after Hive's curren...
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/17344 It looks like this has the wrong JIRA id and should be SPARK-19988? I've marked that jira as resolved by this PR, but let me know if I've misunderstood things here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17302: [SPARK-19959][SQL] Fix to throw NullPointerException in ...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/17302 cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST] Use the database after Hive's curren...
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/17344 Also @gatorsmile don't forget to update the corresponding JIRA when you merge a PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/17338 @srowen I think #17344 was intended to be for a different JIRA and was accidentally assigned the same JIRA# as this PR (although they are related issues). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17344: [SPARK-19990][TEST] Use the database after Hive's curren...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17344 Sorry, I did not realize we have such a JIRA. We just hit it in another PR and simply fix it. Thanks! Originally, I thought it was trivial, and thus I did not open a separate JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17338 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17338 Let us see which cases still failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17338 **[Test build #74834 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74834/testReport)** for PR 17338 at commit [`a02d3d6`](https://github.com/apache/spark/commit/a02d3d664b19301c9d2336a39b087a486dd3f627). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17338 **[Test build #74834 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74834/testReport)** for PR 17338 at commit [`a02d3d6`](https://github.com/apache/spark/commit/a02d3d664b19301c9d2336a39b087a486dd3f627). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17338 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17338 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74834/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/15435 @sethah Don't worry, I will update code ASAP and @yanboliang will also help review it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17338 ok... It was stopped at Spark Core this time. Let me see which PR in Spark Core was merged recently. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17331: [SPARK-19994][SQL] Wrong outputOrdering for right/full o...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/17331 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17290: [SPARK-16599][CORE] java.util.NoSuchElementException: No...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17290 As found in https://github.com/apache/spark/pull/17338, I am afraid this one caused the following test failure. - proactive block replication - 4 replicas - 3 block manager deletions *** FAILED *** 1 did not equal 0 Read locks unreleased! (BlockManagerReplicationSuite.scala:504) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17346: [SPARK-19965][SS] DataFrame batch reader may fail to inf...
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/17346 @zsxwing would you take a look at this? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17331: [SPARK-19994][SQL] Wrong outputOrdering for right/full o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17331 **[Test build #74835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74835/testReport)** for PR 17331 at commit [`e4c41dc`](https://github.com/apache/spark/commit/e4c41dcbca9afdcce5ebe44836f5f8cef0a01bb4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17338 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17338 **[Test build #74836 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74836/testReport)** for PR 17338 at commit [`a02d3d6`](https://github.com/apache/spark/commit/a02d3d664b19301c9d2336a39b087a486dd3f627). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/17338 @gatorsmile we can find all the test failed here [maven test result](https://spark-tests.appspot.com/builds/spark-master-test-maven-hadoop-2.7/2799) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16971: [SPARK-19573][SQL] Make NaN/null handling consistent in ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16971 **[Test build #74837 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74837/testReport)** for PR 16971 at commit [`b1125fd`](https://github.com/apache/spark/commit/b1125fd1a6610d61689f28b6022b4554146ac878). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17290: [SPARK-16599][CORE] java.util.NoSuchElementException: No...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17290 After removing the changes in this PR, the test case passed in my local environment. I think we need to revert it back. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17338 Can you revert the changes made in https://github.com/apache/spark/pull/17290? Then, we can see how many tests still failed after the existing fixes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17338 **[Test build #74838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74838/testReport)** for PR 17338 at commit [`fdc9d2a`](https://github.com/apache/spark/commit/fdc9d2a1088daf738d12bf00257d25d4c6f7e48d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/17338 ok, let me do it~ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17014: [SPARK-18608][ML] Fix double-caching in ML algorithms
Github user zhengruifeng commented on the issue: https://github.com/apache/spark/pull/17014 @hhbyyh I think I misunderstood your comments in jira. I will update this pr with the new plan: directly add `protected var storageLevel` in `Predictor`, without adding setter and getter of it now.. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17352: Miss backslash in python code
GitHub user uncleGen opened a pull request: https://github.com/apache/spark/pull/17352 Miss backslash in python code ## What changes were proposed in this pull request? Add backslash for line continuation in python code. ## How was this patch tested? Jenkins. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uncleGen/spark python-example-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17352.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17352 commit d0a1c9f15288a4af4b3a3e12a89aff94d7104f7a Author: uncleGen Date: 2017-03-13T02:58:23Z fix python example in doc commit 965dce3d8707cadadf59594dc88310e2224ffeef Author: uncleGen Date: 2017-03-20T02:00:06Z Merge branch 'master' into python-example-doc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17352: [SPARK-20021][PySpark] Miss backslash in python code
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17352 **[Test build #74839 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74839/testReport)** for PR 17352 at commit [`965dce3`](https://github.com/apache/spark/commit/965dce3d8707cadadf59594dc88310e2224ffeef). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14617: [SPARK-17019][Core] Expose on-heap and off-heap memory u...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/14617 Thanks @squito , finally get someone's attention :). I will update the code soon. Before this change off-heap on on-heap memory are added together (https://github.com/apache/spark/pull/13920). I think it would be better to expose this details to users. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17338 **[Test build #74840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74840/testReport)** for PR 17338 at commit [`0a05bfa`](https://github.com/apache/spark/commit/0a05bfa66c6e4fcefa109fd03c0505b7a423dbaa). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17338: [SPARK-19990][SQL][test-maven]create a temp file for fil...
Github user windpiger commented on the issue: https://github.com/apache/spark/pull/17338 @gatorsmile I have merge this pr with master and revert #17290, and waiting for the jenkins to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17352: [SPARK-20021][PySpark] Miss backslash in python code
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17352 **[Test build #74839 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74839/testReport)** for PR 17352 at commit [`965dce3`](https://github.com/apache/spark/commit/965dce3d8707cadadf59594dc88310e2224ffeef). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17352: [SPARK-20021][PySpark] Miss backslash in python code
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17352 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15363: [SPARK-17791][SQL] Join reordering using star schema det...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15363 **[Test build #74842 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74842/testReport)** for PR 15363 at commit [`891813f`](https://github.com/apache/spark/commit/891813ff7316ab06acfcf28a7268da65ac9fd4cf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17352: [SPARK-20021][PySpark] Miss backslash in python code
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17352 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74839/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17141: [SPARK-19800][SS][WIP] Implement one kind of streaming s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17141 **[Test build #74841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74841/testReport)** for PR 17141 at commit [`1ddb82e`](https://github.com/apache/spark/commit/1ddb82e5a5aaadebd28080d334d58d082a125da8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17141: [SPARK-19800][SS][WIP] Implement one kind of streaming s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17141 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17141: [SPARK-19800][SS][WIP] Implement one kind of streaming s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17141 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74841/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17141: [SPARK-19800][SS][WIP] Implement one kind of streaming s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17141 **[Test build #74841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74841/testReport)** for PR 17141 at commit [`1ddb82e`](https://github.com/apache/spark/commit/1ddb82e5a5aaadebd28080d334d58d082a125da8). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15363: [SPARK-17791][SQL] Join reordering using star sch...
Github user ioana-delaney commented on a diff in the pull request:
https://github.com/apache/spark/pull/15363#discussion_r106828049
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -20,19 +20,347 @@ package org.apache.spark.sql.catalyst.optimizer
import scala.annotation.tailrec
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.planning.ExtractFiltersAndInnerJoins
+import
org.apache.spark.sql.catalyst.planning.{ExtractFiltersAndInnerJoins,
PhysicalOperation}
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules._
+import org.apache.spark.sql.catalyst.CatalystConf
+
+/**
+ * Encapsulates star-schema join detection.
+ */
+case class StarSchemaDetection(conf: CatalystConf) extends PredicateHelper
{
+
+ /**
+ * Star schema consists of one or more fact tables referencing a number
of dimension
+ * tables. In general, star-schema joins are detected using the
following conditions:
+ * 1. Informational RI constraints (reliable detection)
+ *+ Dimension contains a primary key that is being joined to the
fact table.
+ *+ Fact table contains foreign keys referencing multiple dimension
tables.
+ * 2. Cardinality based heuristics
+ *+ Usually, the table with the highest cardinality is the fact
table.
+ *+ Table being joined with the most number of tables is the fact
table.
+ *
+ * To detect star joins, the algorithm uses a combination of the above
two conditions.
+ * The fact table is chosen based on the cardinality heuristics, and the
dimension
+ * tables are chosen based on the RI constraints. A star join will
consist of the largest
+ * fact table joined with the dimension tables on their primary keys. To
detect that a
+ * column is a primary key, the algorithm uses table and column
statistics.
+ *
+ * Since Catalyst only supports left-deep tree plans, the algorithm
currently returns only
+ * the star join with the largest fact table. Choosing the largest fact
table on the
+ * driving arm to avoid large inners is in general a good heuristic.
This restriction can
+ * be lifted with support for bushy tree plans.
+ *
+ * The highlights of the algorithm are the following:
+ *
+ * Given a set of joined tables/plans, the algorithm first verifies if
they are eligible
+ * for star join detection. An eligible plan is a base table access with
valid statistics.
+ * A base table access represents Project or Filter operators above a
LeafNode. Conservatively,
+ * the algorithm only considers base table access as part of a star join
since they provide
+ * reliable statistics.
+ *
+ * If some of the plans are not base table access, or statistics are not
available, the algorithm
+ * returns an empty star join plan since, in the absence of statistics,
it cannot make
+ * good planning decisions. Otherwise, the algorithm finds the table
with the largest cardinality
+ * (number of rows), which is assumed to be a fact table.
+ *
+ * Next, it computes the set of dimension tables for the current fact
table. A dimension table
+ * is assumed to be in a RI relationship with a fact table. To infer
column uniqueness,
+ * the algorithm compares the number of distinct values with the total
number of rows in the
+ * table. If their relative difference is within certain limits (i.e.
ndvMaxError * 2, adjusted
+ * based on 1TB TPC-DS data), the column is assumed to be unique.
+ */
+ def findStarJoins(
+ input: Seq[LogicalPlan],
+ conditions: Seq[Expression]): Seq[Seq[LogicalPlan]] = {
+
+val emptyStarJoinPlan = Seq.empty[Seq[LogicalPlan]]
+
+if (!conf.starSchemaDetection || input.size < 2) {
+ emptyStarJoinPlan
+} else {
+ // Find if the input plans are eligible for star join detection.
+ // An eligible plan is a base table access with valid statistics.
+ val foundEligibleJoin = input.forall {
+case PhysicalOperation(_, _, t: LeafNode) if
t.stats(conf).rowCount.isDefined => true
+case _ => false
+ }
+
+ if (!foundEligibleJoin) {
+// Some plans don't have stats or are complex plans.
Conservatively,
+// return an empty star join. This restriction can be lifted
+// once statistics are propagated in the plan.
+emptyStarJoinPlan
+ } else {
+// Find the fact table using cardinality based heuristics i.e.
+// the table with the largest number of rows.
+val sortedFactTables = input.map { plan =>
+
[GitHub] spark issue #15363: [SPARK-17791][SQL] Join reordering using star schema det...
Github user ioana-delaney commented on the issue: https://github.com/apache/spark/pull/15363 @gatorsmile @cloud-fan I rewrote the test cases to align to the join reorder suite. Please take a look. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
