[GitHub] spark issue #22467: [SPARK-25465][TEST] Refactor Parquet test suites in proj...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22467 **[Test build #96400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96400/testReport)** for PR 22467 at commit [`813d19c`](https://github.com/apache/spark/commit/813d19c63477b82a76bdd0d1da73cf3cb1d38564). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459] Add viewOriginalText back to CatalogTable
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22458 **[Test build #96399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96399/testReport)** for PR 22458 at commit [`f3d3100`](https://github.com/apache/spark/commit/f3d3100399be442da9fd5e417aeefb9662903c49). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19045 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96398/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19045 **[Test build #96398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96398/testReport)** for PR 19045 at commit [`42a29ab`](https://github.com/apache/spark/commit/42a29abf4d4479f5195eee6324efd181f118535b). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19045 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22407: [SPARK-25416][SQL] ArrayPosition function may return inc...
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22407 @ueshin Wenchen thought it may be risky to backport the fix to tighestCommonType. Given this, can this be looked at now ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19045 **[Test build #96398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96398/testReport)** for PR 19045 at commit [`42a29ab`](https://github.com/apache/spark/commit/42a29abf4d4479f5195eee6324efd181f118535b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19045 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3327/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19045 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22455: [SPARK-24572][SPARKR] "eager execution" for R she...
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/22455#discussion_r219388335
--- Diff: R/pkg/R/DataFrame.R ---
@@ -244,11 +245,15 @@ setMethod("showDF",
#' @note show(SparkDataFrame) since 1.4.0
setMethod("show", "SparkDataFrame",
function(object) {
-cols <- lapply(dtypes(object), function(l) {
- paste(l, collapse = ":")
-})
-s <- paste(cols, collapse = ", ")
-cat(paste(class(object), "[", s, "]\n", sep = ""))
+if (identical(sparkR.conf("spark.sql.repl.eagerEval.enabled",
"false")[[1]], "true")) {
--- End diff --
@adrian555 Thanks for the explanation.
> However, my second point is that I don't think these two configs matter
much or that important/necessary. Since the eager execution is just to show a
snippet data of the SparkDataFrame, our default numRows = 20 and truncate =
TRUE are good enough iMO. If users want to see more or less number of rows,
they should call showDF().
So i just wanted to make sure if its possible to have parity with how it
works for python. It seems to me that in python, we just get the two configs
and call the showstring method.
> And if we think that showDF() can ignore the eager execution setting and
still want the show() to observe eager execution config, we can certainly just
grab the maxNumRows and truncate setting and pass to showDF() call.
What will happen if we grab these config in show() when eager execution is
enabled and then call showDF() by passing these parameters ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22513 **[Test build #96397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96397/testReport)** for PR 22513 at commit [`9288933`](https://github.com/apache/spark/commit/9288933b4a71e646e67f551dcfd80f9ff9a470da). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Ben...
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/22513#discussion_r219388085
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala
---
@@ -27,7 +27,7 @@ import
org.apache.spark.sql.functions.monotonically_increasing_id
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.ParquetOutputTimestampType
import org.apache.spark.sql.types.{ByteType, Decimal, DecimalType,
TimestampType}
-import org.apache.spark.util.{Benchmark, BenchmarkBase =>
FileBenchmarkBase, Utils}
+import org.apache.spark.util.Utils
/**
* Benchmark to measure read performance with Filter pushdown.
--- End diff --
How about change scala doc to below to fix **fails to generate
documentation**?
```scala
* To run this benchmark:
* {{{
* 1. without sbt: bin/spark-submit --class
* 2. build/sbt "sql/test:runMain "
* 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt
"sql/test:runMain "
* Results will be written to
"benchmarks/FilterPushdownBenchmark-results.txt".
* }}}
```
fails to generate documentation error message:
```java
/home/jenkins/workspace/SparkPullRequestBuilder@2/target/javaunidoc/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.html...
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/mllib/target/java/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.java:5:
error: unknown tag: this
[error] * 1. without sbt: bin/spark-submit --class
[error] ^
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/mllib/target/java/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.java:5:
error: unknown tag: spark
[error] * 1. without sbt: bin/spark-submit --class
[error] ^
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/mllib/target/java/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.java:6:
error: unknown tag: this
[error] * 2. build/sbt "mllib/test:runMain "
[error] ^
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/mllib/target/java/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.java:7:
error: unknown tag: this
[error] * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt
"mllib/test:runMain "
[error]
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22512: [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite failures w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22512 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22513 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3326/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22512: [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite failures w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22512 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96395/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22456: [SPARK-19355][SQL] Fix variable names numberOfOut...
Github user rxin closed the pull request at: https://github.com/apache/spark/pull/22456 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22512: [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite failures w...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22512 **[Test build #96395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96395/testReport)** for PR 22512 at commit [`39c5e92`](https://github.com/apache/spark/commit/39c5e92713b86f342e756591235f9cbe25126f90). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` s\"but $` * `case class Literal(value: Any, dataType: DataType) extends LeafExpression ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22513 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22461: [SPARK-25453][SQL][TEST] OracleIntegrationSuite I...
Github user seancxmao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22461#discussion_r219386919
--- Diff:
external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala
---
@@ -442,6 +442,8 @@ class OracleIntegrationSuite extends
DockerJDBCIntegrationSuite with SharedSQLCo
.option("lowerBound", "2018-07-06")
.option("upperBound", "2018-07-20")
.option("numPartitions", 3)
+ .option("oracle.jdbc.mapDateToTimestamp", "false")
--- End diff --
ok. I will add notes to
http://spark.apache.org/docs/latest/sql-programming-guide.html#jdbc-to-other-databases,
and will also add comments to the code.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22471: [SPARK-25469][SQL][Performance] Eval methods of Concat, ...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22471 LGTM Btw, we don't need `[Performance]` in the title, probably. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22513 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96396/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22513 **[Test build #96396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96396/testReport)** for PR 22513 at commit [`89bd830`](https://github.com/apache/spark/commit/89bd8300405a6c7f2ed4d756db66b2d1cc3f7389). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22513 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22512: [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite failures w...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22512 I thought we currently had less tests for interpreted projections, so I was checking if we had no bug caused by these projections. Then, I noticed these two issues when the interpreted mode enabled in `SQLQueryTestSuite`. I'm still digging if we have other bugs about interpreted projections, so I set `WIP`. Btw, we'd be better to split this pr into multiple ones, probably. But, I'd like to make all the related bugs clear first. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22506: [SPARK-25494][SQL] Upgrade Spark's use of Janino ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22506 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22513 **[Test build #96396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96396/testReport)** for PR 22513 at commit [`89bd830`](https://github.com/apache/spark/commit/89bd8300405a6c7f2ed4d756db66b2d1cc3f7389). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22513 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3325/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22513 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/22513 @wangyum @yucai @dongjoon-hyun @cloud-fan @gatorsmile Let's focus on this before we merge other benchmark PRs. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22513: [SPARK-25499][TEST]Refactor BenchmarkBase and Ben...
GitHub user gengliangwang opened a pull request: https://github.com/apache/spark/pull/22513 [SPARK-25499][TEST]Refactor BenchmarkBase and Benchmark ## What changes were proposed in this pull request? Currently there are two classes with the same naming BenchmarkBase: 1. `org.apache.spark.util.BenchmarkBase` 2. `org.apache.spark.sql.execution.benchmark.BenchmarkBase` This is very confusing. And the benchmark object `org.apache.spark.sql.execution.benchmark.FilterPushdownBenchmark` is using the one in `org.apache.spark.util.BenchmarkBase`, while there is another class `BenchmarkBase` in the same package of it. Here I propose: 1. the package org.apache.spark.util.BenchmarkBase should be in test package, move to org.apache.spark.sql.execution.benchmark . 2. Rename the org.apache.spark.sql.execution.benchmark.BenchmarkBase as BenchmarkWithCodegen 3. Move org.apache.spark.util.Benchmark to test package of org.apache.spark.sql.execution.benchmark ## How was this patch tested? Unit test You can merge this pull request into a Git repository by running: $ git pull https://github.com/gengliangwang/spark refactorBenchmarkBase Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22513.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22513 commit 89bd8300405a6c7f2ed4d756db66b2d1cc3f7389 Author: Gengliang Wang Date: 2018-09-21T05:07:01Z refactor BenchmarkBase --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22506: [SPARK-25494][SQL] Upgrade Spark's use of Janino to 3.0....
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22506 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22512: [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite failures w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22512 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3324/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22512: [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite failures w...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22512 **[Test build #96395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96395/testReport)** for PR 22512 at commit [`39c5e92`](https://github.com/apache/spark/commit/39c5e92713b86f342e756591235f9cbe25126f90). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22512: [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite failures w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22512 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22512: [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite fai...
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/22512 [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite failures when the interpreter mode enabled ## What changes were proposed in this pull request? This pr fixed test failures in `SQLQueryTestSuite` when the interpreter mode enabled. This pr addressed the two cases below; - The current `InterpretedMutableProjection` can't handle `UnsafeRow` in the internal buffer `mutableRow`. `AggregationIterator` uses `MutableProjection` in that manner and `GenerateMutableProjection` can handle `UnsafeRow` as buffer internally. - `Literal` returns different a typed value between codegen and interpreter modes in some cases, e.g., `Literal(1, LongType)` returns a long value for the codegen mode and returns an int value for the interpreter mode. So, `InterpretedUnsafeProjection` fails when running `SQLQueryTestSuite`. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/maropu/spark InterpreterTest Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22512.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22512 commit 39c5e92713b86f342e756591235f9cbe25126f90 Author: Takeshi Yamamuro Date: 2018-09-21T04:25:53Z Fix test failures with the interpreter mode enabled --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22455: [SPARK-24572][SPARKR] "eager execution" for R she...
Github user adrian555 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22455#discussion_r219384936
--- Diff: R/pkg/R/DataFrame.R ---
@@ -244,11 +245,15 @@ setMethod("showDF",
#' @note show(SparkDataFrame) since 1.4.0
setMethod("show", "SparkDataFrame",
function(object) {
-cols <- lapply(dtypes(object), function(l) {
- paste(l, collapse = ":")
-})
-s <- paste(cols, collapse = ", ")
-cat(paste(class(object), "[", s, "]\n", sep = ""))
+if (identical(sparkR.conf("spark.sql.repl.eagerEval.enabled",
"false")[[1]], "true")) {
--- End diff --
Had thought about this. First, I consider it is not in the scope of this
jira, because I think they are conflicting with the current `showDF()`
behavior.
Some details: the `showDF()` already takes `numRows` and `truncate`
arguments. So if we are going to respect those two as well, we have to decide
what behavior is best suitable for `showDF()`. For example, whether `showDF()`
should just ignore the eager execution, or it picks the `maxNumRows` and
`truncate` set through eager execution like following:
```
setMethod("showDF",
signature(x = "SparkDataFrame"),
function(x, numRows = 20, truncate = TRUE, vertical = FALSE) {
eagerNumRows <-
as.numeric(sparkR.conf("spark.sql.repl.eagerEval.maxNumRows", "0")[[1]])
numRows <- ifelse(eagerNumRows == 0, numRows, eagerNumRows)
eagerTruncate <-
as.numeric(sparkR.conf("spark.sql.repl.eagerEval.truncate", "0")[[1]])
truncate <- ifelse(eagerTruncate == 0, truncate, eagerTruncate)
if (is.logical(truncate) && truncate) {
s <- callJMethod(x@sdf, "showString", numToInt(numRows),
numToInt(20), vertical)
} else {
truncate2 <- as.numeric(truncate)
s <- callJMethod(x@sdf, "showString", numToInt(numRows),
numToInt(truncate2),
vertical)
}
cat(s)
})
```
And if we think that `showDF()` can ignore the eager execution setting and
still want the `show()` to observe eager execution config, we can certainly
just grab the `maxNumRows` and `truncate` setting and pass to `showDF() call.
However, my second point is that I don't think these two configs matter
much or that important/necessary. Since the eager execution is just to show a
snippet data of the SparkDataFrame, our default `numRows = 20` and `truncate =
TRUE` are good enough iMO. If users want to see more or less number of rows,
they should call `showDF()`.
@felixcheung, your thought?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22511: [SPARK-25422][CORE] Don't memory map blocks streamed to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22511 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3323/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22511: [SPARK-25422][CORE] Don't memory map blocks streamed to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22511 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22511: [SPARK-25422][CORE] Don't memory map blocks streamed to ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22511 **[Test build #96394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96394/testReport)** for PR 22511 at commit [`aee82ab`](https://github.com/apache/spark/commit/aee82abe4cd9fbefa14fb280644276fe491bcf9a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22511: [SPARK-25422][CORE] Don't memory map blocks strea...
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/22511 [SPARK-25422][CORE] Don't memory map blocks streamed to disk. After data has been streamed to disk, the buffers are inserted into the memory store in some cases (eg., with broadcast blocks). But broadcast code also disposes of those buffers when the data has been read, to ensure that we don't leave mapped buffers using up memory, which then leads to garbage data in the memory store. ## How was this patch tested? Ran the old failing test in a loop. Full tests on jenkins You can merge this pull request into a Git repository by running: $ git pull https://github.com/squito/spark SPARK-25422 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22511.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22511 commit aee82abe4cd9fbefa14fb280644276fe491bcf9a Author: Imran Rashid Date: 2018-09-20T19:50:06Z [SPARK-25422][CORE] Don't memory map blocks streamed to disk. After data has been streamed to disk, the buffers are inserted into the memory store in some cases (eg., with broadcast blocks). But broadcast code also disposes of those buffers when the data has been read, to ensure that we don't leave mapped buffers using up memory, which then leads to garbage data in the memory store. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22510: [SPARK-25321][ML] Fix local LDA model constructor
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22510 **[Test build #96393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96393/testReport)** for PR 22510 at commit [`2b2fdaf`](https://github.com/apache/spark/commit/2b2fdaf3f7598fe31161fdd4401728d6b314bbfe). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22510: [SPARK-25321][ML] Fix local LDA model constructor
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22510 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96393/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22510: [SPARK-25321][ML] Fix local LDA model constructor
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22510 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22325: [SPARK-25318]. Add exception handling when wrapping the ...
Github user rezasafi commented on the issue: https://github.com/apache/spark/pull/22325 Flaky again. retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22506: [SPARK-25494][SQL] Upgrade Spark's use of Janino to 3.0....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22506 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22506: [SPARK-25494][SQL] Upgrade Spark's use of Janino to 3.0....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22506 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96386/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22506: [SPARK-25494][SQL] Upgrade Spark's use of Janino to 3.0....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22506 **[Test build #96386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96386/testReport)** for PR 22506 at commit [`c3f8a6b`](https://github.com/apache/spark/commit/c3f8a6b41c9409339bf62fb17cd4cd905853d97f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22510: [SPARK-25321][ML] Fix local LDA model constructor
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22510 **[Test build #96393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96393/testReport)** for PR 22510 at commit [`2b2fdaf`](https://github.com/apache/spark/commit/2b2fdaf3f7598fe31161fdd4401728d6b314bbfe). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22510: [SPARK-25321][ML] Fix local LDA model constructor
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22510 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3322/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22510: [SPARK-25321][ML] Fix local LDA model constructor
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22510 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22510: [SPARK-25321][ML] Fix local LDA model constructor
GitHub user WeichenXu123 opened a pull request: https://github.com/apache/spark/pull/22510 [SPARK-25321][ML] Fix local LDA model constructor ## What changes were proposed in this pull request? change back the constructor to: ``` class LocalLDAModel private[ml] ( uid: String, vocabSize: Int, private[clustering] val oldLocalModel : OldLocalLDAModel, sparkSession: SparkSession) ``` Although it is marked `private[ml]`, it is used in `mleap` and the master change breaks `mleap` building. ## How was this patch tested? Manual. You can merge this pull request into a Git repository by running: $ git pull https://github.com/WeichenXu123/spark LDA_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22510.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22510 commit 2b2fdaf3f7598fe31161fdd4401728d6b314bbfe Author: WeichenXu Date: 2018-09-21T03:03:30Z init pr --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19868: [SPARK-22676] Avoid iterating all partition paths when s...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19868 Basically we need to introduce this new `spark.sql.files.ignoreMissingFiles` config in detail. And them explain how can we use it to replace `spark.sql.hive.verifyPartitionPath`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459] Add viewOriginalText back to CatalogTable
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22458 LGTM. If there are more properties like `originalViewText` which are useless to Spark and only need to be displayed, I'd suggest we create a map for them, instead of adding more and more fields into `CatalogTable`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22451: [SPARK-24777][SQL] Add write benchmark for AVRO
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/22451 Oh I was creating a PR for refactoring BenchmarkBase, I planned to merge this one after that one. Since this is merged, I will create one to refactor both. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22505: Revert "[SPARK-23715][SQL] the input of to/from_u...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22505 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22458: [SPARK-25459] Add viewOriginalText back to Catalo...
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/22458#discussion_r219370221 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala --- @@ -467,9 +467,9 @@ private[hive] class HiveClientImpl( properties = filteredProperties, stats = readHiveStats(properties), comment = comment, -// In older versions of Spark(before 2.2.0), we expand the view original text and store -// that into `viewExpandedText`, and that should be used in view resolution. So we get -// `viewExpandedText` instead of `viewOriginalText` for viewText here. --- End diff -- This comment is for `viewText`, please rephrase and keep it, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22505: Revert "[SPARK-23715][SQL] the input of to/from_utc_time...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22505 thanks, merging to master/2.4! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22494: [SPARK-22036][SQL][followup] DECIMAL_OPERATIONS_ALLOW_PR...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22494 **[Test build #96392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96392/testReport)** for PR 22494 at commit [`1ee9f02`](https://github.com/apache/spark/commit/1ee9f0208a3cb6de373e05366c19bf69967eecd8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22494: [SPARK-22036][SQL][followup] DECIMAL_OPERATIONS_ALLOW_PR...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22494 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22494: [SPARK-22036][SQL][followup] DECIMAL_OPERATIONS_ALLOW_PR...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22494 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3321/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22504 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96384/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22504 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22504 **[Test build #96384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96384/testReport)** for PR 22504 at commit [`d792809`](https://github.com/apache/spark/commit/d792809532ca648d133260c564d33e014a35c135). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22505: Revert "[SPARK-23715][SQL] the input of to/from_utc_time...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22505 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96385/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22505: Revert "[SPARK-23715][SQL] the input of to/from_utc_time...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22505 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22316: [SPARK-25048][SQL] Pivoting by multiple columns i...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22316#discussion_r219368724 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala --- @@ -416,7 +426,7 @@ class RelationalGroupedDataset protected[sql]( new RelationalGroupedDataset( df, groupingExprs, - RelationalGroupedDataset.PivotType(pivotColumn.expr, values.map(Literal.apply))) + RelationalGroupedDataset.PivotType(pivotColumn.expr, values.map(lit(_).expr))) --- End diff -- now we eventually call `Literal.create` instead of `Literal.apply`. I'm not sure if there is a behavior change though. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22316: [SPARK-25048][SQL] Pivoting by multiple columns i...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22316#discussion_r219368791 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala --- @@ -416,7 +426,7 @@ class RelationalGroupedDataset protected[sql]( new RelationalGroupedDataset( df, groupingExprs, - RelationalGroupedDataset.PivotType(pivotColumn.expr, values.map(Literal.apply))) + RelationalGroupedDataset.PivotType(pivotColumn.expr, values.map(lit(_).expr))) --- End diff -- from a quick look, seems `Literal.create` is more powerful and should not have regressions. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22505: Revert "[SPARK-23715][SQL] the input of to/from_utc_time...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22505 **[Test build #96385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96385/testReport)** for PR 22505 at commit [`3908b98`](https://github.com/apache/spark/commit/3908b98564a72500100a0b76466b9abf48cb9cb7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22316: [SPARK-25048][SQL] Pivoting by multiple columns i...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22316#discussion_r219368334
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -330,6 +331,15 @@ class RelationalGroupedDataset protected[sql](
* df.groupBy("year").pivot("course").sum("earnings")
* }}}
*
+ * From Spark 3.0.0, values can be literal columns, for instance,
struct. For pivoting by
--- End diff --
3.0.0 => 2.5.0
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22482: WIP - [SPARK-10816][SS] Support session window natively
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/22482 According to the discussion on SPARK-10816, I'm holding up effort to improve and plan to discuss further from JIRA issue. I guess someone interested for this patch can still review or try this out and share feedback. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22138: [SPARK-25151][SS] Apply Apache Commons Pool to Ka...
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/22138#discussion_r219367280
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -18,222 +18,247 @@
package org.apache.spark.sql.kafka010
import java.{util => ju}
+import java.io.Closeable
import java.util.concurrent.TimeoutException
import scala.collection.JavaConverters._
import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord,
KafkaConsumer, OffsetOutOfRangeException}
import org.apache.kafka.common.TopicPartition
-import org.apache.spark.{SparkEnv, SparkException, TaskContext}
+import org.apache.spark.TaskContext
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.kafka010.KafkaDataConsumer.AvailableOffsetRange
+import
org.apache.spark.sql.kafka010.KafkaDataConsumer.{AvailableOffsetRange,
CacheKey, UNKNOWN_OFFSET}
import org.apache.spark.sql.kafka010.KafkaSourceProvider._
-import org.apache.spark.util.UninterruptibleThread
+import org.apache.spark.util.{ShutdownHookManager, UninterruptibleThread}
+
+/**
+ * This class simplifies the usages of Kafka consumer in Spark SQL Kafka
connector.
+ *
+ * NOTE: Like KafkaConsumer, this class is not thread-safe.
+ * NOTE for contributors: It is possible for the instance to be used from
multiple callers,
+ * so all the methods should not rely on current cursor and use seek
manually.
+ */
+private[kafka010] class InternalKafkaConsumer(
+val topicPartition: TopicPartition,
+val kafkaParams: ju.Map[String, Object]) extends Closeable with
Logging {
+
+ val groupId =
kafkaParams.get(ConsumerConfig.GROUP_ID_CONFIG).asInstanceOf[String]
+
+ private val consumer = createConsumer
-private[kafka010] sealed trait KafkaDataConsumer {
/**
- * Get the record for the given offset if available.
- *
- * If the record is invisible (either a
- * transaction message, or an aborted message when the consumer's
`isolation.level` is
- * `read_committed`), it will be skipped and this method will try to
fetch next available record
- * within [offset, untilOffset).
- *
- * This method also will try its best to detect data loss. If
`failOnDataLoss` is `true`, it will
- * throw an exception when we detect an unavailable offset. If
`failOnDataLoss` is `false`, this
- * method will try to fetch next available record within [offset,
untilOffset).
- *
- * When this method tries to skip offsets due to either invisible
messages or data loss and
- * reaches `untilOffset`, it will return `null`.
+ * Poll messages from Kafka starting from `offset` and returns a pair of
"list of consumer record"
+ * and "offset after poll". The list of consumer record may be empty if
the Kafka consumer fetches
+ * some messages but all of them are not visible messages (either
transaction messages,
+ * or aborted messages when `isolation.level` is `read_committed`).
*
- * @param offset the offset to fetch.
- * @param untilOffsetthe max offset to fetch. Exclusive.
- * @param pollTimeoutMs timeout in milliseconds to poll data from Kafka.
- * @param failOnDataLoss When `failOnDataLoss` is `true`, this method
will either return record at
- * offset if available, or throw exception.when
`failOnDataLoss` is `false`,
- * this method will either return record at offset
if available, or return
- * the next earliest available record less than
untilOffset, or null. It
- * will not throw any exception.
+ * @throws OffsetOutOfRangeException if `offset` is out of range.
+ * @throws TimeoutException if the consumer position is not changed
after polling. It means the
+ * consumer polls nothing before timeout.
*/
- def get(
- offset: Long,
- untilOffset: Long,
- pollTimeoutMs: Long,
- failOnDataLoss: Boolean): ConsumerRecord[Array[Byte], Array[Byte]] =
{
-internalConsumer.get(offset, untilOffset, pollTimeoutMs,
failOnDataLoss)
+ def fetch(offset: Long, pollTimeoutMs: Long)
+ : (ju.List[ConsumerRecord[Array[Byte], Array[Byte]]], Long) = {
+// Seek to the offset because we may call seekToBeginning or seekToEnd
before this.
+seek(offset)
+val p = consumer.poll(pollTimeoutMs)
+val r = p.records(topicPartition)
+logDebug(s"Polled $groupId ${p.partitions()} ${r.size}")
+val offsetAfterPoll = consumer.position(topicPartition)
+logDebug(s"Offset changed from $offset to $offsetAfterPoll after
polling")
+val
[GitHub] spark issue #22509: [SPARK-25384][SQL] Clarify fromJsonForceNullableSchema w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22509 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3320/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22509: [SPARK-25384][SQL] Clarify fromJsonForceNullableSchema w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22509 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22499: [SPARK-25489][ML][TEST] Refactor UDTSerialization...
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/22499#discussion_r219366799
--- Diff:
mllib/src/test/scala/org/apache/spark/mllib/linalg/UDTSerializationBenchmark.scala
---
@@ -18,52 +18,52 @@
package org.apache.spark.mllib.linalg
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
-import org.apache.spark.util.Benchmark
+import org.apache.spark.util.{Benchmark, BenchmarkBase =>
FileBenchmarkBase}
/**
* Serialization benchmark for VectorUDT.
+ * To run this benchmark:
+ * 1. without sbt: bin/spark-submit --class
--- End diff --
I think `<` should replaced to `[`:
```scala
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/target/java/org/apache/spark/sql/DatasetBenchmark.java:5:
error: unknown tag: this
[error] * 1. without sbt: bin/spark-submit --class
[error] ^
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/target/java/org/apache/spark/sql/DatasetBenchmark.java:5:
error: unknown tag: spark
[error] * 1. without sbt: bin/spark-submit --class
[error] ^
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/target/java/org/apache/spark/sql/DatasetBenchmark.java:6:
error: unknown tag: this
[error] * 2. build/sbt "sql/test:runMain "
[error] ^
[error]
/home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/target/java/org/apache/spark/sql/DatasetBenchmark.java:7:
error: unknown tag: this
[error] * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt
"sql/test:runMain "
[error]
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22509: [SPARK-25384][SQL] Clarify fromJsonForceNullableSchema w...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22509 **[Test build #96391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96391/testReport)** for PR 22509 at commit [`8ad50d5`](https://github.com/apache/spark/commit/8ad50d5433ac5a0f888fb5909893317002d5aa51). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22508: [SPARK-23549][SQL] Rename config spark.sql.legacy.compar...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3319/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22509: [SPARK-25384][SQL] Clarify fromJsonForceNullableSchema w...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/22509 cc @dongjoon-hyun @MaxGekk we still need this pr don't we? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22509: [SPARK-25384][SQL] Clarify fromJsonForceNullableS...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/22509 [SPARK-25384][SQL] Clarify fromJsonForceNullableSchema will be removed in Spark 3.0 See above. This should go into the 2.4 release. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark SPARK-25384 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22509.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22509 commit 8ad50d5433ac5a0f888fb5909893317002d5aa51 Author: Reynold Xin Date: 2018-09-21T02:06:28Z x --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22508: [SPARK-23549][SQL] Rename config spark.sql.legacy.compar...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22508 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22508: [SPARK-23549][SQL] Rename config spark.sql.legacy.compar...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/22508 cc @gatorsmile who merged the original pr. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22508: [SPARK-23549][SQL] Rename config spark.sql.legacy.compar...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22508 **[Test build #96390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96390/testReport)** for PR 22508 at commit [`f29dd89`](https://github.com/apache/spark/commit/f29dd8905f0b14c937a47d7abe291828c7de48b9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22508: [SPARK-23549][SQL] Rename config spark.sql.legacy...
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/22508
[SPARK-23549][SQL] Rename config
spark.sql.legacy.compareDateTimestampInTimestamp
## What changes were proposed in this pull request?
See title.
## How was this patch tested?
Make sure all references have been updated:
```
> git grep compareDateTimestampInTimestamp
docs/sql-programming-guide.md: - Since Spark 2.4, Spark compares a DATE
type with a TIMESTAMP type after promotes both sides to TIMESTAMP. To set
`false` to `spark.sql.legacy.compareDateTimestampInTimestamp` restores the
previous behavior. This option will be removed in Spark 3.0.
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
// if conf.compareDateTimestampInTimestamp is true
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
=> if (conf.compareDateTimestampInTimestamp) Some(TimestampType) else
Some(StringType)
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
=> if (conf.compareDateTimestampInTimestamp) Some(TimestampType) else
Some(StringType)
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
buildConf("spark.sql.legacy.compareDateTimestampInTimestamp")
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
def compareDateTimestampInTimestamp : Boolean =
getConf(COMPARE_DATE_TIMESTAMP_IN_TIMESTAMP)
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala:
"spark.sql.legacy.compareDateTimestampInTimestamp" ->
convertToTS.toString) {
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark SPARK-23549
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22508.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22508
commit f29dd8905f0b14c937a47d7abe291828c7de48b9
Author: Reynold Xin
Date: 2018-09-21T02:00:59Z
[SPARK-23549][SQL] Rename config
spark.sql.legacy.compareDateTimestampInTimestamp
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22498: [SPARK-18364] : Expose metrics for YarnShuffleService
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22498 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22498: [SPARK-18364] : Expose metrics for YarnShuffleService
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22498 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96382/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22498: [SPARK-18364] : Expose metrics for YarnShuffleService
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22498 **[Test build #96382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96382/testReport)** for PR 22498 at commit [`1ac18d9`](https://github.com/apache/spark/commit/1ac18d9a8d8269eb73bac1b09e45215a20fa53c5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22506: [SPARK-25494][SQL] Upgrade Spark's use of Janino to 3.0....
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22506 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22163: [SPARK-25166][CORE]Reduce the number of write operations...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22163 **[Test build #96389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96389/testReport)** for PR 22163 at commit [`2dc94a2`](https://github.com/apache/spark/commit/2dc94a24ab06141768413dc2bf6f9c5e29ce7249). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22163: [SPARK-25166][CORE]Reduce the number of write operations...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22163 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3318/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22163: [SPARK-25166][CORE]Reduce the number of write operations...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22163 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22163: [SPARK-25166][CORE]Reduce the number of write operations...
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/22163 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22492: [SPARK-25321][ML] Revert SPARK-14681 to avoid API breaki...
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/22492 @mengxr Should this be put into master ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96378/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22379 **[Test build #96378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96378/testReport)** for PR 22379 at commit [`81ae688`](https://github.com/apache/spark/commit/81ae6881ea09ba2caeafc41ff6a4fa097096deac). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22461: [SPARK-25453][SQL][TEST] OracleIntegrationSuite I...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22461#discussion_r219359831
--- Diff:
external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala
---
@@ -442,6 +442,8 @@ class OracleIntegrationSuite extends
DockerJDBCIntegrationSuite with SharedSQLCo
.option("lowerBound", "2018-07-06")
.option("upperBound", "2018-07-20")
.option("numPartitions", 3)
+ .option("oracle.jdbc.mapDateToTimestamp", "false")
--- End diff --
Also leave comments before line 445 to explain why we need to do this.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22461: [SPARK-25453][SQL][TEST] OracleIntegrationSuite I...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22461#discussion_r219359638
--- Diff:
external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala
---
@@ -442,6 +442,8 @@ class OracleIntegrationSuite extends
DockerJDBCIntegrationSuite with SharedSQLCo
.option("lowerBound", "2018-07-06")
.option("upperBound", "2018-07-20")
.option("numPartitions", 3)
+ .option("oracle.jdbc.mapDateToTimestamp", "false")
--- End diff --
This is good to know. Could you help update the doc and explain we can pass
the options to the underlying JDBC and also give some examples?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22507: [SPARK-25495][SS]FetchedData.reset should reset all fiel...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22507 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22507: [SPARK-25495][SS]FetchedData.reset should reset all fiel...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22507 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96388/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22455: [SPARK-24572][SPARKR] "eager execution" for R shell, IDE
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22455 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96387/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22455: [SPARK-24572][SPARKR] "eager execution" for R shell, IDE
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22455 **[Test build #96387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96387/testReport)** for PR 22455 at commit [`7b121e6`](https://github.com/apache/spark/commit/7b121e65b99e177d8870b4e098797f1f1e86ce65). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
