[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92507/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21684 **[Test build #92507 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92507/testReport)** for PR 21684 at commit [`e2eaa1c`](https://github.com/apache/spark/commit/e2eaa1c47d16549713da9cca7cce06552c384c0c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21649: [SPARK-23648][R][SQL]Adds more types for hint in ...
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21649#discussion_r199338975
--- Diff: R/pkg/R/DataFrame.R ---
@@ -3905,6 +3905,16 @@ setMethod("rollup",
groupedData(sgd)
})
+isTypeAllowedForSqlHint <- function(x) {
+ if (is.character(x) | is.numeric(x)) {
+TRUE
+ } else if (is.list(x)) {
+all (sapply(x, (function (y) is.character(y) | is.numeric(y
+ } else {
+FALSE
+ }
+}
+
#' hint
#'
#' Specifies execution plan hint and return a new SparkDataFrame.
--- End diff --
Scala doesn't have a doc on hint type supported. Actually it has type Any
in the signature.
```
def hint(name: String, parameters: Any*): Dataset[T]
```
I was trying to follow the python implementation in #20788, which allows
```basestring, list, float, int, dict.```Now I am kind of suspect if this is
the right behavior. Should we allow any types?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21535 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/607/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21535 **[Test build #92508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92508/testReport)** for PR 21535 at commit [`250074b`](https://github.com/apache/spark/commit/250074b0377c3fbcf63ebf355b6d61c4f4f9e446). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21535 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21535: [SPARK-23596][SQL][WIP] Test interpreted path on Dataset...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21535 ping @hvanhovell @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92505/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92505/testReport)** for PR 21073 at commit [`4ee7b46`](https://github.com/apache/spark/commit/4ee7b46ef7d337c37b6198104ea37a93a5584e72). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21667: [SPARK-24691][SQL]Add new API `supportDataType` in FileF...
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/21667 @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21685: [SPARK-24707][DSTREAMS] Enable spark-kafka-streaming to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21685 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21685: [SPARK-24707][DSTREAMS] Enable spark-kafka-streaming to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21685 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21685: [SPARK-24707][DSTREAMS] Enable spark-kafka-streaming to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21685 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21685: [SPARK-24707][DSTREAMS] Enable spark-kafka-stream...
GitHub user sidhavratha opened a pull request: https://github.com/apache/spark/pull/21685 [SPARK-24707][DSTREAMS] Enable spark-kafka-streaming to maintain min ⦠â¦buffer using async thread to avoid blocking kafka poll ## What changes were proposed in this pull request? Currently Spark Kafka RDD will block on kafka consumer poll. Specially in Spark-Kafka-streaming job this poll duration adds into batch processing time which result in * Increased batch processing time (which is apart from time taken to process records) * Results in unpredictable batch processing time based on poll time. This PR consists of changes to maintain min records in buffer, so that streaming batches processing do not have to get blocked on kafka poll. ## How was this patch tested? Unit test / manual test. [Before_change.pdf](https://github.com/apache/spark/files/2152353/Before_change.pdf) [After_change_2000_buffer_per_part.pdf](https://github.com/apache/spark/files/2152354/After_change_2000_buffer_per_part.pdf) You can merge this pull request into a Git repository by running: $ git pull https://github.com/sidhavratha/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21685.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21685 commit 35d792a83f13291a99cd1bf3ce89f932614da9c0 Author: s0k00rv Date: 2018-07-01T03:00:45Z [SPARK-24707][DSTREAMS] Enable spark-kafka-streaming to maintain min buffer using async thread to avoid blocking kafka poll --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21684 **[Test build #92507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92507/testReport)** for PR 21684 at commit [`e2eaa1c`](https://github.com/apache/spark/commit/e2eaa1c47d16549713da9cca7cce06552c384c0c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/606/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21684 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21684: [SPARK-24470][Core] RestSubmissionClient to be ro...
GitHub user rekhajoshm opened a pull request:
https://github.com/apache/spark/pull/21684
[SPARK-24470][Core] RestSubmissionClient to be robust against 404 & non
json responses
## What changes were proposed in this pull request?
Added check for 404, to avoid json parsing on not found response and to
avoid returning malformed or bad request when it was a not found http response.
Not sure if I need to add an additional check on non json response
[if(connection.getHeaderField("Content-Type").contains("text/html")) then
exception] as non-json is a subset of malformed json and covered in flow.
## How was this patch tested?
./dev/run-tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rekhajoshm/spark SPARK-24470
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21684.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21684
commit e3677c9fa9697e0d34f9df52442085a6a481c9e9
Author: Rekha Joshi
Date: 2015-05-05T23:10:08Z
Merge pull request #1 from apache/master
Pulling functionality from apache spark
commit 106fd8eee8f6a6f7c67cfc64f57c1161f76d8f75
Author: Rekha Joshi
Date: 2015-05-08T21:49:09Z
Merge pull request #2 from apache/master
pull latest from apache spark
commit 0be142d6becba7c09c6eba0b8ea1efe83d649e8c
Author: Rekha Joshi
Date: 2015-06-22T00:08:08Z
Merge pull request #3 from apache/master
Pulling functionality from apache spark
commit 6c6ee12fd733e3f9902e10faf92ccb78211245e3
Author: Rekha Joshi
Date: 2015-09-17T01:03:09Z
Merge pull request #4 from apache/master
Pulling functionality from apache spark
commit b123c601e459d1ad17511fd91dd304032154882a
Author: Rekha Joshi
Date: 2015-11-25T18:50:32Z
Merge pull request #5 from apache/master
pull request from apache/master
commit c73c32aadd6066e631956923725a48d98a18777e
Author: Rekha Joshi
Date: 2016-03-18T19:13:51Z
Merge pull request #6 from apache/master
pull latest from apache spark
commit 7dbf7320057978526635bed09dabc8cf8657a28a
Author: Rekha Joshi
Date: 2016-04-05T20:26:40Z
Merge pull request #8 from apache/master
pull latest from apache spark
commit 5e9d71827f8e2e4d07027281b80e4e073e7fecd1
Author: Rekha Joshi
Date: 2017-05-01T23:00:30Z
Merge pull request #9 from apache/master
Pull apache spark
commit 63d99b3ce5f222d7126133170a373591f0ac67dd
Author: Rekha Joshi
Date: 2017-09-30T22:26:44Z
Merge pull request #10 from apache/master
pull latest apache spark
commit a7fc787466b71784ff86f9694f617db0f1042da8
Author: Rekha Joshi
Date: 2018-01-21T00:17:58Z
Merge pull request #11 from apache/master
Apache spark pull latest
commit 3a2d45377ed4397de802badd764bc2588cfd275b
Author: Rekha Joshi
Date: 2018-02-09T04:55:12Z
Merge pull request #12 from apache/master
Apache spark latest pull
commit dca3a9e6f4c5fb6d34aeac7d2579c24102834fc2
Author: Rekha Joshi
Date: 2018-05-31T22:27:26Z
Merge pull request #13 from apache/master
apache spark latest pull
commit fe58c3770814543dbe923420f0bf77a74696f8d4
Author: Rekha Joshi
Date: 2018-06-21T05:59:44Z
Merge remote-tracking branch 'upstream/master'
commit ae51f60e9451b4b6be7200f6f142e871c38a1a8a
Author: Rekha Joshi
Date: 2018-06-21T06:10:45Z
Merge pull request #14 from apache/master
latest apache spark
commit 1cf4ed2df52fbb1404ca8ddf015ac799989b7d62
Author: Rekha Joshi
Date: 2018-06-30T17:17:49Z
Merge branch 'master' of https://github.com/rekhajoshm/spark
commit 1c48d4f6e49e5c818609ae1e82ed48ac8426a6d0
Author: Rekha Joshi
Date: 2018-06-30T17:20:10Z
Merge pull request #15 from apache/master
apache spark latest
commit aa735f5e1680b9abf23ce8e40b0b8436fdb61e2b
Author: Rekha Joshi
Date: 2018-06-30T17:21:09Z
Merge branch 'master' of https://github.com/rekhajoshm/spark
commit e2eaa1c47d16549713da9cca7cce06552c384c0c
Author: Rekha Joshi
Date: 2018-07-01T01:39:45Z
[SPARK-24470][Core] RestSubmissionClient to be robust against 404 & non
json responses
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21671: [SPARK-24682] [SQL] from_json / to_json now handl...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21671#discussion_r199336223
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
---
@@ -317,16 +292,52 @@ class JacksonParser(
row
}
+ private def parseTimestamp(stringValue: String): Long = {
+// This one will lose microseconds parts.
+// See https://issues.apache.org/jira/browse/SPARK-10681.x
+Try(options.timestampFormat.parse(stringValue).getTime * 1000L)
+ .getOrElse {
+// If it fails to parse, then tries the way used in 2.0 and 1.x
for backwards
+// compatibility.
+DateTimeUtils.stringToTime(stringValue).getTime * 1000L
+ }
+ }
+
+ private def parseDate(stringValue: String): Int = {
+
Try(DateTimeUtils.millisToDays(options.dateFormat.parse(stringValue).getTime))
+ .orElse {
+// If it fails to parse, then tries the way used in 2.0 and 1.x
for backwards
+// compatibility.
+
Try(DateTimeUtils.millisToDays(DateTimeUtils.stringToTime(stringValue).getTime))
+ }
+ .getOrElse {
+// In Spark 1.5.0, we store the data as number of days since epoch
in string.
+// So, we just convert it to Int.
+stringValue.toInt
+ }
+ }
+
/**
* Parse an object as a Map, preserving all fields.
*/
private def convertMap(
parser: JsonParser,
+ keyType: DataType,
fieldConverter: ValueConverter): MapData = {
-val keys = ArrayBuffer.empty[UTF8String]
+val keys = ArrayBuffer.empty[Any]
val values = ArrayBuffer.empty[Any]
while (nextUntil(parser, JsonToken.END_OBJECT)) {
- keys += UTF8String.fromString(parser.getCurrentName)
+
+ val keyValue = keyType match {
+case DateType =>
+ parseDate(parser.getCurrentName)
+case TimestampType =>
+ parseTimestamp(parser.getCurrentName)
+case _ =>
+ UTF8String.fromString(parser.getCurrentName)
+ }
+
+ keys += keyValue
--- End diff --
Yea, that maybe can be not blocked. However, one thing I am a bit hesitant
of is if we can really support other types with a min fix, and if it's worth.
Otherwise I should see if it's worth of a big change. Let me check again and
left a comment soon again.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21640: [SPARK-24654][BUILD] Update, fix LICENSE and NOTI...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21640 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21681: Pin tag 210
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21681 Close this @zhangchj1990 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21640: [SPARK-24654][BUILD] Update, fix LICENSE and NOTICE, and...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21640 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21660: [SPARK-24683][K8S] Fix k8s no resource
Github user rekhajoshm commented on a diff in the pull request:
https://github.com/apache/spark/pull/21660#discussion_r199335279
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -21,17 +21,17 @@ import java.nio.file.{Path, Paths}
import java.util.UUID
import java.util.regex.Pattern
-import scala.collection.JavaConverters._
-
import com.google.common.io.PatternFilenameFilter
import io.fabric8.kubernetes.api.model.{Container, Pod}
import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
import org.scalatest.concurrent.{Eventually, PatienceConfiguration}
import org.scalatest.time.{Minutes, Seconds, Span}
+import scala.collection.JavaConverters._
--- End diff --
this order does not seem correct.could be an unintentional move.please
check.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21683: [SPARK-24507][Documentation] Update streaming guide
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21683 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21683: [SPARK-24507][Documentation] Update streaming guide
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21683 **[Test build #92506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92506/testReport)** for PR 21683 at commit [`61bec7b`](https://github.com/apache/spark/commit/61bec7bd9cd0fc6a87eaadbc3336df7274cd373d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21683: [SPARK-24507][Documentation] Update streaming guide
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21683 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92506/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21683: [SPARK-24507][Documentation] Update streaming guide
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21683 **[Test build #92506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92506/testReport)** for PR 21683 at commit [`61bec7b`](https://github.com/apache/spark/commit/61bec7bd9cd0fc6a87eaadbc3336df7274cd373d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21683: [SPARK-24507][Documentation] Update streaming guide
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21683 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/605/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21683: [SPARK-24507][Documentation] Update streaming guide
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21683 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21683: [SPARK-24507][Documentation] Update streaming gui...
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/21683 [SPARK-24507][Documentation] Update streaming guide ## What changes were proposed in this pull request? Updated streaming guide for direct stream and link to integration guide. ## How was this patch tested? jekyll build You can merge this pull request into a Git repository by running: $ git pull https://github.com/rekhajoshm/spark SPARK-24507 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21683.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21683 commit e3677c9fa9697e0d34f9df52442085a6a481c9e9 Author: Rekha Joshi Date: 2015-05-05T23:10:08Z Merge pull request #1 from apache/master Pulling functionality from apache spark commit 106fd8eee8f6a6f7c67cfc64f57c1161f76d8f75 Author: Rekha Joshi Date: 2015-05-08T21:49:09Z Merge pull request #2 from apache/master pull latest from apache spark commit 0be142d6becba7c09c6eba0b8ea1efe83d649e8c Author: Rekha Joshi Date: 2015-06-22T00:08:08Z Merge pull request #3 from apache/master Pulling functionality from apache spark commit 6c6ee12fd733e3f9902e10faf92ccb78211245e3 Author: Rekha Joshi Date: 2015-09-17T01:03:09Z Merge pull request #4 from apache/master Pulling functionality from apache spark commit b123c601e459d1ad17511fd91dd304032154882a Author: Rekha Joshi Date: 2015-11-25T18:50:32Z Merge pull request #5 from apache/master pull request from apache/master commit c73c32aadd6066e631956923725a48d98a18777e Author: Rekha Joshi Date: 2016-03-18T19:13:51Z Merge pull request #6 from apache/master pull latest from apache spark commit 7dbf7320057978526635bed09dabc8cf8657a28a Author: Rekha Joshi Date: 2016-04-05T20:26:40Z Merge pull request #8 from apache/master pull latest from apache spark commit 5e9d71827f8e2e4d07027281b80e4e073e7fecd1 Author: Rekha Joshi Date: 2017-05-01T23:00:30Z Merge pull request #9 from apache/master Pull apache spark commit 63d99b3ce5f222d7126133170a373591f0ac67dd Author: Rekha Joshi Date: 2017-09-30T22:26:44Z Merge pull request #10 from apache/master pull latest apache spark commit a7fc787466b71784ff86f9694f617db0f1042da8 Author: Rekha Joshi Date: 2018-01-21T00:17:58Z Merge pull request #11 from apache/master Apache spark pull latest commit 3a2d45377ed4397de802badd764bc2588cfd275b Author: Rekha Joshi Date: 2018-02-09T04:55:12Z Merge pull request #12 from apache/master Apache spark latest pull commit dca3a9e6f4c5fb6d34aeac7d2579c24102834fc2 Author: Rekha Joshi Date: 2018-05-31T22:27:26Z Merge pull request #13 from apache/master apache spark latest pull commit fe58c3770814543dbe923420f0bf77a74696f8d4 Author: Rekha Joshi Date: 2018-06-21T05:59:44Z Merge remote-tracking branch 'upstream/master' commit ae51f60e9451b4b6be7200f6f142e871c38a1a8a Author: Rekha Joshi Date: 2018-06-21T06:10:45Z Merge pull request #14 from apache/master latest apache spark commit 1cf4ed2df52fbb1404ca8ddf015ac799989b7d62 Author: Rekha Joshi Date: 2018-06-30T17:17:49Z Merge branch 'master' of https://github.com/rekhajoshm/spark commit 1c48d4f6e49e5c818609ae1e82ed48ac8426a6d0 Author: Rekha Joshi Date: 2018-06-30T17:20:10Z Merge pull request #15 from apache/master apache spark latest commit aa735f5e1680b9abf23ce8e40b0b8436fdb61e2b Author: Rekha Joshi Date: 2018-06-30T17:21:09Z Merge branch 'master' of https://github.com/rekhajoshm/spark commit 61bec7bd9cd0fc6a87eaadbc3336df7274cd373d Author: Rekha Joshi Date: 2018-06-30T23:45:39Z [SPARK-24507][Docs] Update streaming guide --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92505/testReport)** for PR 21073 at commit [`4ee7b46`](https://github.com/apache/spark/commit/4ee7b46ef7d337c37b6198104ea37a93a5584e72). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21073 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92502/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92502 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92502/testReport)** for PR 21073 at commit [`484ac17`](https://github.com/apache/spark/commit/484ac173d253efcf83a86edc02f61cc3cba57d02). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21677 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21677 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92504/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21677 **[Test build #92504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92504/testReport)** for PR 21677 at commit [`ec62e13`](https://github.com/apache/spark/commit/ec62e13b9034ca81002dffe07a5ffbd1b425240a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21680: [SPARK-24704][WebUI] Fix the order of stages in the DAG ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92501/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21680: [SPARK-24704][WebUI] Fix the order of stages in the DAG ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21680 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21680: [SPARK-24704][WebUI] Fix the order of stages in the DAG ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21680 **[Test build #92501 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92501/testReport)** for PR 21680 at commit [`b3420d6`](https://github.com/apache/spark/commit/b3420d61025f7bb9e17160dfb586bc54fba1a51d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21682: [SPARK-24706][SQL] ByteType and ShortType support pushdo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21682 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92503/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21682: [SPARK-24706][SQL] ByteType and ShortType support pushdo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21682 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21682: [SPARK-24706][SQL] ByteType and ShortType support pushdo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21682 **[Test build #92503 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92503/testReport)** for PR 21682 at commit [`e9d5625`](https://github.com/apache/spark/commit/e9d56252e6c65f5afa207bc98c8c5e008de57e0c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21671: [SPARK-24682] [SQL] from_json / to_json now handl...
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21671#discussion_r199331278
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
---
@@ -317,16 +292,52 @@ class JacksonParser(
row
}
+ private def parseTimestamp(stringValue: String): Long = {
+// This one will lose microseconds parts.
+// See https://issues.apache.org/jira/browse/SPARK-10681.x
+Try(options.timestampFormat.parse(stringValue).getTime * 1000L)
+ .getOrElse {
+// If it fails to parse, then tries the way used in 2.0 and 1.x
for backwards
+// compatibility.
+DateTimeUtils.stringToTime(stringValue).getTime * 1000L
+ }
+ }
+
+ private def parseDate(stringValue: String): Int = {
+
Try(DateTimeUtils.millisToDays(options.dateFormat.parse(stringValue).getTime))
+ .orElse {
+// If it fails to parse, then tries the way used in 2.0 and 1.x
for backwards
+// compatibility.
+
Try(DateTimeUtils.millisToDays(DateTimeUtils.stringToTime(stringValue).getTime))
+ }
+ .getOrElse {
+// In Spark 1.5.0, we store the data as number of days since epoch
in string.
+// So, we just convert it to Int.
+stringValue.toInt
+ }
+ }
+
/**
* Parse an object as a Map, preserving all fields.
*/
private def convertMap(
parser: JsonParser,
+ keyType: DataType,
fieldConverter: ValueConverter): MapData = {
-val keys = ArrayBuffer.empty[UTF8String]
+val keys = ArrayBuffer.empty[Any]
val values = ArrayBuffer.empty[Any]
while (nextUntil(parser, JsonToken.END_OBJECT)) {
- keys += UTF8String.fromString(parser.getCurrentName)
+
+ val keyValue = keyType match {
+case DateType =>
+ parseDate(parser.getCurrentName)
+case TimestampType =>
+ parseTimestamp(parser.getCurrentName)
+case _ =>
+ UTF8String.fromString(parser.getCurrentName)
+ }
+
+ keys += keyValue
--- End diff --
I don't think that support of all types as key types must block this PR.
You can support only date and timestamp types in this PR, and postpone generic
solution for the near future. @HyukjinKwon @viirya What do you think of that?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21677 **[Test build #92504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92504/testReport)** for PR 21677 at commit [`ec62e13`](https://github.com/apache/spark/commit/ec62e13b9034ca81002dffe07a5ffbd1b425240a). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21677 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21677 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/604/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21682: [SPARK-24706][SQL] ByteType and ShortType support pushdo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21682 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/603/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21682: [SPARK-24706][SQL] ByteType and ShortType support pushdo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21682 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21682: [SPARK-24706][SQL] ByteType and ShortType support pushdo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21682 **[Test build #92503 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92503/testReport)** for PR 21682 at commit [`e9d5625`](https://github.com/apache/spark/commit/e9d56252e6c65f5afa207bc98c8c5e008de57e0c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21682: [SPARK-24706][SQL] ByteType and ShortType support...
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/21682 [SPARK-24706][SQL] ByteType and ShortType support pushdown to parquet ## What changes were proposed in this pull request? `ByteType` and `ShortType` support pushdown to parquet data source. ## How was this patch tested? unit tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangyum/spark SPARK-24706 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21682.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21682 commit e9d56252e6c65f5afa207bc98c8c5e008de57e0c Author: Yuming Wang Date: 2018-06-30T19:13:13Z ByteType and ShortType pushdown to parquet --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21671: [SPARK-24682] [SQL] from_json / to_json now handl...
Github user patrickmcgloin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21671#discussion_r199330240
--- Diff: python/pyspark/sql/functions.py ---
@@ -2163,9 +2163,9 @@ def json_tuple(col, *fields):
@since(2.1)
def from_json(col, schema, options={}):
"""
-Parses a column containing a JSON string into a :class:`MapType` with
:class:`StringType`
-as keys type, :class:`StructType` or :class:`ArrayType` of
:class:`StructType`\\s with
-the specified schema. Returns `null`, in the case of an unparseable
string.
+Parses a column containing a JSON string into a :class:`MapType`,
:class:`StructType`
+or :class:`ArrayType` of :class:`StructType`\\s with the specified
schema. Returns
+`null`, in the case of an unparseable string.
--- End diff --
I think that other basic types (e.g Int, Long, etc) were already supported
along with String. As long as they can be directly converted from String to
that type specified in the schema then they were ok. What we have added here
is parsing on the DateType and TimestampType. I don't think complex types
would ever be supported. And I don't think other basic types need the parsing
that date/times do.
Do you think we should specify which types are supported as keys?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and 2.12.6
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21495 In addition to that, it woud be great if we fix `sbt` soon. After this PR, `mvn` works correctly, but `sbt` is still hitting `NoSuchMethodError` in `master` branch. ```scala $ ./build/sbt -Pyarn -Phadoop-2.7 -Phadoop-cloud -Phive -Phive-thriftserver -Psparkr test:package $ bin/spark-shell scala> Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1530385877441). Spark session available as 'spark'. Exception in thread "main" java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21671: [SPARK-24682] [SQL] from_json / to_json now handl...
Github user patrickmcgloin commented on a diff in the pull request: https://github.com/apache/spark/pull/21671#discussion_r199330156 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala --- @@ -43,6 +43,11 @@ private[sql] class JacksonGenerator( // we can directly access data in `ArrayData` without the help of `SpecificMutableRow`. private type ValueWriter = (SpecializedGetters, Int) => Unit + // A `ValueReader` is responsible for reading a field of an `InternalRow` to a String. + // The only purpose of this is to read the key values for a map so that they can be + // written as JSON filed names. --- End diff -- Fixed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21672: [SPARK-24694][K8S] Pass all app args to integration test...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21672 @felixcheung we could call a test with more than one parameters like: https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/MultiBroadcastTest.scala --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21623: [SPARK-24638][SQL] StringStartsWith support push down
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/21623 @cloud-fan . AFAIK, ORC doesn't support custom filter yet. I'll follow up that stuff in ORC. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92502/testReport)** for PR 21073 at commit [`484ac17`](https://github.com/apache/spark/commit/484ac173d253efcf83a86edc02f61cc3cba57d02). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21649 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21649 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92500/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21649 **[Test build #92500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92500/testReport)** for PR 21649 at commit [`d323dd0`](https://github.com/apache/spark/commit/d323dd00916957abb48a31440c3274c3877e5401). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21649: [SPARK-23648][R][SQL]Adds more types for hint in ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21649#discussion_r199328283
--- Diff: R/pkg/R/DataFrame.R ---
@@ -3905,6 +3905,16 @@ setMethod("rollup",
groupedData(sgd)
})
+isTypeAllowedForSqlHint <- function(x) {
+ if (is.character(x) | is.numeric(x)) {
--- End diff --
nit: very minor, both is.* returns a single element, but with `||` you get
shortcut eval if is.character(x) is TRUE
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21649: [SPARK-23648][R][SQL]Adds more types for hint in ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21649#discussion_r199328450
--- Diff: R/pkg/R/DataFrame.R ---
@@ -3905,6 +3905,16 @@ setMethod("rollup",
groupedData(sgd)
})
+isTypeAllowedForSqlHint <- function(x) {
+ if (is.character(x) | is.numeric(x)) {
+TRUE
+ } else if (is.list(x)) {
+all (sapply(x, (function (y) is.character(y) | is.numeric(y
--- End diff --
also, no space after `all` - it's really `all()`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21649: [SPARK-23648][R][SQL]Adds more types for hint in ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21649#discussion_r199328269
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -2370,6 +2370,15 @@ test_that("join(), crossJoin() and merge() on a
DataFrame", {
expect_true(any(grepl("BroadcastHashJoin", execution_plan_broadcast)))
})
+test_that("test hint", {
--- End diff --
ok
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21649: [SPARK-23648][R][SQL]Adds more types for hint in ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21649#discussion_r199328430
--- Diff: R/pkg/R/DataFrame.R ---
@@ -3905,6 +3905,16 @@ setMethod("rollup",
groupedData(sgd)
})
+isTypeAllowedForSqlHint <- function(x) {
+ if (is.character(x) | is.numeric(x)) {
+TRUE
+ } else if (is.list(x)) {
+all (sapply(x, (function (y) is.character(y) | is.numeric(y
--- End diff --
ditto here
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21649: [SPARK-23648][R][SQL]Adds more types for hint in ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21649#discussion_r199328470
--- Diff: R/pkg/R/DataFrame.R ---
@@ -3905,6 +3905,16 @@ setMethod("rollup",
groupedData(sgd)
})
+isTypeAllowedForSqlHint <- function(x) {
+ if (is.character(x) | is.numeric(x)) {
+TRUE
+ } else if (is.list(x)) {
+all (sapply(x, (function (y) is.character(y) | is.numeric(y
+ } else {
+FALSE
+ }
+}
+
#' hint
#'
#' Specifies execution plan hint and return a new SparkDataFrame.
--- End diff --
does scala have any doc on hint type supported?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21640: [SPARK-24654][BUILD] Update, fix LICENSE and NOTICE, and...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21640 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21640: [SPARK-24654][BUILD] Update, fix LICENSE and NOTICE, and...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21640 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92499/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21640: [SPARK-24654][BUILD] Update, fix LICENSE and NOTICE, and...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21640 **[Test build #92499 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92499/testReport)** for PR 21640 at commit [`198b208`](https://github.com/apache/spark/commit/198b2082019d0a17ea327d03816cc7d496df7443). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21631: [SPARK-24645][SQL] Skip parsing when csvColumnPruning en...
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21631 The bug has been already fixed in uniVocity `2.6.5-SNAPSHOT` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21680: [SPARK-24704][WebUI] Fix the order of stages in the DAG ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21680 **[Test build #92501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92501/testReport)** for PR 21680 at commit [`b3420d6`](https://github.com/apache/spark/commit/b3420d61025f7bb9e17160dfb586bc54fba1a51d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21680: [SPARK-24704][WebUI] Fix the order of stages in the DAG ...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/21680 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21649 **[Test build #92500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92500/testReport)** for PR 21649 at commit [`d323dd0`](https://github.com/apache/spark/commit/d323dd00916957abb48a31440c3274c3877e5401). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21649 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21649: [SPARK-23648][R][SQL]Adds more types for hint in SparkR
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21649 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/602/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21649: [SPARK-23648][R][SQL]Adds more types for hint in ...
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/21649#discussion_r199327384
--- Diff: R/pkg/R/DataFrame.R ---
@@ -3905,6 +3905,18 @@ setMethod("rollup",
groupedData(sgd)
})
+isTypeAllowed <- function(x) {
+ if (is.character(x)) {
+TRUE
+ } else if (is.list(x)) {
--- End diff --
Thanks. I will make changes.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21678: [SPARK-23461][R]vignettes should include model predictio...
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/21678 Here is the output for Linear SVM Classifier in sparkr-vignettes.html. ``` prediction <- predict(model, training) head(select(prediction, "Class", "Sex", "Age", "Freq", "Survived", "prediction")) ## ClassSex Age Freq Survived prediction ## 1 1st Male Child0 NoYes ## 2 2nd Male Child0 NoYes ## 3 3rd Male Child 35 NoYes ## 4 Crew Male Child0 NoYes ## 5 1st Female Child0 NoYes ## 6 2nd Female Child0 NoYes ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21681: Pin tag 210
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21681 @zhangchj1990 Looks mistakenly open. Mind closing this please? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21640: [SPARK-24654][BUILD] Update, fix LICENSE and NOTICE, and...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21640 **[Test build #92499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92499/testReport)** for PR 21640 at commit [`198b208`](https://github.com/apache/spark/commit/198b2082019d0a17ea327d03816cc7d496df7443). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21677 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92498/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21677 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21648: [SPARK-24665][PySpark] Use SQLConf in PySpark to manage ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21648 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21648: [SPARK-24665][PySpark] Use SQLConf in PySpark to manage ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21648 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92497/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21648: [SPARK-24665][PySpark] Use SQLConf in PySpark to manage ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21648 **[Test build #92497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92497/testReport)** for PR 21648 at commit [`4fc0ae4`](https://github.com/apache/spark/commit/4fc0ae4410edcf9890a89bb0b47d0d633bfb3dda). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21648: [SPARK-24665][PySpark] Use SQLConf in PySpark to manage ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21648 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21648: [SPARK-24665][PySpark] Use SQLConf in PySpark to manage ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21648 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92496/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21648: [SPARK-24665][PySpark] Use SQLConf in PySpark to manage ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21648 **[Test build #92496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92496/testReport)** for PR 21648 at commit [`b816549`](https://github.com/apache/spark/commit/b8165495f2ad68d54f3f2ba6b4247d58825bd41c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21671: [SPARK-24682] [SQL] from_json / to_json now handl...
Github user patrickmcgloin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21671#discussion_r199319281
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -163,6 +175,22 @@ private[sql] class JacksonGenerator(
s"with the type of $dataType to JSON.")
}
+ private def makeReader(dataType: DataType): ValueReader = dataType match
{
+
+case TimestampType =>
+ (row: SpecializedGetters, ordinal: Int) =>
+
options.timestampFormat.format(DateTimeUtils.toJavaTimestamp(row.getLong(ordinal)))
+
+case DateType =>
+ (row: SpecializedGetters, ordinal: Int) =>
+
options.dateFormat.format(DateTimeUtils.toJavaDate(row.getInt(ordinal)))
+
+case _ =>
+ (row: SpecializedGetters, ordinal: Int) =>
+row.get(ordinal, dataType).toString
--- End diff --
Actually, now that I consider it a bit more I think nested types are not
possible. The result has to be stored as a String, e.g.
"key": "value"
Putting the nested type inside the name of the JSON wouldn't be possible.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21671: [SPARK-24682] [SQL] from_json / to_json now handl...
Github user patrickmcgloin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21671#discussion_r199319246
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -163,6 +175,22 @@ private[sql] class JacksonGenerator(
s"with the type of $dataType to JSON.")
}
+ private def makeReader(dataType: DataType): ValueReader = dataType match
{
+
+case TimestampType =>
+ (row: SpecializedGetters, ordinal: Int) =>
+
options.timestampFormat.format(DateTimeUtils.toJavaTimestamp(row.getLong(ordinal)))
+
+case DateType =>
+ (row: SpecializedGetters, ordinal: Int) =>
+
options.dateFormat.format(DateTimeUtils.toJavaDate(row.getInt(ordinal)))
+
+case _ =>
+ (row: SpecializedGetters, ordinal: Int) =>
+row.get(ordinal, dataType).toString
--- End diff --
I considered this the default case and thought it would be covered by
existing Unit Tests. Which types should I add tests for? Int, String, Long,
etc. All of the children of AtomicType?
Nested types is interesting. I will add some tests for keys with a
Seq[Int], Map[String, Int] to see that they work. If there was something else
you were thinking of please let me know.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21671: [SPARK-24682] [SQL] from_json / to_json now handl...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21671#discussion_r199317878 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala --- @@ -43,6 +43,11 @@ private[sql] class JacksonGenerator( // we can directly access data in `ArrayData` without the help of `SpecificMutableRow`. private type ValueWriter = (SpecializedGetters, Int) => Unit + // A `ValueReader` is responsible for reading a field of an `InternalRow` to a String. + // The only purpose of this is to read the key values for a map so that they can be + // written as JSON filed names. --- End diff -- filed -> field? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21671: [SPARK-24682] [SQL] from_json / to_json now handl...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21671#discussion_r199317586
--- Diff: python/pyspark/sql/functions.py ---
@@ -2163,9 +2163,9 @@ def json_tuple(col, *fields):
@since(2.1)
def from_json(col, schema, options={}):
"""
-Parses a column containing a JSON string into a :class:`MapType` with
:class:`StringType`
-as keys type, :class:`StructType` or :class:`ArrayType` of
:class:`StructType`\\s with
-the specified schema. Returns `null`, in the case of an unparseable
string.
+Parses a column containing a JSON string into a :class:`MapType`,
:class:`StructType`
+or :class:`ArrayType` of :class:`StructType`\\s with the specified
schema. Returns
+`null`, in the case of an unparseable string.
--- End diff --
Do we support all data types as key in `MapType`? With this change, besides
`StringType`, we only support `DateType` and `TimestampType`, right?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21677 **[Test build #92498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92498/testReport)** for PR 21677 at commit [`be5d219`](https://github.com/apache/spark/commit/be5d2197d3d3ff06c5f80ed66d5a0bb20924e130). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21677 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21677: [SPARK-24692][TESTS] Improvement FilterPushdownBenchmark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21677 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/600/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21681: Pin tag 210
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21681 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21681: Pin tag 210
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21681 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21681: Pin tag 210
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21681 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21681: Pin tag 210
GitHub user zhangchj1990 opened a pull request: https://github.com/apache/spark/pull/21681 Pin tag 210 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhangchj1990/spark pin-tag-210 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21681.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21681 commit 6b6eb4e520d07a27aa68d3450f3c7613b233d928 Author: Zheng RuiFeng Date: 2016-11-16T10:46:27Z [SPARK-18434][ML] Add missing ParamValidations for ML algos ## What changes were proposed in this pull request? Add missing ParamValidations for ML algos ## How was this patch tested? existing tests Author: Zheng RuiFeng Closes #15881 from zhengruifeng/arg_checking. (cherry picked from commit c68f1a38af67957ee28889667193da8f64bb4342) Signed-off-by: Yanbo Liang commit 416bc3dd3db7f7ae2cc7b3ffe395decd0c5b73f9 Author: Zheng RuiFeng Date: 2016-11-16T10:53:23Z [SPARK-18446][ML][DOCS] Add links to API docs for ML algos ## What changes were proposed in this pull request? Add links to API docs for ML algos ## How was this patch tested? Manual checking for the API links Author: Zheng RuiFeng Closes #15890 from zhengruifeng/algo_link. (cherry picked from commit a75e3fe923372c56bc1b2f4baeaaf5868ad28341) Signed-off-by: Sean Owen commit b0ae8712358fc8c07aa5efe4d0bd337e7e452078 Author: Xianyang Liu Date: 2016-11-16T11:59:00Z [SPARK-18420][BUILD] Fix the errors caused by lint check in Java Small fix, fix the errors caused by lint check in Java - Clear unused objects and `UnusedImports`. - Add comments around the method `finalize` of `NioBufferedFileInputStream`to turn off checkstyle. - Cut the line which is longer than 100 characters into two lines. Travis CI. ``` $ build/mvn -T 4 -q -DskipTests -Pyarn -Phadoop-2.3 -Pkinesis-asl -Phive -Phive-thriftserver install $ dev/lint-java ``` Before: ``` Checkstyle checks failed at following occurrences: [ERROR] src/main/java/org/apache/spark/network/util/TransportConf.java:[21,8] (imports) UnusedImports: Unused import - org.apache.commons.crypto.cipher.CryptoCipherFactory. [ERROR] src/test/java/org/apache/spark/network/sasl/SparkSaslSuite.java:[516,5] (modifier) RedundantModifier: Redundant 'public' modifier. [ERROR] src/main/java/org/apache/spark/io/NioBufferedFileInputStream.java:[133] (coding) NoFinalizer: Avoid using finalizer method. [ERROR] src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeMapData.java:[71] (sizes) LineLength: Line is longer than 100 characters (found 113). [ERROR] src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java:[112] (sizes) LineLength: Line is longer than 100 characters (found 110). [ERROR] src/test/java/org/apache/spark/sql/catalyst/expressions/HiveHasherSuite.java:[31,17] (modifier) ModifierOrder: 'static' modifier out of order with the JLS suggestions. [ERROR]src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionWithElasticNetExample.java:[64] (sizes) LineLength: Line is longer than 100 characters (found 103). [ERROR] src/main/java/org/apache/spark/examples/ml/JavaInteractionExample.java:[22,8] (imports) UnusedImports: Unused import - org.apache.spark.ml.linalg.Vectors. [ERROR] src/main/java/org/apache/spark/examples/ml/JavaInteractionExample.java:[51] (regexp) RegexpSingleline: No trailing whitespace allowed. ``` After: ``` $ build/mvn -T 4 -q -DskipTests -Pyarn -Phadoop-2.3 -Pkinesis-asl -Phive -Phive-thriftserver install $ dev/lint-java Using `mvn` from path: /home/travis/build/ConeyLiu/spark/build/apache-maven-3.3.9/bin/mvn Checkstyle checks passed. ``` Author: Xianyang Liu Closes #15865 from ConeyLiu/master. (cherry picked from commit 7569cf6cb85bda7d0e76d3e75e286d4796e77e08) Signed-off-by: Sean Owen commit c0dbe08d604dea543eb17ccb802a8a20d6c21a69 Author: gatorsmile Date: 2016-11-16T16:25:15Z [SPARK-18415][SQL] Weird Plan Output when CTE used in RunnableCommand ### What changes were proposed in this pull request? Currently, when CTE is used in RunnableCommand, the Analyzer does not replace the logical n
