[spark] 01/01: change SparkR maintainer
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch r-maintainer
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 5dbe3bbc5d29e2ba6caca022c37eb0b5a2f13202
Author: Felix Cheung
AuthorDate: Sun May 23 16:49:27 2021 -0700
change SparkR maintainer
---
R/pkg/DESCRIPTION | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index f55286d..4a310e6 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -3,11 +3,11 @@ Type: Package
Version: 3.2.0
Title: R Front End for 'Apache Spark'
Description: Provides an R Front end for 'Apache Spark'
<https://spark.apache.org>.
-Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
+Authors@R: c(person("Shivaram", "Venkataraman", role = "aut",
email = "shiva...@cs.berkeley.edu"),
person("Xiangrui", "Meng", role = "aut",
email = "m...@databricks.com"),
- person("Felix", "Cheung", role = "aut",
+ person("Felix", "Cheung", role = c("aut", "cre"),
email = "felixche...@apache.org"),
person(family = "The Apache Software Foundation", role = c("aut",
"cph")))
License: Apache License (== 2.0)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch r-maintainer created (now 5dbe3bb)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a change to branch r-maintainer in repository https://gitbox.apache.org/repos/asf/spark.git. at 5dbe3bb change SparkR maintainer This branch includes the following new commits: new 5dbe3bb change SparkR maintainer The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Add test-hadoop3.2 phrase (#203)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 16052cb Add test-hadoop3.2 phrase (#203) 16052cb is described below commit 16052cbd63d17ce4d74ee2ca302e0a8532d46e80 Author: Yuming Wang AuthorDate: Mon May 13 06:53:29 2019 +0800 Add test-hadoop3.2 phrase (#203) --- developer-tools.md | 1 + 1 file changed, 1 insertion(+) diff --git a/developer-tools.md b/developer-tools.md index e5a66d8..21abf47 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -254,6 +254,7 @@ your pull request to change testing behavior. This includes: - `[test-maven]` - signals to test the pull request using maven - `[test-hadoop2.7]` - signals to test using Spark's Hadoop 2.7 profile +- `[test-hadoop3.2]` - signals to test using Spark's Hadoop 3.2 profile Binary compatibility - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[zeppelin] branch master updated: [ZEPPELIN-4026] document anonymous access
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 05fdfe5 [ZEPPELIN-4026] document anonymous access 05fdfe5 is described below commit 05fdfe5162bb9f38eb927fc68d754370b316a133 Author: Felix Cheung AuthorDate: Sun Mar 3 23:42:11 2019 -0500 [ZEPPELIN-4026] document anonymous access ### What is this PR for? doc update ### What type of PR is it? [Documentation] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-4026 ### Questions: * Does the licenses files need update? N * Is there breaking changes for older versions? N * Does this needs documentation? N --- docs/setup/security/shiro_authentication.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/setup/security/shiro_authentication.md b/docs/setup/security/shiro_authentication.md index c9bf905..40b6e53 100644 --- a/docs/setup/security/shiro_authentication.md +++ b/docs/setup/security/shiro_authentication.md @@ -28,6 +28,9 @@ limitations under the License. When you connect to Apache Zeppelin, you will be asked to enter your credentials. Once you logged in, then you have access to all notes including other user's notes. +## Important Note +By default, Zeppelin allows anonymous access. It is strongly recommended that you consider setting up Apache Shiro for authentication (as described in this document, see 2 Secure the Websocket channel), or only deploy and use Zeppelin in a secured and trusted environment. + ## Security Setup You can setup **Zeppelin notebook authentication** in some simple steps.
[zeppelin] branch master updated: [ZEPPELIN-4012] Save paragraphs after clone
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new dd624a6 [ZEPPELIN-4012] Save paragraphs after clone
dd624a6 is described below
commit dd624a67a4cf27e1778be260d964abc3df96e345
Author: Egor Klimov
AuthorDate: Wed Feb 27 08:25:23 2019 +0300
[ZEPPELIN-4012] Save paragraphs after clone
### What is this PR for?
Paragraphs in cloned note disappear after zeppelin restart if they weren't
executed
* Bug:
* Fix:
### What type of PR is it?
Bug Fix
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4012
### How should this be tested?
* CI pass
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
---
.../src/main/java/org/apache/zeppelin/notebook/Notebook.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
index ccb9b95..ab32059 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
@@ -244,6 +244,7 @@ public class Notebook {
for (Paragraph p : paragraphs) {
newNote.addCloneParagraph(p, subject);
}
+saveNote(newNote, subject);
return newNote;
}
[zeppelin] branch master updated: [ZEPPELIN-4011] Fix note path
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 6dcc2ab [ZEPPELIN-4011] Fix note path
6dcc2ab is described below
commit 6dcc2ab49c1f3d0c9ad4c8d68b005e37690faa7a
Author: Egor Klimov
AuthorDate: Wed Feb 27 08:24:01 2019 +0300
[ZEPPELIN-4011] Fix note path
### What is this PR for?
Now note may move to the root directory after enabling cron in it. It
caused by `Note#setName`
* Bug:
* Fix:
### What type of PR is it?
Bug Fix
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4011
### How should this be tested?
* CI pass
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
---
zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
index 403db06..2632e7d 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
@@ -232,7 +232,7 @@ public class Note implements JsonSerializable {
this.path = "/" + name;
}
} else {
- int pos = this.path.indexOf("/");
+ int pos = this.path.lastIndexOf("/");
this.path = this.path.substring(0, pos + 1) + this.name;
}
}
[spark] branch branch-2.3 updated: [R][BACKPORT-2.4] update package description
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.3 by this push:
new 6691c04 [R][BACKPORT-2.4] update package description
6691c04 is described below
commit 6691c041dcab4c19d362aaff74f56e5beeda85cd
Author: Felix Cheung
AuthorDate: Thu Feb 21 08:42:15 2019 -0800
[R][BACKPORT-2.4] update package description
doesn't port cleanly to 2.4. we need this in branch-2.4 and branch-2.3
Author: Felix Cheung
Closes #23860 from felixcheung/2.4rdesc.
(cherry picked from commit d8576301fd1d33675a9542791e58e7963081ce04)
Signed-off-by: Felix Cheung
---
R/pkg/DESCRIPTION | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index a82446e..136d782 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,8 +1,6 @@
Package: SparkR
Type: Package
Version: 2.3.4
-Title: R Frontend for Apache Spark
-Description: Provides an R Frontend for Apache Spark.
Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
email = "shiva...@cs.berkeley.edu"),
person("Xiangrui", "Meng", role = "aut",
@@ -11,8 +9,8 @@ Authors@R: c(person("Shivaram", "Venkataraman", role =
c("aut", "cre"),
email = "felixche...@apache.org"),
person(family = "The Apache Software Foundation", role = c("aut",
"cph")))
License: Apache License (== 2.0)
-URL: http://www.apache.org/ http://spark.apache.org/
-BugReports: http://spark.apache.org/contributing.html
+URL: https://www.apache.org/ https://spark.apache.org/
+BugReports: https://spark.apache.org/contributing.html
SystemRequirements: Java (== 8)
Depends:
R (>= 3.0),
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [R][BACKPORT-2.4] update package description
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new d8576301 [R][BACKPORT-2.4] update package description
d8576301 is described below
commit d8576301fd1d33675a9542791e58e7963081ce04
Author: Felix Cheung
AuthorDate: Thu Feb 21 08:42:15 2019 -0800
[R][BACKPORT-2.4] update package description
#23852
doesn't port cleanly to 2.4. we need this in branch-2.4 and branch-2.3
Author: Felix Cheung
Closes #23860 from felixcheung/2.4rdesc.
---
R/pkg/DESCRIPTION | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index 2361289..5e3d186 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,8 +1,8 @@
Package: SparkR
Type: Package
Version: 2.4.2
-Title: R Frontend for Apache Spark
-Description: Provides an R Frontend for Apache Spark.
+Title: R Front end for 'Apache Spark'
+Description: Provides an R Front end for 'Apache Spark'
<https://spark.apache.org>.
Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
email = "shiva...@cs.berkeley.edu"),
person("Xiangrui", "Meng", role = "aut",
@@ -11,8 +11,8 @@ Authors@R: c(person("Shivaram", "Venkataraman", role =
c("aut", "cre"),
email = "felixche...@apache.org"),
person(family = "The Apache Software Foundation", role = c("aut",
"cph")))
License: Apache License (== 2.0)
-URL: http://www.apache.org/ http://spark.apache.org/
-BugReports: http://spark.apache.org/contributing.html
+URL: https://www.apache.org/ https://spark.apache.org/
+BugReports: https://spark.apache.org/contributing.html
SystemRequirements: Java (== 8)
Depends:
R (>= 3.0),
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Update Thomas Graves Information (#179)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 6205847 Update Thomas Graves Information (#179) 6205847 is described below commit 6205847f851303dd8eee80dd25236a875ca3256a Author: Thomas Graves AuthorDate: Tue Feb 12 01:29:42 2019 -0600 Update Thomas Graves Information (#179) * Update Thomas Graves Information Update my company info. I ran jekyll build and server and it looked fine. * capitalize --- committers.md| 2 +- site/committers.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/committers.md b/committers.md index c5c4104..97b21a4 100644 --- a/committers.md +++ b/committers.md @@ -26,7 +26,7 @@ navigation: |Robert Evans|Oath| |Wenchen Fan|Databricks| |Joseph Gonzalez|UC Berkeley| -|Thomas Graves|Oath| +|Thomas Graves|NVIDIA| |Stephen Haberman|LinkedIn| |Mark Hamstra|ClearStory Data| |Seth Hendrickson|Cloudera| diff --git a/site/committers.html b/site/committers.html index 42c9171..a690ca7 100644 --- a/site/committers.html +++ b/site/committers.html @@ -276,7 +276,7 @@ Thomas Graves - Oath + NVIDIA Stephen Haberman - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[zeppelin] branch branch-0.7 updated: formatting fix for LDAP realm documentation
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch branch-0.7 in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/branch-0.7 by this push: new 7906a5c formatting fix for LDAP realm documentation 7906a5c is described below commit 7906a5c9443a33f4a21e3e93dccf06844649ce7f Author: Jake Maul <1141442+superawes...@users.noreply.github.com> AuthorDate: Sat Feb 9 22:37:26 2019 -0700 formatting fix for LDAP realm documentation spacing of "```" block causing it to be ignored, and the rest of the contents to be rendered incorrectly. ### What is this PR for? The LDAP section of the documentation for zeppelin-0.7.3 renders incorrectly, here: http://zeppelin.apache.org/docs/0.7.3/security/shiroauthentication.html It looks correct on GitHub, but not on the standalone site. This should fix it. ### What type of PR is it? [Documentation | Bug Fix] ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? none known * Does this needs documentation? no --- docs/security/shiroauthentication.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/security/shiroauthentication.md b/docs/security/shiroauthentication.md index 75a52e1..bf5e8e3 100644 --- a/docs/security/shiroauthentication.md +++ b/docs/security/shiroauthentication.md @@ -143,7 +143,8 @@ ldapRealm.contextFactory.authenticationMechanism = simple The other more flexible option is to use the LdapRealm. It allows for mapping of ldapgroups to roles and also allows for role/group based authentication into the zeppelin server. Sample configuration for this realm is given below. - ``` + +``` [main] ldapRealm=org.apache.zeppelin.realm.LdapRealm @@ -179,7 +180,7 @@ ldapRealm.allowedRolesForAuthentication = admin_role,user_role ldapRealm.permissionsByRole= user_role = *:ToDoItemsJdo:*:*, *:ToDoItem:*:*; admin_role = * securityManager.sessionManager = $sessionManager securityManager.realms = $ldapRealm - ``` +``` ### PAM [PAM](https://en.wikipedia.org/wiki/Pluggable_authentication_module) authentication support allows the reuse of existing authentication
[spark] branch master updated: [SPARK-26603][K8S] Update minikube backend
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 196ca0c [SPARK-26603][K8S] Update minikube backend
196ca0c is described below
commit 196ca0c8f5f5caa2ae588ff4a63ce314aa42aecc
Author: Stavros Kontopoulos
AuthorDate: Sun Feb 3 17:15:20 2019 -0800
[SPARK-26603][K8S] Update minikube backend
## What changes were proposed in this pull request?
- Covers latest minikube versions.
- keeps the older version support
Note: While I was facing disk pressure issues locally on machine, I noticed
minikube status command would report that everything was working fine even if
some kube-system pods were not up. I don't think the output is 100% reliable
but it is good enough for most cases.
## How was this patch tested?
Run it against latest version of minikube (v0.32.0).
Author: Stavros Kontopoulos
Closes #23520 from skonto/update-mini-backend.
---
.../backend/minikube/Minikube.scala| 64 ++
1 file changed, 54 insertions(+), 10 deletions(-)
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala
index 6494cbc..58aa177 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala
@@ -16,7 +16,6 @@
*/
package org.apache.spark.deploy.k8s.integrationtest.backend.minikube
-import java.io.File
import java.nio.file.Paths
import io.fabric8.kubernetes.client.{ConfigBuilder, DefaultKubernetesClient}
@@ -26,8 +25,14 @@ import org.apache.spark.internal.Logging
// TODO support windows
private[spark] object Minikube extends Logging {
-
private val MINIKUBE_STARTUP_TIMEOUT_SECONDS = 60
+ private val HOST_PREFIX = "host:"
+ private val KUBELET_PREFIX = "kubelet:"
+ private val APISERVER_PREFIX = "apiserver:"
+ private val KUBECTL_PREFIX = "kubectl:"
+ private val MINIKUBE_VM_PREFIX = "minikubeVM: "
+ private val MINIKUBE_PREFIX = "minikube: "
+ private val MINIKUBE_PATH = ".minikube"
def getMinikubeIp: String = {
val outputs = executeMinikube("ip")
@@ -38,12 +43,21 @@ private[spark] object Minikube extends Logging {
def getMinikubeStatus: MinikubeStatus.Value = {
val statusString = executeMinikube("status")
- .filter(line => line.contains("minikubeVM: ") ||
line.contains("minikube:"))
- .head
- .replaceFirst("minikubeVM: ", "")
- .replaceFirst("minikube: ", "")
-MinikubeStatus.unapply(statusString)
+logInfo(s"Minikube status command output:\n$statusString")
+// up to minikube version v0.30.0 use this to check for minikube status
+val oldMinikube = statusString
+ .filter(line => line.contains(MINIKUBE_VM_PREFIX) ||
line.contains(MINIKUBE_PREFIX))
+
+if (oldMinikube.isEmpty) {
+ getIfNewMinikubeStatus(statusString)
+} else {
+ val finalStatusString = oldMinikube
+.head
+.replaceFirst(MINIKUBE_VM_PREFIX, "")
+.replaceFirst(MINIKUBE_PREFIX, "")
+ MinikubeStatus.unapply(finalStatusString)
.getOrElse(throw new IllegalStateException(s"Unknown status
$statusString"))
+}
}
def getKubernetesClient: DefaultKubernetesClient = {
@@ -52,13 +66,43 @@ private[spark] object Minikube extends Logging {
val kubernetesConf = new ConfigBuilder()
.withApiVersion("v1")
.withMasterUrl(kubernetesMaster)
- .withCaCertFile(Paths.get(userHome, ".minikube",
"ca.crt").toFile.getAbsolutePath)
- .withClientCertFile(Paths.get(userHome, ".minikube",
"apiserver.crt").toFile.getAbsolutePath)
- .withClientKeyFile(Paths.get(userHome, ".minikube",
"apiserver.key").toFile.getAbsolutePath)
+ .withCaCertFile(Paths.get(userHome, MINIKUBE_PATH,
"ca.crt").toFile.getAbsolutePath)
+ .withClientCertFile(Paths.get(userHome, MINIKUBE_PATH,
"apiserver.crt").toFile.getAbsolutePath)
+ .withClientKeyFile(Paths.get(userHome, MINIKUBE_PATH,
"apiserver.key").toFile.getAbsolutePath)
.build()
new DefaultKubernetesClient(kubernetesConf)
}
+ // Covers minikube status output after Minikube V0.30.
[spark] branch master updated: [SPARK-26642][K8S] Add --num-executors option to spark-submit for Spark on K8S.
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6c18d8d [SPARK-26642][K8S] Add --num-executors option to spark-submit
for Spark on K8S.
6c18d8d is described below
commit 6c18d8d8079ac4d2d6dc7539601ab83fc5b51760
Author: Luca Canali
AuthorDate: Sun Jan 20 12:43:34 2019 -0800
[SPARK-26642][K8S] Add --num-executors option to spark-submit for Spark on
K8S.
## What changes were proposed in this pull request?
This PR proposes to extend the spark-submit option --num-executors to be
applicable to Spark on K8S too. It is motivated by convenience, for example
when migrating jobs written for YARN to run on K8S.
## How was this patch tested?
Manually tested on a K8S cluster.
Author: Luca Canali
Closes #23573 from LucaCanali/addNumExecutorsToK8s.
---
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 4 ++--
.../main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala | 6 +++---
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index b403cc4..d5e17ff 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -537,14 +537,14 @@ private[spark] class SparkSubmit extends Logging {
// Yarn only
OptionAssigner(args.queue, YARN, ALL_DEPLOY_MODES, confKey =
"spark.yarn.queue"),
- OptionAssigner(args.numExecutors, YARN, ALL_DEPLOY_MODES,
-confKey = EXECUTOR_INSTANCES.key),
OptionAssigner(args.pyFiles, YARN, ALL_DEPLOY_MODES, confKey =
"spark.yarn.dist.pyFiles"),
OptionAssigner(args.jars, YARN, ALL_DEPLOY_MODES, confKey =
"spark.yarn.dist.jars"),
OptionAssigner(args.files, YARN, ALL_DEPLOY_MODES, confKey =
"spark.yarn.dist.files"),
OptionAssigner(args.archives, YARN, ALL_DEPLOY_MODES, confKey =
"spark.yarn.dist.archives"),
// Other options
+ OptionAssigner(args.numExecutors, YARN | KUBERNETES, ALL_DEPLOY_MODES,
+confKey = EXECUTOR_INSTANCES.key),
OptionAssigner(args.executorCores, STANDALONE | YARN | KUBERNETES,
ALL_DEPLOY_MODES,
confKey = EXECUTOR_CORES.key),
OptionAssigner(args.executorMemory, STANDALONE | MESOS | YARN |
KUBERNETES, ALL_DEPLOY_MODES,
diff --git
a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
index f5e4c4a..9692d2a 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
@@ -585,15 +585,15 @@ private[deploy] class SparkSubmitArguments(args:
Seq[String], env: Map[String, S
| in standalone mode).
|
| Spark on YARN and Kubernetes only:
+| --num-executors NUM Number of executors to launch (Default:
2).
+| If dynamic allocation is enabled, the
initial number of
+| executors will be at least NUM.
| --principal PRINCIPAL Principal to be used to login to KDC.
| --keytab KEYTAB The full path to the file that contains
the keytab for the
| principal specified above.
|
| Spark on YARN only:
| --queue QUEUE_NAME The YARN queue to submit to (Default:
"default").
-| --num-executors NUM Number of executors to launch (Default:
2).
-| If dynamic allocation is enabled, the
initial number of
-| executors will be at least NUM.
| --archives ARCHIVES Comma separated list of archives to be
extracted into the
| working directory of each executor.
""".stripMargin
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[zeppelin] branch master updated: [ZEPPELIN-3944] Update Dockerfiles of spark_standalone and spark_yarn_cluster (#3282)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 966a392 [ZEPPELIN-3944] Update Dockerfiles of spark_standalone and spark_yarn_cluster (#3282) 966a392 is described below commit 966a39240237edfd72b067afe867a244fae6474a Author: keineahnung2345 AuthorDate: Wed Jan 16 16:10:05 2019 +0800 [ZEPPELIN-3944] Update Dockerfiles of spark_standalone and spark_yarn_cluster (#3282) ### What is this PR for? Upgrade the Dockerfiles of spark_standalone and spark_yarn_cluster to CentOS7 and Spark 2.4.0. Java remains in version 7 since hadoop 2.x depends on Java7. ### What type of PR is it? Improvement ### What is the Jira issue? * [ZEPPELIN-3944](https://issues.apache.org/jira/browse/ZEPPELIN-3944) ### How should this be tested? * Follow the instructions here: [spark-standalone](https://zeppelin.apache.org/docs/0.8.0/setup/deployment/spark_cluster_mode.html#spark-standalone-mode) and [spark-yarn-cluster](https://zeppelin.apache.org/docs/0.8.0/setup/deployment/spark_cluster_mode.html#spark-on-yarn-mode) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No * Update spark_standalone Dockerfile * update yarn dockerfile * back to java 7 since hadoop 2.x depends on java 7 * back to java7 hadoop 2.x depends on java 7 * change the path to /xxx/.../jre-1.7.0-openjdk --- scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile | 4 ++-- .../docker/spark-cluster-managers/spark_standalone/Dockerfile | 10 +- .../spark-cluster-managers/spark_yarn_cluster/Dockerfile | 8 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile b/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile index 0eb26ca..b87838e 100644 --- a/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile +++ b/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile @@ -36,8 +36,8 @@ yum clean all # Remove old jdk RUN yum remove java; yum remove jdk -# install jdk8 -RUN yum install -y java-1.8.0-openjdk-devel +# install jdk7 +RUN yum install -y java-1.7.0-openjdk-devel ENV JAVA_HOME /usr/lib/jvm/java ENV PATH $PATH:$JAVA_HOME/bin diff --git a/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile b/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile index 8bf0f8d..e4fb780 100644 --- a/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile +++ b/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile @@ -12,10 +12,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -FROM centos:centos6 +FROM centos:centos7 -ENV SPARK_PROFILE 2.1 -ENV SPARK_VERSION 2.1.2 +ENV SPARK_PROFILE 2.4 +ENV SPARK_VERSION 2.4.0 ENV HADOOP_PROFILE 2.7 ENV SPARK_HOME /usr/local/spark @@ -33,13 +33,13 @@ yum clean all # Remove old jdk RUN yum remove java; yum remove jdk -# install jdk7 +# install jdk7 RUN yum install -y java-1.7.0-openjdk-devel ENV JAVA_HOME /usr/lib/jvm/java ENV PATH $PATH:$JAVA_HOME/bin # install spark -RUN curl -s http://apache.mirror.cdnetworks.com/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE.tgz | tar -xz -C /usr/local/ +RUN curl -s http://www.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE.tgz | tar -xz -C /usr/local/ RUN cd /usr/local && ln -s spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE spark # update boot script diff --git a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile index cbbda20..da3df1c 100644 --- a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile +++ b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile @@ -12,10 +12,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -FROM centos:centos6 +FROM centos:centos7 -ENV SPARK_PROFILE 2.1 -ENV SPARK_VERSION 2.1.2 +ENV SPARK_PROFILE 2.4 +ENV SPARK_VERSION 2.4.0 ENV HADOOP_PROFILE 2.7 ENV HADOOP_VERSION 2.7.0 @@ -52,7 +52,7 @@ ENV HADOOP_MAPRED_HOME /usr/local/hadoop ENV HADOOP_YARN_HOME /usr/local/hadoop ENV HADOOP_CONF_DIR /usr/local/hadoop/etc/hadoop -RUN sed -i '/^export JAVA_HOME/ s:.*:export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64\nexport HADOOP_PREFIX=/usr/local/hadoop\nexport HADOOP_H
[spark] branch branch-2.3 updated: [SPARK-26120][TESTS][SS][SPARKR] Fix a streaming query leak in Structured Streaming R tests
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.3 by this push:
new 1979712 [SPARK-26120][TESTS][SS][SPARKR] Fix a streaming query leak
in Structured Streaming R tests
1979712 is described below
commit 19797124f1e169138258c8c113874ec6ffedbe3d
Author: Shixiong Zhu
AuthorDate: Wed Nov 21 09:31:12 2018 +0800
[SPARK-26120][TESTS][SS][SPARKR] Fix a streaming query leak in Structured
Streaming R tests
## What changes were proposed in this pull request?
Stop the streaming query in `Specify a schema by using a DDL-formatted
string when reading` to avoid outputting annoying logs.
## How was this patch tested?
Jenkins
Closes #23089 from zsxwing/SPARK-26120.
Authored-by: Shixiong Zhu
Signed-off-by: hyukjinkwon
(cherry picked from commit 4b7f7ef5007c2c8a5090f22c6e08927e9f9a407b)
Signed-off-by: Felix Cheung
---
R/pkg/tests/fulltests/test_streaming.R | 1 +
1 file changed, 1 insertion(+)
diff --git a/R/pkg/tests/fulltests/test_streaming.R
b/R/pkg/tests/fulltests/test_streaming.R
index bfb1a04..6f0d2ae 100644
--- a/R/pkg/tests/fulltests/test_streaming.R
+++ b/R/pkg/tests/fulltests/test_streaming.R
@@ -127,6 +127,7 @@ test_that("Specify a schema by using a DDL-formatted string
when reading", {
expect_false(awaitTermination(q, 5 * 1000))
callJMethod(q@ssq, "processAllAvailable")
expect_equal(head(sql("SELECT count(*) FROM people3"))[[1]], 3)
+ stopQuery(q)
expect_error(read.stream(path = parquetPath, schema = "name stri"),
"DataType stri is not supported.")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: add checker note (#168)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 70a6071 add checker note (#168) 70a6071 is described below commit 70a60716fed85bfcfce4188b26faeb63ecbe2b79 Author: Felix Cheung AuthorDate: Sat Jan 12 20:07:14 2019 -0500 add checker note (#168) dd note to check checker after release --- release-process.md| 4 ++-- site/release-process.html | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/release-process.md b/release-process.md index 14c9c16..31d395c 100644 --- a/release-process.md +++ b/release-process.md @@ -130,7 +130,7 @@ that looks something like `[RESULT] [VOTE]...`. **THIS STEP IS IRREVERSIBLE so make sure you selected the correct staging repository. Once you move the artifacts into the release folder, they cannot be removed.** -After the vote passes, to upload the binaries to Apache mirrors, you move the binaries from dev directory (this should be where they are voted) to release directory. This "moving" is the only way you can add stuff to the actual release directory. +After the vote passes, to upload the binaries to Apache mirrors, you move the binaries from dev directory (this should be where they are voted) to release directory. This "moving" is the only way you can add stuff to the actual release directory. (Note: only PMC can move to release directory) ``` # Move the sub-directory in "dev" to the @@ -146,7 +146,7 @@ curl "https://dist.apache.org/repos/dist/dev/spark/KEYS; > svn-spark/KEYS Verify that the resources are present in https://www.apache.org/dist/spark/;>https://www.apache.org/dist/spark/. It may take a while for them to be visible. This will be mirrored throughout the Apache network. -There are a few remaining steps. +Check the release checker result of the release at https://checker.apache.org/projs/spark.html;>https://checker.apache.org/projs/spark.html. For Maven Central Repository, you can Release from the https://repository.apache.org/;>Apache Nexus Repository Manager. This is already populated by the `release-build.sh publish-release` step. Log in, open Staging Repositories, find the one voted on (eg. orgapachespark-1257 for https://repository.apache.org/content/repositories/orgapachespark-1257/), select and click Release and confirm. If successful, it should show up under https://repository.apache.org/content/repositori [...] diff --git a/site/release-process.html b/site/release-process.html index 2a4b82b..65f8fb4 100644 --- a/site/release-process.html +++ b/site/release-process.html @@ -334,7 +334,7 @@ that looks something like [RESULT] [VOTE] THIS STEP IS IRREVERSIBLE so make sure you selected the correct staging repository. Once you move the artifacts into the release folder, they cannot be removed. -After the vote passes, to upload the binaries to Apache mirrors, you move the binaries from dev directory (this should be where they are voted) to release directory. This moving is the only way you can add stuff to the actual release directory. +After the vote passes, to upload the binaries to Apache mirrors, you move the binaries from dev directory (this should be where they are voted) to release directory. This moving is the only way you can add stuff to the actual release directory. (Note: only PMC can move to release directory) # Move the sub-directory in "dev" to the # corresponding directory in "release" @@ -349,7 +349,7 @@ curl "https://dist.apache.org/repos/dist/dev/spark/KEYS; svn-spark/KEYS Verify that the resources are present in https://www.apache.org/dist/spark/;>https://www.apache.org/dist/spark/. It may take a while for them to be visible. This will be mirrored throughout the Apache network. -There are a few remaining steps. +Check the release checker result of the release at https://checker.apache.org/projs/spark.html;>https://checker.apache.org/projs/spark.html. For Maven Central Repository, you can Release from the https://repository.apache.org/;>Apache Nexus Repository Manager. This is already populated by the release-build.sh publish-release step. Log in, open Staging Repositories, find the one voted on (eg. orgapachespark-1257 for https://repository.apache.org/content/repositories/orgapachespark-1257/), select and click Release and confirm. If successful, it should show up under https://repository.apache.org/cont [...] and the same under https://repository.apache.org/content/groups/maven-staging-group/org/apache/spark/spark-core_2.11/2.2.1/ (look for the correct release version). After some time this will be syncd to https://search.maven.org/;>Maven Central automatically. -
[zeppelin] branch master updated: [ZEPPELIN-3932] spark_mesos Dockerfile should be updated (#3279)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new 7bf79d8 [ZEPPELIN-3932] spark_mesos Dockerfile should be updated (#3279) 7bf79d8 is described below commit 7bf79d80f5beaaafa3c0e18a96654e43a20a41f9 Author: keineahnung2345 AuthorDate: Sat Jan 5 13:15:03 2019 +0800 [ZEPPELIN-3932] spark_mesos Dockerfile should be updated (#3279) ### What is this PR for? In the [spark_mesos example - Dockerfile](https://github.com/apache/zeppelin/blob/master/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile), there are some issues. 1. The original used SPARK_VERSION=2.1.2 is not available now, update it to 2.4.0 2. There is no package named libevent2-devel, it should be corrected to libevent-devel 3. Update from centos6 to centos7 4. Update from jdk7 to jdk8 In the [spark_mesos example - entrypoint.sh](https://github.com/apache/zeppelin/blob/master/scripts/docker/spark-cluster-managers/spark_mesos/entrypoint.sh): 1. Follow the instructions: [zeppelin on spark mesos mode - configure spark interpreter in zeppelin](https://zeppelin.apache.org/docs/0.8.0/setup/deployment/spark_cluster_mode.html#4-configure-spark-interpreter-in-zeppelin-1) add the two environment variables. 2. add --hostname flag to the command mesos-master. This solves the problem of "Failed to connect to xx.xx.xx.xx:5050"(screenshot). ### What type of PR is - Bug Fix - Improvement ### What is the Jira issue? * [ZEPPELIN-3932](https://issues.apache.org/jira/browse/ZEPPELIN-3932) ### How should this be tested? * Follow the instructions here: [spark-on-mesos-mode](https://zeppelin.apache.org/docs/0.8.0/setup/deployment/spark_cluster_mode.html#spark-on-mesos-mode) ### Screenshots (if appropriate)  * update mesos Dockerfile 1. SPARK_VERSION 2.1.2 is not available anymore 2. centos cannot find libevent2-devel, but libevent-devel 3. update to centos7 and jdk8 * update entrypoint.sh according to tutorial 1. add environment variables MASTER and MESOS_NATIVE_JAVA_LIBRARY(from https://zeppelin.apache.org/docs/0.8.0/setup/deployment/spark_cluster_mode.html#4-configure-spark-interpreter-in-zeppelin-1) 2. add --hostname after mesos-master to solve the problem of "Failed to connect to xx.xx.xx.xx:5050" --- .../spark-cluster-managers/spark_mesos/Dockerfile | 26 +++--- .../spark_mesos/entrypoint.sh | 4 +++- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile b/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile index 0afda57..0eb26ca 100644 --- a/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile +++ b/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile @@ -12,10 +12,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -FROM centos:centos6 +FROM centos:centos7 -ENV SPARK_PROFILE 2.1 -ENV SPARK_VERSION 2.1.2 +ENV SPARK_PROFILE 2.4 +ENV SPARK_VERSION 2.4.0 ENV HADOOP_PROFILE 2.7 ENV HADOOP_VERSION 2.7.0 @@ -29,15 +29,15 @@ tar \ curl \ svn \ cyrus-sasl-md5 \ -libevent2-devel \ +libevent-devel \ && \ yum clean all # Remove old jdk RUN yum remove java; yum remove jdk -# install jdk7 -RUN yum install -y java-1.7.0-openjdk-devel +# install jdk8 +RUN yum install -y java-1.8.0-openjdk-devel ENV JAVA_HOME /usr/lib/jvm/java ENV PATH $PATH:$JAVA_HOME/bin @@ -45,14 +45,9 @@ ENV PATH $PATH:$JAVA_HOME/bin RUN curl -s http://www.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE.tgz | tar -xz -C /usr/local/ RUN cd /usr/local && ln -s spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE spark -# update boot script -COPY entrypoint.sh /etc/entrypoint.sh -RUN chown root.root /etc/entrypoint.sh -RUN chmod 700 /etc/entrypoint.sh - # install mesos -RUN wget http://repos.mesosphere.com/el/6/x86_64/RPMS/mesos-1.0.0-2.0.89.centos65.x86_64.rpm -RUN rpm -Uvh mesos-1.0.0-2.0.89.centos65.x86_64.rpm +RUN wget http://repos.mesosphere.com/el/7/x86_64/RPMS/mesos-1.7.0-2.0.1.el7.x86_64.rpm +RUN rpm -Uvh mesos-1.7.0-2.0.1.el7.x86_64.rpm #spark EXPOSE 8080 7077 7072 8081 8082 @@ -60,4 +55,9 @@ EXPOSE 8080 7077 7072 8081 8082 #mesos EXPOSE 5050 5051 +# update boot script +COPY entrypoint.sh /etc/entrypoint.sh +RUN chown root.root /etc/entrypoint.sh +RUN chmod 700 /etc/entrypoint.sh + ENTRYPOINT ["/etc/en
[zeppelin] branch master updated: [ZEPPELIN-3914] upgrade Flink to 1.7.1 (#3266)
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 13fd50e [ZEPPELIN-3914] upgrade Flink to 1.7.1 (#3266)
13fd50e is described below
commit 13fd50ee535cd28b237cb4af740d0f893dbff3ae
Author: Xue Yu <278006...@qq.com>
AuthorDate: Tue Jan 1 09:22:42 2019 +0800
[ZEPPELIN-3914] upgrade Flink to 1.7.1 (#3266)
### What is this PR for?
This PR is for upgrading current Flink to 1.7.1
### What type of PR is it?
[Improvement]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3914
### How should this be tested?
*manual
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
no
* Is there breaking changes for older versions?
no
* Does this needs documentation?
no
---
docs/interpreter/flink.md | 2 +-
flink/pom.xml | 2 +-
.../apache/zeppelin/flink/FlinkSQLScalaInterpreter.scala| 2 +-
.../org/apache/zeppelin/flink/FlinkScalaInterpreter.scala | 13 -
4 files changed, 7 insertions(+), 12 deletions(-)
diff --git a/docs/interpreter/flink.md b/docs/interpreter/flink.md
index 2cf3125..d3f2223 100644
--- a/docs/interpreter/flink.md
+++ b/docs/interpreter/flink.md
@@ -50,7 +50,7 @@ At the "Interpreters" menu, you have to create a new Flink
interpreter and provi
-For more information about Flink configuration, you can find it
[here](https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html).
+For more information about Flink configuration, you can find it
[here](https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html).
## How to test it's working
You can find an example of Flink usage in the Zeppelin Tutorial folder or try
the following word count example, by using the [Zeppelin
notebook](https://www.zeppelinhub.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL05GTGFicy96ZXBwZWxpbi1ub3RlYm9va3MvbWFzdGVyL25vdGVib29rcy8yQVFFREs1UEMvbm90ZS5qc29u)
from Till Rohrmann's presentation [Interactive data analysis with Apache
Flink](http://www.slideshare.net/tillrohrmann/data-analysis-49806564) for
Apache Flink Meetup.
diff --git a/flink/pom.xml b/flink/pom.xml
index 7a374f2..331e19c 100644
--- a/flink/pom.xml
+++ b/flink/pom.xml
@@ -36,7 +36,7 @@
flink
-1.5.2
+1.7.1
2.3.7
2.0.1
2.11
diff --git
a/flink/src/main/scala/org/apache/zeppelin/flink/FlinkSQLScalaInterpreter.scala
b/flink/src/main/scala/org/apache/zeppelin/flink/FlinkSQLScalaInterpreter.scala
index 1694a44..b2d8d16 100644
---
a/flink/src/main/scala/org/apache/zeppelin/flink/FlinkSQLScalaInterpreter.scala
+++
b/flink/src/main/scala/org/apache/zeppelin/flink/FlinkSQLScalaInterpreter.scala
@@ -30,7 +30,7 @@ class FlinkSQLScalaInterpreter(scalaInterpreter:
FlinkScalaInterpreter,
def interpret(code: String, context: InterpreterContext): InterpreterResult
= {
try {
- val table: Table = this.btenv.sql(code)
+ val table: Table = this.btenv.sqlQuery(code)
val result = z.showData(table)
return new InterpreterResult(InterpreterResult.Code.SUCCESS, result)
} catch {
diff --git
a/flink/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala
b/flink/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala
index 14f8959..1d8b27e 100644
--- a/flink/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala
+++ b/flink/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala
@@ -26,7 +26,7 @@ import org.apache.flink.api.scala.FlinkShell._
import org.apache.flink.api.scala.{ExecutionEnvironment, FlinkILoop}
import org.apache.flink.client.program.ClusterClient
import org.apache.flink.configuration.GlobalConfiguration
-import org.apache.flink.runtime.minicluster.{MiniCluster,
StandaloneMiniCluster}
+import org.apache.flink.runtime.minicluster.MiniCluster
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.table.api.TableEnvironment
import org.apache.flink.table.api.scala.{BatchTableEnvironment,
StreamTableEnvironment}
@@ -45,8 +45,7 @@ class FlinkScalaInterpreter(val properties: Properties) {
lazy val LOGGER: Logger = LoggerFactory.getLogger(getClass)
private var flinkILoop: FlinkILoop = _
- private var cluster: Option[Either[Either[StandaloneMiniCluster,
MiniCluster],
-ClusterClient[_]]] = _
+ private var cluster: Option[Either[MiniCluster, ClusterClient[_]]] = _
private var scalaCompleter: ScalaCompleter = _
private val interpreterOutput = new InterpreterOutputStream(LOGGER)
@@ -68,8 +67,7 @@ class FlinkScalaInterpreter(val properties: Proper
[zeppelin] branch master updated: docs: fix punctuation (#3197)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new c20c830 docs: fix punctuation (#3197) c20c830 is described below commit c20c8304bdc9cce2edd931f0d58d6f40a8fed39a Author: Gerard de Melo AuthorDate: Mon Dec 31 00:07:19 2018 +0100 docs: fix punctuation (#3197) ### What is this PR for? Minor update to fix punctuation-related typos in docs/usage/interpreter/overview.md ### What type of PR is it? Documentation --- docs/usage/interpreter/overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/usage/interpreter/overview.md b/docs/usage/interpreter/overview.md index 6580a59..da59076 100644 --- a/docs/usage/interpreter/overview.md +++ b/docs/usage/interpreter/overview.md @@ -139,8 +139,8 @@ This approach works, but is not particularly convenient. `ConfInterpreter` can p `ConfInterpreter` is a generic interpreter that can be used by any interpreter. The input format should be the property file format. It can be used to make custom settings for any interpreter. However, `ConfInterpreter` needs to be run before that interpreter process is launched. When that interpreter process is launched is determined by the interpreter mode setting. -So users need to understand the ([interpreter mode setting ](../usage/interpreter/interpreter_bindings_mode.html) of Zeppelin and be aware of when the interpreter process is launched. E.g., if we set the Spark interpreter setting as isolated per note, then, under this setting, each note will launch one interpreter process. -In this scenario, users need to put `ConfInterpreter` as the first paragraph as in the below example. Otherwise, the customized setting cannot be applied (actually it would report ERROR) +So users need to understand the [interpreter mode setting](../usage/interpreter/interpreter_bindings_mode.html) of Zeppelin and be aware of when the interpreter process is launched. E.g., if we set the Spark interpreter setting as isolated per note, then, under this setting, each note will launch one interpreter process. +In this scenario, users need to put `ConfInterpreter` as the first paragraph as in the below example. Otherwise, the customized setting cannot be applied (actually it would report ERROR).
[incubator-pinot-site] 01/01: updating trademark
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch felixcheung-tm-1 in repository https://gitbox.apache.org/repos/asf/incubator-pinot-site.git commit d11cf4bcaf40f3613ce8231a1a76d9b9d8d5b9b5 Author: Felix Cheung AuthorDate: Fri Dec 28 14:34:25 2018 -0800 updating trademark as per https://www.apache.org/foundation/marks/pmcs#attributions --- index.html | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/index.html b/index.html index 89e2a7a..2a8a4c2 100644 --- a/index.html +++ b/index.html @@ -20,7 +20,7 @@ -Apache Pinot +Apache Pinot™ (Incubating) -Apache Pinot +Apache Pinot™ (Incubating) Pinot is a realtime distributed OLAP datastore, which is used at LinkedIn to deliver scalable real time analytics with low latency. -It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). +It can ingest data from offline data sources (such as Apache Hadoop and flat files) as well as online sources (such as Apache Kafka). Pinot is designed to scale horizontally. http://github.com/linkedin/pinot;>Download @@ -87,9 +87,9 @@ Apache Pinot is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that [...] -Copyright © 2017 http://www.apache.org/; target="_blank">The Apache Software Foundation +Copyright © 2018 http://www.apache.org/; target="_blank">The Apache Software Foundation -Apache, Apache Pinot (incubating), the Apache feather and the Pinot logo are trademarks of The Apache Software Foundation +Apache, Apache Pinot (incubating), the Apache feather and the Pinot logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[zeppelin] branch master updated: [ZEPPELIN-3864] Fix Travis tests
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 6639025 [ZEPPELIN-3864] Fix Travis tests
6639025 is described below
commit 663902518762cbad2fdc4efa3d552ad13effe4b0
Author: Savalek - Aleksey Savelenko <30798933+sava...@users.noreply.github.com>
AuthorDate: Tue Dec 25 22:35:45 2018 +0300
[ZEPPELIN-3864] Fix Travis tests
### What is this PR for?
Maven `frontend plugin v1.4` incorrectly processed `npm` messages with
error.
This PR update `frontend plugin` to 1.6 from 1.4.
### What type of PR is it?
Bug Fix
### What is the Jira issue?
[ZEPPELIN-3864](https://issues.apache.org/jira/browse/ZEPPELIN-3864),
[ZP-70]
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
---
pom.xml| 2 +-
zeppelin-web/e2e/searchBlock.spec.js | 2 ++
zeppelin-web/package.json | 12
zeppelin-web/pom.xml | 2 +-
.../src/app/notebook/notebook.controller.js| 22 +++--
.../src/app/notebook/notebook.controller.test.js | 1 +
.../note-create/note-create.controller.test.js | 10 +++---
.../components/note-list/note-list.factory.test.js | 36 +++---
zeppelin-web/webpack.config.js | 5 +--
9 files changed, 49 insertions(+), 43 deletions(-)
diff --git a/pom.xml b/pom.xml
index 2465009..04a6ad8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -102,7 +102,7 @@
v8.9.3
5.5.1
-1.4
+1.6
1.7.10
diff --git a/zeppelin-web/e2e/searchBlock.spec.js
b/zeppelin-web/e2e/searchBlock.spec.js
index 570673b..a146e47 100644
--- a/zeppelin-web/e2e/searchBlock.spec.js
+++ b/zeppelin-web/e2e/searchBlock.spec.js
@@ -20,6 +20,8 @@ describe('Search block e2e Test', function() {
beforeEach(function() {
browser.get('http://localhost:8080')
+browser.sleep(500);
+waitVisibility(element(by.linkText('Create new note')))
clickOn(element(by.linkText('Create new note')))
waitVisibility(element(by.id('noteCreateModal')))
clickAndWait(element(by.id('createNoteButton')))
diff --git a/zeppelin-web/package.json b/zeppelin-web/package.json
index 69248c6..de83a62 100644
--- a/zeppelin-web/package.json
+++ b/zeppelin-web/package.json
@@ -22,7 +22,7 @@
"pree2e": "webdriver-manager update --gecko false --versions.chrome=2.35",
"e2e": "protractor protractor.conf.js",
"pretest": "npm rebuild phantomjs-prebuilt",
-"test": "karma start karma.conf.js"
+"karma-test": "karma start karma.conf.js"
},
"dependencies": {
"angular-ui-grid": "4.4.6",
@@ -78,15 +78,15 @@
"html-webpack-plugin": "^3.2.0",
"imports-loader": "^0.7.1",
"istanbul-instrumenter-loader": "^0.2.0",
-"jasmine-core": "^2.5.2",
-"jasmine-spec-reporter": "^4.1.1",
-"karma": "~1.3.0",
+"jasmine-core": "^3.3.0",
+"jasmine-spec-reporter": "^4.2.1",
+"karma": "~3.1.3",
"karma-coverage": "^1.1.2",
-"karma-jasmine": "~1.0.2",
+"karma-jasmine": "~2.0.1",
"karma-phantomjs-launcher": "^1.0.4",
"karma-sourcemap-loader": "^0.3.7",
"karma-spec-reporter": "0.0.31",
-"karma-webpack": "^1.8.1",
+"karma-webpack": "^3.0.5",
"load-grunt-tasks": "^0.4.0",
"mini-css-extract-plugin": "^0.4.4",
"ng-annotate-loader": "^0.2.0",
diff --git a/zeppelin-web/pom.xml b/zeppelin-web/pom.xml
index 498803b..7de15dc 100644
--- a/zeppelin-web/pom.xml
+++ b/zeppelin-web/pom.xml
@@ -111,7 +111,7 @@
test
${web.e2e.disabled}
- run test
+ run karma-test
diff --git a/zeppelin-web/src/app/notebook/notebook.controller.js
b/zeppelin-web/src/app/notebook/notebook.controller.js
index 2f1437d..426667d 100644
--- a/zeppelin-web/src/app/notebook/notebook.controller.js
+++ b/zeppelin-web/src/app/notebook/notebook.controller.js
@@ -1579,14 +1579,16 @@ function NotebookCtrl($scope, $route, $routeParams,
$location, $rootScope,
});
let content = document.getElementById('content');
- $scope.addEvent({
-eventID: co
[zeppelin] branch branch-0.8 updated: [ZEPPELIN-3553] Fix URLs on "Multi-user Support" page
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a commit to branch branch-0.8 in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/branch-0.8 by this push: new 2c2deb1 [ZEPPELIN-3553] Fix URLs on "Multi-user Support" page 2c2deb1 is described below commit 2c2deb1891b8700d3d9d68695891e09e3139a48d Author: mebelousov AuthorDate: Tue Jun 19 16:38:10 2018 +0300 [ZEPPELIN-3553] Fix URLs on "Multi-user Support" page ### What is this PR for? On page Setup > Multi-user Support http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/setup/basics/multi_user_support.html there are two urls: * Shiro Authentication - http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/setup/setup/security/shiro_authentication.html * Notebook Permission - http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/setup/setup/security/notebook_authorization.html Need to remove one of the "setup" ### What type of PR is it? Bug Fix ### What is the Jira issue? [ZEPPELIN-3553](https://issues.apache.org/jira/browse/ZEPPELIN-3553) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: mebelousov Closes #3035 from mebelousov/ZEPPELIN-3553 and squashes the following commits: 100a2af06 [mebelousov] ZEPPELIN-3553 Fix URLs for "Notebook Permission" and "Shiro Authentication" (cherry picked from commit efdc16d71c1c66e46ea67de21a4b1185f85c10f5) Signed-off-by: Felix Cheung --- docs/setup/basics/multi_user_support.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/setup/basics/multi_user_support.md b/docs/setup/basics/multi_user_support.md index 15d911c..e61b723 100644 --- a/docs/setup/basics/multi_user_support.md +++ b/docs/setup/basics/multi_user_support.md @@ -25,8 +25,8 @@ limitations under the License. This page describes about multi-user support. -- multiple users login / logout using [Shiro Authentication](../setup/security/shiro_authentication.html) -- managing [Notebook Permission](../setup/security/notebook_authorization.html) +- multiple users login / logout using [Shiro Authentication](../security/shiro_authentication.html) +- managing [Notebook Permission](../security/notebook_authorization.html) - how to setup [impersonation for interpreters](../../usage/interpreter/user_impersonation.html) - different contexts per user / note using [Interpreter Binding Mode](../../usage/interpreter/interpreter_binding_mode.html) - a paragraph in a notebook can be [Personalized](../../usage/other_features/personalized_mode.html)
[zeppelin] branch master updated: Revert "[ZEPPELIN-3882] Neo4jInterpreter - Support Point and Date Types" (#3270)
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 45fa3fd Revert "[ZEPPELIN-3882] Neo4jInterpreter - Support Point and
Date Types" (#3270)
45fa3fd is described below
commit 45fa3fdac0c79f78f6d0416e1bbfce281ef4e0df
Author: Felix Cheung
AuthorDate: Fri Dec 21 02:03:48 2018 -0500
Revert "[ZEPPELIN-3882] Neo4jInterpreter - Support Point and Date Types"
(#3270)
* Revert "fix checkstyle"
This reverts commit 5542262568b3532fd11e2832d3168b386f35b3d6.
* Revert "Added support for Neo4j temporal and point types"
This reverts commit b98ccea98996eda183382a332379f3cfc428c0e9.
* Revert "Added support for Neo4j temporal and point types"
This reverts commit 0fbcddae980ff245eead964e8a5e0ee0f1a6.
---
docs/interpreter/neo4j.md | 3 -
neo4j/pom.xml | 6 +-
.../graph/neo4j/Neo4jCypherInterpreter.java| 14 ---
.../graph/neo4j/Neo4jCypherInterpreterTest.java| 105 -
4 files changed, 20 insertions(+), 108 deletions(-)
diff --git a/docs/interpreter/neo4j.md b/docs/interpreter/neo4j.md
index eec9e07..1b14127 100644
--- a/docs/interpreter/neo4j.md
+++ b/docs/interpreter/neo4j.md
@@ -26,9 +26,6 @@ limitations under the License.
## Overview
[Neo4j](https://neo4j.com/product/) is a native graph database, designed to
store and process graphs from bottom to top.
-### Supported Version
-
-The Neo4j Interpreter supports all Neo4j versions since v3 via the official
[Neo4j Java Driver](https://github.com/neo4j/neo4j-java-driver)
diff --git a/neo4j/pom.xml b/neo4j/pom.xml
index cc39fdc..b8a89ad 100644
--- a/neo4j/pom.xml
+++ b/neo4j/pom.xml
@@ -33,9 +33,9 @@
Zeppelin: Neo4j interpreter
- 1.7.1
- 3.4.10
- 3.4.10
+ 1.4.3
+ 3.2.3
+ 3.2.3
2.8.9
neo4j
diff --git
a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
index d7f8485..bcb9d7b 100644
---
a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
+++
b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
@@ -201,20 +201,6 @@ public class Neo4jCypherInterpreter extends Interpreter {
value = val.asList();
} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.MAP())) {
value = val.asMap();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.POINT())) {
- value = val.asPoint();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE())) {
- value = val.asLocalDate();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.TIME())) {
- value = val.asOffsetTime();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_TIME())) {
- value = val.asLocalTime();
-} else if
(val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_DATE_TIME())) {
- value = val.asLocalDateTime();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE_TIME())) {
- value = val.asZonedDateTime();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DURATION())) {
- value = val.asIsoDuration();
}
}
if (value instanceof Collection) {
diff --git
a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
index 7940d5f..24bd513 100644
---
a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
+++
b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
@@ -55,15 +55,10 @@ public class Neo4jCypherInterpreterTest {
private static final String REL_KNOWS = "KNOWS";
private static final String CYPHER_FOREACH =
- "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age:
%s, " +
- "address: point({ longitude: 56.7, latitude: 12.78, height:
8 }), " +
- "birth: date('1984-04-04')}))";
- private static final String CHPHER_UNWIND = "UNWIND range(1,100) as x "
-+ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 100) "
+ "FOREACH (x in range(1,1000) | CREATE (:%s{name: \"name\" + x, age:
%s}))";
+ private static final String CHPHER_UNWIND = "UNWIND range(1,1000) as x "
++ "MATCH (n), (m) WHERE id(n) = x AND id(m) = to
[zeppelin] branch revert-3244-update-neo4j-datatypes created (now 7ae0e41)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a change to branch revert-3244-update-neo4j-datatypes in repository https://gitbox.apache.org/repos/asf/zeppelin.git. at 7ae0e41 Revert "Added support for Neo4j temporal and point types" This branch includes the following new commits: new 4ddee85 Revert "fix checkstyle" new 0193be5 Revert "Added support for Neo4j temporal and point types" new 7ae0e41 Revert "Added support for Neo4j temporal and point types" The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[zeppelin] 01/03: Revert "fix checkstyle"
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch revert-3244-update-neo4j-datatypes
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
commit 4ddee8579401c4bbf59f714a9972538684f555e5
Author: Felix Cheung
AuthorDate: Thu Dec 20 23:03:14 2018 -0800
Revert "fix checkstyle"
This reverts commit 5542262568b3532fd11e2832d3168b386f35b3d6.
---
.../graph/neo4j/Neo4jCypherInterpreter.java| 14 ---
.../graph/neo4j/Neo4jCypherInterpreterTest.java| 101 +++--
2 files changed, 15 insertions(+), 100 deletions(-)
diff --git
a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
index d7f8485..bcb9d7b 100644
---
a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
+++
b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
@@ -201,20 +201,6 @@ public class Neo4jCypherInterpreter extends Interpreter {
value = val.asList();
} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.MAP())) {
value = val.asMap();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.POINT())) {
- value = val.asPoint();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE())) {
- value = val.asLocalDate();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.TIME())) {
- value = val.asOffsetTime();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_TIME())) {
- value = val.asLocalTime();
-} else if
(val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_DATE_TIME())) {
- value = val.asLocalDateTime();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE_TIME())) {
- value = val.asZonedDateTime();
-} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DURATION())) {
- value = val.asIsoDuration();
}
}
if (value instanceof Collection) {
diff --git
a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
index 7940d5f..f44cd77 100644
---
a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
+++
b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
@@ -55,15 +55,10 @@ public class Neo4jCypherInterpreterTest {
private static final String REL_KNOWS = "KNOWS";
private static final String CYPHER_FOREACH =
- "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age:
%s, " +
- "address: point({ longitude: 56.7, latitude: 12.78, height:
8 }), " +
- "birth: date('1984-04-04')}))";
+ "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age:
%s, address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), birth:
date('1984-04-04')}))";
private static final String CHPHER_UNWIND = "UNWIND range(1,100) as x "
+ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 100) "
+ "CREATE (n)-[:%s]->(m)";
-
- private static final String TABLE_RESULT_PREFIX = "%table ";
- private static final String NETWORK_RESULT_PREFIX = "%network ";
@BeforeClass
public static void setUpNeo4jServer() throws Exception {
@@ -78,7 +73,7 @@ public class Neo4jCypherInterpreterTest {
public static void tearDownNeo4jServer() throws Exception {
server.close();
}
-
+
@Before
public void setUpZeppelin() {
Properties p = new Properties();
@@ -88,7 +83,7 @@ public class Neo4jCypherInterpreterTest {
interpreter = new Neo4jCypherInterpreter(p);
context = InterpreterContext.builder()
.setInterpreterOut(new InterpreterOutput(null))
-.build();
+.build();;
}
@After
@@ -103,15 +98,14 @@ public class Neo4jCypherInterpreterTest {
"return 'a' as colA, 'b' as colB, [1, 2, 3] as colC", context);
assertEquals(Code.SUCCESS, result.code());
final String tableResult = "colA\tcolB\tcolC\n\"a\"\t\"b\"\t[1,2,3]\n";
-assertEquals(tableResult, result.toString().replace(TABLE_RESULT_PREFIX,
StringUtils.EMPTY));
-
+assertEquals(tableResult, result.toString().replace("%table ",
StringUtils.EMPTY));
+
result = interpreter.interpret(
"return 'a' as colA, 'b' as colB, [{key: \"value\"}, {key: 1}] as
colC", context);
assertEquals(Code.SUCCESS, result.code());
final String tableResultWithMap =
"colA\tcolB\tcolC\n\"a\"\t\"b\"\t[{\"key\":\"value\"},{\"key\":1
[zeppelin] 02/03: Revert "Added support for Neo4j temporal and point types"
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch revert-3244-update-neo4j-datatypes
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
commit 0193be5b312845f4ea4fed737a0441b001e1b403
Author: Felix Cheung
AuthorDate: Thu Dec 20 23:03:14 2018 -0800
Revert "Added support for Neo4j temporal and point types"
This reverts commit b98ccea98996eda183382a332379f3cfc428c0e9.
---
docs/interpreter/neo4j.md | 3 ---
1 file changed, 3 deletions(-)
diff --git a/docs/interpreter/neo4j.md b/docs/interpreter/neo4j.md
index eec9e07..1b14127 100644
--- a/docs/interpreter/neo4j.md
+++ b/docs/interpreter/neo4j.md
@@ -26,9 +26,6 @@ limitations under the License.
## Overview
[Neo4j](https://neo4j.com/product/) is a native graph database, designed to
store and process graphs from bottom to top.
-### Supported Version
-
-The Neo4j Interpreter supports all Neo4j versions since v3 via the official
[Neo4j Java Driver](https://github.com/neo4j/neo4j-java-driver)
[zeppelin] 03/03: Revert "Added support for Neo4j temporal and point types"
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch revert-3244-update-neo4j-datatypes
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
commit 7ae0e418a15540fffb2402391d122219e6819234
Author: Felix Cheung
AuthorDate: Thu Dec 20 23:03:14 2018 -0800
Revert "Added support for Neo4j temporal and point types"
This reverts commit 0fbcddae980ff245eead964e8a5e0ee0f1a6.
---
neo4j/pom.xml | 6 +++---
.../org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/neo4j/pom.xml b/neo4j/pom.xml
index cc39fdc..b8a89ad 100644
--- a/neo4j/pom.xml
+++ b/neo4j/pom.xml
@@ -33,9 +33,9 @@
Zeppelin: Neo4j interpreter
- 1.7.1
- 3.4.10
- 3.4.10
+ 1.4.3
+ 3.2.3
+ 3.2.3
2.8.9
neo4j
diff --git
a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
index f44cd77..24bd513 100644
---
a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
+++
b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
@@ -55,9 +55,9 @@ public class Neo4jCypherInterpreterTest {
private static final String REL_KNOWS = "KNOWS";
private static final String CYPHER_FOREACH =
- "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age:
%s, address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), birth:
date('1984-04-04')}))";
- private static final String CHPHER_UNWIND = "UNWIND range(1,100) as x "
-+ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 100) "
+ "FOREACH (x in range(1,1000) | CREATE (:%s{name: \"name\" + x, age:
%s}))";
+ private static final String CHPHER_UNWIND = "UNWIND range(1,1000) as x "
++ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 1000) "
+ "CREATE (n)-[:%s]->(m)";
@BeforeClass
[zeppelin] 02/03: Added support for Neo4j temporal and point types
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
commit b98ccea98996eda183382a332379f3cfc428c0e9
Author: Andrea Santurbano
AuthorDate: Mon Nov 26 23:48:06 2018 +0100
Added support for Neo4j temporal and point types
---
docs/interpreter/neo4j.md | 3 +++
1 file changed, 3 insertions(+)
diff --git a/docs/interpreter/neo4j.md b/docs/interpreter/neo4j.md
index 1b14127..eec9e07 100644
--- a/docs/interpreter/neo4j.md
+++ b/docs/interpreter/neo4j.md
@@ -26,6 +26,9 @@ limitations under the License.
## Overview
[Neo4j](https://neo4j.com/product/) is a native graph database, designed to
store and process graphs from bottom to top.
+### Supported Version
+
+The Neo4j Interpreter supports all Neo4j versions since v3 via the official
[Neo4j Java Driver](https://github.com/neo4j/neo4j-java-driver)
[zeppelin] branch master updated (a6f4562 -> 5542262)
This is an automated email from the ASF dual-hosted git repository. felixcheung pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git. from a6f4562 [ZEPPELIN-3902]upgrade jetty version new 0fbcd11 Added support for Neo4j temporal and point types new b98ccea Added support for Neo4j temporal and point types new 5542262 fix checkstyle The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/interpreter/neo4j.md | 3 + neo4j/pom.xml | 6 +- .../graph/neo4j/Neo4jCypherInterpreter.java| 14 +++ .../graph/neo4j/Neo4jCypherInterpreterTest.java| 105 + 4 files changed, 108 insertions(+), 20 deletions(-)
[zeppelin] 03/03: fix checkstyle
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
commit 5542262568b3532fd11e2832d3168b386f35b3d6
Author: Andrea Santurbano
AuthorDate: Thu Nov 29 09:44:11 2018 +0100
fix checkstyle
---
.../graph/neo4j/Neo4jCypherInterpreter.java| 14 +++
.../graph/neo4j/Neo4jCypherInterpreterTest.java| 101 ++---
2 files changed, 100 insertions(+), 15 deletions(-)
diff --git
a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
index bcb9d7b..d7f8485 100644
---
a/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
+++
b/neo4j/src/main/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreter.java
@@ -201,6 +201,20 @@ public class Neo4jCypherInterpreter extends Interpreter {
value = val.asList();
} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.MAP())) {
value = val.asMap();
+} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.POINT())) {
+ value = val.asPoint();
+} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE())) {
+ value = val.asLocalDate();
+} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.TIME())) {
+ value = val.asOffsetTime();
+} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_TIME())) {
+ value = val.asLocalTime();
+} else if
(val.hasType(InternalTypeSystem.TYPE_SYSTEM.LOCAL_DATE_TIME())) {
+ value = val.asLocalDateTime();
+} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DATE_TIME())) {
+ value = val.asZonedDateTime();
+} else if (val.hasType(InternalTypeSystem.TYPE_SYSTEM.DURATION())) {
+ value = val.asIsoDuration();
}
}
if (value instanceof Collection) {
diff --git
a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
index f44cd77..7940d5f 100644
---
a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
+++
b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
@@ -55,10 +55,15 @@ public class Neo4jCypherInterpreterTest {
private static final String REL_KNOWS = "KNOWS";
private static final String CYPHER_FOREACH =
- "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age:
%s, address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), birth:
date('1984-04-04')}))";
+ "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age:
%s, " +
+ "address: point({ longitude: 56.7, latitude: 12.78, height:
8 }), " +
+ "birth: date('1984-04-04')}))";
private static final String CHPHER_UNWIND = "UNWIND range(1,100) as x "
+ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 100) "
+ "CREATE (n)-[:%s]->(m)";
+
+ private static final String TABLE_RESULT_PREFIX = "%table ";
+ private static final String NETWORK_RESULT_PREFIX = "%network ";
@BeforeClass
public static void setUpNeo4jServer() throws Exception {
@@ -73,7 +78,7 @@ public class Neo4jCypherInterpreterTest {
public static void tearDownNeo4jServer() throws Exception {
server.close();
}
-
+
@Before
public void setUpZeppelin() {
Properties p = new Properties();
@@ -83,7 +88,7 @@ public class Neo4jCypherInterpreterTest {
interpreter = new Neo4jCypherInterpreter(p);
context = InterpreterContext.builder()
.setInterpreterOut(new InterpreterOutput(null))
-.build();;
+.build();
}
@After
@@ -98,14 +103,15 @@ public class Neo4jCypherInterpreterTest {
"return 'a' as colA, 'b' as colB, [1, 2, 3] as colC", context);
assertEquals(Code.SUCCESS, result.code());
final String tableResult = "colA\tcolB\tcolC\n\"a\"\t\"b\"\t[1,2,3]\n";
-assertEquals(tableResult, result.toString().replace("%table ",
StringUtils.EMPTY));
-
+assertEquals(tableResult, result.toString().replace(TABLE_RESULT_PREFIX,
StringUtils.EMPTY));
+
result = interpreter.interpret(
"return 'a' as colA, 'b' as colB, [{key: \"value\"}, {key: 1}] as
colC", context);
assertEquals(Code.SUCCESS, result.code());
final String tableResultWithMap =
"colA\tcolB\tcolC\n\"a\"\t\"b\"\t[{\"key\":\"value\"},{\"key\":1}]\n";
-assertEquals(tableResultWithMap, result.toString().replace("%table ",
StringUtils.
[zeppelin] 01/03: Added support for Neo4j temporal and point types
This is an automated email from the ASF dual-hosted git repository.
felixcheung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
commit 0fbcddae980ff245eead964e8a5e0ee0f1a6
Author: Andrea Santurbano
AuthorDate: Mon Nov 26 23:48:06 2018 +0100
Added support for Neo4j temporal and point types
---
neo4j/pom.xml | 6 +++---
.../org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/neo4j/pom.xml b/neo4j/pom.xml
index b8a89ad..cc39fdc 100644
--- a/neo4j/pom.xml
+++ b/neo4j/pom.xml
@@ -33,9 +33,9 @@
Zeppelin: Neo4j interpreter
- 1.4.3
- 3.2.3
- 3.2.3
+ 1.7.1
+ 3.4.10
+ 3.4.10
2.8.9
neo4j
diff --git
a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
index 24bd513..f44cd77 100644
---
a/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
+++
b/neo4j/src/test/java/org/apache/zeppelin/graph/neo4j/Neo4jCypherInterpreterTest.java
@@ -55,9 +55,9 @@ public class Neo4jCypherInterpreterTest {
private static final String REL_KNOWS = "KNOWS";
private static final String CYPHER_FOREACH =
- "FOREACH (x in range(1,1000) | CREATE (:%s{name: \"name\" + x, age:
%s}))";
- private static final String CHPHER_UNWIND = "UNWIND range(1,1000) as x "
-+ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 1000) "
+ "FOREACH (x in range(1,100) | CREATE (:%s{name: \"name\" + x, age:
%s, address: point({ longitude: 56.7, latitude: 12.78, height: 8 }), birth:
date('1984-04-04')}))";
+ private static final String CHPHER_UNWIND = "UNWIND range(1,100) as x "
++ "MATCH (n), (m) WHERE id(n) = x AND id(m) = toInt(rand() * 100) "
+ "CREATE (n)-[:%s]->(m)";
@BeforeClass
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3258 in that case, can we make it build beam only when it has support for it. ie. build scala 2.10 - beam 2.0.0 build scala 2.11 - beam not included (but not failing) ---
[GitHub] zeppelin issue #3244: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and D...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3244 merging if no more comment ---
[GitHub] zeppelin issue #3244: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and D...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3244 I see it's green. not sure why the report above is not. ---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3258 > that's what i did in: ffca03c ah, that would make different versions of beam with scala-2.10 or scala-2.11. is there a scio version that works with scala-2.11? what if we upgrade the supported version of beam and scio in all profile? ---
[GitHub] zeppelin issue #3265: [ZEPPELIN-3910] IPython delete temp file and close str...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3265 can you check failed tests https://travis-ci.org/yx91490/zeppelin/builds/468316435 ---
[GitHub] zeppelin issue #3264: [ZEPPELIN-3908] Fix Date Sorting issue
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3264 please fix description - remove `First time? Setup Travis CI as described ` ... ---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3258 also we could upgrade beam interpreter to "require" beam 2.3 also (so it does build with scala-2.11) ---
[GitHub] zeppelin issue #3258: fix beam-runners-flink and zeppelin-scio scala version...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3258 what about we don't build beam and scio if scala-2.11? ---
[GitHub] spark pull request #23252: [SPARK-26239] File-based secret key loading for S...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23252#discussion_r240022921
--- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala
---
@@ -440,12 +473,27 @@ class SecurityManagerSuite extends SparkFunSuite with
ResetSystemProperties {
intercept[IllegalArgumentException] {
mgr.getSecretKey()
}
+ case FILE =>
+val secretFile = createTempSecretFile()
+conf.set(AUTH_SECRET_FILE, secretFile.getAbsolutePath)
+mgr.initializeAuth()
+assert(encodeFileAsBase64(secretFile) ===
mgr.getSecretKey())
}
}
}
)
}
}
+ private def encodeFileAsBase64(secretFile: File) = {
+Base64.getEncoder.encodeToString(Files.readAllBytes(secretFile.toPath))
+ }
+
+ private def createTempSecretFile(contents: String = "test-secret"): File
= {
+val secretDir = Utils.createTempDir("temp-secrets")
+val secretFile = new File(secretDir, "temp-secret.txt")
+Files.write(secretFile.toPath,
contents.getBytes(StandardCharsets.UTF_8))
+secretFile
--- End diff --
ah it's fine
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23224: [SPARK-26277][SQL][TEST] WholeStageCodegen metrics shoul...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23224 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23256: [SPARK-24207][R] follow-up PR for SPARK-24207 to fix cod...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23256 ideally, but really not for this PR --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23256: [SPARK-24207][R] follow-up PR for SPARK-24207 to ...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23256#discussion_r239997109
--- Diff: R/pkg/tests/fulltests/test_mllib_fpm.R ---
@@ -84,19 +84,20 @@ test_that("spark.fpGrowth", {
})
test_that("spark.prefixSpan", {
-df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
- list(list(list(1L), list(3L, 2L), list(1L, 2L))),
- list(list(list(1L, 2L), list(5L))),
- list(list(list(6L, schema = c("sequence"))
-result1 <- spark.findFrequentSequentialPatterns(df, minSupport = 0.5,
maxPatternLength = 5L,
-maxLocalProjDBSize =
3200L)
-
-expected_result <- createDataFrame(list(list(list(list(1L)), 3L),
-list(list(list(3L)), 2L),
-list(list(list(2L)), 3L),
-list(list(list(1L, 2L)), 3L),
-list(list(list(1L), list(3L)),
2L)),
-schema = c("sequence", "freq"))
- })
+ df <- createDataFrame(list(list(list(list(1L, 2L), list(3L))),
+ list(list(list(1L), list(3L, 2L), list(1L,
2L))),
+ list(list(list(1L, 2L), list(5L))),
+ list(list(list(6L,
+schema = c("sequence"))
+ result <- spark.findFrequentSequentialPatterns(df, minSupport = 0.5,
maxPatternLength = 5L,
+ maxLocalProjDBSize =
3200L)
+
+ expected_result <- createDataFrame(list(list(list(list(1L)), 3L),
list(list(list(3L)), 2L),
+ list(list(list(2L)), 3L),
list(list(list(1L, 2L)), 3L),
+ list(list(list(1L), list(3L)),
2L)),
+ schema = c("sequence", "freq"))
+
+ expect_equivalent(expected_result, result)
--- End diff --
this is an important fix..
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3235: [ZEPPELIN-3864] Fix Travis tests
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3235 Can you please fix the tests then. We canât merge anything that will cause all PRs to fail in Travis. Even when they âshouldâ be ---
[GitHub] spark issue #23218: [SPARK-26266][BUILD] Update to Scala 2.12.8
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23218 do we need to relnote jvm compatibility? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23252: [SPARK-26239] File-based secret key loading for S...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23252#discussion_r239705869
--- Diff: core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala
---
@@ -440,12 +473,27 @@ class SecurityManagerSuite extends SparkFunSuite with
ResetSystemProperties {
intercept[IllegalArgumentException] {
mgr.getSecretKey()
}
+ case FILE =>
+val secretFile = createTempSecretFile()
+conf.set(AUTH_SECRET_FILE, secretFile.getAbsolutePath)
+mgr.initializeAuth()
+assert(encodeFileAsBase64(secretFile) ===
mgr.getSecretKey())
}
}
}
)
}
}
+ private def encodeFileAsBase64(secretFile: File) = {
+Base64.getEncoder.encodeToString(Files.readAllBytes(secretFile.toPath))
+ }
+
+ private def createTempSecretFile(contents: String = "test-secret"): File
= {
+val secretDir = Utils.createTempDir("temp-secrets")
+val secretFile = new File(secretDir, "temp-secret.txt")
+Files.write(secretFile.toPath,
contents.getBytes(StandardCharsets.UTF_8))
+secretFile
--- End diff --
can this secret be recovered on disk or we trust tempDir ACL is sufficient?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3254: [ZEPPELIN-3575] Add 'Copy Column Name' to table visual...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3254 any comment on this? ---
[GitHub] zeppelin issue #3235: [ZEPPELIN-3864] Fix Travis tests
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3235 as I said, travis is failing with your fix? please check this test failed? https://travis-ci.org/TinkoffCreditSystems/zeppelin/jobs/458386760 ---
[GitHub] spark issue #22305: [SPARK-24561][SQL][Python] User-defined window aggregati...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22305 I can help if this looks good to @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23184#discussion_r238120855
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala
---
@@ -225,4 +225,10 @@ private[sql] object SQLUtils extends Logging {
}
sparkSession.sessionState.catalog.listTables(db).map(_.table).toArray
}
+
+ def createArrayType(elementType: DataType): ArrayType =
DataTypes.createArrayType(elementType)
--- End diff --
yea, it's really minor, but really no one except you is working R APIs :)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23184#discussion_r238120812
--- Diff: R/pkg/R/functions.R ---
@@ -2254,40 +2255,48 @@ setMethod("date_format", signature(y = "Column", x
= "character"),
column(jc)
})
+setClassUnion("characterOrstructTypeOrColumn", c("character",
"structType", "Column"))
--- End diff --
yes
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3245: [ZEPPELIN-3893] Bug Fix that clear paragraphs when exe...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3245 merging if no more comment ---
[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23072#discussion_r238087240
--- Diff:
examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala ---
@@ -64,4 +64,3 @@ object FPGrowthExample {
spark.stop()
}
}
-// scalastyle:on println
--- End diff --
yes, println is not used
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3243: [ZEPPELIN-3886] Remove dependency on flatmap-stream 0....
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3243 could you check again? it seems travis tests are failing for other PRs because of event-stream now ---
[GitHub] zeppelin issue #3244: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and D...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3244 looks like you are blocked by https://github.com/apache/zeppelin/pull/3243 https://api.travis-ci.org/v3/job/461718800/log.txt ---
[GitHub] spark issue #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23161 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
spark git commit: [SPARK-26189][R] Fix unionAll doc in SparkR
Repository: spark
Updated Branches:
refs/heads/master 28d337440 -> 2f6e88fec
[SPARK-26189][R] Fix unionAll doc in SparkR
## What changes were proposed in this pull request?
Fix unionAll doc in SparkR
## How was this patch tested?
Manually ran test
Author: Huaxin Gao
Closes #23161 from huaxingao/spark-26189.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2f6e88fe
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2f6e88fe
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2f6e88fe
Branch: refs/heads/master
Commit: 2f6e88fecb455a02c4c08c41290e2f338e979543
Parents: 28d3374
Author: Huaxin Gao
Authored: Fri Nov 30 23:14:05 2018 -0800
Committer: Felix Cheung
Committed: Fri Nov 30 23:14:05 2018 -0800
--
R/pkg/R/DataFrame.R | 20
R/pkg/R/generics.R | 2 +-
2 files changed, 17 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/2f6e88fe/R/pkg/R/DataFrame.R
--
diff --git a/R/pkg/R/DataFrame.R b/R/pkg/R/DataFrame.R
index 745bb3e..24ed449 100644
--- a/R/pkg/R/DataFrame.R
+++ b/R/pkg/R/DataFrame.R
@@ -2730,13 +2730,25 @@ setMethod("union",
dataFrame(unioned)
})
-#' Return a new SparkDataFrame containing the union of rows
+#' Return a new SparkDataFrame containing the union of rows.
#'
-#' This is an alias for `union`.
+#' This is an alias for \code{union}.
#'
-#' @rdname union
-#' @name unionAll
+#' @param x a SparkDataFrame.
+#' @param y a SparkDataFrame.
+#' @return A SparkDataFrame containing the result of the unionAll operation.
+#' @family SparkDataFrame functions
#' @aliases unionAll,SparkDataFrame,SparkDataFrame-method
+#' @rdname unionAll
+#' @name unionAll
+#' @seealso \link{union}
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' df1 <- read.json(path)
+#' df2 <- read.json(path2)
+#' unionAllDF <- unionAll(df1, df2)
+#' }
#' @note unionAll since 1.4.0
setMethod("unionAll",
signature(x = "SparkDataFrame", y = "SparkDataFrame"),
http://git-wip-us.apache.org/repos/asf/spark/blob/2f6e88fe/R/pkg/R/generics.R
--
diff --git a/R/pkg/R/generics.R b/R/pkg/R/generics.R
index 9d8c24c..eed7646 100644
--- a/R/pkg/R/generics.R
+++ b/R/pkg/R/generics.R
@@ -631,7 +631,7 @@ setGeneric("toRDD", function(x) { standardGeneric("toRDD")
})
#' @rdname union
setGeneric("union", function(x, y) { standardGeneric("union") })
-#' @rdname union
+#' @rdname unionAll
setGeneric("unionAll", function(x, y) { standardGeneric("unionAll") })
#' @rdname unionByName
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] spark issue #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23161 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23184#discussion_r238055143
--- Diff: R/pkg/R/functions.R ---
@@ -2254,40 +2255,48 @@ setMethod("date_format", signature(y = "Column", x
= "character"),
column(jc)
})
+setClassUnion("characterOrstructTypeOrColumn", c("character",
"structType", "Column"))
--- End diff --
we should probably try to pull all the setClassUnion in one place. (to
avoid conflict or duplication)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23184#discussion_r238055087
--- Diff: R/pkg/R/functions.R ---
@@ -2254,40 +2255,48 @@ setMethod("date_format", signature(y = "Column", x
= "character"),
column(jc)
})
+setClassUnion("characterOrstructTypeOrColumn", c("character",
"structType", "Column"))
+
#' @details
#' \code{from_json}: Parses a column containing a JSON string into a
Column of \code{structType}
#' with the specified \code{schema} or array of \code{structType} if
\code{as.json.array} is set
#' to \code{TRUE}. If the string is unparseable, the Column will contain
the value NA.
#'
#' @rdname column_collection_functions
#' @param as.json.array indicating if input string is JSON array of
objects or a single object.
-#' @aliases from_json from_json,Column,characterOrstructType-method
+#' @aliases from_json from_json,Column,characterOrstructTypeOrColumn-method
#' @examples
#'
#' \dontrun{
#' df2 <- sql("SELECT named_struct('date', cast('2000-01-01' as date)) as
d")
#' df2 <- mutate(df2, d2 = to_json(df2$d, dateFormat = 'dd/MM/'))
#' schema <- structType(structField("date", "string"))
#' head(select(df2, from_json(df2$d2, schema, dateFormat = 'dd/MM/')))
-
#' df2 <- sql("SELECT named_struct('name', 'Bob') as people")
#' df2 <- mutate(df2, people_json = to_json(df2$people))
#' schema <- structType(structField("name", "string"))
#' head(select(df2, from_json(df2$people_json, schema)))
-#' head(select(df2, from_json(df2$people_json, "name STRING")))}
+#' head(select(df2, from_json(df2$people_json, "name STRING")))
+#' head(select(df2, from_json(df2$people_json,
schema_of_json(head(df2)$people_json}
#' @note from_json since 2.2.0
-setMethod("from_json", signature(x = "Column", schema =
"characterOrstructType"),
+setMethod("from_json", signature(x = "Column", schema =
"characterOrstructTypeOrColumn"),
function(x, schema, as.json.array = FALSE, ...) {
if (is.character(schema)) {
- schema <- structType(schema)
+ jschema <- structType(schema)$jobj
+} else if (class(schema) == "structType") {
+ jschema <- schema$jobj
+} else {
+ jschema <- schema@jc
}
if (as.json.array) {
- jschema <-
callJStatic("org.apache.spark.sql.types.DataTypes",
- "createArrayType",
- schema$jobj)
-} else {
- jschema <- schema$jobj
+ # This case is R-specifically different. Unlike Scala and
Python side,
--- End diff --
what if `as.json.array` is TRUE but schema is also set?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23184#discussion_r238055126
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala
---
@@ -225,4 +225,10 @@ private[sql] object SQLUtils extends Logging {
}
sparkSession.sessionState.catalog.listTables(db).map(_.table).toArray
}
+
+ def createArrayType(elementType: DataType): ArrayType =
DataTypes.createArrayType(elementType)
--- End diff --
as mentioned before, I kinda have to disagree with you here... I'd prefer
less stuff in r/SQLUtils, and instead, call the scala/jvm method directly from
R...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23184: [SPARK-26227][R] from_[csv|json] should accept sc...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23184#discussion_r238055173
--- Diff: R/pkg/R/functions.R ---
@@ -202,8 +202,9 @@ NULL
#' \itemize{
#' \item \code{from_json}: a structType object to use as the
schema to use
#' when parsing the JSON string. Since Spark 2.3, the
DDL-formatted string is
-#' also supported for the schema.
-#' \item \code{from_csv}: a DDL-formatted string
+#' also supported for the schema. Since Spark 3.0,
\code{schema_of_json} or
+#' a string literal can also be accepted.
--- End diff --
that's true...? they are the same?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22939: [SPARK-25446][R] Add schema_of_json() and schema_of_csv(...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22939 Error looks reasonable... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23161#discussion_r237383462
--- Diff: R/pkg/R/DataFrame.R ---
@@ -2732,13 +2732,24 @@ setMethod("union",
dataFrame(unioned)
})
-#' Return a new SparkDataFrame containing the union of rows
-#'
-#' This is an alias for `union`.
+#' Return a new SparkDataFrame containing the union of rows.
+#' This is an alias for \code{union}.
--- End diff --
actually, we do need a newline - L2735 is the title as this is a new page.
could you build the roxygen2 doc to double check?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3243: [ZEPPELIN-3886] Remove dependency on flatmap-stream 0....
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3243 ouch, thanks for fixing this. @prabhjyotsingh @jongyoul @Leemoonsoo this is important. I checked and don't think we have released with this commit https://github.com/apache/zeppelin/commit/affccab6ab3577a815e02bb4d88aa1e90d973d16 https://issues.apache.org/jira/browse/ZEPPELIN-3781 ---
[GitHub] zeppelin issue #3242: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and D...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3242 can you check the failed tests? ---
[GitHub] spark pull request #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23161#discussion_r236972877
--- Diff: R/pkg/R/DataFrame.R ---
@@ -2732,14 +2732,24 @@ setMethod("union",
dataFrame(unioned)
})
-#' Return a new SparkDataFrame containing the union of rows
-#'
-#' This is an alias for `union`.
+#' Return a new SparkDataFrame containing the union of rows.
+#' This is an alias for \code{union}.
#'
-#' @rdname union
-#' @name unionAll
+#' @param x a SparkDataFrame.
+#' @param y a SparkDataFrame.
+#' @return A SparkDataFrame containing the result of the unionAll
operation.
+#' @family SparkDataFrame functions
#' @aliases unionAll,SparkDataFrame,SparkDataFrame-method
-#' @note unionAll since 1.4.0
+#' @rdname unionAll
+#' @name unionAll
--- End diff --
I'd add a `@seealso` to union
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23161: [SPARK-26189][R]Fix unionAll doc in SparkR
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23161#discussion_r236973169
--- Diff: R/pkg/R/DataFrame.R ---
@@ -2732,14 +2732,24 @@ setMethod("union",
dataFrame(unioned)
})
-#' Return a new SparkDataFrame containing the union of rows
-#'
-#' This is an alias for `union`.
+#' Return a new SparkDataFrame containing the union of rows.
+#' This is an alias for \code{union}.
#'
-#' @rdname union
-#' @name unionAll
+#' @param x a SparkDataFrame.
+#' @param y a SparkDataFrame.
+#' @return A SparkDataFrame containing the result of the unionAll
operation.
+#' @family SparkDataFrame functions
#' @aliases unionAll,SparkDataFrame,SparkDataFrame-method
-#' @note unionAll since 1.4.0
+#' @rdname unionAll
+#' @name unionAll
+#' @examples
+#'\dontrun{
+#' sparkR.session()
+#' df1 <- read.json(path)
+#' df2 <- read.json(path2)
+#' unionAllDF <- unionAll(df1, df2)
+#' }
+#' @note unionAll since 3.0.0
--- End diff --
this should be `1.4.0` as above
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3242: [ZEPPELIN-3882] Neo4jInterpreter - Support Point and D...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3242 perhaps we should have a min version compatible in the interpreter doc? otherwise I think it's fine not to have tested version, supported version etc in doc since then we'd need to keep updating them ---
[GitHub] spark pull request #23025: [SPARK-26024][SQL]: Update documentation for repa...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23025#discussion_r236970732
--- Diff: R/pkg/R/DataFrame.R ---
@@ -767,6 +767,14 @@ setMethod("repartition",
#' using \code{spark.sql.shuffle.partitions} as
number of partitions.}
#'}
#'
+#' At least one partition-by expression must be specified.
--- End diff --
761 is significant also, but correct.
essentially:
1. first line of the blob is the title (L760)
2. second text after "empty line" is the description (L762)
3. third after another "empty line" is the "detail note" which is stashed
all the way to the bottom of the doc page
so generally you want "important" part of the description on top and not in
the "detail" section because it is easily missed.
this is the most common pattern in this code base. there's another, where
multiple function is doc together as a group, eg. collection sql function (in
functions.R). other finer control is possible as well but not used today in
this code base.
similarly L829 is good, L831 is a bit fuzzy - I'd personally prefer without
L831 to keep the whole text in the description section of the doc.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23072#discussion_r236771417 --- Diff: docs/ml-clustering.md --- @@ -265,3 +265,44 @@ Refer to the [R API docs](api/R/spark.gaussianMixture.html) for more details. + +## Power Iteration Clustering (PIC) + +Power Iteration Clustering (PIC) is a scalable graph clustering algorithm --- End diff -- could you open a separate PR with just this file (minus R) and FPGrowthExample.scala on branch-2.4? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22939: [SPARK-25446][R] Add schema_of_json() and schema_...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/22939#discussion_r236770223
--- Diff: R/pkg/R/functions.R ---
@@ -2230,6 +2237,32 @@ setMethod("from_json", signature(x = "Column",
schema = "characterOrstructType")
column(jc)
})
+#' @details
+#' \code{schema_of_json}: Parses a JSON string and infers its schema in
DDL format.
+#'
+#' @rdname column_collection_functions
+#' @aliases schema_of_json schema_of_json,characterOrColumn-method
+#' @examples
+#'
+#' \dontrun{
+#' json <- '{"name":"Bob"}'
+#' df <- sql("SELECT * FROM range(1)")
+#' head(select(df, schema_of_json(json)))}
+#' @note schema_of_json since 3.0.0
+setMethod("schema_of_json", signature(x = "characterOrColumn"),
+ function(x, ...) {
+if (class(x) == "character") {
+ col <- callJStatic("org.apache.spark.sql.functions", "lit",
x)
+} else {
+ col <- x@jc
--- End diff --
maybe to think about the design of API in R and Scala and else where - what
does it look like when the user passes in a column that is not a literal
string? probably worthwhile to follow up separately.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3241: ZEPPELIN-3880: Code refactoring: remove class files us...
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3241 tbh, this seems rather minor. I'd leave this for others to comment. ---
[GitHub] spark pull request #23145: [MINOR][Docs][WIP] Fix Typos
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23145#discussion_r236765511 --- Diff: docs/index.md --- @@ -67,7 +67,7 @@ Example applications are also provided in Python. For example, ./bin/spark-submit examples/src/main/python/pi.py 10 Spark also provides an experimental [R API](sparkr.html) since 1.4 (only DataFrames APIs included). --- End diff -- let's remove `experimental`, WDYT? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support fo...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23098#discussion_r236764795
--- Diff: R/pkg/R/sparkR.R ---
@@ -269,7 +269,7 @@ sparkR.sparkContext <- function(
#' sparkR.session("yarn-client", "SparkR", "/home/spark",
#'list(spark.executor.memory="4g"),
#'c("one.jar", "two.jar", "three.jar"),
-#'c("com.databricks:spark-avro_2.11:2.0.1"))
+#'c("com.databricks:spark-avro_2.12:2.0.1"))
--- End diff --
yes, dummy name is completely fine with me.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23131: [SPARK-25908][SQL][FOLLOW-UP] Add back unionAll
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23131#discussion_r236763355
--- Diff: R/pkg/R/DataFrame.R ---
@@ -2732,6 +2732,20 @@ setMethod("union",
dataFrame(unioned)
})
+#' Return a new SparkDataFrame containing the union of rows
+#'
+#' This is an alias for `union`.
--- End diff --
also backtick doesn't format with roxygen2. this should be
```
This is an alias for \code{union}.
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23025: [SPARK-26024][SQL]: Update documentation for repa...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23025#discussion_r236762465
--- Diff: R/pkg/R/DataFrame.R ---
@@ -767,6 +767,14 @@ setMethod("repartition",
#' using \code{spark.sql.shuffle.partitions} as
number of partitions.}
#'}
#'
+#' At least one partition-by expression must be specified.
--- End diff --
this won't be formatted correctly in R doc due to the fact that "empty
line" is significant. L769 should be removed to ensure it is in description
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23131: [SPARK-25908][SQL][FOLLOW-UP] Add back unionAll
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23131#discussion_r236760822
--- Diff: R/pkg/R/DataFrame.R ---
@@ -2732,6 +2732,20 @@ setMethod("union",
dataFrame(unioned)
})
+#' Return a new SparkDataFrame containing the union of rows
+#'
+#' This is an alias for `union`.
--- End diff --
If the goal is for this to be like other *All, this should go into a
separate doc page, plus seealso, example etc.
The way this was written, as it was a deprecated function, this doc page
merge with union - as it is, none of the text above will show up and also
unionAll will not be listed in method index list.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23145: [MINOR][Docs] "a R interpreter" -> "an R interpre...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23145#discussion_r236546043 --- Diff: docs/index.md --- @@ -67,7 +67,7 @@ Example applications are also provided in Python. For example, ./bin/spark-submit examples/src/main/python/pi.py 10 Spark also provides an experimental [R API](sparkr.html) since 1.4 (only DataFrames APIs included). --- End diff -- tbh, I'm not sure this should be called "an experimental [R API]" --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] zeppelin issue #3235: [ZEPPELIN-3864][ZP-68] Fix Travis tests
Github user felixcheung commented on the issue: https://github.com/apache/zeppelin/pull/3235 this test failed? https://travis-ci.org/TinkoffCreditSystems/zeppelin/jobs/458386760 ---
[GitHub] spark issue #23089: [SPARK-26120][TESTS][SS][SPARKR]Fix a streaming query le...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23089 Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22939: [SPARK-25446][R] Add schema_of_json() and schema_of_csv(...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22939 Sorry for the delay, will do another pass in 1 or 2 days --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23072#discussion_r234432181
--- Diff: R/pkg/R/mllib_clustering.R ---
@@ -610,3 +616,57 @@ setMethod("write.ml", signature(object = "LDAModel",
path = "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PowerIterationClustering
+#'
+#' A scalable graph clustering algorithm. Users can call
\code{spark.assignClusters} to
+#' return a cluster assignment for each input vertex.
+#'
+# Run the PIC algorithm and returns a cluster assignment for each input
vertex.
+#' @param data A SparkDataFrame.
+#' @param k The number of clusters to create.
+#' @param initMode Param for the initialization algorithm.
+#' @param maxIter Param for maximum number of iterations.
+#' @param srcCol Param for the name of the input column for source vertex
IDs.
+#' @param dstCol Name of the input column for destination vertex IDs.
+#' @param weightCol Param for weight column name. If this is not set or
\code{NULL},
+#' we treat all instance weights as 1.0.
+#' @param ... additional argument(s) passed to the method.
+#' @return A dataset that contains columns of vertex id and the
corresponding cluster for the id.
+#' The schema of it will be:
+#' \code{id: Long}
+#' \code{cluster: Int}
+#' @rdname spark.powerIterationClustering
+#' @aliases
assignClusters,PowerIterationClustering-method,SparkDataFrame-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(0L, 1L, 1.0), list(0L, 2L, 1.0),
+#' list(1L, 2L, 1.0), list(3L, 4L, 1.0),
+#' list(4L, 0L, 0.1)), schema = c("src", "dst",
"weight"))
+#' clusters <- spark.assignClusters(df, initMode="degree",
weightCol="weight")
+#' showDF(clusters)
+#' }
+#' @note spark.assignClusters(SparkDataFrame) since 3.0.0
+setMethod("spark.assignClusters",
+ signature(data = "SparkDataFrame"),
+ function(data, k = 2L, initMode = "random", maxIter = 20L,
srcCol = "src",
+dstCol = "dst", weightCol = NULL) {
--- End diff --
I think we try to avoid srcCol dstCol in R (I think there are other R ml
APIs like that)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23072#discussion_r234432019
--- Diff: R/pkg/R/mllib_clustering.R ---
@@ -610,3 +616,57 @@ setMethod("write.ml", signature(object = "LDAModel",
path = "character"),
function(object, path, overwrite = FALSE) {
write_internal(object, path, overwrite)
})
+
+#' PowerIterationClustering
+#'
+#' A scalable graph clustering algorithm. Users can call
\code{spark.assignClusters} to
+#' return a cluster assignment for each input vertex.
+#'
+# Run the PIC algorithm and returns a cluster assignment for each input
vertex.
+#' @param data A SparkDataFrame.
+#' @param k The number of clusters to create.
+#' @param initMode Param for the initialization algorithm.
+#' @param maxIter Param for maximum number of iterations.
+#' @param srcCol Param for the name of the input column for source vertex
IDs.
+#' @param dstCol Name of the input column for destination vertex IDs.
+#' @param weightCol Param for weight column name. If this is not set or
\code{NULL},
+#' we treat all instance weights as 1.0.
+#' @param ... additional argument(s) passed to the method.
+#' @return A dataset that contains columns of vertex id and the
corresponding cluster for the id.
+#' The schema of it will be:
+#' \code{id: Long}
+#' \code{cluster: Int}
+#' @rdname spark.powerIterationClustering
+#' @aliases
assignClusters,PowerIterationClustering-method,SparkDataFrame-method
+#' @examples
+#' \dontrun{
+#' df <- createDataFrame(list(list(0L, 1L, 1.0), list(0L, 2L, 1.0),
+#' list(1L, 2L, 1.0), list(3L, 4L, 1.0),
+#' list(4L, 0L, 0.1)), schema = c("src", "dst",
"weight"))
+#' clusters <- spark.assignClusters(df, initMode="degree",
weightCol="weight")
+#' showDF(clusters)
+#' }
+#' @note spark.assignClusters(SparkDataFrame) since 3.0.0
+setMethod("spark.assignClusters",
+ signature(data = "SparkDataFrame"),
+ function(data, k = 2L, initMode = "random", maxIter = 20L,
srcCol = "src",
--- End diff --
set valid values for initMode and check for it - eg.
https://github.com/apache/spark/pull/23072/files#diff-d9f92e07db6424e2527a7f9d7caa9013R355
and `match.arg(initMode)`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23072: [SPARK-19827][R]spark.ml R API for PIC
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23072#discussion_r234432049
--- Diff: R/pkg/vignettes/sparkr-vignettes.Rmd ---
@@ -968,6 +970,17 @@ predicted <- predict(model, df)
head(predicted)
```
+ Power Iteration Clustering
+
+Power Iteration Clustering (PIC) is a scalable graph clustering algorithm.
`spark.assignClusters` method runs the PIC algorithm and returns a cluster
assignment for each input vertex.
+
+```{r}
+df <- createDataFrame(list(list(0L, 1L, 1.0), list(0L, 2L, 1.0),
+ list(1L, 2L, 1.0), list(3L, 4L, 1.0),
+ list(4L, 0L, 0.1)), schema = c("src", "dst",
"weight"))
+head(spark.assignClusters(df, initMode="degree", weightCol="weight"))
--- End diff --
spacing: `initMode = "degree", weightCol = "weight"`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23073: [SPARK-26104] expose pci info to task scheduler
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23073#discussion_r234431864
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorData.scala ---
@@ -27,12 +27,14 @@ import org.apache.spark.rpc.{RpcAddress, RpcEndpointRef}
* @param executorHost The hostname that this executor is running on
* @param freeCores The current number of cores available for work on the
executor
* @param totalCores The total number of cores available to the executor
+ * @param pcis The external devices avaliable to the executor
--- End diff --
available
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23073: [SPARK-26104] expose pci info to task scheduler
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23073 please put ^ comment into PR description (because comment is not included in commit message once the PR is merged) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 Yea there are some problem with some packages we depend on that are not installable from CRAN (eg too old) so it will be hard to a new version of R and new installation. So to clarify, depreciation as is we still test on R 3.1? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 Hey shane I donât think we are saying to test multiple R version at all. In fact quite the opposite, just the new(er) version at some point in the future. (We donât have a better solution for packages though. Thereâs another PR for R arrow package for example) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 I think it's easier to say unsupported if we are not testing it in jenkins or appveyer. I don't know if we any coverage at release for older R version anyway, so it's better to unsupported then deprecate. but agree maybe the way to do this is deprecate without updating R in jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23017: [WIP][SPARK-26015][K8S] Set a default UID for Spark on K...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23017 noted test issue. let's kick off test though --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23017: [WIP][SPARK-26015][K8S] Set a default UID for Spark on K...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23017 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23007: [SPARK-26010][R] fix vignette eval with Java 11
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23007 merged to master/2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
spark git commit: [SPARK-26010][R] fix vignette eval with Java 11
Repository: spark
Updated Branches:
refs/heads/branch-2.4 3bc4c3330 -> 65e5b2659
[SPARK-26010][R] fix vignette eval with Java 11
## What changes were proposed in this pull request?
changes in vignette only to disable eval
## How was this patch tested?
Jenkins
Author: Felix Cheung
Closes #23007 from felixcheung/rjavavervig.
(cherry picked from commit 88c82627267a9731b2438f0cc28dd656eb3dc834)
Signed-off-by: Felix Cheung
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/65e5b265
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/65e5b265
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/65e5b265
Branch: refs/heads/branch-2.4
Commit: 65e5b26590e66ac4220b5f60e11b7966746c8b08
Parents: 3bc4c33
Author: Felix Cheung
Authored: Mon Nov 12 19:03:30 2018 -0800
Committer: Felix Cheung
Committed: Mon Nov 12 19:03:56 2018 -0800
--
R/pkg/vignettes/sparkr-vignettes.Rmd | 14 ++
1 file changed, 14 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/65e5b265/R/pkg/vignettes/sparkr-vignettes.Rmd
--
diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd
b/R/pkg/vignettes/sparkr-vignettes.Rmd
index 090363c..b13f338 100644
--- a/R/pkg/vignettes/sparkr-vignettes.Rmd
+++ b/R/pkg/vignettes/sparkr-vignettes.Rmd
@@ -57,6 +57,20 @@ First, let's load and attach the package.
library(SparkR)
```
+```{r, include=FALSE}
+# disable eval if java version not supported
+override_eval <- tryCatch(!is.numeric(SparkR:::checkJavaVersion()),
+ error = function(e) { TRUE },
+ warning = function(e) { TRUE })
+
+if (override_eval) {
+ opts_hooks$set(eval = function(options) {
+options$eval = FALSE
+options
+ })
+}
+```
+
`SparkSession` is the entry point into SparkR which connects your R program to
a Spark cluster. You can create a `SparkSession` using `sparkR.session` and
pass in options such as the application name, any Spark packages depended on,
etc.
We use default settings in which it runs in local mode. It auto downloads
Spark package in the background if no previous installation is found. For more
details about setup, see [Spark Session](#SetupSparkSession).
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] spark issue #22866: WIP [SPARK-12172][SPARKR] Remove internal-only RDD metho...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/22866 thx, but DO NOT MERGE - there's some nasty bug I'm still investigating.. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
spark git commit: [SPARK-26010][R] fix vignette eval with Java 11
Repository: spark
Updated Branches:
refs/heads/master f9ff75653 -> 88c826272
[SPARK-26010][R] fix vignette eval with Java 11
## What changes were proposed in this pull request?
changes in vignette only to disable eval
## How was this patch tested?
Jenkins
Author: Felix Cheung
Closes #23007 from felixcheung/rjavavervig.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/88c82627
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/88c82627
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/88c82627
Branch: refs/heads/master
Commit: 88c82627267a9731b2438f0cc28dd656eb3dc834
Parents: f9ff756
Author: Felix Cheung
Authored: Mon Nov 12 19:03:30 2018 -0800
Committer: Felix Cheung
Committed: Mon Nov 12 19:03:30 2018 -0800
--
R/pkg/vignettes/sparkr-vignettes.Rmd | 14 ++
1 file changed, 14 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/88c82627/R/pkg/vignettes/sparkr-vignettes.Rmd
--
diff --git a/R/pkg/vignettes/sparkr-vignettes.Rmd
b/R/pkg/vignettes/sparkr-vignettes.Rmd
index 7d924ef..f80b45b 100644
--- a/R/pkg/vignettes/sparkr-vignettes.Rmd
+++ b/R/pkg/vignettes/sparkr-vignettes.Rmd
@@ -57,6 +57,20 @@ First, let's load and attach the package.
library(SparkR)
```
+```{r, include=FALSE}
+# disable eval if java version not supported
+override_eval <- tryCatch(!is.numeric(SparkR:::checkJavaVersion()),
+ error = function(e) { TRUE },
+ warning = function(e) { TRUE })
+
+if (override_eval) {
+ opts_hooks$set(eval = function(options) {
+options$eval = FALSE
+options
+ })
+}
+```
+
`SparkSession` is the entry point into SparkR which connects your R program to
a Spark cluster. You can create a `SparkSession` using `sparkR.session` and
pass in options such as the application name, any Spark packages depended on,
etc.
We use default settings in which it runs in local mode. It auto downloads
Spark package in the background if no previous installation is found. For more
details about setup, see [Spark Session](#SetupSparkSession).
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23012#discussion_r232881732
--- Diff: R/pkg/R/sparkR.R ---
@@ -283,6 +283,10 @@ sparkR.session <- function(
enableHiveSupport = TRUE,
...) {
+ if (utils::compareVersion(paste0(R.version$major, ".", R.version$minor),
"3.4.0") == -1) {
+warning("R prior to version 3.4 is deprecated as of Spark 3.0.")
+ }
--- End diff --
ditto
`Support for R prior to version 3.4 is deprecated since Spark 3.0.0`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23012#discussion_r232882419
--- Diff: docs/index.md ---
@@ -31,7 +31,8 @@ Spark runs on both Windows and UNIX-like systems (e.g.
Linux, Mac OS). It's easy
locally on one machine --- all you need is to have `java` installed on
your system `PATH`,
or the `JAVA_HOME` environment variable pointing to a Java installation.
-Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API,
Spark {{site.SPARK_VERSION}}
+Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. R prior to version 3.4
is deprecated as of Spark 3.0.
--- End diff --
`R prior to version 3.4 support is deprecated as of Spark 3.0.0.`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/23012#discussion_r232882178
--- Diff: docs/index.md ---
@@ -31,7 +31,8 @@ Spark runs on both Windows and UNIX-like systems (e.g.
Linux, Mac OS). It's easy
locally on one machine --- all you need is to have `java` installed on
your system `PATH`,
or the `JAVA_HOME` environment variable pointing to a Java installation.
-Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API,
Spark {{site.SPARK_VERSION}}
+Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. R prior to version 3.4
is deprecated as of Spark 3.0.
--- End diff --
with all the other changes, we haven't listed all deprecation here, or have
we?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/23012#discussion_r232881594 --- Diff: R/WINDOWS.md --- @@ -3,7 +3,7 @@ To build SparkR on Windows, the following steps are required 1. Install R (>= 3.1) and [Rtools](http://cran.r-project.org/bin/windows/Rtools/). Make sure to -include Rtools and R in `PATH`. +include Rtools and R in `PATH`. Note that R prior to version 3.4 is deprecated as of Spark 3.0. --- End diff -- I really would prefer "unsupported" but if we go with this it should say `Note that support for R prior to version 3.4 is deprecated as of Spark 3.0.0.` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23012: [SPARK-26014][R] Deprecate R prior to version 3.4 in Spa...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/23012 FYI This is unused code Iâm going to remove it https://github.com/apache/spark/blob/master/R/pkg/src-native/string_hash_code.c --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
