[spark] branch branch-3.4 updated: [SPARK-42656][CONNECT] Adding SCALA REPL shell script for JVM client
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 268a96c715e [SPARK-42656][CONNECT] Adding SCALA REPL shell script for
JVM client
268a96c715e is described below
commit 268a96c715e03b2751e882583f148c053f476302
Author: Zhen Li
AuthorDate: Fri Mar 3 14:55:03 2023 +0900
[SPARK-42656][CONNECT] Adding SCALA REPL shell script for JVM client
### What changes were proposed in this pull request?
Adding a simple script to start the Scala client in the Scala REPL. As well
as a script to start the spark connect server for the client to connect to.
### Why are the changes needed?
Make the JVM client more easy to be used.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually tested.
Closes #40257 from zhenlineo/jshell.
Lead-authored-by: Zhen Li
Co-authored-by: Herman van Hovell
Signed-off-by: Hyukjin Kwon
(cherry picked from commit 86e578ac49eff1b1ad3ca49f921a0f45a843da10)
Signed-off-by: Hyukjin Kwon
---
connector/connect/bin/spark-connect| 32 ++
connector/connect/bin/spark-connect-scala-client | 50 ++
.../connect/bin/spark-connect-scala-client.sc | 33 ++
3 files changed, 115 insertions(+)
diff --git a/connector/connect/bin/spark-connect
b/connector/connect/bin/spark-connect
new file mode 100755
index 000..008209c8440
--- /dev/null
+++ b/connector/connect/bin/spark-connect
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Go to the Spark project root directory
+FWDIR="$(cd "`dirname "$0"`"/../../..; pwd)"
+cd "$FWDIR"
+export SPARK_HOME=$FWDIR
+
+# Build the jars needed for spark submit and spark connect
+build/sbt package
+
+SCALA_BINARY_VER=`grep "scala.binary.version" "${SPARK_HOME}/pom.xml" | head
-n1 | awk -F '[<>]' '{print $3}'`
+
+CONNECT_JAR=`ls
"${SPARK_HOME}"/connector/connect/server/target/scala-"${SCALA_BINARY_VER}"/spark-connect-assembly*.jar
| paste -sd ',' -`
+
+exec "${SPARK_HOME}"/bin/spark-submit --class
org.apache.spark.sql.connect.SimpleSparkConnectService "$CONNECT_JAR"
\ No newline at end of file
diff --git a/connector/connect/bin/spark-connect-scala-client
b/connector/connect/bin/spark-connect-scala-client
new file mode 100755
index 000..902091a74de
--- /dev/null
+++ b/connector/connect/bin/spark-connect-scala-client
@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Use the spark connect JVM client to connect to a spark connect server.
+#
+# Start a local server:
+# A local spark-connect server with default settings can be started using the
following command:
+# `connector/connect/bin/spark-connect`
+# The client should be able to connect to this server directly with the
default client settings.
+#
+# Connect to a remote server:
+# To connect to a remote server, use env var `SPARK_REMOTE` to configure the
client connection
+# string. e.g.
+# `export SPARK_REMOTE="sc://:/;token=;="`
+
+# Go to the Spark project root directory
+FWDIR="$(cd "`dirname "$0"`"/../../..; pwd)"
+cd "$FWDIR"
+export SPARK_HOME=$FWDIR
+
+# Build the jars needed for spark connect JVM client
+build/sbt "sql/package;connect-c
[spark] branch branch-3.4 updated: [SPARK-42656][CONNECT] Adding SCALA REPL shell script for JVM client
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 9023f301c78 [SPARK-42656][CONNECT] Adding SCALA REPL shell script for
JVM client
9023f301c78 is described below
commit 9023f301c78f0faa68b555389f40a35df9b50437
Author: Zhen Li
AuthorDate: Thu Mar 2 22:00:32 2023 -0400
[SPARK-42656][CONNECT] Adding SCALA REPL shell script for JVM client
### What changes were proposed in this pull request?
Adding a simple script to start the Scala client in the Scala REPL. As well
as a script to start the spark connect server for the client to connect to.
### Why are the changes needed?
Make the JVM client more easy to be used.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually tested.
Closes #40257 from zhenlineo/jshell.
Authored-by: Zhen Li
Signed-off-by: Herman van Hovell
(cherry picked from commit 7505e5cdd12661dd5c96bfd004185bf6fc4eb33b)
Signed-off-by: Herman van Hovell
---
connector/connect/bin/spark-connect| 32 ++
connector/connect/bin/spark-connect-scala-client | 50 ++
.../connect/bin/spark-connect-scala-client.sc | 15 +++
3 files changed, 97 insertions(+)
diff --git a/connector/connect/bin/spark-connect
b/connector/connect/bin/spark-connect
new file mode 100755
index 000..008209c8440
--- /dev/null
+++ b/connector/connect/bin/spark-connect
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Go to the Spark project root directory
+FWDIR="$(cd "`dirname "$0"`"/../../..; pwd)"
+cd "$FWDIR"
+export SPARK_HOME=$FWDIR
+
+# Build the jars needed for spark submit and spark connect
+build/sbt package
+
+SCALA_BINARY_VER=`grep "scala.binary.version" "${SPARK_HOME}/pom.xml" | head
-n1 | awk -F '[<>]' '{print $3}'`
+
+CONNECT_JAR=`ls
"${SPARK_HOME}"/connector/connect/server/target/scala-"${SCALA_BINARY_VER}"/spark-connect-assembly*.jar
| paste -sd ',' -`
+
+exec "${SPARK_HOME}"/bin/spark-submit --class
org.apache.spark.sql.connect.SimpleSparkConnectService "$CONNECT_JAR"
\ No newline at end of file
diff --git a/connector/connect/bin/spark-connect-scala-client
b/connector/connect/bin/spark-connect-scala-client
new file mode 100755
index 000..902091a74de
--- /dev/null
+++ b/connector/connect/bin/spark-connect-scala-client
@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Use the spark connect JVM client to connect to a spark connect server.
+#
+# Start a local server:
+# A local spark-connect server with default settings can be started using the
following command:
+# `connector/connect/bin/spark-connect`
+# The client should be able to connect to this server directly with the
default client settings.
+#
+# Connect to a remote server:
+# To connect to a remote server, use env var `SPARK_REMOTE` to configure the
client connection
+# string. e.g.
+# `export SPARK_REMOTE="sc://:/;token=;="`
+
+# Go to the Spark project root directory
+FWDIR="$(cd "`dirname "$0"`"/../../..; pwd)"
+cd "$FWDIR"
+export SPARK_HOME=$FWDIR
+
+# Build the jars needed for spark connect JVM client
+build/sbt "sql/package;connect-client-jvm/assembly"
+
+CONNECT_CLASSPATH=
