sekikn opened a new pull request, #1330:
URL: https://github.com/apache/bigtop/pull/1330
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'BIGTOP-3638: Your PR title ...'.
-->
### Description of PR
This PR fixes bigpetstore-spark to use Spark 3.5.4 and Scala 2.12, instead
of 1.3.0 and 2.10 respectively.
Note that it is intentional to built against Spark 3.5.4 while our stack
currently has Spark 3.5.3. I came across some dependency problem when building
it with Spark 3.5.3.
### How was this patch tested?
Before building bigpetstore-spark itself, we have to build the
com.github.rnowling.bigpetstore package and install it locally due to the
Jcenter and Bintray shutdown (BIGTOP-1961 will resolve it).
```
$ git clone --branch=v0.2.1 --depth=1
https://github.com/rnowling/bigpetstore-data-generator.git
$ cd bigpetstore-data-generator
$ vi build.gradle # add the following lines
$ git diff
diff --git a/build.gradle b/build.gradle
index 23cae6b..ee35e00 100644
--- a/build.gradle
+++ b/build.gradle
@@ -1,6 +1,15 @@
apply plugin: 'eclipse'
apply plugin: 'groovy'
apply plugin: 'java'
+apply plugin: 'maven-publish'
+
+publishing {
+ publications {
+ mavenJava(MavenPublication) {
+ artifact jar
+ }
+ }
+}
group = 'com.github.rnowling.bigpetstore'
version = '0.2.1'
$ gradle -version
------------------------------------------------------------
Gradle 5.6.4
------------------------------------------------------------
Build time: 2019-11-01 20:42:00 UTC
Revision: dd870424f9bd8e195d614dc14bb140f43c22da98
Kotlin: 1.3.41
Groovy: 2.5.4
Ant: Apache Ant(TM) version 1.9.14 compiled on March 12 2019
JVM: 1.8.0_432 (Amazon.com Inc. 25.432-b06)
OS: Linux 6.11.0-17-generic amd64
$ gradle clean publishToMavenLocal
```
Then we can build bigpetstore-spark and run it.
```
$ cd /path/to/bigpetstore-spark
$ ./gradlew clean shadowJar
$ spark-submit --master local[*] --class
org.apache.bigtop.bigpetstore.spark.generator.SparkDriver
build/libs/bigpetstore-spark-3.4.0-SNAPSHOT-all.jar generated_data 10 1000
365.0 345
25/02/19 15:37:42 WARN Utils: Your hostname, sekikn-MotherBoard-Series
resolves to a loopback address: 127.0.1.1; using 192.168.2.155 instead (on
interface enp6s0)
25/02/19 15:37:42 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
another address
25/02/19 15:37:42 INFO SparkContext: Running Spark version 3.5.3
25/02/19 15:37:42 INFO SparkContext: OS info Linux, 6.11.0-17-generic, amd64
25/02/19 15:37:42 INFO SparkContext: Java version 1.8.0_432
(snip)
25/02/19 15:37:45 INFO SparkContext: Successfully stopped SparkContext
25/02/19 15:37:45 INFO ShutdownHookManager: Shutdown hook called
25/02/19 15:37:45 INFO ShutdownHookManager: Deleting directory
/tmp/spark-b3100c2f-bc59-4d4d-8221-72c99d662ba8
25/02/19 15:37:45 INFO ShutdownHookManager: Deleting directory
/tmp/spark-a70e001c-f810-44ed-acba-54e735d608bf
$ ls generated_data/transactions
part-00000 part-00003 part-00006 part-00009 part-00012 part-00015
part-00018 part-00021 part-00024 part-00027 part-00030
part-00001 part-00004 part-00007 part-00010 part-00013 part-00016
part-00019 part-00022 part-00025 part-00028 part-00031
part-00002 part-00005 part-00008 part-00011 part-00014 part-00017
part-00020 part-00023 part-00026 part-00029 _SUCCESS
```
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'BIGTOP-3638. Your PR title ...')?
- [x] Make sure that newly added files do not have any licensing issues.
When in doubt refer to https://www.apache.org/licenses/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
