[
https://issues.apache.org/jira/browse/BEAM-3587?focusedWorklogId=154059&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-154059
]
ASF GitHub Bot logged work on BEAM-3587:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Oct/18 23:36
Start Date: 12/Oct/18 23:36
Worklog Time Spent: 10m
Work Description: swegner closed pull request #384: [BEAM-3587] Add a
note to Gradle shadowJar for merge service files
URL: https://github.com/apache/beam-site/pull/384
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/src/documentation/runners/flink.md
b/src/documentation/runners/flink.md
index 6dc6e7b69d..df64ba46f9 100644
--- a/src/documentation/runners/flink.md
+++ b/src/documentation/runners/flink.md
@@ -51,7 +51,7 @@ For more information, the [Flink
Documentation](https://ci.apache.org/projects/f
```java
<dependency>
<groupId>org.apache.beam</groupId>
- <artifactId>beam-runners-flink_2.10</artifactId>
+ <artifactId>beam-runners-flink_2.11</artifactId>
<version>{{ site.release_latest }}</version>
<scope>runtime</scope>
</dependency>
@@ -81,6 +81,62 @@ $ mvn exec:java
-Dexec.mainClass=org.apache.beam.examples.WordCount \
If you have a Flink `JobManager` running on your local machine you can give
`localhost:6123` for
`flinkMaster`.
+Behind the hood, to create your shaded jar (containing your pipeline and the
Flink runner dependencies), you have to use the `maven-shade-plugin`:
+
+```java
+ <dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-runners-flink_2.10</artifactId>
+ <version>{{ site.release_latest }}</version>
+ </dependency>
+```
+
+```java
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>${maven-shade-plugin.version}</version>
+ <configuration>
+
<createDependencyReducedPom>false</createDependencyReducedPom>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ </configuration>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+
<shadedArtifactAttached>true</shadedArtifactAttached>
+ <shadedClassifierName>shaded</shadedClassifierName>
+ <transformers>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
+ </transformers>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+```
+
+Then, Maven build will create the shaded jar.
+
+If you prefer to use Gradle, you can achieve the same using `shadowJar`:
+
+```java
+shadowJar {
+ mergeServiceFiles()
+}
+```
+
## Pipeline options for the Flink Runner
When executing your pipeline with the Flink Runner, you can set these pipeline
options.
diff --git a/src/documentation/runners/spark.md
b/src/documentation/runners/spark.md
index 1502f242c0..4b4479e0e3 100644
--- a/src/documentation/runners/spark.md
+++ b/src/documentation/runners/spark.md
@@ -37,7 +37,7 @@ You can add a dependency on the latest version of the Spark
runner by adding to
### Deploying Spark with your application
-In some cases, such as running in local mode/Standalone, your (self-contained)
application would be required to pack Spark by explicitly adding the following
dependencies in your pom.xml:
+Most of the time (running in local mode/Standalone or using `spark-submit`),
your (self-contained) application would be required to pack Spark by explicitly
adding the following dependencies in your pom.xml:
```java
<dependency>
<groupId>org.apache.spark</groupId>
@@ -94,6 +94,17 @@ After running <code>mvn package</code>, run <code>ls
target</code> and you shoul
beam-examples-1.0.0-shaded.jar
```
+If you are using gradle, you have to use `shadowJar` to create the shaded jar
enabling `mergeServiceFiles()`:
+```java
+shadowJar {
+ transform(AppendingTransformer) {
+ resource = 'reference.conf'
+ }
+ relocate 'com.google.protobuf', 'shaded.protobuf'
+ mergeServiceFiles()
+}
+```
+
To run against a Standalone cluster simply run:
```
spark-submit --class com.beam.examples.BeamPipeline --master spark://HOST:PORT
target/beam-examples-1.0.0-shaded.jar --runner=SparkRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 154059)
Time Spent: 1h 20m (was: 1h 10m)
> User reports TextIO failure in FlinkRunner on master
> ----------------------------------------------------
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
> Issue Type: Bug
> Components: website
> Reporter: Kenneth Knowles
> Assignee: Jean-Baptiste Onofré
> Priority: Minor
> Fix For: Not applicable
>
> Attachments: screen1.png, screen2.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Reported here:
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink
> cluster, using the latest Beam git revision (ff37337). The job fails to start
> with the Exception:
> {{java.lang.UnsupportedOperationException: The transform is currently not
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps
> [https://github.com/pelletier/beam-flink-example]";
> My initial thoughts: I have a guess that this has to do with switching to
> running from a portable pipeline representation, and it looks like there's a
> non-composite transform with an empty URN and it threw a bad error message.
> We can try to root cause but may also mitigate short-term by removing the
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working -
> they only run on a local Flink, yet this seems to be a translation issue that
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the
> bottom of it quickly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
