[
https://issues.apache.org/jira/browse/BEAM-12770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17401257#comment-17401257
]
Luke Cwik commented on BEAM-12770:
----------------------------------
I took a quick stab at it and we need to change 3 things based upon how things
are setup right now:
1) The repository section needs to be updated to contain Kafka deps
{noformat}
repositories {
mavenLocal()
maven {
url =
uri('https://repository.apache.org/content/repositories/snapshots/')
}
maven {
url = uri('https://repo.maven.apache.org/maven2/')
}
// For Confluent Kafka dependencies
maven {
url "https://packages.confluent.io/maven/";
content { includeGroup "io.confluent" }
}
}
{noformat}
2) Add the additional profiles that were missing, I filled out all the easy ones
{noformat}
if (project.hasProperty("dataflow-runner")) {
dependencies {
runtimeOnly
'org.apache.beam:beam-runners-google-cloud-dataflow-java:2.31.0'
}
}
if (project.hasProperty("flink-runner")) {
dependencies {
// Please see the Flink Runner page for an up-to-date list
// of supported Flink versions and their artifact names:
// https://beam.apache.org/documentation/runners/flink/
runtimeOnly "org.apache.beam:${flink.artifact.name}:2.31.0"
}
}
if (project.hasProperty("spark-runner")) {
// TODO(BEAM-???)
}
if (project.hasProperty("samza-runner")) {
dependencies {
runtimeOnly 'org.apache.beam:beam-runners-samza:2.31.0'
}
}
if (project.hasProperty("twister2-runner")) {
dependencies {
runtimeOnly 'org.apache.beam:beam-runners-twister2:2.31.0'
}
}
if (project.hasProperty("nemo-runner")) {
// TODO(BEAM-???)
}
if (project.hasProperty("jet-runner")) {
dependencies {
runtimeOnly 'org.apache.beam:beam-runners-jet:2.31.0'
}
}
{noformat}
3) Update the command to run WordCount for DataflowRunner to:
{noformat}
gradle clean execute -DmainClass=org.apache.beam.examples.WordCount \
-Dexec.args="--runner=DataflowRunner --project=<your-gcp-project> `
--region=<your-gcp-region> \
--gcpTempLocation=gs://<your-gcs-bucket>/tmp `
--inputFile=gs://apache-beam-samples/shakespeare/*
--output=gs://<your-gcs-bucket>/counts" -Pdataflow-runner
{noformat}
> Java Quickstart: Gradle/Dataflow runner command is incorrect
> ------------------------------------------------------------
>
> Key: BEAM-12770
> URL: https://issues.apache.org/jira/browse/BEAM-12770
> Project: Beam
> Issue Type: Bug
> Components: website
> Reporter: David Huntsperger
> Priority: P2
>
> The Gradle command for running the WordCount example on Dataflow does not
> actually use the Dataflow Runner. See:
> https://beam.apache.org/get-started/quickstart-java/#run-wordcount-using-gradle
> But if you add `--runner=DataflowRunner` to -Dexec.args, you get an error:
> `Exception in thread "main" java.lang.IllegalArgumentException: Unknown
> 'runner' specified 'DataflowRunner', supported pipeline runners
> [DirectRunner, PortableRunner, TestUniversalRunner]`
> Per [~lcwik]: "It is likely that the profiles in Maven aren't being converted
> over correctly to Gradle. (e.g. the additional stuff that -Pdataflow-runner
> is doing such as adding the DataflowRunner dep to the runtime classpath)"
> So it looks like we actually need to finish implementing the [Maven-to-Gradle
> conversion|https://docs.gradle.org/current/userguide/migrating_from_maven.html].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
