[
https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112645#comment-17112645
]
Chamikara Madhusanka Jayalath commented on BEAM-9383:
-----------------------------------------------------
I tried running a Kafka pipeline on Dataflow and I see a lot of jars being
staged during pipeline submission.
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/pipeline.pb
in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar
in 24 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar
in 22 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/2704b169-8874-4163-9f3c-ab8765f3c330.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/2704b169-8874-4163-9f3c-ab8765f3c330.jar
in 69 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/40bd912f-ce2f-45a8-9625-019b85c46cc7.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/40bd912f-ce2f-45a8-9625-019b85c46cc7.jar
in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/9d1bfb42-518d-4cc7-9a3a-7a8ea792ce6f.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/9d1bfb42-518d-4cc7-9a3a-7a8ea792ce6f.jar
in 8 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/7e1e7095-32d6-4ea6-b9a0-aa5e2ffdbb31.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/7e1e7095-32d6-4ea6-b9a0-aa5e2ffdbb31.jar
in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/926b735c-552f-4a3a-9e81-f0fe8162ce26.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/926b735c-552f-4a3a-9e81-f0fe8162ce26.jar
in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to
gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/e9f829ba-eadf-4ae4-98c4-492238cb9998.jar...
...
Ideally there should be only one jar,
beam-sdks-java-io-expansion-service-2.22.0-SNAPSHOT.ja
Any idea where additional jars are coming from. Also can we use names of jars
instread of URLs so that we can easily identify what these are ?
cc: [~robertwb] [~lcwik]
> Staging Dataflow artifacts from environment
> -------------------------------------------
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
> Reporter: Heejong Lee
> Assignee: Heejong Lee
> Priority: P0
> Fix For: 2.22.0
>
> Time Spent: 12h
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
