Re: Launch Dataflow Flex Templates from Go
Hi Shivam,
Thanks a lot for your response. I did check the http request. But I wanted
to see if I can use the Google API client Library.
The docs show a Python example for it shown below. I wanted to know if
there is something similar with Go.
from googleapiclient.discovery import build
# project = 'your-gcp-project'
# job = 'unique-job-name'
# template = 'gs://dataflow-templates/latest/Word_Count'
# parameters = {
# 'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
# 'output': 'gs:///wordcount/outputs',
# }
dataflow = build('dataflow', 'v1b3')
request = dataflow.projects().templates().launch(
projectId=project,
gcsPath=template,
body={
'jobName': job,
'parameters': parameters,
}
)
response = request.execute()
Regards,
Ashok
On Wed, Feb 15, 2023 at 4:22 PM Shivam Singhal
wrote:
> There shouldn’t be much change in the API request irrespective of the SDK
> language
>
> On Wed, 15 Feb 2023 at 10:50, Shivam Singhal
> wrote:
>
>> Hey Ashok,
>>
>> If you already have a flex template file and the docker image built, you
>> can use the Dataflow API to run the template.
>>
>> https://cloud.google.com/dataflow/docs/reference/rest
>>
>>
>> On Wed, 15 Feb 2023 at 04:49, Ashok KS wrote:
>>
>>> Hello Beam Community,
>>>
>>> I have written a Dataflow pipeline using Python SDK and I would be
>>> creating a Flex template with it.
>>>
>>> My task is to launch this Flex Template from Cloud Functions which would
>>> be in Go. I found the package below but couldn't find any sample.
>>>
>>>
>>> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>>>
>>> I could find examples in Python to launch templates.
>>> Can someone please share an example in Go to launch a Dataflow Flex
>>> template?
>>>
>>> Thank you in advance.
>>>
>>> Regards,
>>> Ashok
>>>
>>
Re: Launch Dataflow Flex Templates from Go
There shouldn’t be much change in the API request irrespective of the SDK language On Wed, 15 Feb 2023 at 10:50, Shivam Singhal wrote: > Hey Ashok, > > If you already have a flex template file and the docker image built, you > can use the Dataflow API to run the template. > > https://cloud.google.com/dataflow/docs/reference/rest > > > On Wed, 15 Feb 2023 at 04:49, Ashok KS wrote: > >> Hello Beam Community, >> >> I have written a Dataflow pipeline using Python SDK and I would be >> creating a Flex template with it. >> >> My task is to launch this Flex Template from Cloud Functions which would >> be in Go. I found the package below but couldn't find any sample. >> >> >> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch >> >> I could find examples in Python to launch templates. >> Can someone please share an example in Go to launch a Dataflow Flex >> template? >> >> Thank you in advance. >> >> Regards, >> Ashok >> >
Re: Launch Dataflow Flex Templates from Go
Hey Ashok, If you already have a flex template file and the docker image built, you can use the Dataflow API to run the template. https://cloud.google.com/dataflow/docs/reference/rest On Wed, 15 Feb 2023 at 04:49, Ashok KS wrote: > Hello Beam Community, > > I have written a Dataflow pipeline using Python SDK and I would be > creating a Flex template with it. > > My task is to launch this Flex Template from Cloud Functions which would > be in Go. I found the package below but couldn't find any sample. > > > https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch > > I could find examples in Python to launch templates. > Can someone please share an example in Go to launch a Dataflow Flex > template? > > Thank you in advance. > > Regards, > Ashok >
Re: OpenJDK8 / OpenJDK11 container deprecation
SGTM. I asked on the PR if this could impact users, but having read the docker release calendar I am not concerned. The last update to the old version was in 2019, and the introduction of compatible versions was 2020. On Tue, Feb 14, 2023 at 3:01 PM Byron Ellis via user wrote: > FWIW I am Team Upgrade Docker :-) > > On Tue, Feb 14, 2023 at 2:53 PM Luke Cwik via user > wrote: > >> I made some progress in testing the container and did hit an issue where >> Ubuntu 22.04 "Jammy" is dependent on the version of Docker installed. It >> turns out that our boot.go crashes with "runtime/cgo: pthread_create >> failed: Operation not permitted" because the Ubuntu 22.04 is using new >> syscalls that Docker 18.09.4 doesn't have a seccomp policy for (and uses a >> default of deny). We have a couple of choices here: >> 1) upgrade the version of docker on Jenkins and require users to >> similarly use a new enough version of Docker so that this isn't an issue >> for them >> 2) use Ubuntu 20.04 "Focal" as the docker container >> >> I was using Docker 20.10.21 which is why I didn't hit this issue when >> testing the change locally. >> >> We could also do these but they same strictly worse then either of the >> two options discussed above: >> A) disable the seccomp policy on Jenkins >> B) use a custom seccomp policy on Jenkins >> >> My suggestion is to upgrade Docker versions on Jenkins and use Ubuntu >> 22.04 as it will have LTS releases till 2027 and then security patches till >> 2032 which gives everyone the longest runway till we need to swap OS >> versions again for users of Apache Beam. Any concerns or ideas? >> >> >> >> On Thu, Feb 9, 2023 at 10:20 AM Luke Cwik wrote: >> >>> Our current container java 8 container is 262 MiBs and layers on top of >>> openjdk:8-bullseye which is 226 MiBs compressed while eclipse-temurin:8 is >>> 92 MiBs compressed and eclipse-temurin:8-alpine is 65 MiBs compressed. >>> >>> I would rather not get into issues with C library differences caused by >>> the alpine project so I would stick with the safer option and let users >>> choose alpine when building their custom container if they feel it provides >>> a large win for them. We can always swap to alpine in the future as well if >>> the C library differences become a non-issue. >>> >>> So swapping to eclipse-temurin will save us a bunch on the container >>> size which should help with container transfer and hopefully for startup >>> times as well. >>> >>> On Tue, Feb 7, 2023 at 5:41 PM Andrew Pilloud >>> wrote: >>> This sounds reasonable to me as well. I've made swaps like this in the past, the base image of each is probably a bigger factor than the JDK. The openjdk images were based on Debian 11. The default eclipse-temurin images are based on Ubuntu 22.04 with an alpine option. Ubuntu is a Debian derivative but the versions and package names aren't exact matches and Ubuntu tends to update a little faster. For most users I don't think this will matter but users building custom containers may need to make minor changes. The alpine option will be much smaller (which could be a significant improvement) but would be a more significant change to the environment. On Tue, Feb 7, 2023 at 5:18 PM Robert Bradshaw via dev < d...@beam.apache.org> wrote: > Seams reasonable to me. > > On Tue, Feb 7, 2023 at 4:19 PM Luke Cwik via user < > user@beam.apache.org> wrote: > > > > As per [1], the JDK8 and JDK11 containers that Apache Beam uses have > stopped being built and supported since July 2022. I have filed [2] to > track the resolution of this issue. > > > > Based upon [1], almost everyone is swapping to the eclipse-temurin > container[3] as their base based upon the linked issues from the > deprecation notice[1]. The eclipse-temurin container is released under > these licenses: > > Apache License, Version 2.0 > > Eclipse Distribution License 1.0 (BSD) > > Eclipse Public License 2.0 > > 一 (Secondary) GNU General Public License, version 2 with OpenJDK > Assembly Exception > > 一 (Secondary) GNU General Public License, version 2 with the GNU > Classpath Exception > > > > I propose that we swap all our containers to the eclipse-temurin > containers[3]. > > > > Open to other ideas and also would be great to hear about your > experience in any other projects that you have had to make a similar > decision. > > > > 1: https://github.com/docker-library/openjdk/issues/505 > > 2: https://github.com/apache/beam/issues/25371 > > 3: https://hub.docker.com/_/eclipse-temurin >
Launch Dataflow Flex Templates from Go
Hello Beam Community, I have written a Dataflow pipeline using Python SDK and I would be creating a Flex template with it. My task is to launch this Flex Template from Cloud Functions which would be in Go. I found the package below but couldn't find any sample. https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch I could find examples in Python to launch templates. Can someone please share an example in Go to launch a Dataflow Flex template? Thank you in advance. Regards, Ashok
Re: OpenJDK8 / OpenJDK11 container deprecation
FWIW I am Team Upgrade Docker :-) On Tue, Feb 14, 2023 at 2:53 PM Luke Cwik via user wrote: > I made some progress in testing the container and did hit an issue where > Ubuntu 22.04 "Jammy" is dependent on the version of Docker installed. It > turns out that our boot.go crashes with "runtime/cgo: pthread_create > failed: Operation not permitted" because the Ubuntu 22.04 is using new > syscalls that Docker 18.09.4 doesn't have a seccomp policy for (and uses a > default of deny). We have a couple of choices here: > 1) upgrade the version of docker on Jenkins and require users to similarly > use a new enough version of Docker so that this isn't an issue for them > 2) use Ubuntu 20.04 "Focal" as the docker container > > I was using Docker 20.10.21 which is why I didn't hit this issue when > testing the change locally. > > We could also do these but they same strictly worse then either of the two > options discussed above: > A) disable the seccomp policy on Jenkins > B) use a custom seccomp policy on Jenkins > > My suggestion is to upgrade Docker versions on Jenkins and use Ubuntu > 22.04 as it will have LTS releases till 2027 and then security patches till > 2032 which gives everyone the longest runway till we need to swap OS > versions again for users of Apache Beam. Any concerns or ideas? > > > > On Thu, Feb 9, 2023 at 10:20 AM Luke Cwik wrote: > >> Our current container java 8 container is 262 MiBs and layers on top of >> openjdk:8-bullseye which is 226 MiBs compressed while eclipse-temurin:8 is >> 92 MiBs compressed and eclipse-temurin:8-alpine is 65 MiBs compressed. >> >> I would rather not get into issues with C library differences caused by >> the alpine project so I would stick with the safer option and let users >> choose alpine when building their custom container if they feel it provides >> a large win for them. We can always swap to alpine in the future as well if >> the C library differences become a non-issue. >> >> So swapping to eclipse-temurin will save us a bunch on the container size >> which should help with container transfer and hopefully for startup times >> as well. >> >> On Tue, Feb 7, 2023 at 5:41 PM Andrew Pilloud >> wrote: >> >>> This sounds reasonable to me as well. >>> >>> I've made swaps like this in the past, the base image of each is >>> probably a bigger factor than the JDK. The openjdk images were based on >>> Debian 11. The default eclipse-temurin images are based on Ubuntu 22.04 >>> with an alpine option. Ubuntu is a Debian derivative but the versions and >>> package names aren't exact matches and Ubuntu tends to update a little >>> faster. For most users I don't think this will matter but users building >>> custom containers may need to make minor changes. The alpine option will be >>> much smaller (which could be a significant improvement) but would be a more >>> significant change to the environment. >>> >>> On Tue, Feb 7, 2023 at 5:18 PM Robert Bradshaw via dev < >>> d...@beam.apache.org> wrote: >>> Seams reasonable to me. On Tue, Feb 7, 2023 at 4:19 PM Luke Cwik via user wrote: > > As per [1], the JDK8 and JDK11 containers that Apache Beam uses have stopped being built and supported since July 2022. I have filed [2] to track the resolution of this issue. > > Based upon [1], almost everyone is swapping to the eclipse-temurin container[3] as their base based upon the linked issues from the deprecation notice[1]. The eclipse-temurin container is released under these licenses: > Apache License, Version 2.0 > Eclipse Distribution License 1.0 (BSD) > Eclipse Public License 2.0 > 一 (Secondary) GNU General Public License, version 2 with OpenJDK Assembly Exception > 一 (Secondary) GNU General Public License, version 2 with the GNU Classpath Exception > > I propose that we swap all our containers to the eclipse-temurin containers[3]. > > Open to other ideas and also would be great to hear about your experience in any other projects that you have had to make a similar decision. > > 1: https://github.com/docker-library/openjdk/issues/505 > 2: https://github.com/apache/beam/issues/25371 > 3: https://hub.docker.com/_/eclipse-temurin >>>
Re: OpenJDK8 / OpenJDK11 container deprecation
I made some progress in testing the container and did hit an issue where Ubuntu 22.04 "Jammy" is dependent on the version of Docker installed. It turns out that our boot.go crashes with "runtime/cgo: pthread_create failed: Operation not permitted" because the Ubuntu 22.04 is using new syscalls that Docker 18.09.4 doesn't have a seccomp policy for (and uses a default of deny). We have a couple of choices here: 1) upgrade the version of docker on Jenkins and require users to similarly use a new enough version of Docker so that this isn't an issue for them 2) use Ubuntu 20.04 "Focal" as the docker container I was using Docker 20.10.21 which is why I didn't hit this issue when testing the change locally. We could also do these but they same strictly worse then either of the two options discussed above: A) disable the seccomp policy on Jenkins B) use a custom seccomp policy on Jenkins My suggestion is to upgrade Docker versions on Jenkins and use Ubuntu 22.04 as it will have LTS releases till 2027 and then security patches till 2032 which gives everyone the longest runway till we need to swap OS versions again for users of Apache Beam. Any concerns or ideas? On Thu, Feb 9, 2023 at 10:20 AM Luke Cwik wrote: > Our current container java 8 container is 262 MiBs and layers on top of > openjdk:8-bullseye which is 226 MiBs compressed while eclipse-temurin:8 is > 92 MiBs compressed and eclipse-temurin:8-alpine is 65 MiBs compressed. > > I would rather not get into issues with C library differences caused by > the alpine project so I would stick with the safer option and let users > choose alpine when building their custom container if they feel it provides > a large win for them. We can always swap to alpine in the future as well if > the C library differences become a non-issue. > > So swapping to eclipse-temurin will save us a bunch on the container size > which should help with container transfer and hopefully for startup times > as well. > > On Tue, Feb 7, 2023 at 5:41 PM Andrew Pilloud wrote: > >> This sounds reasonable to me as well. >> >> I've made swaps like this in the past, the base image of each is probably >> a bigger factor than the JDK. The openjdk images were based on Debian 11. >> The default eclipse-temurin images are based on Ubuntu 22.04 with an alpine >> option. Ubuntu is a Debian derivative but the versions and package names >> aren't exact matches and Ubuntu tends to update a little faster. For most >> users I don't think this will matter but users building custom containers >> may need to make minor changes. The alpine option will be much smaller >> (which could be a significant improvement) but would be a more significant >> change to the environment. >> >> On Tue, Feb 7, 2023 at 5:18 PM Robert Bradshaw via dev < >> d...@beam.apache.org> wrote: >> >>> Seams reasonable to me. >>> >>> On Tue, Feb 7, 2023 at 4:19 PM Luke Cwik via user >>> wrote: >>> > >>> > As per [1], the JDK8 and JDK11 containers that Apache Beam uses have >>> stopped being built and supported since July 2022. I have filed [2] to >>> track the resolution of this issue. >>> > >>> > Based upon [1], almost everyone is swapping to the eclipse-temurin >>> container[3] as their base based upon the linked issues from the >>> deprecation notice[1]. The eclipse-temurin container is released under >>> these licenses: >>> > Apache License, Version 2.0 >>> > Eclipse Distribution License 1.0 (BSD) >>> > Eclipse Public License 2.0 >>> > 一 (Secondary) GNU General Public License, version 2 with OpenJDK >>> Assembly Exception >>> > 一 (Secondary) GNU General Public License, version 2 with the GNU >>> Classpath Exception >>> > >>> > I propose that we swap all our containers to the eclipse-temurin >>> containers[3]. >>> > >>> > Open to other ideas and also would be great to hear about your >>> experience in any other projects that you have had to make a similar >>> decision. >>> > >>> > 1: https://github.com/docker-library/openjdk/issues/505 >>> > 2: https://github.com/apache/beam/issues/25371 >>> > 3: https://hub.docker.com/_/eclipse-temurin >>> >>
