Congrats! Looking forward to start using it for tests (I am going to use it also for some performance testing soon).
J. On Fri, Jun 19, 2020 at 4:09 PM Greg Neiheisel <g...@astronomer.io> wrote: > Hey everyone, > > If you haven't been following along on the PR > <https://github.com/apache/airflow/pull/8777>, it was recently merged! The > helm chart now lives under the chart directory inside the Airflow repo. > > Thanks to everyone who's contributed over the years to this chart or any of > the other open source charts. I think they all ended up pulling parts from > each other. Happy to have a place where we can all work together on this > together now. > > There's plenty of work to do still, so if you have any improvements, please > send a PR! > > > > On Fri, May 8, 2020 at 8:10 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Thanks, Greg 🚀 > > > > On Fri, May 8, 2020 at 4:19 AM Greg Neiheisel <g...@astronomer.io.invalid > > > > wrote: > > > > > Awesome Jarek. I just opened up a draft PR that adds the chart into the > > > Airflow repo. https://github.com/apache/airflow/pull/8777 > > > > > > It still contains some of our packaging, testing and releasing > scripts. I > > > figure they can serve as a starting point to discuss how we'd want to > > > replicate into Airflow infrastructure. > > > > > > Looking forward to getting some feedback and moving this along! > > > > > > On Tue, Apr 28, 2020 at 8:00 AM Jarek Potiuk <jarek.pot...@polidea.com > > > > > wrote: > > > > > > > Hey Greg, > > > > > > > > I just wanted to update you with some recent changes I've done to > > > automate > > > > most of the image builds stuff we have in Airflow. That might be > useful > > > for > > > > your efforts. > > > > > > > > I am adding (and I will fix some of them very soon) issues based on > > > > feedback from people on the #prod-docker-image channel: > > > > > > > > > > https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image > > > > But in the meantime all the build/CI infrastructure has nicely fallen > > > into > > > > place and It's almost complete (one thing missing but very simple is > > > > automated builds of release images - for now this is manual but it is > > > super > > > > easy change and I will add it before 1.10.11 for sure). > > > > > > > > I described the whole build infrastructure here: > > > > https://github.com/apache/airflow/blob/master/CI.rst including how > > image > > > > caching works, how nightly builds and images are triggered. I hope it > > > will > > > > be helpful in the helm-chart work. Any questions or asks for the > image > > - > > > > please let me know, it is super easy to update the image now when we > > have > > > > infrastructure to build it automatically integrated well with Github > > > > Actions and DockerHub. > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk < > jarek.pot...@polidea.com > > > > > > > wrote: > > > > > > > >> Awesome, thanks for the update! > > > >> > > > >> On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <g...@astronomer.io> > > > >> wrote: > > > >> > > > >>> Hey everyone, just wanted to post a quick update on the helm chart. > > > >>> We're in the process of removing any references to Astronomer > > specific > > > >>> things and making them more general features. That work is > happening > > on > > > >>> this PR here https://github.com/astronomer/airflow-chart/pull/66. > > > >>> Hoping to wrap up the last few outstanding items and open up a PR > to > > > >>> Airflow next week sometime! > > > >>> > > > >>> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman < > > > >>> daniel.imber...@gmail.com> wrote: > > > >>> > > > >>>> Hi Jarek, > > > >>>> > > > >>>> Congrats on getting the image out! This is going to make a lot of > > > >>>> things easier :). We’ve actually been working on separating out > KinD > > > as > > > >>>> well (and in fact moving it out of travis and into GitHub issues). > > > Let me > > > >>>> circle back with Steven (the engineer who’s been working on that) > > and > > > get a > > > >>>> status report from him. > > > >>>> > > > >>>> I will also take some time today to get our helm chart to work > with > > > >>>> the airflow master image. It’s been a bit crazy around here in the > > > last few > > > >>>> weeks but I am consistently pushing to make sure we make the final > > > changes > > > >>>> of ripping out any remaining platform components before we donate > > :). > > > >>>> > > > >>>> I agree that docker hub automation will be great. This will also > > allow > > > >>>> us to have a nightly build for any integration/UI tests we build. > > > >>>> > > > >>>> Thanks again Jarek! > > > >>>> > > > >>>> Daniel > > > >>>> > > > >>>> via Newton Mail > > > >>>> < > > > > > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2 > > > > > > > >>>> > > > >>>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk < > > > jarek.pot...@polidea.com> > > > >>>> wrote: > > > >>>> > > > >>>> Hello Greg and Daniel, > > > >>>> > > > >>>> Production image is merged and it's also going to be released in > > > >>>> 1.10.10. For now, I will manually tag the release in DockerHub > when > > we > > > >>>> release but we will automate that in the future. Currently, we > have > > > >>>> *airflow:master* and *airflow:v1-10-test* images (for all > supported > > > >>>> python versions) that you can use for testing with the Helm chart! > > > >>>> > > > >>>> The next things from my side: > > > >>>> > > > >>>> * I will automate DockerHub building for the prod image from > master > > - > > > >>>> for now only CI is automated but we need to add prod image as well > > > >>>> * I will move kind-cluster creation outside of the container and > use > > > >>>> production image for our CI Kubernetes tests. I have a WIP branch > on > > > it and > > > >>>> I think I should have it early next week > > > >>>> * The next step will be to switch our CI to Github Actions > finally. > > I > > > >>>> think we will run both Travis and GA for a short time to make sure > > it > > > works > > > >>>> * I will ask people for options they would like to have in the > > image - > > > >>>> additional packages etc. so that I can work on the next iteration > of > > > the > > > >>>> image. Also if you have any needs to see for the Helm > integration, I > > > am > > > >>>> super happy to incorporate them in the next version. I plan to > keep > > > the > > > >>>> "master" and "v1-10-test" images updated with any new changes and > > push > > > >>>> images regularly > > > >>>> * I am happy to help with the Helm Chart, so let me know if you > need > > > >>>> anything! > > > >>>> > > > >>>> J. > > > >>>> > > > >>>> > > > >>>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk < > > > jarek.pot...@polidea.com> > > > >>>> wrote: > > > >>>> > > > >>>>> Hello Daniel and Greg. > > > >>>>> > > > >>>>> I think the production image for Airflow is quite ready for the > > final > > > >>>>> review and merge. I did some tests and I think it's quite > complete > > > now. > > > >>>>> > > > >>>>> The PR is running it's Travis build: > > > >>>>> https://github.com/apache/airflow/pull/7832 > > > >>>>> > > > >>>>> I already pushed the image to DockerHub - so that the next time > it > > is > > > >>>>> merged It will use it as a base and will be rather quickly built > > > using > > > >>>>> those images as a cache. Once it is merged, I will add the > > automated > > > build > > > >>>>> in DockerHub and merge it to 1.10. As the next step I also plan > to > > > use the > > > >>>>> prod image for our Kubernetes tests (that will help with moving > to > > > the > > > >>>>> GitHub Actions from Travis). > > > >>>>> > > > >>>>> I added quite a comprehensive documentation - explaining all ins- > > and > > > >>>>> outs- of both CI and PROD images > > > >>>>> > > > > > > https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst > > > >>>>> . > > > >>>>> > > > >>>>> You can build it yourself using manual build process described in > > the > > > >>>>> doc or (easier) using Breeze. That includes using the same > > > Dockerfile to > > > >>>>> build older Airflow version from the 1.10.* line. > > > >>>>> > > > >>>>> Feel free to use the new image in the Helm Chart you have - happy > > to > > > >>>>> review the PRs. The image is available at > > > 'apache/airflow:master-python3.6' > > > >>>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 > > and > > > >>>>> release 1.10.10 we will also have it automatically published for > > > 1.10.10 > > > >>>>> and beyond. > > > >>>>> > > > >>>>> Happy to hear your review comments and hope to merge it very > soon. > > > >>>>> J. > > > >>>>> > > > >>>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk < > > > jarek.pot...@polidea.com> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> Absolutely! Please do :). The more eyes on this the better! > > > >>>>>> > > > >>>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman < > > > >>>>>> daniel.imber...@gmail.com> wrote: > > > >>>>>> > > > >>>>>>> @Jarek Steven and I will look into getting KinD to work on > GitHub > > > >>>>>>> actions today. If we can get that working, we can donate the > helm > > > chart and > > > >>>>>>> possibly even deprecate the old Kubernetes tests in the > process. > > > >>>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk < > > > >>>>>>> jarek.pot...@polidea.com>, wrote: > > > >>>>>>> > We have A POC for Github Actions done by Tomek, and the > blocker > > > >>>>>>> there > > > >>>>>>> > was precisely Kubernetes tests and specifically running kind > > > >>>>>>> cluster > > > >>>>>>> > in the Github Actions > > > >>>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the > > > >>>>>>> meantime > > > >>>>>>> > Jiaje Zhong is trying to make it works on GA we discussed > this > > > very > > > >>>>>>> > thing today at slack: > > > >>>>>>> > > > > >>>>>>> > > > https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200 > > > >>>>>>> > > > > >>>>>>> > For me this is the next thing I will look at after merging > > > >>>>>>> > requirements PR and Prod image PR (and those two are a bit > > > >>>>>>> > pre-requisites to make it stable and fast). I want to switch > to > > > the > > > >>>>>>> > Production Image to run Kubernetes tests (they will be much > > > faster) > > > >>>>>>> > and then we can work out much faster how to run them on > Github > > > >>>>>>> > Actions. Then we can add Helm charts to run all those > different > > > >>>>>>> > deployments. I am really looking forward to it! > > > >>>>>>> > > > > >>>>>>> > I hope - this weekend I move forward all of that and have a > PR > > > >>>>>>> that we > > > >>>>>>> > can start running tests on and replacing the resources with > > helm > > > >>>>>>> > chart. > > > >>>>>>> > > > > >>>>>>> > J. > > > >>>>>>> > > > > >>>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman > > > >>>>>>> > <daniel.imber...@gmail.com> wrote: > > > >>>>>>> > > > > > >>>>>>> > > @Steven glad to help you out on that. > > > >>>>>>> > > > > > >>>>>>> > > For the moment we only use travis, but I believe @jarek has > > > been > > > >>>>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our > > > first > > > >>>>>>> GitHub tests? > > > >>>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller > > > >>>>>>> <ste...@astronomer.io.invalid>, wrote: > > > >>>>>>> > > > Hey I’m happy to PR in the chart and get the CI working > to > > > >>>>>>> test it. Where > > > >>>>>>> > > > do we want to run it? Do you all just use Travis only? > > > >>>>>>> > > > > > > >>>>>>> > > > Steven > > > >>>>>>> > > > > > > >>>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman < > > > >>>>>>> daniel.imber...@gmail.com> > > > >>>>>>> > > > wrote: > > > >>>>>>> > > > > > > >>>>>>> > > > > @Jarek I think with the helm chart + prod image we can > go > > > >>>>>>> even further > > > >>>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA > > > >>>>>>> autoscaling, and a > > > >>>>>>> > > > > bunch of other configurations. > > > >>>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk < > > > >>>>>>> jarek.pot...@polidea.com>, > > > >>>>>>> > > > > wrote: > > > >>>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my > original > > > >>>>>>> email :) > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman < > > > >>>>>>> > > > > daniel.imber...@gmail.com> wrote: > > > >>>>>>> > > > > > > > We’re not using CRDs for the tests at the moment. > > We > > > >>>>>>> just have > > > >>>>>>> > > > > deployment files. If anything having the helm chart as > a > > > >>>>>>> part of the > > > >>>>>>> > > > > airflow repo could mean that the helm chart becomes the > > > >>>>>>> defacto system for > > > >>>>>>> > > > > testing airflow on kubernetes (we can get rid of all > the > > > >>>>>>> yams files and run > > > >>>>>>> > > > > multiple k8s tests with different settings). > > > >>>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel > > > >>>>>>> > > > > <g...@astronomer.io.invalid>, wrote: > > > >>>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow > > > >>>>>>> repo. We've also > > > >>>>>>> > > > > been > > > >>>>>>> > > > > > > > > building up a test suite that currently runs on > > > >>>>>>> CircleCI and uses > > > >>>>>>> > > > > kind > > > >>>>>>> > > > > > > > > (Kubernetes in Docker) to test several > kubernetes > > > >>>>>>> versions with > > > >>>>>>> > > > > some > > > >>>>>>> > > > > > > > > different settings. Right now we're mostly > > testing > > > >>>>>>> the different > > > >>>>>>> > > > > executors > > > >>>>>>> > > > > > > > > since that has the biggest impact on what gets > > > >>>>>>> deployed, but that > > > >>>>>>> > > > > can be > > > >>>>>>> > > > > > > > > expanded. > > > >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > What CRDs are currently being used to run > Airflow > > > >>>>>>> for the tests? > > > >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk < > > > >>>>>>> > > > > jarek.pot...@polidea.com> > > > >>>>>>> > > > > > > > > wrote: > > > >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > One thing for the donation. > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > Did you you want to have separate repository > > > Greg ? > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > I think we should simply create a folder in > > > >>>>>>> Airflow repo and > > > >>>>>>> > > > > keep it > > > >>>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am > > > >>>>>>> going to switch our > > > >>>>>>> > > > > > > > > > Kubernetes Tests to the production image > (will > > > >>>>>>> make the tests > > > >>>>>>> > > > > much > > > >>>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile > > > >>>>>>> automatically in > > > >>>>>>> > > > > CI - > > > >>>>>>> > > > > > > > > > for now we are using some custom Resource > > > >>>>>>> definitions to start > > > >>>>>>> > > > > Airflow > > > >>>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we > > could > > > >>>>>>> switch to > > > >>>>>>> > > > > using the > > > >>>>>>> > > > > > > > > > helm chart - this way we can test all three > > > things > > > >>>>>>> at once: > > > >>>>>>> > > > > > > > > > - Kubernetes Executor > > > >>>>>>> > > > > > > > > > - Dockerfile > > > >>>>>>> > > > > > > > > > - Helm Chart > > > >>>>>>> > > > > > > > > > and we could also add more tests - for > example > > > >>>>>>> testing different > > > >>>>>>> > > > > > > > > > deployment options for the helm chart. > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would > > help > > > >>>>>>> with that - > > > >>>>>>> > > > > > > > > > especially in terms of future changes and > > testing > > > >>>>>>> them > > > >>>>>>> > > > > automatically. > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > J. > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal > > Nurmamat > > > >>>>>>> kyzy > > > >>>>>>> > > > > > > > > > <aizha...@apache.org> wrote: > > > >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > +1 on the donation. Always happy to see > more > > > >>>>>>> useful stuff for > > > >>>>>>> > > > > the > > > >>>>>>> > > > > > > > > > > community :) > > > >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg > > Neiheisel > > > < > > > >>>>>>> > > > > g...@astronomer.io> > > > >>>>>>> > > > > > > > > > wrote: > > > >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up > now > > as > > > >>>>>>> an optional > > > >>>>>>> > > > > Kubernetes > > > >>>>>>> > > > > > > > > > > > CronJob ( > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml > > > >>>>>>> > > > > > > > > > ) > > > >>>>>>> > > > > > > > > > > > that we have run periodically to clean > > failed > > > >>>>>>> pods up and > > > >>>>>>> > > > > could stay > > > >>>>>>> > > > > > > > > > > > separate. > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > The wait_for_migrations script could > > > >>>>>>> definitely be pulled > > > >>>>>>> > > > > into Airflow. > > > >>>>>>> > > > > > > > > > > > For context, we deploy an initContainer > on > > > the > > > >>>>>>> scheduler ( > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84 > > > >>>>>>> > > > > > > > > > ) > > > >>>>>>> > > > > > > > > > > > that runs the upgradedb command before > > > booting > > > >>>>>>> the > > > >>>>>>> > > > > scheduler. This new > > > >>>>>>> > > > > > > > > > > > wait_for_migration script runs in an > > > >>>>>>> initContainer on the > > > >>>>>>> > > > > webserver and > > > >>>>>>> > > > > > > > > > > > workers ( > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65 > > > >>>>>>> > > > > > > > > > ) > > > >>>>>>> > > > > > > > > > > > so that they don't boot up ahead of a > > > >>>>>>> potentially > > > >>>>>>> > > > > long-running > > > >>>>>>> > > > > > > > > > migration > > > >>>>>>> > > > > > > > > > > > and attempt to operate on new or missing > > > >>>>>>> columns/tables > > > >>>>>>> > > > > before the > > > >>>>>>> > > > > > > > > > > > migrations run. This prevents these pods > > from > > > >>>>>>> entering a > > > >>>>>>> > > > > CrashLoop. > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek > > > Potiuk < > > > >>>>>>> > > > > > > > > > jarek.pot...@polidea.com> > > > >>>>>>> > > > > > > > > > > > wrote: > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images > are > > > >>>>>>> currently > > > >>>>>>> > > > > generated from > > > >>>>>>> > > > > > > > > > > > > Dockerfiles > > > >>>>>>> > > > > > > > > > > > > > in this repo > > > >>>>>>> https://github.com/astronomer/ap-airflow > > > >>>>>>> > > > > and get > > > >>>>>>> > > > > > > > > > > > > published to > > > >>>>>>> > > > > > > > > > > > > > DockerHub > > > >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow. > > > >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > For the most part those are typical > > > >>>>>>> Airflow images. > > > >>>>>>> > > > > There's an > > > >>>>>>> > > > > > > > > > > > > entrypoint > > > >>>>>>> > > > > > > > > > > > > > script that we include in the image > > that > > > >>>>>>> handles waiting > > > >>>>>>> > > > > for the > > > >>>>>>> > > > > > > > > > > > > database > > > >>>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which > > is > > > >>>>>>> pretty generic. > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > I already added waiting for the > database > > > >>>>>>> (both metadata > > > >>>>>>> > > > > and celery > > > >>>>>>> > > > > > > > > > URL) in > > > >>>>>>> > > > > > > > > > > > > the PR: > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741 > > > >>>>>>> > > > > > > > > > > > > . > > > >>>>>>> > > > > > > > > > > > > It's functionally the same but more > > > generic. > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > The only other > > > >>>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart > uses > > > >>>>>>> would be the > > > >>>>>>> > > > > scripts in this > > > >>>>>>> > > > > > > > > > repo > > > >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> https://github.com/astronomer/astronomer-airflow-scripts. > > > >>>>>>> > > > > Our > > > >>>>>>> > > > > > > > > > > > > Dockerfiles > > > >>>>>>> > > > > > > > > > > > > > pull this package in. These scripts > are > > > >>>>>>> used to > > > >>>>>>> > > > > coordinate running > > > >>>>>>> > > > > > > > > > > > > > migrations and cleaning up failed > pods. > > > >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > I see two scripts: > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) > not > > > >>>>>>> needed to run in > > > >>>>>>> > > > > airflow - > > > >>>>>>> > > > > > > > > > this > > > >>>>>>> > > > > > > > > > > > > could be run as a separate > pod/container? > > > >>>>>>> > > > > > > > > > > > > * waiting for migrations -> I think > this > > is > > > >>>>>>> a good > > > >>>>>>> > > > > candidate to add > > > >>>>>>> > > > > > > > > > > > > *airflow > > > >>>>>>> > > > > > > > > > > > > db wait_for_migration* command and make > > it > > > >>>>>>> part of airflow > > > >>>>>>> > > > > itself. > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > I think we also have to agree on the > > > Airflow > > > >>>>>>> version > > > >>>>>>> > > > > supported by the > > > >>>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we > > support > > > >>>>>>> 1.10.10+ and we > > > >>>>>>> > > > > > > > > > incorporate > > > >>>>>>> > > > > > > > > > > > > all > > > >>>>>>> > > > > > > > > > > > > the changes needed to airflow (like the > > "db > > > >>>>>>> > > > > wait_for_migration") > > > >>>>>>> > > > > > > > > > into 2.0 > > > >>>>>>> > > > > > > > > > > > > and 1.10 and we support both - image > and > > > >>>>>>> helm chart for > > > >>>>>>> > > > > those versions > > > >>>>>>> > > > > > > > > > > > > only. That would help with people > > migrating > > > >>>>>>> to the latest > > > >>>>>>> > > > > version. > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > WDYT? > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM > Daniel > > > >>>>>>> Imberman < > > > >>>>>>> > > > > > > > > > > > > > daniel.imber...@gmail.com> > > > >>>>>>> > > > > > > > > > > > > > wrote: > > > >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think > > that > > > >>>>>>> pairing an > > > >>>>>>> > > > > official helm > > > >>>>>>> > > > > > > > > > chart > > > >>>>>>> > > > > > > > > > > > > > > with the official image would make > > for > > > a > > > >>>>>>> REALLY > > > >>>>>>> > > > > powerful “up and > > > >>>>>>> > > > > > > > > > > > > running > > > >>>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I > > > have > > > >>>>>>> also been > > > >>>>>>> > > > > looking into > > > >>>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability > to > > > >>>>>>> create custom > > > >>>>>>> > > > > controllers > > > >>>>>>> > > > > > > > > > from > > > >>>>>>> > > > > > > > > > > > > helm > > > >>>>>>> > > > > > > > > > > > > > > charts. We might even able to get a > > 1-2 > > > >>>>>>> punch from the > > > >>>>>>> > > > > same code > > > >>>>>>> > > > > > > > > > base > > > >>>>>>> > > > > > > > > > > > > :). > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if > > there’s > > > >>>>>>> no issues, can > > > >>>>>>> > > > > we please > > > >>>>>>> > > > > > > > > > start > > > >>>>>>> > > > > > > > > > > > > > the > > > >>>>>>> > > > > > > > > > > > > > > process of donation? > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > +1 on my part, of course :) > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > Daniel > > > >>>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, > Jarek > > > >>>>>>> Potiuk < > > > >>>>>>> > > > > > > > > > > > > jarek.pot...@polidea.com>, > > > >>>>>>> > > > > > > > > > > > > > > wrote: > > > >>>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with > > the > > > >>>>>>> official image > > > >>>>>>> > > > > we have > > > >>>>>>> > > > > > > > > > work in > > > >>>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at > the > > > >>>>>>> Astronomer's > > > >>>>>>> > > > > image while > > > >>>>>>> > > > > > > > > > > > > preparing > > > >>>>>>> > > > > > > > > > > > > > my > > > >>>>>>> > > > > > > > > > > > > > > > draft and we can make any > > adjustments > > > >>>>>>> needed to make > > > >>>>>>> > > > > it works > > > >>>>>>> > > > > > > > > > with > > > >>>>>>> > > > > > > > > > > > > the > > > >>>>>>> > > > > > > > > > > > > > > helm > > > >>>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to > > > >>>>>>> collaborate on that. > > > >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > PR here: > > > >>>>>>> https://github.com/apache/airflow/pull/7832 > > > >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > J. > > > >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM > > Kaxil > > > >>>>>>> Naik < > > > >>>>>>> > > > > kaxiln...@gmail.com > > > >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > wrote: > > > >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek < > > > >>>>>>> tomasz.urbas...@polidea.com> : > > > >>>>>>> > > > > > > > > > > > > > > > > Helm Chart Link: > > > >>>>>>> > > > > https://github.com/astronomer/airflow-chart > > > >>>>>>> > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM > > > >>>>>>> Tomasz Urbaszek < > > > >>>>>>> > > > > > > > > > > > > > turbas...@apache.org> > > > >>>>>>> > > > > > > > > > > > > > > > > wrote: > > > >>>>>>> > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > An official helm chart is > > > >>>>>>> something our > > > >>>>>>> > > > > community needs! > > > >>>>>>> > > > > > > > > > Using > > > >>>>>>> > > > > > > > > > > > > your > > > >>>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a > > lot > > > >>>>>>> of sens to me > > > >>>>>>> > > > > because as > > > >>>>>>> > > > > > > > > > you > > > >>>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle > tested. > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow > > image > > > >>>>>>> do you use? > > > >>>>>>> > > > > Also, would you > > > >>>>>>> > > > > > > > > > > > > mind > > > >>>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart? > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > Tomek > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 > PM > > > >>>>>>> Greg Neiheisel > > > >>>>>>> > > > > > > > > > > > > > > > > > <g...@astronomer.io.invalid> > > > >>>>>>> wrote: > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > Hey everyone, > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at > > > >>>>>>> Astronomer, we’ve > > > >>>>>>> > > > > created, > > > >>>>>>> > > > > > > > > > managed, > > > >>>>>>> > > > > > > > > > > > > > and > > > >>>>>>> > > > > > > > > > > > > > > > > > hardened > > > >>>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm > Chart > > > >>>>>>> for Airflow ( > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> https://github.com/astronomer/airflow-chart) > > > >>>>>>> > > > > that is > > > >>>>>>> > > > > > > > > > being > > > >>>>>>> > > > > > > > > > > > > used > > > >>>>>>> > > > > > > > > > > > > > by > > > >>>>>>> > > > > > > > > > > > > > > > > both > > > >>>>>>> > > > > > > > > > > > > > > > > > our > > > >>>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise > > customers. > > > >>>>>>> This chart is > > > >>>>>>> > > > > > > > > > battle-tested and > > > >>>>>>> > > > > > > > > > > > > > > running > > > >>>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow > > deployments > > > >>>>>>> of varying > > > >>>>>>> > > > > sizes and > > > >>>>>>> > > > > > > > > > runtime > > > >>>>>>> > > > > > > > > > > > > > > > > > environments. > > > >>>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to > > > >>>>>>> encapsulate the issues > > > >>>>>>> > > > > that Airflow > > > >>>>>>> > > > > > > > > > > > > users > > > >>>>>>> > > > > > > > > > > > > > run > > > >>>>>>> > > > > > > > > > > > > > > > > into > > > >>>>>>> > > > > > > > > > > > > > > > > > in > > > >>>>>>> > > > > > > > > > > > > > > > > > > the real world. > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > While this chart was > > originally > > > >>>>>>> developed > > > >>>>>>> > > > > internally for > > > >>>>>>> > > > > > > > > > our > > > >>>>>>> > > > > > > > > > > > > > > Astronomer > > > >>>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently > > > >>>>>>> decoupled the chart > > > >>>>>>> > > > > from the > > > >>>>>>> > > > > > > > > > rest of > > > >>>>>>> > > > > > > > > > > > > our > > > >>>>>>> > > > > > > > > > > > > > > > > > platform > > > >>>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the > > > greater > > > >>>>>>> Airflow > > > >>>>>>> > > > > community. With > > > >>>>>>> > > > > > > > > > these > > > >>>>>>> > > > > > > > > > > > > > > changes > > > >>>>>>> > > > > > > > > > > > > > > > > in > > > >>>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a > > > >>>>>>> conversation about > > > >>>>>>> > > > > donating this > > > >>>>>>> > > > > > > > > > > > > chart > > > >>>>>>> > > > > > > > > > > > > > to > > > >>>>>>> > > > > > > > > > > > > > > the > > > >>>>>>> > > > > > > > > > > > > > > > > > > Airflow community. > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > Some of the main features > of > > > the > > > >>>>>>> chart are: > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box. > > With > > > >>>>>>> zero > > > >>>>>>> > > > > configuration, a user > > > >>>>>>> > > > > > > > > > > > > will > > > >>>>>>> > > > > > > > > > > > > > get > > > >>>>>>> > > > > > > > > > > > > > > > > a > > > >>>>>>> > > > > > > > > > > > > > > > > > > postgres database, a > default > > > >>>>>>> user and the > > > >>>>>>> > > > > > > > > > KubernetesExecutor > > > >>>>>>> > > > > > > > > > > > > > ready > > > >>>>>>> > > > > > > > > > > > > > > > > to > > > >>>>>>> > > > > > > > > > > > > > > > > > run > > > >>>>>>> > > > > > > > > > > > > > > > > > > DAGs. > > > >>>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery > > (w/ > > > >>>>>>> optional KEDA > > > >>>>>>> > > > > > > > > > autoscaling) and > > > >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors. > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > Support for optional > > pgbouncer. > > > >>>>>>> We use this to > > > >>>>>>> > > > > share a > > > >>>>>>> > > > > > > > > > > > > > configurable > > > >>>>>>> > > > > > > > > > > > > > > > > > > connection pool size per > > > >>>>>>> deployment. Useful > > > >>>>>>> > > > > for limiting > > > >>>>>>> > > > > > > > > > > > > > > connections to > > > >>>>>>> > > > > > > > > > > > > > > > > > the > > > >>>>>>> > > > > > > > > > > > > > > > > > > metadata database. > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration > support. > > A > > > >>>>>>> user can push a > > > >>>>>>> > > > > newer > > > >>>>>>> > > > > > > > > > version > > > >>>>>>> > > > > > > > > > > > > of > > > >>>>>>> > > > > > > > > > > > > > > > > > Airflow > > > >>>>>>> > > > > > > > > > > > > > > > > > > into an existing release > and > > > >>>>>>> migrations will > > > >>>>>>> > > > > > > > > > automatically run > > > >>>>>>> > > > > > > > > > > > > > > > > > cleanly. > > > >>>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support. > > > Optionally > > > >>>>>>> install and > > > >>>>>>> > > > > configure a > > > >>>>>>> > > > > > > > > > > > > > > > > > statsd-exporter > > > >>>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics > and > > > >>>>>>> expose them to > > > >>>>>>> > > > > Prometheus > > > >>>>>>> > > > > > > > > > > > > > > > > automatically. > > > >>>>>>> > > > > > > > > > > > > > > > > > > - Resource control. > > Optionally > > > >>>>>>> control the > > > >>>>>>> > > > > ResourceQuotas > > > >>>>>>> > > > > > > > > > and > > > >>>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each > > deployment > > > >>>>>>> so that no > > > >>>>>>> > > > > deployment can > > > >>>>>>> > > > > > > > > > > > > > overload > > > >>>>>>> > > > > > > > > > > > > > > a > > > >>>>>>> > > > > > > > > > > > > > > > > > > cluster. > > > >>>>>>> > > > > > > > > > > > > > > > > > > - Simple optional > > Elasticsearch > > > >>>>>>> support. > > > >>>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace > cleanup. > > > >>>>>>> Sometimes > > > >>>>>>> > > > > > > > > > KubernetesExecutor and > > > >>>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods > > fail > > > >>>>>>> for reasons > > > >>>>>>> > > > > other than the > > > >>>>>>> > > > > > > > > > > > > actual > > > >>>>>>> > > > > > > > > > > > > > > > > > task. > > > >>>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep > > things > > > >>>>>>> clean in > > > >>>>>>> > > > > Kubernetes. > > > >>>>>>> > > > > > > > > > > > > > > > > > > - Support for running > locally > > > in > > > >>>>>>> KIND > > > >>>>>>> > > > > (Kubernetes in > > > >>>>>>> > > > > > > > > > Docker). > > > >>>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested > across > > > >>>>>>> many Kubernetes > > > >>>>>>> > > > > versions > > > >>>>>>> > > > > > > > > > with > > > >>>>>>> > > > > > > > > > > > > Helm > > > >>>>>>> > > > > > > > > > > > > > 2 > > > >>>>>>> > > > > > > > > > > > > > > > > > and 3 > > > >>>>>>> > > > > > > > > > > > > > > > > > > support. > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the > cleanest > > > >>>>>>> and most > > > >>>>>>> > > > > reliable way to > > > >>>>>>> > > > > > > > > > deploy > > > >>>>>>> > > > > > > > > > > > > > DAGs > > > >>>>>>> > > > > > > > > > > > > > > to > > > >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them > at > > > >>>>>>> scale is to > > > >>>>>>> > > > > package them > > > >>>>>>> > > > > > > > > > into > > > >>>>>>> > > > > > > > > > > > > the > > > >>>>>>> > > > > > > > > > > > > > > actual > > > >>>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have > > geared > > > >>>>>>> this chart > > > >>>>>>> > > > > towards that > > > >>>>>>> > > > > > > > > > > > > method of > > > >>>>>>> > > > > > > > > > > > > > > > > > > operation, though adding > > other > > > >>>>>>> methods should > > > >>>>>>> > > > > be > > > >>>>>>> > > > > > > > > > > > > straightforward. > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from > > the > > > >>>>>>> community and > > > >>>>>>> > > > > would love > > > >>>>>>> > > > > > > > > > to > > > >>>>>>> > > > > > > > > > > > > see > > > >>>>>>> > > > > > > > > > > > > > > this > > > >>>>>>> > > > > > > > > > > > > > > > > > chart > > > >>>>>>> > > > > > > > > > > > > > > > > > > help others to get up and > > > >>>>>>> running on > > > >>>>>>> > > > > Kubernetes! > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > -- > > > >>>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief > > > >>>>>>> Architect > > > >>>>>>> > > > > Astronomer.io > > > >>>>>>> > > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > -- > > > >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > Jarek Potiuk > > > >>>>>>> > > > > > > > > > > > > > > > Polidea < > https://www.polidea.com/> > > | > > > >>>>>>> Principal > > > >>>>>>> > > > > Software > > > >>>>>>> > > > > > > > > > Engineer > > > >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129 > > > >>>>>>> <+48%20660%20796%20129> > > > >>>>>>> > > > > <+48660796129 > > > >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>> > > > >>>>>>> > > > > > > > > > > > > > > > [image: Polidea] < > > > >>>>>>> https://www.polidea.com/> > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > -- > > > >>>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect > > > >>>>>>> Astronomer.io > > > >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > -- > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > Jarek Potiuk > > > >>>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> | > > > >>>>>>> Principal Software > > > >>>>>>> > > > > Engineer > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > M: +48 660 796 129 > > <+48%20660%20796%20129> > > > >>>>>>> <+48660796129 > > > >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>> > > > >>>>>>> > > > > > > > > > > > > [image: Polidea] < > > https://www.polidea.com/ > > > > > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > -- > > > >>>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect > > > >>>>>>> Astronomer.io > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > -- > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > Jarek Potiuk > > > >>>>>>> > > > > > > > > > Polidea | Principal Software Engineer > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > M: +48 660 796 129 > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > -- > > > >>>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect > Astronomer.io > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > -- > > > >>>>>>> > > > > > Jarek Potiuk > > > >>>>>>> > > > > > Polidea | Principal Software Engineer > > > >>>>>>> > > > > > M: +48 660 796 129 > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > -- > > > >>>>>>> > > > > >>>>>>> > Jarek Potiuk > > > >>>>>>> > Polidea | Principal Software Engineer > > > >>>>>>> > > > > >>>>>>> > M: +48 660 796 129 > > > >>>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> -- > > > >>>>>> > > > >>>>>> Jarek Potiuk > > > >>>>>> Polidea <https://www.polidea.com/> | Principal Software > Engineer > > > >>>>>> > > > >>>>>> M: +48 660 796 129 <+48660796129> > > > >>>>>> [image: Polidea] <https://www.polidea.com/> > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>>> -- > > > >>>>> > > > >>>>> Jarek Potiuk > > > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > > >>>>> > > > >>>>> M: +48 660 796 129 <+48660796129> > > > >>>>> [image: Polidea] <https://www.polidea.com/> > > > >>>>> > > > >>>>> > > > >>>> > > > >>>> -- > > > >>>> > > > >>>> Jarek Potiuk > > > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > > >>>> > > > >>>> M: +48 660 796 129 <+48660796129> > > > >>>> [image: Polidea] <https://www.polidea.com/> > > > >>>> > > > >>>> > > > >>> > > > >>> -- > > > >>> *Greg Neiheisel* / Chief Architect Astronomer.io > > > >>> > > > >> > > > >> > > > >> -- > > > >> > > > >> Jarek Potiuk > > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > > > >> > > > >> M: +48 660 796 129 <+48660796129> > > > >> [image: Polidea] <https://www.polidea.com/> > > > >> > > > >> > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > -- > > > *Greg Neiheisel* / Chief Architect Astronomer.io > > > > > > > > -- > *Greg Neiheisel* / Chief Architect Astronomer.io > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
