Hello everyone, Great news! I've been able to provision some Google Cloud credits to start trying out a CI solution (~3k USD). They should last for a couple months (let's optimize it so it lasts as long as possible :)) while Gris and I work on a longer-term solution to provision the credits.
The name of project is apache-airflow-testing. I will be glad to add any PMC members as owners of the project. Please contact me directly with the email account that I should add to the project. Also, Jarek, please send your preferred email address so you'll have access to the project as well. Thanks so much for working on this : ) Best, Aizhamal On Fri, Jul 12, 2019 at 12:48 PM Jarek Potiuk <[email protected]> wrote: > Just FYI - I opened a ticket to get stats of the machine usage for Travis > to infra: https://issues.apache.org/jira/browse/INFRA-18742 > > On Thu, Jul 11, 2019 at 1:48 PM Jarek Potiuk <[email protected]> > wrote: > > > Absolutely! I thought about it today and GKE cluster would be perfect for > > us - especially that we can also use it to run Kubernetes tests on it ! > > That's still a major pain having to setup minikube for the tests and > having > > a GKE cluster that we can simply use would simplify this part a LOT. > > > > Principal Software Engineer > > Phone: +48660796129 > > > > czw., 11 lip 2019, 09:26 użytkownik Driesprong, Fokko > <[email protected]> > > napisał: > > > >> Yes, Gitlab works very well with GCP. A Kubernetes cluster with > >> autoscaling > >> for the runners would be perfect, and will also minimize the resources > >> provided by Google. > >> > >> Cheers, Fokko > >> > >> Op do 11 jul. 2019 om 07:13 schreef Jarek Potiuk < > >> [email protected]> > >> > >> > Since more than few people (including myself) are in favour of GitLab > >> CI, > >> > and since Apache Infra is talking to GitLab CI, I will make sure to > >> check > >> > if we can combine the two approaches - workers from Google and > managed, > >> > central GitlabCI interface to manage it (likely managed by the Infra > >> team). > >> > Airflow can easily be a "guinea pig" for GitLab CI / Apache > >> integration. > >> > We also have quite an expertise in managin GitLab in my company (we > use > >> > GitLab in Polidea for most of our mobile project CI and all the cloud > >> > builds that we run internally). > >> > > >> > I will make an AIP for that soon and involve the right people :). > >> > > >> > J. > >> > > >> > On Thu, Jul 11, 2019 at 8:01 AM Driesprong, Fokko > <[email protected] > >> > > >> > wrote: > >> > > >> > > Regardings the numbers, I believe that INFRA has an overview of the > >> usage > >> > > per project. I think they are happy to share these numbers with you. > >> > Also, > >> > > it seems like there is also a queue in Jenkins: > >> > https://status.apache.org/ > >> > > > >> > > Talking about Jenkins. I'm not a big fan of it. For example, Spark > >> uses > >> > it, > >> > > and it is rather difficult to set up the environment yourself, in > >> > contrast > >> > > with Travis. I also have good experiences with Gitlab since that is > >> the > >> > > only Docker native CI in my personal opinion. > >> > > > >> > > > But we can try both of course. And even switch later. > >> > > There is nothing as permanent as a temporary solution :-) However, > I'm > >> > not > >> > > against trying. I've checked the beam project, and the integration > >> with > >> > > Github looks good. > >> > > > >> > > Thanks again Jarek and Aizhamal for all the work an effort. > >> > > > >> > > Cheers, Fokko > >> > > > >> > > > >> > > > >> > > > >> > > Op wo 10 jul. 2019 om 23:11 schreef Aizhamal Nurmamat kyzy < > >> > > [email protected]>: > >> > > > >> > > > Hi all, > >> > > > > >> > > > I am still working on trying to get approvals for this, so this is > >> not > >> > > yet > >> > > > a done deal. I'll keep y'all updated. > >> > > > > >> > > > As for the CI solution to use, we have no particular inclination. > As > >> > long > >> > > > as the community supports it, and it is consistent with any Apache > >> > > > guidelines for CI from their projects. Jenkins and GitLab CI both > >> sound > >> > > > sensible. > >> > > > > >> > > > The email from INFRA says that Airflow runs 2600 hours of tests > per > >> > > month, > >> > > > or the equivalent of about 4 machines. Can the community help > with a > >> > > > reasonable estimate for this, so I can use it as a reference for > the > >> > > > request? > >> > > > > >> > > > Thanks! > >> > > > > >> > > > On Wed, Jul 10, 2019 at 2:43 PM Jarek Potiuk < > >> [email protected] > >> > > > >> > > > wrote: > >> > > > > >> > > > > Yeah. Gitlab CI is definitely what I would prefer as well from > the > >> > > > > "modernity" point of view (and one of my very close friends is > >> Gitlab > >> > > CI > >> > > > > maintainer and actually The person who introduced CI to GitLab > >> > > > offering). I > >> > > > > also actually already catalysed discussion between GitLab and > >> Apache > >> > > > > infrastructure to introduce GitLab CI on the "Apache" level > (they > >> are > >> > > > > talking about it now I believe). > >> > > > > > >> > > > > But from Google <> Apache/Procedural point of view it might > >> simply be > >> > > > > easier to follow footsteps of Apache Beam. It might simply be > few > >> > > clicks > >> > > > > away for the Apache Infrastructure to add more machines and > >> connect > >> > > them > >> > > > to > >> > > > > the Apache Jenkins for our project. If we have a path cleared by > >> > > others, > >> > > > > following it might be simply much faster. > >> > > > > > >> > > > > But we can try both of course. And even switch later. The Docker > >> CI > >> > > > > approach I am about to merge is designed to be super-easy to > >> switch > >> > > > betwen > >> > > > > CI systems. Virtually ALL the build logic is in scripts in > shared > >> > > Docker > >> > > > > images. There is basically one file per CI system to add and we > >> can > >> > > > support > >> > > > > Travis/Jenkins/CloudBuild/CircleCI - whatever we imaging. We can > >> even > >> > > > > support all of them at the same time :) > >> > > > > > >> > > > > J. > >> > > > > > >> > > > > On Wed, Jul 10, 2019 at 11:32 PM Bolke de Bruin < > >> [email protected]> > >> > > > wrote: > >> > > > > > >> > > > > > If you need an alternative why not use a couple of gitlab-ci > >> > runners? > >> > > > > Much > >> > > > > > easier to maintain, light weight, and much closer to what we > use > >> > now. > >> > > > > > > >> > > > > > B. > >> > > > > > > >> > > > > > Verstuurd vanaf mijn iPad > >> > > > > > > >> > > > > > > Op 10 jul. 2019 om 23:27 heeft Bolke de Bruin < > >> [email protected] > >> > > > >> > > > het > >> > > > > > volgende geschreven: > >> > > > > > > > >> > > > > > > Awesome! But I hope you are not serious about using Jenkins > >> > right? > >> > > > If I > >> > > > > > need to start a Holy War it would be against Jenkins. > >> > > > > > > > >> > > > > > > B. > >> > > > > > > > >> > > > > > > Verstuurd vanaf mijn iPad > >> > > > > > > > >> > > > > > >> Op 10 jul. 2019 om 22:55 heeft Jarek Potiuk < > >> > > > [email protected] > >> > > > > > > >> > > > > > het volgende geschreven: > >> > > > > > >> > >> > > > > > >> Hello Everyone, > >> > > > > > >> > >> > > > > > >> I have some really good news. I just had a call with Google > >> OSS > >> > > team > >> > > > > > (Gris, > >> > > > > > >> Aizhamal) and they are willing to donate VMs on Google > Cloud > >> > > > Platform > >> > > > > to > >> > > > > > >> run CI for Airflow. In order to simplify the setup (and > make > >> > sure > >> > > it > >> > > > > is > >> > > > > > ok > >> > > > > > >> according to Apache regulations) we think we should go > >> exactly > >> > the > >> > > > > same > >> > > > > > >> route as Apache Beam project (Google donated 16x 16CPU > >> machines > >> > > for > >> > > > > > them). > >> > > > > > >> The route of Apache Beam is to use the machines as workers > >> for > >> > > > Apache > >> > > > > > >> Jenkins (https://builds.apache.org/). Apache Jenkins is > one > >> of > >> > > the > >> > > > > > >> encouraged CI solutions by Apache and if we can have > workers > >> > > > connected > >> > > > > > to > >> > > > > > >> the existing Jenkins master of Apache, it means that the > >> > > maintenance > >> > > > > > >> overhead will be pretty minimal. And we can follow Apache > >> Beam > >> > > setup > >> > > > > so > >> > > > > > I > >> > > > > > >> do not expect any legal problems. > >> > > > > > >> > >> > > > > > >> I also work very closely with the team that uses Apache > Beam > >> > > Jenkins > >> > > > > > >> heavily so I have access to all the necessary experts to > help > >> > with > >> > > > the > >> > > > > > >> setup (and I am happy to help with that). > >> > > > > > >> > >> > > > > > >> I really hope everyone in the community will be really > happy > >> to > >> > go > >> > > > in > >> > > > > > that > >> > > > > > >> direction - it's. Please let me know if you have any > >> concerns ! > >> > > > > > >> > >> > > > > > >> We do not need as many machines as Beam for sure (Beam uses > >> the > >> > > > > > machines to > >> > > > > > >> process a lot of data for tests including some load > testing) > >> but > >> > > we > >> > > > > > need to > >> > > > > > >> estimate the number/types of machines that we are going to > >> need. > >> > > > > > >> Fokko, Ash, others - do you have some recent numbers for > the > >> > > current > >> > > > > > usage > >> > > > > > >> or should I open an Infrastructure ticket for it? > >> > > > > > >> > >> > > > > > >> J > >> > > > > > >> > >> > > > > > >> On Fri, Jun 28, 2019 at 4:50 PM Jarek Potiuk < > >> > > > > [email protected]> > >> > > > > > >> wrote: > >> > > > > > >> > >> > > > > > >>> Thanks Aizhamal! I spoke already to Gris and she confirmed > >> that > >> > > as > >> > > > > well > >> > > > > > >>> and the 8th of July date is ok for us as we will have to > >> > evaluate > >> > > > and > >> > > > > > >>> prepare as well. Have a nice trip. > >> > > > > > >>> > >> > > > > > >>> J. > >> > > > > > >>> > >> > > > > > >>> On Fri, Jun 28, 2019 at 4:25 PM Aizhamal Nurmamat kyzy > >> > > > > > >>> <[email protected]> wrote: > >> > > > > > >>> > >> > > > > > >>>> Hi all, > >> > > > > > >>>> > >> > > > > > >>>> On Thu, Jun 27, 2019 at 15:28 Jarek Potiuk < > >> > > > > [email protected]> > >> > > > > > >>>> wrote: > >> > > > > > >>>> > >> > > > > > >>>>> Yeah. I also have a working version of Cloud build > >> > > configuration > >> > > > > and > >> > > > > > we > >> > > > > > >>>> can > >> > > > > > >>>>> run the tests on cloud build if we can get some credits > >> from > >> > > > > Google. > >> > > > > > >>>> > >> > > > > > >>>> > >> > > > > > >>>> I can look into getting a small amount of credits > approved > >> for > >> > > > this, > >> > > > > > to > >> > > > > > >>>> see > >> > > > > > >>>> if it’s useful to offload some tests to Cloud Build, or > to > >> > > > provision > >> > > > > > some > >> > > > > > >>>> VMs to run on Apache Infra. > >> > > > > > >>>> > >> > > > > > >>>> I am traveling at the moment, but I’ll be back in the > >> office > >> > on > >> > > > July > >> > > > > > 8, > >> > > > > > >>>> and > >> > > > > > >>>> I’ll try to get this done. > >> > > > > > >>>> > >> > > > > > >>>> > >> > > > > > >>>> Thanks, > >> > > > > > >>>> Aizhamal > >> > > > > > >>>> > >> > > > > > >>>> And > >> > > > > > >>>>> the changes from the upcoming CI image will make it much > >> > easier > >> > > > to > >> > > > > > run > >> > > > > > >>>>> tests on any CI provider. Except Kubernetes tests they > are > >> > > pretty > >> > > > > > much > >> > > > > > >>>>> CI-agnostic. Kubernetes tests will likely be also fixed > >> soon. > >> > > > > > >>>>> > >> > > > > > >>>>> Another idea: I thought that in the future we can also > run > >> > only > >> > > > > > subset > >> > > > > > >>>> of > >> > > > > > >>>>> postgres/mysql/sqlite tests on all combinations. I think > >> > there > >> > > > are > >> > > > > > just > >> > > > > > >>>>> handful of tests that are specific for backend (and we > >> > already > >> > > > know > >> > > > > > >>>> which > >> > > > > > >>>>> ones they are - they are skipped-if). > >> > > > > > >>>>> > >> > > > > > >>>>> J. > >> > > > > > >>>>> > >> > > > > > >>>>> Principal Software Engineer > >> > > > > > >>>>> Phone: +48660796129 > >> > > > > > >>>>> > >> > > > > > >>>>> czw., 27 cze 2019, 15:12 użytkownik Philippe Gagnon < > >> > > > > > >>>> [email protected] > >> > > > > > >>>>>> > >> > > > > > >>>>> napisał: > >> > > > > > >>>>> > >> > > > > > >>>>>> I think the combinations that you are proposing are > >> sensible > >> > > for > >> > > > > > >>>>> pre-merge > >> > > > > > >>>>>> checks. > >> > > > > > >>>>>> > >> > > > > > >>>>>> I am working on a proposal to offload extra > combinations > >> to > >> > > > > another > >> > > > > > CI > >> > > > > > >>>>>> provider (Azure DevOps specifically seems like a good > >> > > > candidate), > >> > > > > > >>>> either > >> > > > > > >>>>>> pre or post merge. Ideally I'd like to run more > >> combinations > >> > > > > > pre-merge > >> > > > > > >>>>> but > >> > > > > > >>>>>> there is a trade-off to be conscious of here between > >> > > development > >> > > > > > >>>> velocity > >> > > > > > >>>>>> and quality assurance, which I think this issue > >> highlights > >> > > quite > >> > > > > > well. > >> > > > > > >>>>>> > >> > > > > > >>>>>> Please let me know your thoughts > >> > > > > > >>>>>> > >> > > > > > >>>>>> Philippe > >> > > > > > >>>>>> > >> > > > > > >>>>>> On Thu, Jun 27, 2019 at 9:05 AM Jarek Potiuk < > >> > > > > > >>>> [email protected]> > >> > > > > > >>>>>> wrote: > >> > > > > > >>>>>> > >> > > > > > >>>>>>> Agree that we should be thoughtful about others as > >> well: In > >> > > the > >> > > > > > >>>> latest > >> > > > > > >>>>>> push > >> > > > > > >>>>>>> (few minutes ago) of the upcoming official CI image i > >> > > > implemented > >> > > > > > >>>> the > >> > > > > > >>>>>>> change we discussed in the Github where we limit the > >> number > >> > > of > >> > > > > > >>>>>> combinations > >> > > > > > >>>>>>> we test: > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> You can see it yourself: > >> > > > > > >>>>>>> https://travis-ci.org/apache/airflow/builds/551305240 > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> Those are the combinations I propose: > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> Python: 3.6 > >> > > > > > >>>>>>> BACKEND=mysql ENV=docker > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> Python: 3.6 > >> > > > > > >>>>>>> BACKEND=postgres ENV=docker > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> Python: 3.5 > >> > > > > > >>>>>>> BACKEND=sqlite ENV=docker > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> Python: 3.6 > >> > > > > > >>>>>>> BACKEND=postgres ENV=kubernetes > >> KUBERNETES_VERSION=v1.13.0 > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> J, > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> On Thu, Jun 27, 2019 at 11:00 AM Driesprong, Fokko > >> > > > > > >>>>> <[email protected] > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> wrote: > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> We got this message last year: > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>>> Hello, Airflow PPMC. > >> > > > > > >>>>>>>>> While going through the usage statistics for our > >> Travis > >> > CI > >> > > > > > >>>>> service, I > >> > > > > > >>>>>>>>> have noticed that the Airflow project is using an > >> > > abnormally > >> > > > > > >>>> large > >> > > > > > >>>>>>>>> amount of resources, 2600 hours per month or the > >> > equivalent > >> > > > of > >> > > > > > >>>>> having > >> > > > > > >>>>>>>>> almost 4 machines building airflow non-stop 24/7. As > >> this > >> > > is > >> > > > > not > >> > > > > > >>>>>> free, > >> > > > > > >>>>>>>>> but rather costing us money, I'm contacting you with > >> the > >> > > > > > >>>> intention > >> > > > > > >>>>> of > >> > > > > > >>>>>>>>> figuring out ways to reduce the use of Travis for > the > >> > > > project. > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>>> We would greatly prefer that the project itself > comes > >> up > >> > > > with a > >> > > > > > >>>>>>> solution > >> > > > > > >>>>>>>>> to lower the usage of Travis, as we'd hate to simply > >> turn > >> > > it > >> > > > > off > >> > > > > > >>>>> for > >> > > > > > >>>>>>>>> you, but the usage is at a rather severe level, > >> totaling > >> > > more > >> > > > > > >>>> than > >> > > > > > >>>>>> 21% > >> > > > > > >>>>>>>>> of the total build time of all projects using > Travis, > >> so > >> > > > > > >>>> something > >> > > > > > >>>>>>>>> actionable should be decided upon and (preferably) > >> > > completed > >> > > > by > >> > > > > > >>>> the > >> > > > > > >>>>>> end > >> > > > > > >>>>>>>>> of May that will reduce the consumption of Travis > >> > > resources. > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>>> Alternately, if you are unable to lower the pressure > >> on > >> > > > Travis, > >> > > > > > >>>> the > >> > > > > > >>>>>>>>> podling and/or IPMC may ask the board of directors > >> for a > >> > > > > > >>>> separate > >> > > > > > >>>>>>> budget > >> > > > > > >>>>>>>>> for additional build nodes to cope with the added > >> load - > >> > > I'll > >> > > > > > >>>> leave > >> > > > > > >>>>>>> this > >> > > > > > >>>>>>>>> for the podling and IPMC to decide on. > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>>> Please let us know when you have decided on a plan > to > >> > > remedy > >> > > > > > >>>> this > >> > > > > > >>>>>>>> situation. > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>>> With regards, > >> > > > > > >>>>>>>>> Daniel on behalf of ASF Infrastructure. > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>> I think more and more projects are still migrating to > >> the > >> > > ASF > >> > > > > > >>>> Travis, > >> > > > > > >>>>>> so > >> > > > > > >>>>>>> I > >> > > > > > >>>>>>>> think natural that there is more load. However, this > >> still > >> > > > > leaves > >> > > > > > >>>> the > >> > > > > > >>>>>>>> question if we have to run the full matrix. > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>> Cheers, Fokko > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>> Op do 27 jun. 2019 om 10:56 schreef Jarek Potiuk < > >> > > > > > >>>>>>> [email protected] > >> > > > > > >>>>>>>>> : > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>>> I think we should really involve infra to increase > the > >> > slot > >> > > > > > >>>> number > >> > > > > > >>>>> or > >> > > > > > >>>>>>>> maybe > >> > > > > > >>>>>>>>> even somehow allocate slots per project. > >> > > > > > >>>>>>>>> The problem is that we cannot control what other > >> apache > >> > > > > projects > >> > > > > > >>>>> are > >> > > > > > >>>>>>>> doing, > >> > > > > > >>>>>>>>> so even if we decrease our runtime, it's the other > >> > projects > >> > > > > that > >> > > > > > >>>>>> might > >> > > > > > >>>>>>>> hold > >> > > > > > >>>>>>>>> us in the queue :( > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>>> J. > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>>> On Thu, Jun 27, 2019 at 10:19 AM Driesprong, Fokko > >> > > > > > >>>>>>> <[email protected] > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>>> wrote: > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>>>> I've noticed this at other Apache projects as well, > >> > > > sometimes > >> > > > > > >>>> it > >> > > > > > >>>>>>> takes > >> > > > > > >>>>>>>> up > >> > > > > > >>>>>>>>>> to 7-8 hours. The only thing we can do, is reduce > the > >> > > > runtime > >> > > > > > >>>> of > >> > > > > > >>>>>> the > >> > > > > > >>>>>>>> jobs > >> > > > > > >>>>>>>>>> so we take less slots :-) > >> > > > > > >>>>>>>>>> > >> > > > > > >>>>>>>>>> Cheers, Fokko > >> > > > > > >>>>>>>>>> > >> > > > > > >>>>>>>>>> Op wo 26 jun. 2019 om 21:59 schreef Jarek Potiuk < > >> > > > > > >>>>>>>>> [email protected] > >> > > > > > >>>>>>>>>>> : > >> > > > > > >>>>>>>>>> > >> > > > > > >>>>>>>>>>> Yep. That's what I suggested as the reason in the > >> > ticket > >> > > - > >> > > > I > >> > > > > > >>>>>> guess > >> > > > > > >>>>>>>>> INFRA > >> > > > > > >>>>>>>>>>> are the only people who can do anything about it > >> > > (increase > >> > > > > > >>>>>>>> concurrency > >> > > > > > >>>>>>>>> ? > >> > > > > > >>>>>>>>>>> pay more for Travis :)? ). > >> > > > > > >>>>>>>>>>> > >> > > > > > >>>>>>>>>>> On Wed, Jun 26, 2019 at 9:51 PM Ash Berlin-Taylor > < > >> > > > > > >>>>>> [email protected]> > >> > > > > > >>>>>>>>>> wrote: > >> > > > > > >>>>>>>>>>> > >> > > > > > >>>>>>>>>>>> I asked Travis on twitter and they said it was > due > >> to > >> > > the > >> > > > > > >>>>>> Apache > >> > > > > > >>>>>>>>> other > >> > > > > > >>>>>>>>>>>> projects build queues > >> > > > > > >>>>>>>>>>>> > >> > > > > > >>>>>>>>>>>> > >> > https://twitter.com/travisci/status/1143893051460526080 > >> > > > > > >>>>>>>>>>>> > >> > > > > > >>>>>>>>>>>> -ash > >> > > > > > >>>>>>>>>>>> > >> > > > > > >>>>>>>>>>>> On 26 June 2019 20:48:33 BST, Jarek Potiuk < > >> > > > > > >>>>>>>> [email protected] > >> > > > > > >>>>>>>>>> > >> > > > > > >>>>>>>>>>>> wrote: > >> > > > > > >>>>>>>>>>>>> > >> > > > > > >>>>>>>>>>>>> Hello everyone, > >> > > > > > >>>>>>>>>>>>> > >> > > > > > >>>>>>>>>>>>> For the last few days the Travis builds for > >> > > > > > >>>> apache/airflow > >> > > > > > >>>>>>> project > >> > > > > > >>>>>>>>> are > >> > > > > > >>>>>>>>>>>>> waiting in a queue for hours. This is not a > normal > >> > > > > > >>>>> situation. > >> > > > > > >>>>>>> I've > >> > > > > > >>>>>>>>>>> opened > >> > > > > > >>>>>>>>>>>>> INFRA ticket for that: > >> > > > > > >>>>>>>>>>> https://issues.apache.org/jira/browse/INFRA-18657 > >> > > > > > >>>>>>>>>>>>> > >> > > > > > >>>>>>>>>>>>> J. > >> > > > > > >>>>>>>>>>>>> > >> > > > > > >>>>>>>>>>>>> > >> > > > > > >>>>>>>>>>> > >> > > > > > >>>>>>>>>>> -- > >> > > > > > >>>>>>>>>>> > >> > > > > > >>>>>>>>>>> Jarek Potiuk > >> > > > > > >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal > >> > Software > >> > > > > > >>>>> Engineer > >> > > > > > >>>>>>>>>>> > >> > > > > > >>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >> > > > > > >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >> > > > > > >>>>>>>>>>> > >> > > > > > >>>>>>>>>> > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>>> -- > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>>> Jarek Potiuk > >> > > > > > >>>>>>>>> Polidea <https://www.polidea.com/> | Principal > >> Software > >> > > > > > >>>> Engineer > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>>> M: +48 660 796 129 <+48660796129> > >> > > > > > >>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> -- > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> Jarek Potiuk > >> > > > > > >>>>>>> Polidea <https://www.polidea.com/> | Principal > Software > >> > > > Engineer > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> M: +48 660 796 129 <+48660796129> > >> > > > > > >>>>>>> [image: Polidea] <https://www.polidea.com/> > >> > > > > > >>>>>>> > >> > > > > > >>>>>> > >> > > > > > >>>>> > >> > > > > > >>>> > >> > > > > > >>> > >> > > > > > >>> > >> > > > > > >>> -- > >> > > > > > >>> > >> > > > > > >>> Jarek Potiuk > >> > > > > > >>> Polidea <https://www.polidea.com/> | Principal Software > >> > Engineer > >> > > > > > >>> > >> > > > > > >>> M: +48 660 796 129 <+48660796129> > >> > > > > > >>> [image: Polidea] <https://www.polidea.com/> > >> > > > > > >>> > >> > > > > > >>> > >> > > > > > >> > >> > > > > > >> -- > >> > > > > > >> > >> > > > > > >> Jarek Potiuk > >> > > > > > >> Polidea <https://www.polidea.com/> | Principal Software > >> > Engineer > >> > > > > > >> > >> > > > > > >> M: +48 660 796 129 <+48660796129> > >> > > > > > >> [image: Polidea] <https://www.polidea.com/> > >> > > > > > > >> > > > > > >> > > > > > >> > > > > -- > >> > > > > > >> > > > > Jarek Potiuk > >> > > > > Polidea <https://www.polidea.com/> | Principal Software > Engineer > >> > > > > > >> > > > > M: +48 660 796 129 <+48660796129> > >> > > > > [image: Polidea] <https://www.polidea.com/> > >> > > > > > >> > > > > >> > > > >> > > >> > > >> > -- > >> > > >> > Jarek Potiuk > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > > >> > M: +48 660 796 129 <+48660796129> > >> > [image: Polidea] <https://www.polidea.com/> > >> > > >> > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
