Just FYI - I opened a ticket to get stats of the machine usage for Travis to infra: https://issues.apache.org/jira/browse/INFRA-18742
On Thu, Jul 11, 2019 at 1:48 PM Jarek Potiuk <[email protected]> wrote: > Absolutely! I thought about it today and GKE cluster would be perfect for > us - especially that we can also use it to run Kubernetes tests on it ! > That's still a major pain having to setup minikube for the tests and having > a GKE cluster that we can simply use would simplify this part a LOT. > > Principal Software Engineer > Phone: +48660796129 > > czw., 11 lip 2019, 09:26 użytkownik Driesprong, Fokko <[email protected]> > napisał: > >> Yes, Gitlab works very well with GCP. A Kubernetes cluster with >> autoscaling >> for the runners would be perfect, and will also minimize the resources >> provided by Google. >> >> Cheers, Fokko >> >> Op do 11 jul. 2019 om 07:13 schreef Jarek Potiuk < >> [email protected]> >> >> > Since more than few people (including myself) are in favour of GitLab >> CI, >> > and since Apache Infra is talking to GitLab CI, I will make sure to >> check >> > if we can combine the two approaches - workers from Google and managed, >> > central GitlabCI interface to manage it (likely managed by the Infra >> team). >> > Airflow can easily be a "guinea pig" for GitLab CI / Apache >> integration. >> > We also have quite an expertise in managin GitLab in my company (we use >> > GitLab in Polidea for most of our mobile project CI and all the cloud >> > builds that we run internally). >> > >> > I will make an AIP for that soon and involve the right people :). >> > >> > J. >> > >> > On Thu, Jul 11, 2019 at 8:01 AM Driesprong, Fokko <[email protected] >> > >> > wrote: >> > >> > > Regardings the numbers, I believe that INFRA has an overview of the >> usage >> > > per project. I think they are happy to share these numbers with you. >> > Also, >> > > it seems like there is also a queue in Jenkins: >> > https://status.apache.org/ >> > > >> > > Talking about Jenkins. I'm not a big fan of it. For example, Spark >> uses >> > it, >> > > and it is rather difficult to set up the environment yourself, in >> > contrast >> > > with Travis. I also have good experiences with Gitlab since that is >> the >> > > only Docker native CI in my personal opinion. >> > > >> > > > But we can try both of course. And even switch later. >> > > There is nothing as permanent as a temporary solution :-) However, I'm >> > not >> > > against trying. I've checked the beam project, and the integration >> with >> > > Github looks good. >> > > >> > > Thanks again Jarek and Aizhamal for all the work an effort. >> > > >> > > Cheers, Fokko >> > > >> > > >> > > >> > > >> > > Op wo 10 jul. 2019 om 23:11 schreef Aizhamal Nurmamat kyzy < >> > > [email protected]>: >> > > >> > > > Hi all, >> > > > >> > > > I am still working on trying to get approvals for this, so this is >> not >> > > yet >> > > > a done deal. I'll keep y'all updated. >> > > > >> > > > As for the CI solution to use, we have no particular inclination. As >> > long >> > > > as the community supports it, and it is consistent with any Apache >> > > > guidelines for CI from their projects. Jenkins and GitLab CI both >> sound >> > > > sensible. >> > > > >> > > > The email from INFRA says that Airflow runs 2600 hours of tests per >> > > month, >> > > > or the equivalent of about 4 machines. Can the community help with a >> > > > reasonable estimate for this, so I can use it as a reference for the >> > > > request? >> > > > >> > > > Thanks! >> > > > >> > > > On Wed, Jul 10, 2019 at 2:43 PM Jarek Potiuk < >> [email protected] >> > > >> > > > wrote: >> > > > >> > > > > Yeah. Gitlab CI is definitely what I would prefer as well from the >> > > > > "modernity" point of view (and one of my very close friends is >> Gitlab >> > > CI >> > > > > maintainer and actually The person who introduced CI to GitLab >> > > > offering). I >> > > > > also actually already catalysed discussion between GitLab and >> Apache >> > > > > infrastructure to introduce GitLab CI on the "Apache" level (they >> are >> > > > > talking about it now I believe). >> > > > > >> > > > > But from Google <> Apache/Procedural point of view it might >> simply be >> > > > > easier to follow footsteps of Apache Beam. It might simply be few >> > > clicks >> > > > > away for the Apache Infrastructure to add more machines and >> connect >> > > them >> > > > to >> > > > > the Apache Jenkins for our project. If we have a path cleared by >> > > others, >> > > > > following it might be simply much faster. >> > > > > >> > > > > But we can try both of course. And even switch later. The Docker >> CI >> > > > > approach I am about to merge is designed to be super-easy to >> switch >> > > > betwen >> > > > > CI systems. Virtually ALL the build logic is in scripts in shared >> > > Docker >> > > > > images. There is basically one file per CI system to add and we >> can >> > > > support >> > > > > Travis/Jenkins/CloudBuild/CircleCI - whatever we imaging. We can >> even >> > > > > support all of them at the same time :) >> > > > > >> > > > > J. >> > > > > >> > > > > On Wed, Jul 10, 2019 at 11:32 PM Bolke de Bruin < >> [email protected]> >> > > > wrote: >> > > > > >> > > > > > If you need an alternative why not use a couple of gitlab-ci >> > runners? >> > > > > Much >> > > > > > easier to maintain, light weight, and much closer to what we use >> > now. >> > > > > > >> > > > > > B. >> > > > > > >> > > > > > Verstuurd vanaf mijn iPad >> > > > > > >> > > > > > > Op 10 jul. 2019 om 23:27 heeft Bolke de Bruin < >> [email protected] >> > > >> > > > het >> > > > > > volgende geschreven: >> > > > > > > >> > > > > > > Awesome! But I hope you are not serious about using Jenkins >> > right? >> > > > If I >> > > > > > need to start a Holy War it would be against Jenkins. >> > > > > > > >> > > > > > > B. >> > > > > > > >> > > > > > > Verstuurd vanaf mijn iPad >> > > > > > > >> > > > > > >> Op 10 jul. 2019 om 22:55 heeft Jarek Potiuk < >> > > > [email protected] >> > > > > > >> > > > > > het volgende geschreven: >> > > > > > >> >> > > > > > >> Hello Everyone, >> > > > > > >> >> > > > > > >> I have some really good news. I just had a call with Google >> OSS >> > > team >> > > > > > (Gris, >> > > > > > >> Aizhamal) and they are willing to donate VMs on Google Cloud >> > > > Platform >> > > > > to >> > > > > > >> run CI for Airflow. In order to simplify the setup (and make >> > sure >> > > it >> > > > > is >> > > > > > ok >> > > > > > >> according to Apache regulations) we think we should go >> exactly >> > the >> > > > > same >> > > > > > >> route as Apache Beam project (Google donated 16x 16CPU >> machines >> > > for >> > > > > > them). >> > > > > > >> The route of Apache Beam is to use the machines as workers >> for >> > > > Apache >> > > > > > >> Jenkins (https://builds.apache.org/). Apache Jenkins is one >> of >> > > the >> > > > > > >> encouraged CI solutions by Apache and if we can have workers >> > > > connected >> > > > > > to >> > > > > > >> the existing Jenkins master of Apache, it means that the >> > > maintenance >> > > > > > >> overhead will be pretty minimal. And we can follow Apache >> Beam >> > > setup >> > > > > so >> > > > > > I >> > > > > > >> do not expect any legal problems. >> > > > > > >> >> > > > > > >> I also work very closely with the team that uses Apache Beam >> > > Jenkins >> > > > > > >> heavily so I have access to all the necessary experts to help >> > with >> > > > the >> > > > > > >> setup (and I am happy to help with that). >> > > > > > >> >> > > > > > >> I really hope everyone in the community will be really happy >> to >> > go >> > > > in >> > > > > > that >> > > > > > >> direction - it's. Please let me know if you have any >> concerns ! >> > > > > > >> >> > > > > > >> We do not need as many machines as Beam for sure (Beam uses >> the >> > > > > > machines to >> > > > > > >> process a lot of data for tests including some load testing) >> but >> > > we >> > > > > > need to >> > > > > > >> estimate the number/types of machines that we are going to >> need. >> > > > > > >> Fokko, Ash, others - do you have some recent numbers for the >> > > current >> > > > > > usage >> > > > > > >> or should I open an Infrastructure ticket for it? >> > > > > > >> >> > > > > > >> J >> > > > > > >> >> > > > > > >> On Fri, Jun 28, 2019 at 4:50 PM Jarek Potiuk < >> > > > > [email protected]> >> > > > > > >> wrote: >> > > > > > >> >> > > > > > >>> Thanks Aizhamal! I spoke already to Gris and she confirmed >> that >> > > as >> > > > > well >> > > > > > >>> and the 8th of July date is ok for us as we will have to >> > evaluate >> > > > and >> > > > > > >>> prepare as well. Have a nice trip. >> > > > > > >>> >> > > > > > >>> J. >> > > > > > >>> >> > > > > > >>> On Fri, Jun 28, 2019 at 4:25 PM Aizhamal Nurmamat kyzy >> > > > > > >>> <[email protected]> wrote: >> > > > > > >>> >> > > > > > >>>> Hi all, >> > > > > > >>>> >> > > > > > >>>> On Thu, Jun 27, 2019 at 15:28 Jarek Potiuk < >> > > > > [email protected]> >> > > > > > >>>> wrote: >> > > > > > >>>> >> > > > > > >>>>> Yeah. I also have a working version of Cloud build >> > > configuration >> > > > > and >> > > > > > we >> > > > > > >>>> can >> > > > > > >>>>> run the tests on cloud build if we can get some credits >> from >> > > > > Google. >> > > > > > >>>> >> > > > > > >>>> >> > > > > > >>>> I can look into getting a small amount of credits approved >> for >> > > > this, >> > > > > > to >> > > > > > >>>> see >> > > > > > >>>> if it’s useful to offload some tests to Cloud Build, or to >> > > > provision >> > > > > > some >> > > > > > >>>> VMs to run on Apache Infra. >> > > > > > >>>> >> > > > > > >>>> I am traveling at the moment, but I’ll be back in the >> office >> > on >> > > > July >> > > > > > 8, >> > > > > > >>>> and >> > > > > > >>>> I’ll try to get this done. >> > > > > > >>>> >> > > > > > >>>> >> > > > > > >>>> Thanks, >> > > > > > >>>> Aizhamal >> > > > > > >>>> >> > > > > > >>>> And >> > > > > > >>>>> the changes from the upcoming CI image will make it much >> > easier >> > > > to >> > > > > > run >> > > > > > >>>>> tests on any CI provider. Except Kubernetes tests they are >> > > pretty >> > > > > > much >> > > > > > >>>>> CI-agnostic. Kubernetes tests will likely be also fixed >> soon. >> > > > > > >>>>> >> > > > > > >>>>> Another idea: I thought that in the future we can also run >> > only >> > > > > > subset >> > > > > > >>>> of >> > > > > > >>>>> postgres/mysql/sqlite tests on all combinations. I think >> > there >> > > > are >> > > > > > just >> > > > > > >>>>> handful of tests that are specific for backend (and we >> > already >> > > > know >> > > > > > >>>> which >> > > > > > >>>>> ones they are - they are skipped-if). >> > > > > > >>>>> >> > > > > > >>>>> J. >> > > > > > >>>>> >> > > > > > >>>>> Principal Software Engineer >> > > > > > >>>>> Phone: +48660796129 >> > > > > > >>>>> >> > > > > > >>>>> czw., 27 cze 2019, 15:12 użytkownik Philippe Gagnon < >> > > > > > >>>> [email protected] >> > > > > > >>>>>> >> > > > > > >>>>> napisał: >> > > > > > >>>>> >> > > > > > >>>>>> I think the combinations that you are proposing are >> sensible >> > > for >> > > > > > >>>>> pre-merge >> > > > > > >>>>>> checks. >> > > > > > >>>>>> >> > > > > > >>>>>> I am working on a proposal to offload extra combinations >> to >> > > > > another >> > > > > > CI >> > > > > > >>>>>> provider (Azure DevOps specifically seems like a good >> > > > candidate), >> > > > > > >>>> either >> > > > > > >>>>>> pre or post merge. Ideally I'd like to run more >> combinations >> > > > > > pre-merge >> > > > > > >>>>> but >> > > > > > >>>>>> there is a trade-off to be conscious of here between >> > > development >> > > > > > >>>> velocity >> > > > > > >>>>>> and quality assurance, which I think this issue >> highlights >> > > quite >> > > > > > well. >> > > > > > >>>>>> >> > > > > > >>>>>> Please let me know your thoughts >> > > > > > >>>>>> >> > > > > > >>>>>> Philippe >> > > > > > >>>>>> >> > > > > > >>>>>> On Thu, Jun 27, 2019 at 9:05 AM Jarek Potiuk < >> > > > > > >>>> [email protected]> >> > > > > > >>>>>> wrote: >> > > > > > >>>>>> >> > > > > > >>>>>>> Agree that we should be thoughtful about others as >> well: In >> > > the >> > > > > > >>>> latest >> > > > > > >>>>>> push >> > > > > > >>>>>>> (few minutes ago) of the upcoming official CI image i >> > > > implemented >> > > > > > >>>> the >> > > > > > >>>>>>> change we discussed in the Github where we limit the >> number >> > > of >> > > > > > >>>>>> combinations >> > > > > > >>>>>>> we test: >> > > > > > >>>>>>> >> > > > > > >>>>>>> You can see it yourself: >> > > > > > >>>>>>> https://travis-ci.org/apache/airflow/builds/551305240 >> > > > > > >>>>>>> >> > > > > > >>>>>>> Those are the combinations I propose: >> > > > > > >>>>>>> >> > > > > > >>>>>>> Python: 3.6 >> > > > > > >>>>>>> BACKEND=mysql ENV=docker >> > > > > > >>>>>>> >> > > > > > >>>>>>> Python: 3.6 >> > > > > > >>>>>>> BACKEND=postgres ENV=docker >> > > > > > >>>>>>> >> > > > > > >>>>>>> Python: 3.5 >> > > > > > >>>>>>> BACKEND=sqlite ENV=docker >> > > > > > >>>>>>> >> > > > > > >>>>>>> Python: 3.6 >> > > > > > >>>>>>> BACKEND=postgres ENV=kubernetes >> KUBERNETES_VERSION=v1.13.0 >> > > > > > >>>>>>> >> > > > > > >>>>>>> J, >> > > > > > >>>>>>> >> > > > > > >>>>>>> >> > > > > > >>>>>>> On Thu, Jun 27, 2019 at 11:00 AM Driesprong, Fokko >> > > > > > >>>>> <[email protected] >> > > > > > >>>>>>> >> > > > > > >>>>>>> wrote: >> > > > > > >>>>>>> >> > > > > > >>>>>>>> We got this message last year: >> > > > > > >>>>>>>> >> > > > > > >>>>>>>>> Hello, Airflow PPMC. >> > > > > > >>>>>>>>> While going through the usage statistics for our >> Travis >> > CI >> > > > > > >>>>> service, I >> > > > > > >>>>>>>>> have noticed that the Airflow project is using an >> > > abnormally >> > > > > > >>>> large >> > > > > > >>>>>>>>> amount of resources, 2600 hours per month or the >> > equivalent >> > > > of >> > > > > > >>>>> having >> > > > > > >>>>>>>>> almost 4 machines building airflow non-stop 24/7. As >> this >> > > is >> > > > > not >> > > > > > >>>>>> free, >> > > > > > >>>>>>>>> but rather costing us money, I'm contacting you with >> the >> > > > > > >>>> intention >> > > > > > >>>>> of >> > > > > > >>>>>>>>> figuring out ways to reduce the use of Travis for the >> > > > project. >> > > > > > >>>>>>>> >> > > > > > >>>>>>>>> We would greatly prefer that the project itself comes >> up >> > > > with a >> > > > > > >>>>>>> solution >> > > > > > >>>>>>>>> to lower the usage of Travis, as we'd hate to simply >> turn >> > > it >> > > > > off >> > > > > > >>>>> for >> > > > > > >>>>>>>>> you, but the usage is at a rather severe level, >> totaling >> > > more >> > > > > > >>>> than >> > > > > > >>>>>> 21% >> > > > > > >>>>>>>>> of the total build time of all projects using Travis, >> so >> > > > > > >>>> something >> > > > > > >>>>>>>>> actionable should be decided upon and (preferably) >> > > completed >> > > > by >> > > > > > >>>> the >> > > > > > >>>>>> end >> > > > > > >>>>>>>>> of May that will reduce the consumption of Travis >> > > resources. >> > > > > > >>>>>>>> >> > > > > > >>>>>>>>> Alternately, if you are unable to lower the pressure >> on >> > > > Travis, >> > > > > > >>>> the >> > > > > > >>>>>>>>> podling and/or IPMC may ask the board of directors >> for a >> > > > > > >>>> separate >> > > > > > >>>>>>> budget >> > > > > > >>>>>>>>> for additional build nodes to cope with the added >> load - >> > > I'll >> > > > > > >>>> leave >> > > > > > >>>>>>> this >> > > > > > >>>>>>>>> for the podling and IPMC to decide on. >> > > > > > >>>>>>>> >> > > > > > >>>>>>>>> Please let us know when you have decided on a plan to >> > > remedy >> > > > > > >>>> this >> > > > > > >>>>>>>> situation. >> > > > > > >>>>>>>> >> > > > > > >>>>>>>>> With regards, >> > > > > > >>>>>>>>> Daniel on behalf of ASF Infrastructure. >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> I think more and more projects are still migrating to >> the >> > > ASF >> > > > > > >>>> Travis, >> > > > > > >>>>>> so >> > > > > > >>>>>>> I >> > > > > > >>>>>>>> think natural that there is more load. However, this >> still >> > > > > leaves >> > > > > > >>>> the >> > > > > > >>>>>>>> question if we have to run the full matrix. >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> Cheers, Fokko >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> >> > > > > > >>>>>>>> Op do 27 jun. 2019 om 10:56 schreef Jarek Potiuk < >> > > > > > >>>>>>> [email protected] >> > > > > > >>>>>>>>> : >> > > > > > >>>>>>>> >> > > > > > >>>>>>>>> I think we should really involve infra to increase the >> > slot >> > > > > > >>>> number >> > > > > > >>>>> or >> > > > > > >>>>>>>> maybe >> > > > > > >>>>>>>>> even somehow allocate slots per project. >> > > > > > >>>>>>>>> The problem is that we cannot control what other >> apache >> > > > > projects >> > > > > > >>>>> are >> > > > > > >>>>>>>> doing, >> > > > > > >>>>>>>>> so even if we decrease our runtime, it's the other >> > projects >> > > > > that >> > > > > > >>>>>> might >> > > > > > >>>>>>>> hold >> > > > > > >>>>>>>>> us in the queue :( >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> J. >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> On Thu, Jun 27, 2019 at 10:19 AM Driesprong, Fokko >> > > > > > >>>>>>> <[email protected] >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> wrote: >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>>> I've noticed this at other Apache projects as well, >> > > > sometimes >> > > > > > >>>> it >> > > > > > >>>>>>> takes >> > > > > > >>>>>>>> up >> > > > > > >>>>>>>>>> to 7-8 hours. The only thing we can do, is reduce the >> > > > runtime >> > > > > > >>>> of >> > > > > > >>>>>> the >> > > > > > >>>>>>>> jobs >> > > > > > >>>>>>>>>> so we take less slots :-) >> > > > > > >>>>>>>>>> >> > > > > > >>>>>>>>>> Cheers, Fokko >> > > > > > >>>>>>>>>> >> > > > > > >>>>>>>>>> Op wo 26 jun. 2019 om 21:59 schreef Jarek Potiuk < >> > > > > > >>>>>>>>> [email protected] >> > > > > > >>>>>>>>>>> : >> > > > > > >>>>>>>>>> >> > > > > > >>>>>>>>>>> Yep. That's what I suggested as the reason in the >> > ticket >> > > - >> > > > I >> > > > > > >>>>>> guess >> > > > > > >>>>>>>>> INFRA >> > > > > > >>>>>>>>>>> are the only people who can do anything about it >> > > (increase >> > > > > > >>>>>>>> concurrency >> > > > > > >>>>>>>>> ? >> > > > > > >>>>>>>>>>> pay more for Travis :)? ). >> > > > > > >>>>>>>>>>> >> > > > > > >>>>>>>>>>> On Wed, Jun 26, 2019 at 9:51 PM Ash Berlin-Taylor < >> > > > > > >>>>>> [email protected]> >> > > > > > >>>>>>>>>> wrote: >> > > > > > >>>>>>>>>>> >> > > > > > >>>>>>>>>>>> I asked Travis on twitter and they said it was due >> to >> > > the >> > > > > > >>>>>> Apache >> > > > > > >>>>>>>>> other >> > > > > > >>>>>>>>>>>> projects build queues >> > > > > > >>>>>>>>>>>> >> > > > > > >>>>>>>>>>>> >> > https://twitter.com/travisci/status/1143893051460526080 >> > > > > > >>>>>>>>>>>> >> > > > > > >>>>>>>>>>>> -ash >> > > > > > >>>>>>>>>>>> >> > > > > > >>>>>>>>>>>> On 26 June 2019 20:48:33 BST, Jarek Potiuk < >> > > > > > >>>>>>>> [email protected] >> > > > > > >>>>>>>>>> >> > > > > > >>>>>>>>>>>> wrote: >> > > > > > >>>>>>>>>>>>> >> > > > > > >>>>>>>>>>>>> Hello everyone, >> > > > > > >>>>>>>>>>>>> >> > > > > > >>>>>>>>>>>>> For the last few days the Travis builds for >> > > > > > >>>> apache/airflow >> > > > > > >>>>>>> project >> > > > > > >>>>>>>>> are >> > > > > > >>>>>>>>>>>>> waiting in a queue for hours. This is not a normal >> > > > > > >>>>> situation. >> > > > > > >>>>>>> I've >> > > > > > >>>>>>>>>>> opened >> > > > > > >>>>>>>>>>>>> INFRA ticket for that: >> > > > > > >>>>>>>>>>> https://issues.apache.org/jira/browse/INFRA-18657 >> > > > > > >>>>>>>>>>>>> >> > > > > > >>>>>>>>>>>>> J. >> > > > > > >>>>>>>>>>>>> >> > > > > > >>>>>>>>>>>>> >> > > > > > >>>>>>>>>>> >> > > > > > >>>>>>>>>>> -- >> > > > > > >>>>>>>>>>> >> > > > > > >>>>>>>>>>> Jarek Potiuk >> > > > > > >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal >> > Software >> > > > > > >>>>> Engineer >> > > > > > >>>>>>>>>>> >> > > > > > >>>>>>>>>>> M: +48 660 796 129 <+48660796129> >> > > > > > >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> >> > > > > > >>>>>>>>>>> >> > > > > > >>>>>>>>>> >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> -- >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> Jarek Potiuk >> > > > > > >>>>>>>>> Polidea <https://www.polidea.com/> | Principal >> Software >> > > > > > >>>> Engineer >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>>> M: +48 660 796 129 <+48660796129> >> > > > > > >>>>>>>>> [image: Polidea] <https://www.polidea.com/> >> > > > > > >>>>>>>>> >> > > > > > >>>>>>>> >> > > > > > >>>>>>> >> > > > > > >>>>>>> >> > > > > > >>>>>>> -- >> > > > > > >>>>>>> >> > > > > > >>>>>>> Jarek Potiuk >> > > > > > >>>>>>> Polidea <https://www.polidea.com/> | Principal Software >> > > > Engineer >> > > > > > >>>>>>> >> > > > > > >>>>>>> M: +48 660 796 129 <+48660796129> >> > > > > > >>>>>>> [image: Polidea] <https://www.polidea.com/> >> > > > > > >>>>>>> >> > > > > > >>>>>> >> > > > > > >>>>> >> > > > > > >>>> >> > > > > > >>> >> > > > > > >>> >> > > > > > >>> -- >> > > > > > >>> >> > > > > > >>> Jarek Potiuk >> > > > > > >>> Polidea <https://www.polidea.com/> | Principal Software >> > Engineer >> > > > > > >>> >> > > > > > >>> M: +48 660 796 129 <+48660796129> >> > > > > > >>> [image: Polidea] <https://www.polidea.com/> >> > > > > > >>> >> > > > > > >>> >> > > > > > >> >> > > > > > >> -- >> > > > > > >> >> > > > > > >> Jarek Potiuk >> > > > > > >> Polidea <https://www.polidea.com/> | Principal Software >> > Engineer >> > > > > > >> >> > > > > > >> M: +48 660 796 129 <+48660796129> >> > > > > > >> [image: Polidea] <https://www.polidea.com/> >> > > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > >> > > > > Jarek Potiuk >> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > > > > >> > > > > M: +48 660 796 129 <+48660796129> >> > > > > [image: Polidea] <https://www.polidea.com/> >> > > > > >> > > > >> > > >> > >> > >> > -- >> > >> > Jarek Potiuk >> > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > >> > M: +48 660 796 129 <+48660796129> >> > [image: Polidea] <https://www.polidea.com/> >> > >> > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
