RE: Pros and cons of Oracle db as Airflow SqlAlchemy backend

2019-12-13 Thread Daniel Imberman
Mehmet, for what it’s worth. I don’t know anyone who uses oracledb for an airflow backend, and I’m pretty certain it’s not high on the committer/PMC radar so you might be on your own for any in-depth debugging. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&sour

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

2019-12-13 Thread Jarek Potiuk
Yeah. All at once seems more than reasonable. J. On Fri, Dec 13, 2019 at 4:50 PM Kaxil Naik wrote: > Agree with Daniel, let's do it all at once. > > On Fri, Dec 13, 2019 at 3:49 PM Daniel Imberman > > wrote: > > > I’d rather we do it all at once and not need to maintain two builds. > > Most/al

Re: Test fixtures for repeatable (performance) tests?

2019-12-13 Thread Kaxil Naik
https://github.com/FactoryBoy/factory_boy looks good for creating Fixtures. Good example and article about it here: https://medium.com/@vittorio.camisa/agile-database-integration-tests-with-python-sqlalchemy-and-factory-boy-6824e8fe33a1 PS: I have tried or tested it yet On Fri, Dec 13, 2019 at 3

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

2019-12-13 Thread Kaxil Naik
Agree with Daniel, let's do it all at once. On Fri, Dec 13, 2019 at 3:49 PM Daniel Imberman wrote: > I’d rather we do it all at once and not need to maintain two builds. > Most/all of our CI/CD is dockerized at this point so there shouldn’t be a > huge difference. > > > On Fri, Dec 13, 2019 at 1

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

2019-12-13 Thread Daniel Imberman
I’d rather we do it all at once and not need to maintain two builds. Most/all of our CI/CD is dockerized at this point so there shouldn’t be a huge difference. On Fri, Dec 13, 2019 at 1:23 AM, Tomasz Urbaszek wrote: Hi all, I have to solve a problem with our kubernetes test but for your info

Test fixtures for repeatable (performance) tests?

2019-12-13 Thread Ash Berlin-Taylor
As part of my performance testing work I created a simple driver script, one of the steps it did was to "reset" the DB (at least for a given dag) so that the code was operating on the same rows in the same state each time -- I ran all my tests about 10 times to get mean and variance. My "reset"

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

2019-12-13 Thread Tomasz Urbaszek
Hi all, I have to solve a problem with our kubernetes test but for your information I have never experienced a flaky or timeouted build on Github Actions. Maybe I am lucky or maybe there's something different. If we agree to move to Github Actions, would we like to migrate incrementally or fully?

Re: Scheduler Stuck when using LocalExecutor and KubernetesPodOperator

2019-12-13 Thread Maulik Soneji
Hi Tao, We have an internal tool called Optimus that we use for ETL transformations. Optimus takes in the Bigquery transformation requests and creates dags and leverages ExternalTaskSensor to manage the dependencies between dags. We have used KubernetesOperator for this tool, as it has to run an i