kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-542431579
@mik-laj Agree with most of your comments, will update it by end of this
week.
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-535914795
I am going to squash this PR.
I have created a branch on my fork
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-532172943
> > * Do we want to let users use old versions of MySQL (and Mariadb that
doesn't support JSON with
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-530789821
Few things to do:
- [ ] Add
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-530785443
Few things to do:
- [ ] Add
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-529024391
@ashb @coufon With the above commit:
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-528982114
> That last point is interesting - cos at the SQL level we still need to
write it as a string. I wonder
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-528697431
Some more testing:
For another trial, I am completely removing the need to load JSON from a str
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-528654768
Have done some more benchmarking:
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-52749
> How about instead of a background thread (I'm wary of using threads in
python) could we instead query
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-523847270
> fileloc is not stored in DB for subdag, because subdag does not have a row
in DB. subdag's fileloc is
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-523416060
> Bug:
>
> ```
> [2019-08-21 12:19:17,625] {app.py:1891} ERROR - Exception on /task [GET]
>
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-523239054
As far as I can tell, this PR is now working & ready for Code Review.
**Notes**:
-
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-521358954
> > Pending Issues:
> >
> > * We still have the issue of `SerializedBaseOperator` being displayed
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-520101071
Pending Issues:
- ~Add Timezone support to `serialized_dag` table~
- We still have the issue of
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-519723159
Just rebased
This is an automated
kaxil commented on issue #5743: [AIRFLOW-5088][AIP-24] Persisting serialized
DAG in DB for webserver scalability
URL: https://github.com/apache/airflow/pull/5743#issuecomment-519647789
> hmm. It still failing on rat. This is strange. I am taking a look.
Thanks
17 matches
Mail list logo