As promised, here is a quick write-up to help kick-start the discussion
https://docs.google.com/document/d/1jqmXO6mGZzHkhuvISedQVnhyc37V_iz00hokdMuoFaI/
Elad, Asif, Paola, Pete, and apologies to anyone I missed,
Let's collaborate on this. We can use the slack channel Ash setup
(#issue-triage) to
Are there any use-cases that REQUIRE pickle? Do we have any sense of what % of
the Airflow community depends on Pickle? I’m all for killing it if possible but
I want to make sure we’re not setting up a major hurdle for migration.
via Newton Mail
I'm getting bad flashbacks of fighting with pickles early on in the history
of the project. I've learned since then to stay away. Almost all solutions
that involve pickles are bad solutions. Beyond but related to the security
implication are the issues of pickle entanglement, not really knowing
11 +1 votes, no disenters. I will ask Infra to make this change.
-a
On Sep 15 2020, at 8:21 am, Chao-Han Tsai wrote:
> +1
>
> On Mon, Sep 14, 2020 at 10:44 PM Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
>
>> +1
>>
>> how about +1000² to "let's make html email a default thing
I agree with Gerard on all fronts.
SubDags are difficult, slow, and can cause a lot of strange edge cases. The
difficulties in SubDags is a sticking point for Airflow competitors that I want
to remove as quickly as possible. I think that 2.0 is a perfect time to
introduce TaskGroups as people
Internally, we have blocked people from using SubDags due to its slowness
and sometimes instability. I would vote for adding TaskGroups earliest as
possible + remove documentation for SubDags (users can go to old versions
of docs to find that) + add a notice of deprecation on new TaskGroup
Hi Kevin,
I'm not sure if it's exactly the same issue, but I recently had a similar
issue where airflow stopped reading logs from certain pods and thought they
were still running when in fact they were completed. It only affected pods
that have gaps of a few minutes between writing a log line.
I
Hi all,
One of the thing that we skipped in last Airflow 2.0 dev was around
"documentation". The main reason for skipping it in the meeting was to
gather good data-points and feedbacks from the community.
So my question to everyone reading this email is "What improvements do you
want in the docs
Hi all,
One of the things we discussed on Monday's Airflow 2.0 Dev call was around
*TaskGroups* - a new concept introduced by
https://github.com/apache/airflow/pull/10153 (AIP-34
Hi all,
We briefly discussed how pickling is currently used in Airflow codebase and
whether or not we should remove it for 2.0 in the Airflow 2.0 Dev call this
Monday.
Currently, AFAIK only *CeleryExecutor* supports pickling (code
Hello Team,
Currently we are using the airflow version - 1.10.10 to data ingest.
In our DAG, we create tasks dynamically based on data volume , i.e if data
volume is high, number of parallel tasks increases and if the data volume
is less the number of parallel tasks reduces in the next run
After reviewing some issues I think we might need a few more labels like:
area:scheduling options
missing information
can't reproduce
maybe we can create a dedicated slack channel to work on the small details.
Elad
On Wed, Sep 16, 2020 at 9:12 PM Kaxil Naik wrote:
> @paola - Unfortunately
12 matches
Mail list logo
