Should we just replace the
import_errors table in place?
If I can get a sense of high-level direction I can put together an AIP / PR.
Cheers,
Jake
--
*Jacob Ferriero*
Strategic Cloud Engineer: Data Engineering
jferri...@google.com
617-714-2509
> expected
> > > > > > to
> > > > > > > be compatible,
> > > > > > > but changes to task structure within the DAG may cause the tree
> > > view
> > > > > not
> > > > > > to
> > > > > &g
t; > > > >> > UI).
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > There was a previous thread about this FYI with
> more
> > > > > context
> > > > >
l.com>
> > > > > > >> wrote:
> > > > > > >> > >It's a good idea.
> > > > > > >> > >
> > > > > > >> > >It will make reading the codebase easier, and besides the
> > whole
> &
> operation (set of events)?
> >
> > By doing this I would like to work on a set of how-to rules for
> > designing the logic of `execute` method. I would like to encourage you
> > to share your experiences with desiging and working with complex
> > operators :)
> >
> > Hope you are good,
> > Tomek
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> <+48%20660%20796%20129>>
> [image: Polidea] <https://www.polidea.com/>
>
--
*Jacob Ferriero*
Strategic Cloud Engineer: Data Engineering
jferri...@google.com
617-714-2509
gt; > On Fri, Jun 5, 2020 at 8:32 PM Jarek Potiuk
> > wrote:
> >
> >> Hey Jacob,
> >>
> >> All times in Pretalx are in Bay Area time - Pacific Time ( PDT as we
> >> observe PDT till November :)
> >>
> >>
> >> J.
> >&g
Hi Airflow Community,
I was to excited to receive pretalx acceptance acceptance for Airflow
Summit talk today and it asks to fill in availability in a 24 hr slots for
each day.
What timezone are the sign up times?
Seems like it'd be fitting if it were UTC because that is the airflow
default but I
ause data
> > in
> > > > source can change). Perhaps in your initial load you don't want to
> > pull
> > > by
> > > > day (too slow) but you also don't want to pull in one batch -- so you
> > > carve
> > > > up batches that are appropriate to the situation. And this is where
> > it's
> > > > helpful to have a state persistence mechanism: you can use this to
> > store
> > > > your progress on initial load, and in the event of failure, resume
> from
> > > > point of failure. Yes you _could_ parse it from s3 or wherever, but
> > > doing
> > > > so presents its own challenges and risks, and it is convenient to
> just
> > > > store it in the database -- and not necessarily any more problematic.
> > > >
> > >
> > > Same here - if your data source is not providing data in fixed
> > intervals, I
> > > think Apache Airflow might not be the best choice.
> > >
> > >
> > > >
> > > > *Scenario 4: no access*
> > > > As pointed out earlier, sometimes you might not have access to
> target.
> > > > E.g. i am pushing to someone elses s3 bucket and we only have
> PutObject
> > > but
> > > > can't read what's there. So we can't look at target to infer state.
> > > >
> > > > I'm sure there are other use cases out there. Anything "incremental"
> > > > implies a state.
> > > >
> > >
> > > That's the point I think that there might be a problem. Airflow is not
> > > designed to support incremental source of data. And trying to convert
> > > Airflow into such use case is probably not a good idea. Maybe it's just
> > the
> > > same as trying to use an axe to hammer a nail. It will work sometimes,
> > but
> > > maybe it's better to use a hammer instead.
> > >
> > > J.
> > >
> >
>
--
*Jacob Ferriero*
Strategic Cloud Engineer: Data Engineering
jferri...@google.com
617-714-2509
Image not working on dev list here is link to the github review comment
containing said image:
https://github.com/apache/airflow/pull/6370#issuecomment-546582724.
On Tue, Jan 7, 2020 at 5:40 PM Jacob Ferriero wrote:
> Hello Dev List,
>
> The inspiration for this is to allow operators t
DO_NOT_CLEAR_PREFIX = "_STATEFUL_"` or similar.
[image: image.png]
--
*Jacob Ferriero*
Strategic Cloud Engineer: Data Engineering
jferri...@google.com
617-714-2509 <(617)%20714-2509>
10 matches
Mail list logo
