Hey Fokko!
Sounds like a great event! Will any of the talks/workshops be
streamed/livecast/recorded for those of us who can't make it to Amsterdam?
- Ben
On Fri, Oct 12, 2018 at 12:40 PM Driesprong, Fokko
wrote:
> Hi all,
>
> From October 15-19, 2018, GoDataFest takes place in Amsterdam, The
>
Hi all,
>From October 15-19, 2018, GoDataFest takes place in Amsterdam, The
Netherlands. This week is dedicated to data technology and features free
talks, training sessions and workshops.
Leading tech companies, like AWS (Monday, October 15), Dataiku (Tuesday,
October 16), Databricks (Wednesday,
For something to add to Airflow itself: I would love a more flexible
mapping between data time and processing time. The default is "n-1" (day
over day, you're aiming to process yesterday's data) but people post other
use cases on this mailing list quite frequently.
On Fri, Oct 12, 2018 at 7:46 AM
What about an exponential back off on the poke interval?
On Fri, 12 Oct 2018, 13:01 Ash Berlin-Taylor, wrote:
> That would work for some of our other uses cases (and has been an idea in
> our backlog for months) but not this case as we're reading from someone
> else's bucket so can't set up noti
That would work for some of our other uses cases (and has been an idea in our
backlog for months) but not this case as we're reading from someone else's
bucket so can't set up notifications etc. :(
-ash
> On 12 Oct 2018, at 11:57, Bolke de Bruin wrote:
>
> S3 Bucket notification that triggers
S3 Bucket notification that triggers a dag?
Verstuurd vanaf mijn iPad
> Op 12 okt. 2018 om 12:42 heeft Ash Berlin-Taylor het
> volgende geschreven:
>
> A lot of our dags are ingesting data (usually daily or weekly) from
> suppliers, and they are universally late.
>
> In the case I'm setting
A lot of our dags are ingesting data (usually daily or weekly) from suppliers,
and they are universally late.
In the case I'm setting up now the delivery lag is about 30hours - data for
2018-10-10 turned up at 2018-10-12 05:43.
I was going to just set this up with an S3KeySensor and a daily sch
We're working hard to get rid of the tight Travis integration and moving to
a Docker based setup. I think it should be very easy to get a Docker up and
running which is packed with the required dependencies. Unfortunately we're
not there yet. Also the tox layer feels a bit redundant to me, since we
