A suggestion for maintaining stability: as a "test mode" item, write
database triggers for MySQL or Postgres that fail if a database transaction
puts the database in a bogus state.
On Fri, May 13, 2016 at 5:37 PM, siddharth anand wrote:
> I'm not familiar enough with Celery -- refer to my commen
I'm not familiar enough with Celery -- refer to my comment about giving up
after a day of playing with it -- to discount it totally. I'd actually feel
better informed once I got it running and could publish a "take these
steps", which I'm surprised that no one has done.
I'm all for simple, though
Hey Sid,
I question the need for both local and celery executors (leaving sequential
out of this). I think all we need is a scheduler + distributed executor. If
you run only one each, then you have the LocalExecutor. The main thing that
I care about is that this one thing is easy out of the box, a
> Op 13 mei 2016, om 23:06 heeft harish singh het
> volgende geschreven:
>
> we are seeing this in production. I wont be able to update the version
> right now. But I will try to test this out over the weekend.
> But if I consider 1.7.0, am I doing something incorrect? or did something
> change
we are seeing this in production. I wont be able to update the version
right now. But I will try to test this out over the weekend.
But if I consider 1.7.0, am I doing something incorrect? or did something
change in .1.rc6?
One thing I forgot to mention was that - we do run a backfill before we
tu
> Op 13 mei 2016, om 22:51 heeft harish singh het
> volgende geschreven:
>
> Bolke, its 1.7.0
>
>
> On Fri, May 13, 2016 at 1:35 PM, Bolke de Bruin wrote:
>
>>
>>> Op 13 mei 2016, om 22:19 heeft harish singh
>> het volgende geschreven:
>>>
>>> Hi guys,
>>>
>>> I am having an issue with
Bolke, its 1.7.0
On Fri, May 13, 2016 at 1:35 PM, Bolke de Bruin wrote:
>
> > Op 13 mei 2016, om 22:19 heeft harish singh
> het volgende geschreven:
> >
> > Hi guys,
> >
> > I am having an issue with making 'depends_on_past=true' work
> >
> > This my pipeline:
> >
> > a -> b -> c -> d -> e
> >
> Op 13 mei 2016, om 22:19 heeft harish singh het
> volgende geschreven:
>
> Hi guys,
>
> I am having an issue with making 'depends_on_past=true' work
>
> This my pipeline:
>
> a -> b -> c -> d -> e
>
> a -> x -> e
>
> a -> y -> e
>
> I have default args for all Tasks:
>
> scheduling_sta
Hi guys,
I am having an issue with making 'depends_on_past=true' work
This my pipeline:
a -> b -> c -> d -> e
a -> x -> e
a -> y -> e
I have default args for all Tasks:
scheduling_start_date = (datetime.utcnow() -
datetime.timedelta(hours=1)).replace(minute=0, second=0,
microsecond=0)
defau
Bolke, Thanks for providing the document and for generally driving a path
forward.
Regarding Local vs. Celery I think the project benefits greatly from having
multiple executors. Widespread adoption of Airflow involves keeping the
barriers to adoption as low as possible. We ship Airflow with a S
@Maxime, I take your point. I think what I'd prefer is to have one stable,
first class citizen for a way to do distributed execution. I would also
like for that solution to not peg me to RabbigMQ or something wacky like
that--at least initially.
My concerns with Celery as it is currently:
1. It s
+1 to what Sid said.
On Fri, May 13, 2016 at 10:33 AM, Siddharth Anand wrote:
> I mentioned this on the call yesterday as well. Going forward, all
> meetings will be community-inclusive. We can follow what Apache Beam is
> doing ( they have 10-15+ video windows at a time ) in this respect. We wi
A few thoughts on moving away from Celery and around the Executor
interface. To me LocalExecutor means local as "in-process" and it's
implemented as a local multiprocess pool/queue, so making it remote or "out
of process" changes its definition or premise. Let's then refer to what
we're really talk
I mentioned this on the call yesterday as well. Going forward, all meetings
will be community-inclusive. We can follow what Apache Beam is doing ( they
have 10-15+ video windows at a time ) in this respect. We will need a topic and
agenda for each meetings, so that they are not misconstrued as "
On 13 May 2016 at 00:40, Bolke de Bruin wrote:
> The question is how to keep the trust of that first group - they are vital to
> the work - while growing the community.
Another perspective is for the first group to trust the Apache Way.
The procedures, norms, votes, requirements, whole Incubator
Sent from my iPhone
> On 13 mei 2016, at 19:02, Jakob Homan wrote:
>
>> On 13 May 2016 at 00:40, Bolke de Bruin wrote:
>> The question is how to keep the trust of that first group - they are vital
>> to the work - while growing the community.
>
> Another perspective is for the first group t
It was but it wasn't broadly communicated. We will repeat it, with an open
invitation, every week or two weeks.
Now to figure out how to share a video link that works continuously without me
or someone else being there every time...
B.
Sent from my iPhone
> On 13 mei 2016, at 18:55, Jakob
Cool. Was this a public meeting? Will the next one be?
On 13 May 2016 at 08:20, Chris Riccomini wrote:
> Hey Bolke,
>
> Thanks for writing this up. I don't have a ton of feedback, as I'm not
> terribly familiar with the internals of the scheduler, but two notes:
>
> 1. A major +1 for the celery
Hey Bolke,
Thanks for writing this up. I don't have a ton of feedback, as I'm not
terribly familiar with the internals of the scheduler, but two notes:
1. A major +1 for the celery/local executor discussion. IMO, Celery is a
net-negative on this project, and should be fully removed in favor of th
Hi,
We did a video conference on the scheduler with a couple of the committers
yesterday. The meeting was not there to finalize any roadmap but more to get a
general understanding of each other's work. To keep it as transparent as
possible hereby a summary:
Who were attending:
Max, Paul, Arthu
Hi Hitesh et al,
I was trying to look at it from the perspective of a group of people that did
the work (specifically not naming Airbnb here) and have given that work to a
very young community. Suddenly, a lot more people are involved and have started
adjusting. Obviously, that first group gets
21 matches
Mail list logo
