Re: [DISCUSS] Future of Pendulum in Airflow

An option would be to fork Pendulum? It is MIT Licensed, I don't know whether that poses problems to get in ASF? If forking (?) which is somewhat non-ideal, would we want that 'in' airflow? If not 'in' airflow, I wonder if ASF incubator would accept a forked project? [ anecdotally Linux Foundat

Re: [DISCUSS] Maciej Obuchowski for committer?

should this have gone to private@ ? On Wed, Jun 14, 2023 at 1:52 PM Jarek Potiuk wrote: > Dear PMC members, > > How about Maciej for committer? > > He is mostly committing in OpenLineage - of course, but I think with his > experience and skills and the experience he brings from years of > integr

Re: [ANNOUNCE] New committer Niko Oliveira (o-nikolas)

Thanks for all you do, @niko! On Mon, Dec 19, 2022, 2:42 PM Robert Karish wrote: > Congrats Niko! 🍾 > > On Mon, Dec 19, 2022 at 4:41 PM Felix Uellendall > wrote: > >> Congrats! >> >> >> >> Sent from Proton Mail for iOS >> >> >> On Mon, Dec 19, 2022 at 22:23, Jarek Potiuk wrote: >> >> Congrats

Re: [VOTE] August 2022 PR of the Month

[25888] Prefer the local Quick Start in docs and [25610] Grid logs for mapped instances Both are likely to help users immensely, but target different stages of use. On Wed, Aug 31, 2022 at 10:20 AM Brent Bovenzi wrote: > +1 for 25610 > > On Wed, Aug 31, 2022 at 1:05 PM Vikram Koka > wrote: >

Re: Wiki access please?

If much is out, rather than in, is there a different pool from where you will draw contributors and eventually committers/pmc? Sounds somewhat like a question of whether to grow the tent of contributors, committers, pmc of what is deemed to be "Airflow" (capital "A" and in)? Or err towards things

Re: [VOTE] Airflow Providers prepared on June 15, 2022

I could probably help verify some of these things by early/mid-week ( via https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors ), but also wouldn't be casting a binding vote so no need to be heldup on me. On Sun, Jun 19, 2022 at 8:36 AM Jarek

Re: Create official Apache Airflow publication on Medium.com

I started looking into the Medium API. Looks like you could manage, and if wiring the automation scripts solution up auto-publish to medium from one of the airflow repos: https://github.com/Medium/medium-api-docs that would at least make it straightforward how to manage. Ex: github.../repo/mediu

Re: [VOTE] Enable Github Discussions on Apache Airflow Github Repo

+1 just make sure to appropriate self-police, to keep use for intended purposes. On Tue, Sep 22, 2020 at 12:05 PM Deng Xiaodong wrote: > +1. > > I did share some concerns with folks, but I think mostly already clarified > well in the earlier discussion. > > > XD > > On 22 Sep 2020, at 9:00 PM,

Re: Generic Transfer Operator

there's a version > clash > > > > >> >between dependencies? > > > > >> > > > > > >> >2. The initial approach using `DataSource` concept allowed users > to > > > > >> >use it in any operator (not only transfe

Re: Generic Transfer Operator

t; > Best, > Tomek > > > On Wed, Sep 2, 2020 at 12:37 AM Austin Bennett > wrote: > > > > Are there IOs that would be desired for a generic transfer operator that > > don't exist in: https://beam.apache.org/documentation/io/built-in/ <- > > th

Re: Generic Transfer Operator

ystem with a java/python SDK? I think @jarek and Polidea > do a > >> > lot of > >> > > > work with Beam as well so I’d love their thoughts if this a good > >> > use-case. > >> > > > > >> > > > via

Re: Generic Transfer Operator

Are you guys familiar with Beam ? Esp. if not doing transforms, it might rather straightforward to rely on the ecosystem of connectors in that Apache Project to use as the foundations for a generic transfer operator. On Tue, Sep 1, 2020 at 11:05 AM Jarek Potiuk wrote: >

Re: [DISCUSS] Airflow Website Blog Guidelines

Depends on your aims. @Kaxil "... for it to be meaningful to serve any purpose." is quite loaded, and *any* seems a bit exaggerated. There is value in being the go-to content aggregator, and therefore initials stop, even if whole of content lives elsewhere. That is different from trying to keep

Re: [Discuss] Airflow Summits 2020

As an airflow user, and based in Bay Area (proposed first location), with experience organizing events -- happy to contribute. On Tue, Nov 12, 2019 at 12:31 PM Jarek Potiuk wrote: > I am looking forward to it and happy to help with the organization. I am > sure both will be great community event

Airflow @ApacheCon?

ApacheCon: https://www.apachecon.com/acna19/ is coming up next week. Looking at the schedule, I don't see anything on Airflow... Am I not finding, or is nobody talking/going from Airflow?

Re: Working on a new Airflow website!

Hi Aizhamal and others working on this effort, Please keep in mind tagging content appropriately to facilitate ease of translations/multi-language site. See conversation: https://www.mail-archive.com/dev@airflow.apache.org/msg01727.html If multi-language isn't on scope of this redesign, it'd be

Re: [DISCUSS] Apache Airflow website requirements

6, 2019 at 6:29 PM Austin Bennett < > whatwouldausti...@gmail.com> > wrote: > > > Seems a suitable and reasonable approach, esp given current > circumstances. > > > > I would imagine the desire would be to then update anything in other > > languages eac

Re: [DISCUSS] Apache Airflow website requirements

Seems a suitable and reasonable approach, esp given current circumstances. I would imagine the desire would be to then update anything in other languages each time English updated? Or on a daily basis? Ongoing would absolutely be important - in a predictable manner - rather than just running thi

Re: Operator for syncing s3 buckets (and recursively copying)?

open a Jira for that and then submit your PR. > There is a similar request for GCP: > https://issues.apache.org/jira/browse/AIRFLOW-2842 > > > Sent with ProtonMail Secure Email. > > ‐‐‐ Original Message ‐‐‐ > On Thursday, May 23, 2019 6:32 AM, Austin Bennett < >

Re: Proposal: Restructure slack

Slack has a search capability, which I've used to find "old' issues. I did loose track of whether airflow's slack maintains history persistently. If it doesn't do I remember that the ASFs overall does (which airflow could benefit from/opt-into, if desired)? On Sat, May 25, 2019 at 3:16 AM Jarek

Re: [DISCUSS] Time for a user@ list?

+1, these are useful separations. On Fri, May 24, 2019 at 10:25 PM Prajwal Tuladhar wrote: > Makes sense. > > +1 for this. > > On Fri, May 24, 2019 at 10:06 PM Jason Rich .invalid> > wrote: > > > +1 > > > > Quick question, could we also push user@ threads to a specified Slack > > channel? > > >

Re: Operator for syncing s3 buckets (and recursively copying)?

Hi All, I need s3 sync functionality, and currently use it via bash. Would love for an operator to exist. I see nothing in Jira. If not here, it looks like that would be the place to add a ticket, get any feedback and then build? Thanks, Austin On Sat, May 4, 2019 at 6:23 PM Austin Bennett

Operator for syncing s3 buckets (and recursively copying)?

Hi Airflow Devs, I am not having any luck finding some S3 functionality that I would like to use. Specifically the ability to sync buckets. And, the ability to recursively copy. `aws s3 sync s3://source-bucket/ s3://destination-bucket` or `aws s3 cp s3://source-bucket/folder/ s3://destination

Re: [2.0 spring cleaning] Deprecate contrib folder?

It seems naming of contrib is used in practice with what I thought it would be (as a novice), but incubator not bad either -- yes, it gets messy if anything can get contributed. If using for that purpose, maybe there is some sort of timeframe for graduation or it winds up being removed? Indeed, c

Re: Hangout: Learn how to hack on Airflow (CHANGE OF DAY)

Would be great if recorded/available later -- have a meeting during the 'live' time. On Thu, Mar 28, 2019 at 9:09 AM Gabriele Di Bernardo wrote: > Hi Ash, > > Are you going to record the session in the end? > > Thank you so much! > > Best, > > > Gabriele > > > On 28 Mar 2019, at 11:35, Ash Berli

Re: [DISCUSS] Moving to the ASF slack?

Hi Kaxil, Not suggesting that a migration to ASF slack is the right move (don't have a strong opinion). I disagree with your premise that separation of channels per topic is impossible in the ASF Seems could have channels in the ASF Slack : #Airflow-announce #Airflow-users-bay-area #Airflow-gith

Re: IAM Permissions for AWS / S3_to_Redshift_Operator

Harmon > wrote: > > > I think what @ash is referring to is if you have an IAM role associated > > with an EC2 instance, and your AWS connection in Airflow is left blank, > > Boto3 will default tonthst role for any calls made by Boto3. However, in > > this instance,

Re: IAM Permissions for AWS / S3_to_Redshift_Operator

of the > > connection. Works well so far. I’m not sure if that helps or if you’re > > looking for an out of the box solution, but I’d be happy to share my code > > with you. > > > > Andrew > > > > On Wed, Jan 30, 2019 at 5:57 PM Austin Bennett < > &g

Re: IAM Permissions for AWS / S3_to_Redshift_Operator

, was trying to get ahead of things. I can start with an empty connection type and reading up on instance profiles. Thank you for the input/feedback! On Wed, Jan 30, 2019 at 1:56 PM Ash Berlin-Taylor wrote: > If you create an "empty" connection of type "AWS" (i.e. don't

IAM Permissions for AWS / S3_to_Redshift_Operator

Have started to push our group to standardizing on airflow. We still have a few large Redshift clusters. The s3_to_redshift_operator.py only appears to be written to authenticate via secret/access-keys. We no longer use Key Based authentication and rely upon Role Based, therefore IAM groups. Wh