Re: Meetup Interest?

+1 at SF meetup Would be interested in seeing that progress of airflow + k8s and any other advancements the community has made. On Sun, Oct 15, 2017 at 9:24 AM Feng Lu wrote: > +1 > > We can give an update on task secret management in K8SExecutor and also > want to

Re: Meetup Interest?

+1 We can give an update on task secret management in K8SExecutor and also want to share our thoughts and get feedback on Airflow CI/CD with the set of GCP operators/hooks as an example. On Sat, Oct 14, 2017 at 7:06 PM, Marc Bollinger wrote: > +1 > > We'd definitely be in.

Re: Return results optionally from spark_sql_hook

Thanks Fokko. Do you know if it is better to use pyspark directly within python operator or invoke submit-job instead? My understanding in both cases airflow uses yarn-client deployment mode, not yarn-cluster and spark driver always runs on the same node with airflow worker. Not sure it is the

Re: Return results optionally from spark_sql_hook

Hi Boris, Instead of writing it to a file, you can also write it to xcom, this will keep everything inside of Airflow. My personal opinion on this; spark-sql is a bit limited by nature, it only support SQL. If you want to do more dynamic stuff, you will eventually have to move to spark-submit