This is a good case for using conda environments or virtualenv environments.
With conda you would create a new conda environment that uses python 3.6,
install your dependencies including airflow in that environment, then use
the airflow version in that environment.
To figure out the versions of
A task is assigned to a pool by the task specifying the name of the pool.
The docs suggest that the pool argument is a string, not a list of strings.
https://airflow.apache.org/code.html#baseoperator
And looking at the code it does seem like this relationship of one task
assigned to zero or one
Hi Vivian,
What happens when you run the command in the crontab from the perspective
of the cron user? Are you able to run the cron job under the account of
the ec2-user?
I've used systemd on an ec2 where airflow is run as the user 'airflow'. I
needed to provide an EnvironmentFile in order for
tes
> > the next based off interval; this is also why update to start date have
> no
> > affect (doesn't try to fill gaps)
> >
> > On Mon, Apr 2, 2018, 11:26 AM Dennis O'Brien <den...@dennisobrien.net>
> > wrote:
> >
> > > Hi folks,
> >
Hi folks,
I recently asked this question on gitter but didn't get any feedback.
Anyone know if there is a way to get the scheduler to reverse the order of
the dag runs? By default a new DAG starts at start_date then moves
sequentially forward in time until it is caught up (assuming
Hi Kalyan,
We use conda for managing the python environment. Everything including
airflow is installed in this conda environment (via conda when available,
or pip otherwise). Then the path to that conda env is added to the PATH
environment variable. The same approach should work if you are
On Mon, Feb 5, 2018 at 7:17 AM Dennis O'Brien <den...@dennisobrien.net>
> wrote:
>
> > Thanks for the input! I'll take a look at using queues for this.
> >
> > thanks,
> > Dennis
> >
> > On Tue, Jan 30, 2018 at 4:17 PM Hbw <br...@heisenbergwo
g
> the same af core.
>
> B
>
> Sent from a device with less than stellar autocorrect
>
> > On Jan 30, 2018, at 9:13 AM, Dennis O'Brien <den...@dennisobrien.net>
> wrote:
> >
> > Hi All,
> >
> > I have a number of jobs that use scikit-learn
Hi All,
I have a number of jobs that use scikit-learn for scoring players.
Occasionally I need to upgrade scikit-learn to take advantage of some new
features. We have a single conda environment that specifies all the
dependencies for Airflow as well as for all of our DAGs. So currently
Hi Luke,
I've been running Airflow on Python 3.5 with Celery. I'm using Redis as
the message broker (and ElastiCache Redis in production). I haven't tried
RabbitMQ so I can't speak to its compatibility with Python 3.5.
On Mon, Nov 28, 2016 at 9:18 AM Maycock, Luke <
rflow.contrib.operators.vertica_operator import VerticaOperator
I think my problem is already covered in those tickets, but if you think it
is separate and deserves an issue, I'll file one.
thanks,
Dennis
On Thu, Jun 23, 2016 at 6:17 PM Dennis O'Brien <den...@dennisobrien.net>
wrote:
in
> > sys.path . However, the names "lib" and "db_connect" are quite generic.
> I'd
> > consider renaming lib (sth. like etl_lib) and adding just etl/ to
> sys.path
> > , and an __init__.py to the lib folder to avoid namespace pollution.
> You'd
> >
Hi folks
I'm looking for some advice here on how others separate their DAGs and the
code those DAGs call and any PYTHONPATH fixups that may be necessary.
I have a project that looks like this
.
├── airflow
│ ├── dags
│ │ ├── reports
│ │ └── sql
│ └── deploy
│└── templates
├── etl
│ ├── lib
13 matches
Mail list logo
