It's quite hard to share my complete dags. I don't have this locally, but I have it in my production environment where I use Celery. I rolled back to beta 4 to make it work again.
Also @bolke the scheduler logs don't respect the log path. On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davy...@airbnb.com.invalid> wrote: > @Alex > I'm not able to reproduce locally (assuming the two python files are in the > same folder or is on your PYTHONPATH). I don't see that import error > anyways. > > Just in case, what is your complete DAG definition? Is anyone else able to > repro? > > On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <a...@vanboxel.be> wrote: > > > Well this means none of my DAG's work anymore: > > > > you just can do this anymore: > > > > file bqschema.py with > > > > def marketing_segment(): > > return [ > > {"name": "user_id", "type": "integer", "mode": "nullable"}, > > {"name": "bucket_date", "type": "timestamp", "mode": "nullable"}, > > {"name": "segment_main", "type": "string", "mode": "nullable"}, > > {"name": "segment_sub", "type": "integer", "mode": "nullable"}, > > > > > > In marketing_segmentation.py: > > > > > > import bqschema > > > > Gives an error: > > > > Traceback (most recent call last): > > File > > "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+ > > apache.incubating-py2.7.egg/airflow/models.py", > > line 264, in process_file > > m = imp.load_source(mod_name, filepath) > > File "/home/airflow/dags/marketing_segmentation.py", line 17, in > > <module> > > import bqschema > > ImportError: No module named bqschema > > > > *I don't think this is incorrect?!* > > > > > > > > On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davy...@airbnb.com. > > invalid> > > wrote: > > > > > The latest commit fixed a regression since 1.7 that files with parsing > > > errors no longer showed up on the UI. > > > > > > On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <a...@vanboxel.be> > > wrote: > > > > > > > Just installed beta 5 on our dev environment it lighted up as a > > christmas > > > > tree. I got a a screen full of import errors. I see that the latest > > > commit > > > > did something with import errors... is it coorect?! > > > > > > > > On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbr...@gmail.com> > > > wrote: > > > > > > > > > Hey Boris > > > > > > > > > > The scheduler is a bit more aggressive and can use multiple > > processors, > > > > so > > > > > higher CPU usage is actually a good thing. > > > > > > > > > > I case it is really out of hand look at the new scheduler options > and > > > > > heartbeat options (see PR for updating.md not in the beta yet). > > > > > > > > > > Bolke > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com> > > > wrote: > > > > > > > > > > > > I am not sure if it is my config or something, but looks like > after > > > the > > > > > > upgrade and start of scheduler, airflow would totally hose CPU. > The > > > > > reason > > > > > > is two new examples that start running right away - latest only > and > > > > > latest > > > > > > with trigger. Once I pause them, CPU goes back to idle. Is this > > > because > > > > > now > > > > > > dags are not paused by default like it was before? > > > > > > > > > > > > As I mentioned before, I also had to upgrade mysql to 5.7 - if > > > someone > > > > > > needs a step by step instruction, make sure to follow all steps > > > > precisely > > > > > > here for in-place upgrade or you will have heck of the time (like > > > me). > > > > > > > > > > > https://dev.mysql.com/doc/refman/5.7/en/upgrading.html# > > > > upgrade-procedure-inplace > > > > > > > > > > > > BTW official Oracle repository for Oracle Linux only has MySql > 5.6 > > - > > > > for > > > > > > 5.7 you have to use MySql community repo. > > > > > > > > > > > >> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin < > > bdbr...@gmail.com > > > > > > > > > wrote: > > > > > >> > > > > > >> Hi All, > > > > > >> > > > > > >> I have made the FIFTH beta of Airflow 1.8.0 available at: > > > > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ < > > > > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , > > public > > > > > keys > > > > > >> are available at https://dist.apache.org/repos/ > > > > dist/release/incubator/ > > > > > >> airflow/ <https://dist.apache.org/repos/dist/release/incubator/ > > > > airflow/ > > > > > > > > > > > >> . It is tagged with a local version “apache.incubating” so it > > allows > > > > > >> upgrading from earlier releases. > > > > > >> > > > > > >> Issues fixed: > > > > > >> * Parsing errors not showing up in UI fixing a regression** > > > > > >> * Scheduler would terminate immediately if no dag files present > > > > > >> > > > > > >> ** As this touches the scheduler logic I though it warranted > > another > > > > > beta. > > > > > >> > > > > > >> This should be the last beta in my opinion and we can prepare > > > > changelog, > > > > > >> upgrade notes and release notes for the RC (Feb 2). > > > > > >> > > > > > >> Cheers > > > > > >> Bolke > > > > > > > > > -- > > > > _/ > > > > _/ Alex Van Boxel > > > > > > > > > -- > > _/ > > _/ Alex Van Boxel > > > -- _/ _/ Alex Van Boxel