Hey Alex Can you provide some info on the scheduler paths thing. I don't have/see that issue. Do you mean cli paths or by cfg? Jira would be nice in any case.
I don't think the dag processor respects cli parameters. Bolke Sent from my iPhone > On 31 Jan 2017, at 15:10, Alex Van Boxel <a...@vanboxel.be> wrote: > > It's quite hard to share my complete dags. I don't have this locally, but I > have it in my production environment where I use Celery. I rolled back to > beta 4 to make it work again. > > Also @bolke the scheduler logs don't respect the log path. > > On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davy...@airbnb.com.invalid> > wrote: > >> @Alex >> I'm not able to reproduce locally (assuming the two python files are in the >> same folder or is on your PYTHONPATH). I don't see that import error >> anyways. >> >> Just in case, what is your complete DAG definition? Is anyone else able to >> repro? >> >>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <a...@vanboxel.be> wrote: >>> >>> Well this means none of my DAG's work anymore: >>> >>> you just can do this anymore: >>> >>> file bqschema.py with >>> >>> def marketing_segment(): >>> return [ >>> {"name": "user_id", "type": "integer", "mode": "nullable"}, >>> {"name": "bucket_date", "type": "timestamp", "mode": "nullable"}, >>> {"name": "segment_main", "type": "string", "mode": "nullable"}, >>> {"name": "segment_sub", "type": "integer", "mode": "nullable"}, >>> >>> >>> In marketing_segmentation.py: >>> >>> >>> import bqschema >>> >>> Gives an error: >>> >>> Traceback (most recent call last): >>> File >>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+ >>> apache.incubating-py2.7.egg/airflow/models.py", >>> line 264, in process_file >>> m = imp.load_source(mod_name, filepath) >>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in >>> <module> >>> import bqschema >>> ImportError: No module named bqschema >>> >>> *I don't think this is incorrect?!* >>> >>> >>> >>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davy...@airbnb.com. >>> invalid> >>> wrote: >>> >>>> The latest commit fixed a regression since 1.7 that files with parsing >>>> errors no longer showed up on the UI. >>>> >>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <a...@vanboxel.be> >>> wrote: >>>> >>>>> Just installed beta 5 on our dev environment it lighted up as a >>> christmas >>>>> tree. I got a a screen full of import errors. I see that the latest >>>> commit >>>>> did something with import errors... is it coorect?! >>>>> >>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbr...@gmail.com> >>>> wrote: >>>>> >>>>>> Hey Boris >>>>>> >>>>>> The scheduler is a bit more aggressive and can use multiple >>> processors, >>>>> so >>>>>> higher CPU usage is actually a good thing. >>>>>> >>>>>> I case it is really out of hand look at the new scheduler options >> and >>>>>> heartbeat options (see PR for updating.md not in the beta yet). >>>>>> >>>>>> Bolke >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com> >>>> wrote: >>>>>>> >>>>>>> I am not sure if it is my config or something, but looks like >> after >>>> the >>>>>>> upgrade and start of scheduler, airflow would totally hose CPU. >> The >>>>>> reason >>>>>>> is two new examples that start running right away - latest only >> and >>>>>> latest >>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this >>>> because >>>>>> now >>>>>>> dags are not paused by default like it was before? >>>>>>> >>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if >>>> someone >>>>>>> needs a step by step instruction, make sure to follow all steps >>>>> precisely >>>>>>> here for in-place upgrade or you will have heck of the time (like >>>> me). >>>>>>> >>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html# >>>>> upgrade-procedure-inplace >>>>>>> >>>>>>> BTW official Oracle repository for Oracle Linux only has MySql >> 5.6 >>> - >>>>> for >>>>>>> 5.7 you have to use MySql community repo. >>>>>>> >>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin < >>> bdbr...@gmail.com >>>>> >>>>>> wrote: >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at: >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ < >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , >>> public >>>>>> keys >>>>>>>> are available at https://dist.apache.org/repos/ >>>>> dist/release/incubator/ >>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/ >>>>> airflow/ >>>>>>> >>>>>>>> . It is tagged with a local version “apache.incubating” so it >>> allows >>>>>>>> upgrading from earlier releases. >>>>>>>> >>>>>>>> Issues fixed: >>>>>>>> * Parsing errors not showing up in UI fixing a regression** >>>>>>>> * Scheduler would terminate immediately if no dag files present >>>>>>>> >>>>>>>> ** As this touches the scheduler logic I though it warranted >>> another >>>>>> beta. >>>>>>>> >>>>>>>> This should be the last beta in my opinion and we can prepare >>>>> changelog, >>>>>>>> upgrade notes and release notes for the RC (Feb 2). >>>>>>>> >>>>>>>> Cheers >>>>>>>> Bolke >>>>>> >>>>> -- >>>>> _/ >>>>> _/ Alex Van Boxel >>>>> >>>> >>> -- >>> _/ >>> _/ Alex Van Boxel >>> >> > -- > _/ > _/ Alex Van Boxel