I'll try to identify the core problem On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bdbr...@gmail.com> wrote:
> Hey Alex > > Can you provide some info on the scheduler paths thing. I don't have/see > that issue. Do you mean cli paths or by cfg? Jira would be nice in any case. > > I don't think the dag processor respects cli parameters. > > Bolke > > Sent from my iPhone > > > On 31 Jan 2017, at 15:10, Alex Van Boxel <a...@vanboxel.be> wrote: > > > > It's quite hard to share my complete dags. I don't have this locally, > but I > > have it in my production environment where I use Celery. I rolled back to > > beta 4 to make it work again. > > > > Also @bolke the scheduler logs don't respect the log path. > > > > On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davy...@airbnb.com > .invalid> > > wrote: > > > >> @Alex > >> I'm not able to reproduce locally (assuming the two python files are in > the > >> same folder or is on your PYTHONPATH). I don't see that import error > >> anyways. > >> > >> Just in case, what is your complete DAG definition? Is anyone else able > to > >> repro? > >> > >>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <a...@vanboxel.be> > wrote: > >>> > >>> Well this means none of my DAG's work anymore: > >>> > >>> you just can do this anymore: > >>> > >>> file bqschema.py with > >>> > >>> def marketing_segment(): > >>> return [ > >>> {"name": "user_id", "type": "integer", "mode": "nullable"}, > >>> {"name": "bucket_date", "type": "timestamp", "mode": > "nullable"}, > >>> {"name": "segment_main", "type": "string", "mode": "nullable"}, > >>> {"name": "segment_sub", "type": "integer", "mode": "nullable"}, > >>> > >>> > >>> In marketing_segmentation.py: > >>> > >>> > >>> import bqschema > >>> > >>> Gives an error: > >>> > >>> Traceback (most recent call last): > >>> File > >>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+ > >>> apache.incubating-py2.7.egg/airflow/models.py", > >>> line 264, in process_file > >>> m = imp.load_source(mod_name, filepath) > >>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in > >>> <module> > >>> import bqschema > >>> ImportError: No module named bqschema > >>> > >>> *I don't think this is incorrect?!* > >>> > >>> > >>> > >>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davy...@airbnb.com. > >>> invalid> > >>> wrote: > >>> > >>>> The latest commit fixed a regression since 1.7 that files with parsing > >>>> errors no longer showed up on the UI. > >>>> > >>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <a...@vanboxel.be> > >>> wrote: > >>>> > >>>>> Just installed beta 5 on our dev environment it lighted up as a > >>> christmas > >>>>> tree. I got a a screen full of import errors. I see that the latest > >>>> commit > >>>>> did something with import errors... is it coorect?! > >>>>> > >>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbr...@gmail.com> > >>>> wrote: > >>>>> > >>>>>> Hey Boris > >>>>>> > >>>>>> The scheduler is a bit more aggressive and can use multiple > >>> processors, > >>>>> so > >>>>>> higher CPU usage is actually a good thing. > >>>>>> > >>>>>> I case it is really out of hand look at the new scheduler options > >> and > >>>>>> heartbeat options (see PR for updating.md not in the beta yet). > >>>>>> > >>>>>> Bolke > >>>>>> > >>>>>> Sent from my iPhone > >>>>>> > >>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com> > >>>> wrote: > >>>>>>> > >>>>>>> I am not sure if it is my config or something, but looks like > >> after > >>>> the > >>>>>>> upgrade and start of scheduler, airflow would totally hose CPU. > >> The > >>>>>> reason > >>>>>>> is two new examples that start running right away - latest only > >> and > >>>>>> latest > >>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this > >>>> because > >>>>>> now > >>>>>>> dags are not paused by default like it was before? > >>>>>>> > >>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if > >>>> someone > >>>>>>> needs a step by step instruction, make sure to follow all steps > >>>>> precisely > >>>>>>> here for in-place upgrade or you will have heck of the time (like > >>>> me). > >>>>>>> > >>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html# > >>>>> upgrade-procedure-inplace > >>>>>>> > >>>>>>> BTW official Oracle repository for Oracle Linux only has MySql > >> 5.6 > >>> - > >>>>> for > >>>>>>> 5.7 you have to use MySql community repo. > >>>>>>> > >>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin < > >>> bdbr...@gmail.com > >>>>> > >>>>>> wrote: > >>>>>>>> > >>>>>>>> Hi All, > >>>>>>>> > >>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at: > >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ < > >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , > >>> public > >>>>>> keys > >>>>>>>> are available at https://dist.apache.org/repos/ > >>>>> dist/release/incubator/ > >>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/ > >>>>> airflow/ > >>>>>>> > >>>>>>>> . It is tagged with a local version “apache.incubating” so it > >>> allows > >>>>>>>> upgrading from earlier releases. > >>>>>>>> > >>>>>>>> Issues fixed: > >>>>>>>> * Parsing errors not showing up in UI fixing a regression** > >>>>>>>> * Scheduler would terminate immediately if no dag files present > >>>>>>>> > >>>>>>>> ** As this touches the scheduler logic I though it warranted > >>> another > >>>>>> beta. > >>>>>>>> > >>>>>>>> This should be the last beta in my opinion and we can prepare > >>>>> changelog, > >>>>>>>> upgrade notes and release notes for the RC (Feb 2). > >>>>>>>> > >>>>>>>> Cheers > >>>>>>>> Bolke > >>>>>> > >>>>> -- > >>>>> _/ > >>>>> _/ Alex Van Boxel > >>>>> > >>>> > >>> -- > >>> _/ > >>> _/ Alex Van Boxel > >>> > >> > > -- > > _/ > > _/ Alex Van Boxel > -- _/ _/ Alex Van Boxel