So bumped to RC1 and this seems fine. I don't get it. So it's a go. Sorry for the noise, but better safe then sorry. I also found that the scheduler logs setting are not equal from the normal logging so that's also a go.
You get green light from me. On Tue, Jan 31, 2017 at 8:32 PM Bolke de Bruin <bdbr...@gmail.com> wrote: > And the scheduler doesn't log anything in beta 4? > > Sent from my iPhone > > > On 31 Jan 2017, at 19:26, Alex Van Boxel <a...@vanboxel.be> wrote: > > > > I see it in the scheduler and in the UI. Currently trying to do a new > > upgrade. > > > >> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin <bdbr...@gmail.com> > wrote: > >> > >> Please note I will be holding off on the RC, before we understand this > >> issue better. > >> > >> Bolke > >> > >>> On 31 Jan 2017, at 18:06, Bolke de Bruin <bdbr...@gmail.com> wrote: > >>> > >>> Hey Alex, > >>> > >>> Could it actually be that Airflow is doing the right thing? Earlier it > >> was swallowing the errors. Where do you see the errors? In the scheduler > >> logs, UI, processor logs? > >>> > >>> - Bolke > >>> > >>>> On 31 Jan 2017, at 16:25, Alex Van Boxel <a...@vanboxel.be> wrote: > >>>> > >>>> I'll try to identify the core problem > >>>> > >>>>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bdbr...@gmail.com> > wrote: > >>>>> > >>>>> Hey Alex > >>>>> > >>>>> Can you provide some info on the scheduler paths thing. I don't > >> have/see > >>>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in > any > >> case. > >>>>> > >>>>> I don't think the dag processor respects cli parameters. > >>>>> > >>>>> Bolke > >>>>> > >>>>> Sent from my iPhone > >>>>> > >>>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <a...@vanboxel.be> wrote: > >>>>>> > >>>>>> It's quite hard to share my complete dags. I don't have this > locally, > >>>>> but I > >>>>>> have it in my production environment where I use Celery. I rolled > >> back to > >>>>>> beta 4 to make it work again. > >>>>>> > >>>>>> Also @bolke the scheduler logs don't respect the log path. > >>>>>> > >>>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davy...@airbnb.com > >>>>> .invalid> > >>>>>> wrote: > >>>>>> > >>>>>>> @Alex > >>>>>>> I'm not able to reproduce locally (assuming the two python files > are > >> in > >>>>> the > >>>>>>> same folder or is on your PYTHONPATH). I don't see that import > error > >>>>>>> anyways. > >>>>>>> > >>>>>>> Just in case, what is your complete DAG definition? Is anyone else > >> able > >>>>> to > >>>>>>> repro? > >>>>>>> > >>>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <a...@vanboxel.be > > > >>>>> wrote: > >>>>>>>> > >>>>>>>> Well this means none of my DAG's work anymore: > >>>>>>>> > >>>>>>>> you just can do this anymore: > >>>>>>>> > >>>>>>>> file bqschema.py with > >>>>>>>> > >>>>>>>> def marketing_segment(): > >>>>>>>> return [ > >>>>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"}, > >>>>>>>> {"name": "bucket_date", "type": "timestamp", "mode": > >>>>> "nullable"}, > >>>>>>>> {"name": "segment_main", "type": "string", "mode": > "nullable"}, > >>>>>>>> {"name": "segment_sub", "type": "integer", "mode": > "nullable"}, > >>>>>>>> > >>>>>>>> > >>>>>>>> In marketing_segmentation.py: > >>>>>>>> > >>>>>>>> > >>>>>>>> import bqschema > >>>>>>>> > >>>>>>>> Gives an error: > >>>>>>>> > >>>>>>>> Traceback (most recent call last): > >>>>>>>> File > >>>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+ > >>>>>>>> apache.incubating-py2.7.egg/airflow/models.py", > >>>>>>>> line 264, in process_file > >>>>>>>> m = imp.load_source(mod_name, filepath) > >>>>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in > >>>>>>>> <module> > >>>>>>>> import bqschema > >>>>>>>> ImportError: No module named bqschema > >>>>>>>> > >>>>>>>> *I don't think this is incorrect?!* > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov < > >> dan.davy...@airbnb.com. > >>>>>>>> invalid> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> The latest commit fixed a regression since 1.7 that files with > >> parsing > >>>>>>>>> errors no longer showed up on the UI. > >>>>>>>>> > >>>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel < > a...@vanboxel.be> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Just installed beta 5 on our dev environment it lighted up as a > >>>>>>>> christmas > >>>>>>>>>> tree. I got a a screen full of import errors. I see that the > >> latest > >>>>>>>>> commit > >>>>>>>>>> did something with import errors... is it coorect?! > >>>>>>>>>> > >>>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin < > bdbr...@gmail.com > >>> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hey Boris > >>>>>>>>>>> > >>>>>>>>>>> The scheduler is a bit more aggressive and can use multiple > >>>>>>>> processors, > >>>>>>>>>> so > >>>>>>>>>>> higher CPU usage is actually a good thing. > >>>>>>>>>>> > >>>>>>>>>>> I case it is really out of hand look at the new scheduler > options > >>>>>>> and > >>>>>>>>>>> heartbeat options (see PR for updating.md not in the beta > yet). > >>>>>>>>>>> > >>>>>>>>>>> Bolke > >>>>>>>>>>> > >>>>>>>>>>> Sent from my iPhone > >>>>>>>>>>> > >>>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com > > > >>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> I am not sure if it is my config or something, but looks like > >>>>>>> after > >>>>>>>>> the > >>>>>>>>>>>> upgrade and start of scheduler, airflow would totally hose > CPU. > >>>>>>> The > >>>>>>>>>>> reason > >>>>>>>>>>>> is two new examples that start running right away - latest > only > >>>>>>> and > >>>>>>>>>>> latest > >>>>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is > this > >>>>>>>>> because > >>>>>>>>>>> now > >>>>>>>>>>>> dags are not paused by default like it was before? > >>>>>>>>>>>> > >>>>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if > >>>>>>>>> someone > >>>>>>>>>>>> needs a step by step instruction, make sure to follow all > steps > >>>>>>>>>> precisely > >>>>>>>>>>>> here for in-place upgrade or you will have heck of the time > >> (like > >>>>>>>>> me). > >>>>>>>>>>>> > >>>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html# > >>>>>>>>>> upgrade-procedure-inplace > >>>>>>>>>>>> > >>>>>>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql > >>>>>>> 5.6 > >>>>>>>> - > >>>>>>>>>> for > >>>>>>>>>>>> 5.7 you have to use MySql community repo. > >>>>>>>>>>>> > >>>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin < > >>>>>>>> bdbr...@gmail.com > >>>>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi All, > >>>>>>>>>>>>> > >>>>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at: > >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ < > >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , > >>>>>>>> public > >>>>>>>>>>> keys > >>>>>>>>>>>>> are available at https://dist.apache.org/repos/ > >>>>>>>>>> dist/release/incubator/ > >>>>>>>>>>>>> airflow/ < > >> https://dist.apache.org/repos/dist/release/incubator/ > >>>>>>>>>> airflow/ > >>>>>>>>>>>> > >>>>>>>>>>>>> . It is tagged with a local version “apache.incubating” so it > >>>>>>>> allows > >>>>>>>>>>>>> upgrading from earlier releases. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Issues fixed: > >>>>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression** > >>>>>>>>>>>>> * Scheduler would terminate immediately if no dag files > present > >>>>>>>>>>>>> > >>>>>>>>>>>>> ** As this touches the scheduler logic I though it warranted > >>>>>>>> another > >>>>>>>>>>> beta. > >>>>>>>>>>>>> > >>>>>>>>>>>>> This should be the last beta in my opinion and we can prepare > >>>>>>>>>> changelog, > >>>>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Cheers > >>>>>>>>>>>>> Bolke > >>>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> _/ > >>>>>>>>>> _/ Alex Van Boxel > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> -- > >>>>>>>> _/ > >>>>>>>> _/ Alex Van Boxel > >>>>>>>> > >>>>>>> > >>>>>> -- > >>>>>> _/ > >>>>>> _/ Alex Van Boxel > >>>>> > >>>> -- > >>>> _/ > >>>> _/ Alex Van Boxel > >>> > >> > >> -- > > _/ > > _/ Alex Van Boxel > -- _/ _/ Alex Van Boxel