Please note I will be holding off on the RC, before we understand this issue
better.
Bolke
> On 31 Jan 2017, at 18:06, Bolke de Bruin <bdbr...@gmail.com> wrote:
>
> Hey Alex,
>
> Could it actually be that Airflow is doing the right thing? Earlier it was
> swallowing the errors. Where do you see the errors? In the scheduler logs,
> UI, processor logs?
>
> - Bolke
>
>> On 31 Jan 2017, at 16:25, Alex Van Boxel <a...@vanboxel.be> wrote:
>>
>> I'll try to identify the core problem
>>
>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bdbr...@gmail.com> wrote:
>>
>>> Hey Alex
>>>
>>> Can you provide some info on the scheduler paths thing. I don't have/see
>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in any case.
>>>
>>> I don't think the dag processor respects cli parameters.
>>>
>>> Bolke
>>>
>>> Sent from my iPhone
>>>
>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <a...@vanboxel.be> wrote:
>>>>
>>>> It's quite hard to share my complete dags. I don't have this locally,
>>> but I
>>>> have it in my production environment where I use Celery. I rolled back to
>>>> beta 4 to make it work again.
>>>>
>>>> Also @bolke the scheduler logs don't respect the log path.
>>>>
>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davy...@airbnb.com
>>> .invalid>
>>>> wrote:
>>>>
>>>>> @Alex
>>>>> I'm not able to reproduce locally (assuming the two python files are in
>>> the
>>>>> same folder or is on your PYTHONPATH). I don't see that import error
>>>>> anyways.
>>>>>
>>>>> Just in case, what is your complete DAG definition? Is anyone else able
>>> to
>>>>> repro?
>>>>>
>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <a...@vanboxel.be>
>>> wrote:
>>>>>>
>>>>>> Well this means none of my DAG's work anymore:
>>>>>>
>>>>>> you just can do this anymore:
>>>>>>
>>>>>> file bqschema.py with
>>>>>>
>>>>>> def marketing_segment():
>>>>>> return [
>>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
>>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
>>> "nullable"},
>>>>>> {"name": "segment_main", "type": "string", "mode": "nullable"},
>>>>>> {"name": "segment_sub", "type": "integer", "mode": "nullable"},
>>>>>>
>>>>>>
>>>>>> In marketing_segmentation.py:
>>>>>>
>>>>>>
>>>>>> import bqschema
>>>>>>
>>>>>> Gives an error:
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>> File
>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
>>>>>> apache.incubating-py2.7.egg/airflow/models.py",
>>>>>> line 264, in process_file
>>>>>> m = imp.load_source(mod_name, filepath)
>>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
>>>>>> <module>
>>>>>> import bqschema
>>>>>> ImportError: No module named bqschema
>>>>>>
>>>>>> *I don't think this is incorrect?!*
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davy...@airbnb.com.
>>>>>> invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> The latest commit fixed a regression since 1.7 that files with parsing
>>>>>>> errors no longer showed up on the UI.
>>>>>>>
>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <a...@vanboxel.be>
>>>>>> wrote:
>>>>>>>
>>>>>>>> Just installed beta 5 on our dev environment it lighted up as a
>>>>>> christmas
>>>>>>>> tree. I got a a screen full of import errors. I see that the latest
>>>>>>> commit
>>>>>>>> did something with import errors... is it coorect?!
>>>>>>>>
>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbr...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hey Boris
>>>>>>>>>
>>>>>>>>> The scheduler is a bit more aggressive and can use multiple
>>>>>> processors,
>>>>>>>> so
>>>>>>>>> higher CPU usage is actually a good thing.
>>>>>>>>>
>>>>>>>>> I case it is really out of hand look at the new scheduler options
>>>>> and
>>>>>>>>> heartbeat options (see PR for updating.md not in the beta yet).
>>>>>>>>>
>>>>>>>>> Bolke
>>>>>>>>>
>>>>>>>>> Sent from my iPhone
>>>>>>>>>
>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I am not sure if it is my config or something, but looks like
>>>>> after
>>>>>>> the
>>>>>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
>>>>> The
>>>>>>>>> reason
>>>>>>>>>> is two new examples that start running right away - latest only
>>>>> and
>>>>>>>>> latest
>>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
>>>>>>> because
>>>>>>>>> now
>>>>>>>>>> dags are not paused by default like it was before?
>>>>>>>>>>
>>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
>>>>>>> someone
>>>>>>>>>> needs a step by step instruction, make sure to follow all steps
>>>>>>>> precisely
>>>>>>>>>> here for in-place upgrade or you will have heck of the time (like
>>>>>>> me).
>>>>>>>>>>
>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
>>>>>>>> upgrade-procedure-inplace
>>>>>>>>>>
>>>>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
>>>>> 5.6
>>>>>> -
>>>>>>>> for
>>>>>>>>>> 5.7 you have to use MySql community repo.
>>>>>>>>>>
>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
>>>>>> bdbr...@gmail.com
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
>>>>>> public
>>>>>>>>> keys
>>>>>>>>>>> are available at https://dist.apache.org/repos/
>>>>>>>> dist/release/incubator/
>>>>>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
>>>>>>>> airflow/
>>>>>>>>>>
>>>>>>>>>>> . It is tagged with a local version “apache.incubating” so it
>>>>>> allows
>>>>>>>>>>> upgrading from earlier releases.
>>>>>>>>>>>
>>>>>>>>>>> Issues fixed:
>>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
>>>>>>>>>>> * Scheduler would terminate immediately if no dag files present
>>>>>>>>>>>
>>>>>>>>>>> ** As this touches the scheduler logic I though it warranted
>>>>>> another
>>>>>>>>> beta.
>>>>>>>>>>>
>>>>>>>>>>> This should be the last beta in my opinion and we can prepare
>>>>>>>> changelog,
>>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>> Bolke
>>>>>>>>>
>>>>>>>> --
>>>>>>>> _/
>>>>>>>> _/ Alex Van Boxel
>>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> _/
>>>>>> _/ Alex Van Boxel
>>>>>>
>>>>>
>>>> --
>>>> _/
>>>> _/ Alex Van Boxel
>>>
>> --
>> _/
>> _/ Alex Van Boxel
>
