Hey Alex
Can you provide some info on the scheduler paths thing. I don't have/see that
issue. Do you mean cli paths or by cfg? Jira would be nice in any case.
I don't think the dag processor respects cli parameters.
Bolke
Sent from my iPhone
> On 31 Jan 2017, at 15:10, Alex Van Boxel <a...@vanboxel.be> wrote:
>
> It's quite hard to share my complete dags. I don't have this locally, but I
> have it in my production environment where I use Celery. I rolled back to
> beta 4 to make it work again.
>
> Also @bolke the scheduler logs don't respect the log path.
>
> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davy...@airbnb.com.invalid>
> wrote:
>
>> @Alex
>> I'm not able to reproduce locally (assuming the two python files are in the
>> same folder or is on your PYTHONPATH). I don't see that import error
>> anyways.
>>
>> Just in case, what is your complete DAG definition? Is anyone else able to
>> repro?
>>
>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <a...@vanboxel.be> wrote:
>>>
>>> Well this means none of my DAG's work anymore:
>>>
>>> you just can do this anymore:
>>>
>>> file bqschema.py with
>>>
>>> def marketing_segment():
>>> return [
>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
>>> {"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
>>> {"name": "segment_main", "type": "string", "mode": "nullable"},
>>> {"name": "segment_sub", "type": "integer", "mode": "nullable"},
>>>
>>>
>>> In marketing_segmentation.py:
>>>
>>>
>>> import bqschema
>>>
>>> Gives an error:
>>>
>>> Traceback (most recent call last):
>>> File
>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
>>> apache.incubating-py2.7.egg/airflow/models.py",
>>> line 264, in process_file
>>> m = imp.load_source(mod_name, filepath)
>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
>>> <module>
>>> import bqschema
>>> ImportError: No module named bqschema
>>>
>>> *I don't think this is incorrect?!*
>>>
>>>
>>>
>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davy...@airbnb.com.
>>> invalid>
>>> wrote:
>>>
>>>> The latest commit fixed a regression since 1.7 that files with parsing
>>>> errors no longer showed up on the UI.
>>>>
>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <a...@vanboxel.be>
>>> wrote:
>>>>
>>>>> Just installed beta 5 on our dev environment it lighted up as a
>>> christmas
>>>>> tree. I got a a screen full of import errors. I see that the latest
>>>> commit
>>>>> did something with import errors... is it coorect?!
>>>>>
>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbr...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> Hey Boris
>>>>>>
>>>>>> The scheduler is a bit more aggressive and can use multiple
>>> processors,
>>>>> so
>>>>>> higher CPU usage is actually a good thing.
>>>>>>
>>>>>> I case it is really out of hand look at the new scheduler options
>> and
>>>>>> heartbeat options (see PR for updating.md not in the beta yet).
>>>>>>
>>>>>> Bolke
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
>>>> wrote:
>>>>>>>
>>>>>>> I am not sure if it is my config or something, but looks like
>> after
>>>> the
>>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
>> The
>>>>>> reason
>>>>>>> is two new examples that start running right away - latest only
>> and
>>>>>> latest
>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
>>>> because
>>>>>> now
>>>>>>> dags are not paused by default like it was before?
>>>>>>>
>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
>>>> someone
>>>>>>> needs a step by step instruction, make sure to follow all steps
>>>>> precisely
>>>>>>> here for in-place upgrade or you will have heck of the time (like
>>>> me).
>>>>>>>
>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
>>>>> upgrade-procedure-inplace
>>>>>>>
>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
>> 5.6
>>> -
>>>>> for
>>>>>>> 5.7 you have to use MySql community repo.
>>>>>>>
>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
>>> bdbr...@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
>>> public
>>>>>> keys
>>>>>>>> are available at https://dist.apache.org/repos/
>>>>> dist/release/incubator/
>>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
>>>>> airflow/
>>>>>>>
>>>>>>>> . It is tagged with a local version “apache.incubating” so it
>>> allows
>>>>>>>> upgrading from earlier releases.
>>>>>>>>
>>>>>>>> Issues fixed:
>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
>>>>>>>> * Scheduler would terminate immediately if no dag files present
>>>>>>>>
>>>>>>>> ** As this touches the scheduler logic I though it warranted
>>> another
>>>>>> beta.
>>>>>>>>
>>>>>>>> This should be the last beta in my opinion and we can prepare
>>>>> changelog,
>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Bolke
>>>>>>
>>>>> --
>>>>> _/
>>>>> _/ Alex Van Boxel
>>>>>
>>>>
>>> --
>>> _/
>>> _/ Alex Van Boxel
>>>
>>
> --
> _/
> _/ Alex Van Boxel
