Hey Alex,
Could it actually be that Airflow is doing the right thing? Earlier it was
swallowing the errors. Where do you see the errors? In the scheduler logs, UI,
processor logs?
- Bolke
> On 31 Jan 2017, at 16:25, Alex Van Boxel <a...@vanboxel.be> wrote:
>
> I'll try to identify the core problem
>
> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bdbr...@gmail.com> wrote:
>
>> Hey Alex
>>
>> Can you provide some info on the scheduler paths thing. I don't have/see
>> that issue. Do you mean cli paths or by cfg? Jira would be nice in any case.
>>
>> I don't think the dag processor respects cli parameters.
>>
>> Bolke
>>
>> Sent from my iPhone
>>
>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <a...@vanboxel.be> wrote:
>>>
>>> It's quite hard to share my complete dags. I don't have this locally,
>> but I
>>> have it in my production environment where I use Celery. I rolled back to
>>> beta 4 to make it work again.
>>>
>>> Also @bolke the scheduler logs don't respect the log path.
>>>
>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davy...@airbnb.com
>> .invalid>
>>> wrote:
>>>
>>>> @Alex
>>>> I'm not able to reproduce locally (assuming the two python files are in
>> the
>>>> same folder or is on your PYTHONPATH). I don't see that import error
>>>> anyways.
>>>>
>>>> Just in case, what is your complete DAG definition? Is anyone else able
>> to
>>>> repro?
>>>>
>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <a...@vanboxel.be>
>> wrote:
>>>>>
>>>>> Well this means none of my DAG's work anymore:
>>>>>
>>>>> you just can do this anymore:
>>>>>
>>>>> file bqschema.py with
>>>>>
>>>>> def marketing_segment():
>>>>> return [
>>>>> {"name": "user_id", "type": "integer", "mode": "nullable"},
>>>>> {"name": "bucket_date", "type": "timestamp", "mode":
>> "nullable"},
>>>>> {"name": "segment_main", "type": "string", "mode": "nullable"},
>>>>> {"name": "segment_sub", "type": "integer", "mode": "nullable"},
>>>>>
>>>>>
>>>>> In marketing_segmentation.py:
>>>>>
>>>>>
>>>>> import bqschema
>>>>>
>>>>> Gives an error:
>>>>>
>>>>> Traceback (most recent call last):
>>>>> File
>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
>>>>> apache.incubating-py2.7.egg/airflow/models.py",
>>>>> line 264, in process_file
>>>>> m = imp.load_source(mod_name, filepath)
>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
>>>>> <module>
>>>>> import bqschema
>>>>> ImportError: No module named bqschema
>>>>>
>>>>> *I don't think this is incorrect?!*
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davy...@airbnb.com.
>>>>> invalid>
>>>>> wrote:
>>>>>
>>>>>> The latest commit fixed a regression since 1.7 that files with parsing
>>>>>> errors no longer showed up on the UI.
>>>>>>
>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <a...@vanboxel.be>
>>>>> wrote:
>>>>>>
>>>>>>> Just installed beta 5 on our dev environment it lighted up as a
>>>>> christmas
>>>>>>> tree. I got a a screen full of import errors. I see that the latest
>>>>>> commit
>>>>>>> did something with import errors... is it coorect?!
>>>>>>>
>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbr...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey Boris
>>>>>>>>
>>>>>>>> The scheduler is a bit more aggressive and can use multiple
>>>>> processors,
>>>>>>> so
>>>>>>>> higher CPU usage is actually a good thing.
>>>>>>>>
>>>>>>>> I case it is really out of hand look at the new scheduler options
>>>> and
>>>>>>>> heartbeat options (see PR for updating.md not in the beta yet).
>>>>>>>>
>>>>>>>> Bolke
>>>>>>>>
>>>>>>>> Sent from my iPhone
>>>>>>>>
>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I am not sure if it is my config or something, but looks like
>>>> after
>>>>>> the
>>>>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
>>>> The
>>>>>>>> reason
>>>>>>>>> is two new examples that start running right away - latest only
>>>> and
>>>>>>>> latest
>>>>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
>>>>>> because
>>>>>>>> now
>>>>>>>>> dags are not paused by default like it was before?
>>>>>>>>>
>>>>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
>>>>>> someone
>>>>>>>>> needs a step by step instruction, make sure to follow all steps
>>>>>>> precisely
>>>>>>>>> here for in-place upgrade or you will have heck of the time (like
>>>>>> me).
>>>>>>>>>
>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
>>>>>>> upgrade-procedure-inplace
>>>>>>>>>
>>>>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
>>>> 5.6
>>>>> -
>>>>>>> for
>>>>>>>>> 5.7 you have to use MySql community repo.
>>>>>>>>>
>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
>>>>> bdbr...@gmail.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
>>>>> public
>>>>>>>> keys
>>>>>>>>>> are available at https://dist.apache.org/repos/
>>>>>>> dist/release/incubator/
>>>>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
>>>>>>> airflow/
>>>>>>>>>
>>>>>>>>>> . It is tagged with a local version “apache.incubating” so it
>>>>> allows
>>>>>>>>>> upgrading from earlier releases.
>>>>>>>>>>
>>>>>>>>>> Issues fixed:
>>>>>>>>>> * Parsing errors not showing up in UI fixing a regression**
>>>>>>>>>> * Scheduler would terminate immediately if no dag files present
>>>>>>>>>>
>>>>>>>>>> ** As this touches the scheduler logic I though it warranted
>>>>> another
>>>>>>>> beta.
>>>>>>>>>>
>>>>>>>>>> This should be the last beta in my opinion and we can prepare
>>>>>>> changelog,
>>>>>>>>>> upgrade notes and release notes for the RC (Feb 2).
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> Bolke
>>>>>>>>
>>>>>>> --
>>>>>>> _/
>>>>>>> _/ Alex Van Boxel
>>>>>>>
>>>>>>
>>>>> --
>>>>> _/
>>>>> _/ Alex Van Boxel
>>>>>
>>>>
>>> --
>>> _/
>>> _/ Alex Van Boxel
>>
> --
> _/
> _/ Alex Van Boxel
