Sounds good for your use, certainly.
I mainly wanted to make sure other people knew before blindly equipping a
foot-canon :)
-ash
> On 29 Sep 2018, at 00:09, Sean Carey <sean.ca...@aurishealth.com> wrote:
>
> Thanks, Ash. I understand what you're saying. The warnings are coming from
> the Qubole operator. We get a lot of this:
>
> PendingDeprecationWarning: Invalid arguments were passed to QuboleOperator.
> Support for passing such arguments will be dropped in Airflow 2.0. Invalid
> arguments were:
> *args: ()
> **kwargs: {...}
> category=PendingDeprecationWarning
>
> We've spoken to Qubole about this and they plan to address it. In the
> meantime, it generates a ton of noise in our logs which we'd like to suppress
> -- we are aware of the issue, and don't need to be told about it every 5
> seconds.
>
> As for it suddenly breaking, being that this is pending Airflow 2.0 I feel
> the risk is low and when we do upgrade it will be thoroughly tested.
>
> Thanks!
>
> Sean
>
>
> On 9/28/18, 5:01 PM, "Ash Berlin-Taylor" <a...@apache.org> wrote:
>
> What deprecation warnings are you getting? Are they from Airflow itself
> (i.e. things Airflow is calling like flask_wtf, etc) or of your use of
> Airflow?
>
> If it is the form could you check and see if someone has already reported
> a Jira issue so we can fix them?
> https://issues.apache.org/jira/issues/?jql=project%3DAIRFLOW
>
> If it is the latter PLEASE DO NOT IGNORE THESE.
>
> Deprecation warnings are how we, the Airflow community tell users that you
> need to make a change to your DAG/code/config to upgrade things. If you
> silence these warnings you will have a much harder time upgrading to new
> versions of Airflow (read: you might suddenly find that things stop working
> because you turned of the warnings.)
>
> -ash
>
>> On 28 Sep 2018, at 22:52, Sean Carey <sean.ca...@aurishealth.com> wrote:
>>
>> Hello,
>>
>> I’ve been looking for a way to suppress the PendingDeprecationWarning
>> messages cluttering our airflow logs and I have a working solution which I
>> thought I would share.
>>
>> In order to do this, you first need to configure airflow for custom logging
>> using steps 1-4 here:
>>
>> https://airflow.readthedocs.io/en/stable/howto/write-logs.html#writing-logs-to-azure-blob-storage
>>
>> (note that although the document is for Azure remote logging you don’t
>> actually need azure for this)
>>
>> Next, modify the log_config.py script created in the step above as follows:
>>
>>
>> 1. Import logging
>> 2. Define the filter class:
>>
>>
>>
>> class DeprecationWarningFilter(logging.Filter):
>>
>> def filter(self, record):
>>
>> allow = 'DeprecationWarning' not in record.msg
>>
>> return allow
>>
>>
>> 1. Add a “filters” section to the LOGGING_CONFIG beneath “formatters:
>>
>>
>>
>> 'filters': {
>>
>> 'noDepWarn': {
>>
>> '()': DeprecationWarningFilter,
>>
>> }
>>
>> },
>>
>>
>> 1. For each of the handlers where you want to suppress the warnings
>> (console, task, processor, or any of the remote log handlers you may be
>> using) add the following line to its configuration:
>>
>>
>>
>> 'filters': ['noDepWarn'],
>>
>> Restart airflow and your logs should be clean.
>>
>>
>> Sean Carey
>>
>
>
>
