Doesn't this warning come from the BaseOperator class -
https://github.com/apache/incubator-airflow/blob/master/airflow/models.py#L2511
?
Are you passing unused arguments to the QuboleOperator, or do you not
control the instantiation of those operators?
Chris
On Fri, Sep 28, 2018 at 7:39 PM Sean Carey <sean.ca...@aurishealth.com>
wrote:
> Haha yes, good point. Thanks!
>
> Sean
>
>
> On 9/28/18, 6:13 PM, "Ash Berlin-Taylor" <a...@apache.org> wrote:
>
> Sounds good for your use, certainly.
>
> I mainly wanted to make sure other people knew before blindly
> equipping a foot-canon :)
>
> -ash
>
> > On 29 Sep 2018, at 00:09, Sean Carey <sean.ca...@aurishealth.com>
> wrote:
> >
> > Thanks, Ash. I understand what you're saying. The warnings are
> coming from the Qubole operator. We get a lot of this:
> >
> > PendingDeprecationWarning: Invalid arguments were passed to
> QuboleOperator. Support for passing such arguments will be dropped in
> Airflow 2.0. Invalid arguments were:
> > *args: ()
> > **kwargs: {...}
> > category=PendingDeprecationWarning
> >
> > We've spoken to Qubole about this and they plan to address it. In
> the meantime, it generates a ton of noise in our logs which we'd like to
> suppress -- we are aware of the issue, and don't need to be told about it
> every 5 seconds.
> >
> > As for it suddenly breaking, being that this is pending Airflow 2.0
> I feel the risk is low and when we do upgrade it will be thoroughly tested.
> >
> > Thanks!
> >
> > Sean
> >
> >
> > On 9/28/18, 5:01 PM, "Ash Berlin-Taylor" <a...@apache.org> wrote:
> >
> > What deprecation warnings are you getting? Are they from Airflow
> itself (i.e. things Airflow is calling like flask_wtf, etc) or of your use
> of Airflow?
> >
> > If it is the form could you check and see if someone has already
> reported a Jira issue so we can fix them?
> https://issues.apache.org/jira/issues/?jql=project%3DAIRFLOW
> >
> > If it is the latter PLEASE DO NOT IGNORE THESE.
> >
> > Deprecation warnings are how we, the Airflow community tell users
> that you need to make a change to your DAG/code/config to upgrade things.
> If you silence these warnings you will have a much harder time upgrading to
> new versions of Airflow (read: you might suddenly find that things stop
> working because you turned of the warnings.)
> >
> > -ash
> >
> >> On 28 Sep 2018, at 22:52, Sean Carey <sean.ca...@aurishealth.com>
> wrote:
> >>
> >> Hello,
> >>
> >> I’ve been looking for a way to suppress the
> PendingDeprecationWarning messages cluttering our airflow logs and I have a
> working solution which I thought I would share.
> >>
> >> In order to do this, you first need to configure airflow for custom
> logging using steps 1-4 here:
> >>
> >>
> https://airflow.readthedocs.io/en/stable/howto/write-logs.html#writing-logs-to-azure-blob-storage
> >>
> >> (note that although the document is for Azure remote logging you
> don’t actually need azure for this)
> >>
> >> Next, modify the log_config.py script created in the step above as
> follows:
> >>
> >>
> >> 1. Import logging
> >> 2. Define the filter class:
> >>
> >>
> >>
> >> class DeprecationWarningFilter(logging.Filter):
> >>
> >> def filter(self, record):
> >>
> >> allow = 'DeprecationWarning' not in record.msg
> >>
> >> return allow
> >>
> >>
> >> 1. Add a “filters” section to the LOGGING_CONFIG beneath
> “formatters:
> >>
> >>
> >>
> >> 'filters': {
> >>
> >> 'noDepWarn': {
> >>
> >> '()': DeprecationWarningFilter,
> >>
> >> }
> >>
> >> },
> >>
> >>
> >> 1. For each of the handlers where you want to suppress the
> warnings (console, task, processor, or any of the remote log handlers you
> may be using) add the following line to its configuration:
> >>
> >>
> >>
> >> 'filters': ['noDepWarn'],
> >>
> >> Restart airflow and your logs should be clean.
> >>
> >>
> >> Sean Carey
> >>
> >
> >
> >
>
>
>
>
