I used requests instead of kube as an example, but what do you think about
doing something like this? I'm happy to put this into a PR if it would
solve the pain point today.
import logging
try:
import requests
except ModuleNotFoundError:
import sys
logging.warning('kube not installed - skipping kube examples')
sys.exit()
resp = requests.get('http://example.com')
print(resp)
...
Taylor
*Taylor Edmiston*
Blog <https://blog.tedmiston.com/> | CV
<https://stackoverflow.com/cv/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor> | Stack Overflow
<https://stackoverflow.com/users/149428/taylor-edmiston>
On Wed, May 30, 2018 at 4:40 AM, Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Hi Craig,
>
> This is something that needs to be fixed. I agree with you this is very
> dirty. In your installation you're not installing the kubernetes stuff, so
> the KubernetesPodOperator is ignored. We need to figure out how to have
> example dags that are not compatible with the vanilla installation, or we
> need to remove the kubernetes example for now, and move it to the
> documentation.
>
> Cheers, Fokko
>
> 2018-05-30 2:11 GMT+02:00 Craig Rodrigues <crodr...@gmail.com>:
>
> > I tested master branch by putting the following in my requirements.txt:
> >
> > git+https://github.com/rodrigc/incubator-airflow@
> > master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,
> > ldap,mysql,postgres,redis,slack,s3]
> >
> > and did a pip install -r requirements.txt
> >
> > When I started the airflow webserver, I saw deprecation warnings. I
> > put some additional debugging in models.py to through an exception so
> that
> > I could see the
> > full stacktrace:
> >
> > [2018-05-29 14:00:34,419] {models.py:307} ERROR - Failed to import:
> > /Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/example_dags/example_kubernetes_operator.py
> > Traceback (most recent call last):
> > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> airflow/models.py",
> > line 304, in process_file
> > m = imp.load_source(mod_name, filepath)
> > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/example_dags/example_kubernetes_operator.py", line 53, in
> <module>
> > dag=dag)
> > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> airflow/utils/decorators.py",
> > line 98, in wrapper
> > result = func(*args, **kwargs)
> > File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> airflow/models.py",
> > line 2308, in __init__
> > raise Exception("Invalid use of args or kwargs")
> > Exception: Invalid use of args or kwargs
> >
> >
> > If looks like example_kubernetes_operator.py, this code is the source of
> > the exception:
> >
> > k = KubernetesPodOperator(
> > namespace='default',
> > image="ubuntu:16.04",
> > cmds=["bash", "-cx"],
> > arguments=["echo", "10"],
> > labels={"foo": "bar"},
> > name="airflow-test-pod",
> > in_cluster=False,
> > task_id="task",
> > get_logs=True,
> > dag=dag)
> >
> >
> > Without my extra debugging, the deprecation warning looks like this:
> >
> > [2018-05-29 14:06:27,567] {example_kubernetes_operator.py:30} WARNING -
> > Could not import KubernetesPodOperator
> > /Users/c-craigr/airflow2/lib/python2.7/site-packages/
> airflow/models.py:2315:
> > PendingDeprecationWarning: Invalid arguments were passed to BaseOperator.
> > Support for passing such arguments will be dropped in Airflow 2.0.
> Invalid
> > arguments were:
> > *args: ()
> > **kwargs: {'name': 'airflow-test-pod', 'image': 'ubuntu:16.04', 'labels':
> > {'foo': 'bar'}, 'namespace': 'default', 'cmds': ['bash', '-cx'],
> > 'arguments': ['echo', '10'], 'in_cluster': False, 'get_logs': True}
> > category=PendingDeprecationWarning
> >
> >
> >
> > What is the correct fix for this? It looks like a lot of operators pass
> > in arguments which are not
> > processed by BaseOperator, and thus trip over this deprecation warning.
> >
> > --
> > Craig
> >
>
