Snakebite is not py3 compatible. We should move to libhdfs. Happy to take patches!
B. Verstuurd vanaf mijn iPad > Op 16 mei 2018 om 18:57 heeft Cindy Rottinghuis <cindyrottingh...@gmail.com> > het volgende geschreven: > > Here is a sample of what I am seeing. > I am running on ubuntu, python 3.5. This is a subset of the full message. > Once it went into the snakebite package, checked and the library wasn’t > supported in python3. > …. > File "/usr/local/airflow/dags/dim_date.py", line 5, in <module> > from airflow.operators.sensors import S3KeySensor > File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py", > line 34, in <module> > from airflow.hooks.hdfs_hook import HDFSHook > File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py", > line 20, in <module> > from snakebite.client import Client, HAClient, Namenode, AutoConfigClient > File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", line 1473 > baseTime = min(time * (1L << retries), cap); > ^ > > >> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis <cindyrottingh...@gmail.com> >> wrote: >> >> Hi, >> >> Yes, it looks like all of the other sensors will work under Python3, but not >> hdfs. I am planning to use the s3 sensor, which is wrapped up in the >> sensors.py module. My issue is that when I test my dag or airflow >> installation under python3, I get errors about the hdfs_hook, which I’m not >> using. Other then creating my own version of the sensors.py file and >> removing the hdfs related functions/libraries, is there any thing else I can >> do to work around this? >> >> >> >>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko <fo...@driesprong.frl> wrote: >>> >>> Hi Cindy, >>> >>> The other sensors should work under Python3. We try to support Python3 as >>> much as possible, but sometimes libraries are used that are not compatible. >>> Could you describe what you are running into? >>> >>> Cheers, Fokko >>> >>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis <cindyrottingh...@gmail.com>: >>> >>>> Hi, >>>> >>>> Are there any plans to update the HDFS_hook.py script to remove the >>>> reference to the snakebite python library? I’d like to run airflow on >>>> python3, and this is causing some issues. The hdfs_hook script is >>>> referenced in the sensors module. >>>> >>>> Any suggestions? >>>> >>>> Thanks, >>>> Cindy >> >
