Not a full patch to replace snakebite with libhdfs, but should temporarily unblock ci/cd: https://github.com/apache/incubator-airflow/pull/3365
On Wed, May 16, 2018 at 10:00 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: > Snakebite is not py3 compatible. We should move to libhdfs. Happy to take > patches! > > B. > > Verstuurd vanaf mijn iPad > > > Op 16 mei 2018 om 18:57 heeft Cindy Rottinghuis < > cindyrottingh...@gmail.com> het volgende geschreven: > > > > Here is a sample of what I am seeing. > > I am running on ubuntu, python 3.5. This is a subset of the full > message. Once it went into the snakebite package, checked and the library > wasn’t supported in python3. > > …. > > File "/usr/local/airflow/dags/dim_date.py", line 5, in <module> > > from airflow.operators.sensors import S3KeySensor > > File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py", > line 34, in <module> > > from airflow.hooks.hdfs_hook import HDFSHook > > File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py", > line 20, in <module> > > from snakebite.client import Client, HAClient, Namenode, > AutoConfigClient > > File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", > line 1473 > > baseTime = min(time * (1L << retries), cap); > > ^ > > > > > >> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis < > cindyrottingh...@gmail.com> wrote: > >> > >> Hi, > >> > >> Yes, it looks like all of the other sensors will work under Python3, > but not hdfs. I am planning to use the s3 sensor, which is wrapped up in > the sensors.py module. My issue is that when I test my dag or airflow > installation under python3, I get errors about the hdfs_hook, which I’m not > using. Other then creating my own version of the sensors.py file and > removing the hdfs related functions/libraries, is there any thing else I > can do to work around this? > >> > >> > >> > >>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko <fo...@driesprong.frl> > wrote: > >>> > >>> Hi Cindy, > >>> > >>> The other sensors should work under Python3. We try to support Python3 > as > >>> much as possible, but sometimes libraries are used that are not > compatible. > >>> Could you describe what you are running into? > >>> > >>> Cheers, Fokko > >>> > >>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis < > cindyrottingh...@gmail.com>: > >>> > >>>> Hi, > >>>> > >>>> Are there any plans to update the HDFS_hook.py script to remove the > >>>> reference to the snakebite python library? I’d like to run airflow on > >>>> python3, and this is causing some issues. The hdfs_hook script is > >>>> referenced in the sensors module. > >>>> > >>>> Any suggestions? > >>>> > >>>> Thanks, > >>>> Cindy > >> > > > >
