Re: Hadoop FS when running standalone

2020-07-16 Thread Lorenzo Nicora
Thanks Alessandro, I think I solved it. I cannot set any HADOOP_HOME as I have no Hadoop installed on the machine running my tests. But adding *org.apache.flink:flink-shaded-hadoop-2:2.8.3-10.0* as a compile dependency to the Maven profile building the standalone version fixed the issue. Lorenzo

Re: Hadoop FS when running standalone

2020-07-16 Thread Alessandro Solimando
Hi Lorenzo, IIRC I had the same error message when trying to write snappified parquet on HDFS with a standalone fat jar. Flink could not "find" the hadoop native/binary libraries (specifically I think for me the issue was related to snappy), because my HADOOP_HOME was not (properly) set. I have n

Hadoop FS when running standalone

2020-07-16 Thread Lorenzo Nicora
Hi I need to run my streaming job as a *standalone* Java application, for testing The job uses the Hadoop S3 FS and I need to test it (not a unit test). The job works fine when deployed (I am using AWS Kinesis Data Analytics, so Flink 1.8.2) I have *org.apache.flink:flink-s3-fs-hadoop* as a "com