Re: Testing Flink 1.4 unable to write to s3 locally

Which dependencies do I need to put in the lib directory? I have the flink-shaded-hadoop2.jar and parquet-hadoop.jar marked as "provided" and both are in the lib/ directory and I still get the error? I've also set classloader.resolve-order: parent-first in my flink-conf.yaml. val

Re: Testing Flink 1.4 unable to write to s3 locally

Hi! For the exception in the "local file path" case, we already had a reason: - You were having Hadoop code in your user code jar. Either not putting the Hadoop dependency into your jar (rather in Flink's lib directory) or setting classloading to "parent-first" should do the trick there. For

Re: Testing Flink 1.4 unable to write to s3 locally

When I change the path from an s3 path to a local path I get the following error: Cluster configuration: Standalone cluster with JobManager at localhost/ 127.0.0.1:6123 Using address localhost:6123 to connect to JobManager. JobManager web interface address http://localhost:8082 Starting execution

Re: Testing Flink 1.4 unable to write to s3 locally

Here is the task manager log: 2018-01-08 16:16:13,406 INFO org.apache.flink.runtime.taskmanager.TaskManager - Received task Source: Kafka -> Sink: S3 (1/1) 2018-01-08 16:16:13,407 INFO org.apache.flink.runtime.taskmanager.Task - Source: Kafka -> Sink: S3 (1/1) (bc932736c6526eb1bd41f6aaa73b2997)

Re: Testing Flink 1.4 unable to write to s3 locally

This looks like the output from the client - do you have some TaskManager log files with more log entries? That would be helpful... On Wed, Jan 3, 2018 at 5:26 PM, Kyle Hamlin wrote: > Hello Stephan & Nico, > > Here is the full stacktrace, its not much more than what I

Re: Testing Flink 1.4 unable to write to s3 locally

Hello Stephan & Nico, Here is the full stacktrace, its not much more than what I originally posted. I remember seeing an XMLInputFactory input error at one point, but I haven't seen that again. Is there any other information I can provide that will help resolve? [flink-1.4.0] ./bin/flink run

Re: Testing Flink 1.4 unable to write to s3 locally

The error is not a the missing Class "S3ErrorResponseHandler", but the initialization of that class. It could be missing classes that are statically referenced by the "S3ErrorResponseHandler". The ones I see are "XMLInputFactory" (part of Java itself, should always be there) and

Re: Testing Flink 1.4 unable to write to s3 locally

Hi Kyle, except for putting the jar into the lib/ folder and setting up credentials, nothing else should be required [1]. The S3ErrorResponseHandler class itself is in the jar, as you can see with jar tf flink-s3-fs-presto-1.4.0.jar | grep

Re: Testing Flink 1.4 unable to write to s3 locally

Hi Kyle! Is there more of the stack trace available, like an original exception cause? Best, Stephan On Sun, Dec 31, 2017 at 5:10 PM, Kyle Hamlin wrote: > Hi, > > When testing Flink 1.4 locally the error below keeps getting thrown. I've > followed the setup by moving the

Testing Flink 1.4 unable to write to s3 locally

Hi, When testing Flink 1.4 locally the error below keeps getting thrown. I've followed the setup by moving the flink-s3-fs-presto.jar from the opt/ folder to the lib/ folder. Is there something additional I need to do? java.lang.NoClassDefFoundError: Could not initialize class