Re: Flink Table API and not recognizing s3 plugins

Sweet, this was the issue. I got this to work by copying the s3 jar over to plugins for the client container. Thanks for all of the help! The Table API is sweet! On Mon, Sep 14, 2020 at 11:14 PM Dan Hill wrote: > Yes, the client runs in K8. It uses a different K8 config than the Helm > chart

Re: Flink Table API and not recognizing s3 plugins

Yes, the client runs in K8. It uses a different K8 config than the Helm chart and does not load the plugins. Does the client use the same plugin structure as the Flink job/task manager? I can try using it tomorrow. Cool, that link would work too. Thanks, Arvid! On Mon, Sep 14, 2020 at 10:59

Re: Flink Table API and not recognizing s3 plugins

Hi Dan, Are you running the client also in K8s? If so you need an initialization step, where you add the library to the plugins directory. Putting it into lib or into the user jar doesn't work anymore as we removed the shading in s3 in Flink 1.10. The official Flink docker image has an easy way t

Re: Flink Table API and not recognizing s3 plugins

Thanks for the update! I'm trying a bunch of combinations on the client side to get the S3 Filesystem to be picked up correctly. Most of my attempts involved building into the job jar (which I'm guessing won't work). I then started getting issues with ClassCastExceptions. I might try a little m

Re: Flink Table API and not recognizing s3 plugins

Hi Dan, I think Arvid and Dawid are right, as a workaround, you can try making S3Filesystem works in the client. But for a long term solution, we can fix it. I created https://issues.apache.org/jira/browse/FLINK-19228 for tracking this. Best, Jingsong On Mon, Sep 14, 2020 at 3:57 PM Dawid Wysak

Re: Flink Table API and not recognizing s3 plugins

Hi Dan, As far as I checked in the code, the FileSystemSink will try to create staging directories from the client. I think it might be problematic, as your case shows. We might need to revisit that part. I am cc'ing Jingsong who worked on the FileSystemSink. As a workaround you might try putting

Re: Flink Table API and not recognizing s3 plugins

This is running on my local minikube and is trying to hit minio. On Thu, Sep 10, 2020 at 1:10 PM Dan Hill wrote: > I'm using this Helm chart > . I > start the job by building an image with the job jar and using kubectl apply > t

Re: Flink Table API and not recognizing s3 plugins

In general, I'd assume that JM and TM are enough. However, it seems like the query planner is doing some path sanitization for which it needs the filesystem. Since I don't know this part too well, I'm pulling in Jark and Dawid that may know more. I'm also not sure if this is intentional or a bug.

Re: Flink Table API and not recognizing s3 plugins

I'm using this Helm chart . I start the job by building an image with the job jar and using kubectl apply to do a flink run with the jar. The log4j.properties on jobmanager and taskmanager have debug level set and are pretty embed

Re: Flink Table API and not recognizing s3 plugins

Copying more of the log 2020-09-10 19:50:17,712 INFO org.apache.flink.client.cli.CliFrontend [] - 2020-09-10 19:50:17,718 INFO org.apache.flink.client.cli.CliFrontend [] - Starting

Re: Flink Table API and not recognizing s3 plugins

I was able to get more info to output on jobmanager. 2020-09-10 19:50:17,722 INFO org.apache.flink.client.cli.CliFrontend [] - 2020-09-10 19:50:17,731 INFO org.apache.flink.configuration.GlobalConfig

Re: Flink Table API and not recognizing s3 plugins

Hi Dan, somehow enabling debug statements did not work. However, the logs helps to narrow down the issue. The exception occurs neither on jobmanager nor on taskmanager. It occurs wherever you execute the command line interface. How do you execute the job? Do you start it from your machine? Can y

Re: Flink Table API and not recognizing s3 plugins

I changed the levels to DEBUG. I don't see useful data in the logs. https://drive.google.com/file/d/1ua1zsr3BInY_8xdsWwA__F0uloAqy-vG/view?usp=sharing On Thu, Sep 10, 2020 at 8:45 AM Arvid Heise wrote: > Could you try 1) or 2) and enable debug logging* and share the log with us? > > *Usually b

Re: Flink Table API and not recognizing s3 plugins

Could you try 1) or 2) and enable debug logging* and share the log with us? *Usually by adjusting FLINK_HOME/conf/log4j.properties. On Thu, Sep 10, 2020 at 5:38 PM Dan Hill wrote: > Ah, sorry, it's a copy/paste issue with this email. I've tried both: > 1) using s3a uri with flink-s3-fs-hadoop

Re: Flink Table API and not recognizing s3 plugins

Ah, sorry, it's a copy/paste issue with this email. I've tried both: 1) using s3a uri with flink-s3-fs-hadoop jar in /opt/flink/plugins/s3-fs-hadoop. 2) using s3p uri with flink-s3-fs-presto jar in /opt/flink/plugins/s3-fs-presto. 3) loading both 1 and 2 4) trying s3 uri. When doing 1) Caused by

Re: Flink Table API and not recognizing s3 plugins

Hi Dan, s3p is only provided by flink-s3-fs-presto plugin. The plugin you used provides s3a. (and both provide s3, but it's good to use the more specific prefix). Best, Arvid On Thu, Sep 10, 2020 at 9:24 AM Dan Hill wrote: > *Background* > I'm converting some prototype Flink v1.11.1 code that

Flink Table API and not recognizing s3 plugins

*Background* I'm converting some prototype Flink v1.11.1 code that uses DataSet/DataTable APIs to use the Table API. *Problem* When switching to using the Table API, my s3 plugins stopped working. I don't know why. I've added the required maven table dependencies to the job. I've tried us movin