Thanks. I opened https://issues.apache.org/jira/browse/BEAM-12739
And submitted a patch https://github.com/apache/beam/pull/15313 On Fri, Aug 6, 2021 at 7:57 AM Chamikara Jayalath <chamik...@google.com> wrote: > Hi Jeremy, > > On Thu, Aug 5, 2021 at 7:36 PM Jeremy Lewi <jeremy.l...@primer.ai> wrote: > >> Hi Folks, >> >> I'm running Beam Python (2.31.0) on Flink on Kubernetes. I'm using the >> PortableRunner and a beam job server. >> I'm using S3 for the artifacts dir. The job server is throwing exceptions >> like the one below complaining that filesystem scheme S3 isn't registered. >> > > I think this is because "S3FileSystem" was not in the CLASSPATH (and hence > was not registered). > >> >> I'm using the stock job server >> container apache/beam_flink1.13_job_server:2.31.0. It doesn't look like the >> jar that ships in that server includes the AWS Filesystem classes. >> >> So I tried adding the >> jar beam-sdks-java-io-amazon-web-services-2.31.0.jar. Now when the job >> server loads I get a ClassNotFound exception for AWSCredentialsProvider >> because >> I'm missing some of the dependencies for that. >> >> Does anyone have recommendations on the easiest path to obtaining the >> jar(s) needed to support S3 for the artifact service? Is there an uber jar >> published somewhere or will I have to build it myself? >> > > I don't think such an uber jar is published so probably you'll have to > build it or include all required dependencies separately. > > Thanks, > Cham > > >> >> Thanks. >> J >> >> >> >> Aug 06, 2021 1:52:51 AM >> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2 >> finishStaging >> >> SEVERE: Error staging artifacts >> >> java.util.concurrent.ExecutionException: >> java.lang.IllegalArgumentException: No filesystem found for scheme s3 >> >> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >> >> at java.util.concurrent.FutureTask.get(FutureTask.java:192) >> >> at >> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.finishStaging(ArtifactStagingService.java:461) >> >> at >> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.resolveNextEnvironment(ArtifactStagingService.java:439) >> >> at >> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:417) >> >> at >> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:303) >> >> at >> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:255) >> >> at >> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) >> >> at >> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) >> >> at >> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309) >> >> at >> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292) >> >> at >> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:765) >> >> at >> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) >> >> at >> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> at java.lang.Thread.run(Thread.java:748) >> >> Caused by: java.lang.IllegalArgumentException: No filesystem found for >> scheme s3 >> >> at >> org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:497) >> >> at >> org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:567) >> >> at >> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.stagingDir(ArtifactStagingService.java:193) >> >> at >> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.getDestination(ArtifactStagingService.java:169) >> >> at >> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:271) >> >> at >> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:247) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> ... 3 more >> >> >> >> >> --- >> >> >> Here's the exception after adding the amazon web services Jar >> >> >> Module could not be instantiated >> >> at java.util.ServiceLoader.fail(ServiceLoader.java:232) >> >> at java.util.ServiceLoader.access$100(ServiceLoader.java:185) >> >> at >> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) >> >> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) >> >> at java.util.ServiceLoader$1.next(ServiceLoader.java:480) >> >> at >> com.fasterxml.jackson.databind.ObjectMapper.findModules(ObjectMapper.java:1081) >> >> at >> org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:478) >> >> at >> org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:72) >> >> Caused by: java.lang.NoClassDefFoundError: >> com/amazonaws/auth/AWSCredentialsProvider >> >> at org.apache.beam.sdk.io.aws.options.AwsModule.<init>(AwsModule.java:86) >> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> >> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> >> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) >> >> at java.lang.Class.newInstance(Class.java:442) >> >> at >> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) >> >> ... 5 more >> >> Caused by: java.lang.ClassNotFoundException: >> com.amazonaws.auth.AWSCredentialsProvider >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:382) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:418) >> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:351) >> >> ... 12 more >> >> >>
