Hello, I am a developer trying to use Apache Beam in Java, and I am having an issue with my TestPipeline taking a long time to get created for my unit tests. When I create a brand new TestPipeline() and call my pipeline.run() method, there are a lot of calls to get AWS information, even though I'm just creating a bare TestPipeline by calling TestPipeline.create(). Here are some stack traces I see in my code: poll:-1, Net (sun.nio.ch) park:191, NioSocketImpl (sun.nio.ch) timedRead:280, NioSocketImpl (sun.nio.ch) implRead:304, NioSocketImpl ( sun.nio.ch) read:346, NioSocketImpl (sun.nio.ch) read:796, NioSocketImpl$1 ( sun.nio.ch) read:1099, Socket$SocketInputStream (java.net) fill:291, BufferedInputStream (java.io) read1:347, BufferedInputStream (java.io) implRead:420, BufferedInputStream (java.io) read:399, BufferedInputStream ( java.io) parseHTTPHeader:827, HttpClient (sun.net.www.http) parseHTTP:759, HttpClient (sun.net.www.http) getInputStream0:1705, HttpURLConnection (sun.net.www.protocol.http) getInputStream:1614, HttpURLConnection (sun.net.www.protocol.http) getResponseCode:531, HttpURLConnection (java.net) readResource:112, HttpResourcesUtils (software.amazon.awssdk.regions.util) getToken:436, EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) getItems:426, EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) getData:362, EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) getData:358, EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) getEC2InstanceRegion:242, EC2MetadataUtils (software.amazon.awssdk.regions.internal.util) tryDetectRegion:72, InstanceProfileRegionProvider (software.amazon.awssdk.regions.providers) getRegion:56, InstanceProfileRegionProvider (software.amazon.awssdk.regions.providers) getRegion:51, AwsRegionProviderChain (software.amazon.awssdk.regions.providers) create:60, AwsOptions$AwsRegionFactory (org.apache.beam.sdk.io.aws2.options) create:55, AwsOptions$AwsRegionFactory (org.apache.beam.sdk.io.aws2.options) returnDefaultHelper:689, ProxyInvocationHandler (org.apache.beam.sdk.options) getDefault:630, ProxyInvocationHandler (org.apache.beam.sdk.options) invoke:227, ProxyInvocationHandler (org.apache.beam.sdk.options) valueOrElse:235, ClientBuilderFactory$DefaultClientBuilder (org.apache.beam.sdk.io.aws2.common) create:142, ClientBuilderFactory$DefaultClientBuilder (org.apache.beam.sdk.io.aws2.common) create:56, ClientBuilderFactory (org.apache.beam.sdk.io.aws2.common) createBuilder:40, DefaultS3ClientBuilderFactory (org.apache.beam.sdk.io.aws2.s3) createBuilder:35, DefaultS3ClientBuilderFactory (org.apache.beam.sdk.io.aws2.s3) getBuilder:100, S3FileSystemConfiguration (org.apache.beam.sdk.io.aws2.s3) builderFrom:88, S3FileSystemConfiguration (org.apache.beam.sdk.io.aws2.s3) fromS3Options:92, S3FileSystemConfiguration (org.apache.beam.sdk.io.aws2.s3) fromOptions:35, DefaultS3FileSystemSchemeRegistrar (org.apache.beam.sdk.io.aws2.s3) lambda$fromOptions$0:49, S3FileSystemRegistrar (org.apache.beam.sdk.io.aws2.s3) accept:273, ReferencePipeline$7$1 (java.util.stream) forEachRemaining:133, Iterator (java.util) forEachRemaining:1939, Spliterators$IteratorSpliterator (java.util) copyInto:509, AbstractPipeline (java.util.stream) wrapAndCopyInto:499, AbstractPipeline (java.util.stream) evaluateSequential:921, ReduceOps$ReduceOp (java.util.stream) evaluate:234, AbstractPipeline (java.util.stream) collect:682, ReferencePipeline (java.util.stream) fromOptions:50, S3FileSystemRegistrar (org.apache.beam.sdk.io.aws2.s3) verifySchemesAreUnique:567, FileSystems (org.apache.beam.sdk.io) setDefaultPipelineOptions:554, FileSystems (org.apache.beam.sdk.io) fromOptions:47, PipelineRunner (org.apache.beam.sdk) run:317, Pipeline (org.apache.beam.sdk) run:398, TestPipeline (org.apache.beam.sdk.testing) run:335, TestPipeline (org.apache.beam.sdk.testing) Regardless of how I create my TestPipeline(), will it always make calls to S3 File System Configuration, or is there a way to bypass this behavior since I don't need to use S3 in my pipeline for my unit tests?
Any help would be appreciated! Thanks and sincerely, Ramya ______________________________________________________________________ The information contained in this e-mail may be confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
