[ https://issues.apache.org/jira/browse/FLINK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Metzger updated FLINK-11459: ----------------------------------- Component/s: (was: Connectors / FileSystem) FileSystems > Presto S3 does not show errors due to missing credentials with minio > -------------------------------------------------------------------- > > Key: FLINK-11459 > URL: https://issues.apache.org/jira/browse/FLINK-11459 > Project: Flink > Issue Type: Bug > Components: FileSystems > Affects Versions: 1.6.2 > Reporter: Nico Kruber > Priority: Major > > It seems that when using minio for S3-like storage and with > mis-configurations such as missing (maybe also wrong) credentials gets into a > failing state but with no reason for it: > {code} > ... > 2019-01-29 15:43:27,676 INFO > org.apache.flink.configuration.GlobalConfiguration [] - Loading > configuration property: taskmanager.heap.mb, 353 > 2019-01-29 15:43:27,738 INFO > org.apache.flink.configuration.GlobalConfiguration [] - Loading > configuration property: jobmanager.heap.mb, 429 > 2019-01-29 15:43:27,758 INFO org.apache.flink.api.java.ExecutionEnvironment > [] - The job has 0 registered types and 0 default Kryo > serializers > 2019-01-29 15:43:29,943 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961], > 2019-01-29 15:43:29,956 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2115.551], > 2019-01-29 15:43:31,946 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788], > 2019-01-29 15:43:31,958 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[3610.417], > 2019-01-29 15:43:33,954 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853], > 2019-01-29 15:43:33,963 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2917.786], > 2019-01-29 15:43:36,133 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942], > 2019-01-29 15:43:36,156 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2029.473], > 2019-01-29 15:43:38,142 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05], > 2019-01-29 15:43:38,164 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2092.878], > 2019-01-29 15:43:42,181 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164], > 2019-01-29 15:43:42,186 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2011.204], > 2019-01-29 15:43:44,262 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165], > 2019-01-29 15:43:44,276 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - > ClientExecuteTime=[2024.312], > 2019-01-29 15:43:44,585 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - RECEIVED > SIGNAL 15: SIGTERM. Shutting down as requested. > 2019-01-29 15:43:44,628 INFO > org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting > down BLOB cache > 2019-01-29 15:43:44,661 INFO org.apache.flink.runtime.blob.BlobServer > [] - Stopped BLOB server at 0.0.0.0:6124 > {code} > With AWS S3, it is actually printing an exception instead: > {code} > 2019-01-29 19:24:39,968 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: rest.port, 8081 > 2019-01-29 19:24:39,990 INFO org.apache.flink.api.java.ExecutionEnvironment > - The job has 0 registered types and 0 default Kryo serializers > 2019-01-29 19:24:43,117 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2047.535], CredentialsRequestTime=[2033.619], > 2019-01-29 19:24:43,118 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2049.826], > 2019-01-29 19:24:46,215 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2003.168], CredentialsRequestTime=[2002.836], > 2019-01-29 19:24:46,216 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2004.182], > 2019-01-29 19:24:50,384 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2003.15], CredentialsRequestTime=[2002.803], > 2019-01-29 19:24:50,384 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2004.308], > 2019-01-29 19:24:56,691 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2002.596], CredentialsRequestTime=[2002.45], > 2019-01-29 19:24:56,691 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2003.177], > 2019-01-29 19:25:07,058 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2003.26], CredentialsRequestTime=[2002.948], > 2019-01-29 19:25:07,058 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2004.175], > 2019-01-29 19:25:25,472 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2001.772], CredentialsRequestTime=[2001.611], > 2019-01-29 19:25:25,473 INFO > org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - > ClientExecuteTime=[2002.873], > 2019-01-29 19:25:25,475 ERROR > org.apache.flink.api.common.io.DelimitedInputFormat - Unexpected > problem while getting the file statistics for files > '[s3://flink/LICENSE.gz]': Unable to load credentials from service endpoint > org.apache.flink.fs.s3presto.shaded.com.amazonaws.SdkClientException: Unable > to load credentials from service endpoint > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:180) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:159) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:141) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) > at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559) > at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138) > at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556) > at > org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307) > at > org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:85) > at > org.apache.flink.api.common.io.FileInputFormat.getFileStats(FileInputFormat.java:526) > at > org.apache.flink.api.common.io.FileInputFormat.getFileStats(FileInputFormat.java:505) > at > org.apache.flink.api.common.io.DelimitedInputFormat.getStatistics(DelimitedInputFormat.java:356) > at > org.apache.flink.api.common.io.DelimitedInputFormat.getStatistics(DelimitedInputFormat.java:47) > at > org.apache.flink.optimizer.dag.DataSourceNode.computeOperatorSpecificDefaultEstimates(DataSourceNode.java:166) > at > org.apache.flink.optimizer.dag.OptimizerNode.computeOutputEstimates(OptimizerNode.java:589) > at > org.apache.flink.optimizer.traversals.IdAndEstimatesVisitor.postVisit(IdAndEstimatesVisitor.java:61) > at > org.apache.flink.optimizer.traversals.IdAndEstimatesVisitor.postVisit(IdAndEstimatesVisitor.java:32) > at > org.apache.flink.optimizer.dag.DataSourceNode.accept(DataSourceNode.java:250) > at > org.apache.flink.optimizer.dag.SingleInputNode.accept(SingleInputNode.java:515) > at > org.apache.flink.optimizer.dag.DataSinkNode.accept(DataSinkNode.java:248) > at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:478) > at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:399) > at > org.apache.flink.client.program.OptimizerPlanEnvironment.execute(OptimizerPlanEnvironment.java:51) > at > org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:817) > at test.FlinkReadS3Test.main(FlinkReadS3Test.java:36) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421) > at > org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83) > at > org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78) > at > org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120) > at > org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$10(JarRunHandler.java:226) > at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:589) > at sun.net.NetworkClient.doConnect(NetworkClient.java:175) > at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) > at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) > at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) > at sun.net.www.http.HttpClient.New(HttpClient.java:339) > at sun.net.www.http.HttpClient.New(HttpClient.java:357) > at > sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) > at > sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) > at > sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) > at > sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:47) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:106) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:77) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:156) > at > org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:121) > ... 52 more > ... > {code} > The job itself is rather simple: > {code} > ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); > exEnv.readTextFile("s3://...").map(...).writeAsCsv(outputLocation).execute(); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)