[jira] [Updated] (FLINK-11459) Presto S3 does not show errors due to missing credentials with minio
[
https://issues.apache.org/jira/browse/FLINK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-11459:
---
Labels: auto-deprioritized-major auto-deprioritized-minor (was:
auto-deprioritized-major stale-minor)
Priority: Not a Priority (was: Minor)
This issue was labeled "stale-minor" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Minor, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Presto S3 does not show errors due to missing credentials with minio
>
>
> Key: FLINK-11459
> URL: https://issues.apache.org/jira/browse/FLINK-11459
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.6.2
>Reporter: Nico Kruber
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> It seems that when using minio for S3-like storage and with
> mis-configurations such as missing (maybe also wrong) credentials gets into a
> failing state but with no reason for it:
> {code}
> ...
> 2019-01-29 15:43:27,676 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.heap.mb, 353
> 2019-01-29 15:43:27,738 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.heap.mb, 429
> 2019-01-29 15:43:27,758 INFO org.apache.flink.api.java.ExecutionEnvironment
> [] - The job has 0 registered types and 0 default Kryo
> serializers
> 2019-01-29 15:43:29,943 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961],
> 2019-01-29 15:43:29,956 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2115.551],
> 2019-01-29 15:43:31,946 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788],
> 2019-01-29 15:43:31,958 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3610.417],
> 2019-01-29 15:43:33,954 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853],
> 2019-01-29 15:43:33,963 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2917.786],
> 2019-01-29 15:43:36,133 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942],
> 2019-01-29 15:43:36,156 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2029.473],
> 2019-01-29 15:43:38,142 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05],
> 2019-01-29 15:43:38,164 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2092.878],
> 2019-01-29 15:43:42,181 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164],
> 2019-01-29 15:43:42,186 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2011.204],
> 2019-01-29 15:43:44,262 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165],
> 2019-01-29 15:43:44,276 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2024.312],
> 2019-01-29 15:43:44,585 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - RECEIVED
> SIGNAL 15: SIGTERM. Shutting down as requested.
> 2019-01-29 15:43:44,628 INFO
> org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting
> down BLOB cache
> 2019-01-29 15:43:44,661 INFO org.apache.flink.runtime.blob.BlobServer
> [] - Stopped BLOB server at 0.0.0.0:6124
> {code}
> With AWS S3, it is actually printing an exception instead:
> {code}
> 2019-01-29 19:24:39,968 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: rest.port, 8081
> 2019-01-29 19:24:39,990 INFO org.apache.flink.api.java.ExecutionEnvironment
> - The job has 0 registered types and 0 default Kryo serializers
> 2019-01-29 19:24:43,117 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2047.535], CredentialsRequestTime=[2033.619],
> 2019-01-29 19:24:43,118 INFO
>
[jira] [Updated] (FLINK-11459) Presto S3 does not show errors due to missing credentials with minio
[
https://issues.apache.org/jira/browse/FLINK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-11459:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Presto S3 does not show errors due to missing credentials with minio
>
>
> Key: FLINK-11459
> URL: https://issues.apache.org/jira/browse/FLINK-11459
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.6.2
>Reporter: Nico Kruber
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> It seems that when using minio for S3-like storage and with
> mis-configurations such as missing (maybe also wrong) credentials gets into a
> failing state but with no reason for it:
> {code}
> ...
> 2019-01-29 15:43:27,676 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.heap.mb, 353
> 2019-01-29 15:43:27,738 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.heap.mb, 429
> 2019-01-29 15:43:27,758 INFO org.apache.flink.api.java.ExecutionEnvironment
> [] - The job has 0 registered types and 0 default Kryo
> serializers
> 2019-01-29 15:43:29,943 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961],
> 2019-01-29 15:43:29,956 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2115.551],
> 2019-01-29 15:43:31,946 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788],
> 2019-01-29 15:43:31,958 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3610.417],
> 2019-01-29 15:43:33,954 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853],
> 2019-01-29 15:43:33,963 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2917.786],
> 2019-01-29 15:43:36,133 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942],
> 2019-01-29 15:43:36,156 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2029.473],
> 2019-01-29 15:43:38,142 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05],
> 2019-01-29 15:43:38,164 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2092.878],
> 2019-01-29 15:43:42,181 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164],
> 2019-01-29 15:43:42,186 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2011.204],
> 2019-01-29 15:43:44,262 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165],
> 2019-01-29 15:43:44,276 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2024.312],
> 2019-01-29 15:43:44,585 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - RECEIVED
> SIGNAL 15: SIGTERM. Shutting down as requested.
> 2019-01-29 15:43:44,628 INFO
> org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting
> down BLOB cache
> 2019-01-29 15:43:44,661 INFO org.apache.flink.runtime.blob.BlobServer
> [] - Stopped BLOB server at 0.0.0.0:6124
> {code}
> With AWS S3, it is actually printing an exception instead:
> {code}
> 2019-01-29 19:24:39,968 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: rest.port, 8081
> 2019-01-29 19:24:39,990 INFO org.apache.flink.api.java.ExecutionEnvironment
> - The job has 0 registered types and 0 default Kryo serializers
> 2019-01-29 19:24:43,117 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
>
[jira] [Updated] (FLINK-11459) Presto S3 does not show errors due to missing credentials with minio
[
https://issues.apache.org/jira/browse/FLINK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-11459:
---
Priority: Minor (was: Major)
> Presto S3 does not show errors due to missing credentials with minio
>
>
> Key: FLINK-11459
> URL: https://issues.apache.org/jira/browse/FLINK-11459
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.6.2
>Reporter: Nico Kruber
>Priority: Minor
> Labels: auto-deprioritized-major
>
> It seems that when using minio for S3-like storage and with
> mis-configurations such as missing (maybe also wrong) credentials gets into a
> failing state but with no reason for it:
> {code}
> ...
> 2019-01-29 15:43:27,676 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.heap.mb, 353
> 2019-01-29 15:43:27,738 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.heap.mb, 429
> 2019-01-29 15:43:27,758 INFO org.apache.flink.api.java.ExecutionEnvironment
> [] - The job has 0 registered types and 0 default Kryo
> serializers
> 2019-01-29 15:43:29,943 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961],
> 2019-01-29 15:43:29,956 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2115.551],
> 2019-01-29 15:43:31,946 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788],
> 2019-01-29 15:43:31,958 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3610.417],
> 2019-01-29 15:43:33,954 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853],
> 2019-01-29 15:43:33,963 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2917.786],
> 2019-01-29 15:43:36,133 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942],
> 2019-01-29 15:43:36,156 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2029.473],
> 2019-01-29 15:43:38,142 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05],
> 2019-01-29 15:43:38,164 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2092.878],
> 2019-01-29 15:43:42,181 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164],
> 2019-01-29 15:43:42,186 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2011.204],
> 2019-01-29 15:43:44,262 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165],
> 2019-01-29 15:43:44,276 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2024.312],
> 2019-01-29 15:43:44,585 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - RECEIVED
> SIGNAL 15: SIGTERM. Shutting down as requested.
> 2019-01-29 15:43:44,628 INFO
> org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting
> down BLOB cache
> 2019-01-29 15:43:44,661 INFO org.apache.flink.runtime.blob.BlobServer
> [] - Stopped BLOB server at 0.0.0.0:6124
> {code}
> With AWS S3, it is actually printing an exception instead:
> {code}
> 2019-01-29 19:24:39,968 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: rest.port, 8081
> 2019-01-29 19:24:39,990 INFO org.apache.flink.api.java.ExecutionEnvironment
> - The job has 0 registered types and 0 default Kryo serializers
> 2019-01-29 19:24:43,117 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2047.535], CredentialsRequestTime=[2033.619],
> 2019-01-29 19:24:43,118 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2049.826],
> 2019-01-29 19:24:46,215 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2003.168], CredentialsRequestTime=[2002.836],
> 2019-01-29 19:24:46,216 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2004.182],
> 2019-01-29 19:24:50,384 INFO
>
[jira] [Updated] (FLINK-11459) Presto S3 does not show errors due to missing credentials with minio
[
https://issues.apache.org/jira/browse/FLINK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-11459:
---
Labels: auto-deprioritized-major (was: stale-major)
> Presto S3 does not show errors due to missing credentials with minio
>
>
> Key: FLINK-11459
> URL: https://issues.apache.org/jira/browse/FLINK-11459
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.6.2
>Reporter: Nico Kruber
>Priority: Major
> Labels: auto-deprioritized-major
>
> It seems that when using minio for S3-like storage and with
> mis-configurations such as missing (maybe also wrong) credentials gets into a
> failing state but with no reason for it:
> {code}
> ...
> 2019-01-29 15:43:27,676 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.heap.mb, 353
> 2019-01-29 15:43:27,738 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.heap.mb, 429
> 2019-01-29 15:43:27,758 INFO org.apache.flink.api.java.ExecutionEnvironment
> [] - The job has 0 registered types and 0 default Kryo
> serializers
> 2019-01-29 15:43:29,943 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961],
> 2019-01-29 15:43:29,956 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2115.551],
> 2019-01-29 15:43:31,946 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788],
> 2019-01-29 15:43:31,958 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3610.417],
> 2019-01-29 15:43:33,954 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853],
> 2019-01-29 15:43:33,963 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2917.786],
> 2019-01-29 15:43:36,133 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942],
> 2019-01-29 15:43:36,156 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2029.473],
> 2019-01-29 15:43:38,142 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05],
> 2019-01-29 15:43:38,164 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2092.878],
> 2019-01-29 15:43:42,181 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164],
> 2019-01-29 15:43:42,186 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2011.204],
> 2019-01-29 15:43:44,262 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165],
> 2019-01-29 15:43:44,276 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2024.312],
> 2019-01-29 15:43:44,585 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - RECEIVED
> SIGNAL 15: SIGTERM. Shutting down as requested.
> 2019-01-29 15:43:44,628 INFO
> org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting
> down BLOB cache
> 2019-01-29 15:43:44,661 INFO org.apache.flink.runtime.blob.BlobServer
> [] - Stopped BLOB server at 0.0.0.0:6124
> {code}
> With AWS S3, it is actually printing an exception instead:
> {code}
> 2019-01-29 19:24:39,968 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: rest.port, 8081
> 2019-01-29 19:24:39,990 INFO org.apache.flink.api.java.ExecutionEnvironment
> - The job has 0 registered types and 0 default Kryo serializers
> 2019-01-29 19:24:43,117 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2047.535], CredentialsRequestTime=[2033.619],
> 2019-01-29 19:24:43,118 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2049.826],
> 2019-01-29 19:24:46,215 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2003.168], CredentialsRequestTime=[2002.836],
> 2019-01-29 19:24:46,216 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2004.182],
> 2019-01-29
[jira] [Updated] (FLINK-11459) Presto S3 does not show errors due to missing credentials with minio
[
https://issues.apache.org/jira/browse/FLINK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-11459:
---
Labels: stale-major (was: )
> Presto S3 does not show errors due to missing credentials with minio
>
>
> Key: FLINK-11459
> URL: https://issues.apache.org/jira/browse/FLINK-11459
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.6.2
>Reporter: Nico Kruber
>Priority: Major
> Labels: stale-major
>
> It seems that when using minio for S3-like storage and with
> mis-configurations such as missing (maybe also wrong) credentials gets into a
> failing state but with no reason for it:
> {code}
> ...
> 2019-01-29 15:43:27,676 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.heap.mb, 353
> 2019-01-29 15:43:27,738 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.heap.mb, 429
> 2019-01-29 15:43:27,758 INFO org.apache.flink.api.java.ExecutionEnvironment
> [] - The job has 0 registered types and 0 default Kryo
> serializers
> 2019-01-29 15:43:29,943 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961],
> 2019-01-29 15:43:29,956 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2115.551],
> 2019-01-29 15:43:31,946 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788],
> 2019-01-29 15:43:31,958 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3610.417],
> 2019-01-29 15:43:33,954 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853],
> 2019-01-29 15:43:33,963 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2917.786],
> 2019-01-29 15:43:36,133 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942],
> 2019-01-29 15:43:36,156 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2029.473],
> 2019-01-29 15:43:38,142 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05],
> 2019-01-29 15:43:38,164 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2092.878],
> 2019-01-29 15:43:42,181 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164],
> 2019-01-29 15:43:42,186 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2011.204],
> 2019-01-29 15:43:44,262 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165],
> 2019-01-29 15:43:44,276 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2024.312],
> 2019-01-29 15:43:44,585 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - RECEIVED
> SIGNAL 15: SIGTERM. Shutting down as requested.
> 2019-01-29 15:43:44,628 INFO
> org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting
> down BLOB cache
> 2019-01-29 15:43:44,661 INFO org.apache.flink.runtime.blob.BlobServer
> [] - Stopped BLOB server at 0.0.0.0:6124
> {code}
> With AWS S3, it is actually printing an exception instead:
> {code}
> 2019-01-29 19:24:39,968 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: rest.port, 8081
> 2019-01-29 19:24:39,990 INFO org.apache.flink.api.java.ExecutionEnvironment
> - The job has 0 registered types and 0 default Kryo serializers
> 2019-01-29 19:24:43,117 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2047.535], CredentialsRequestTime=[2033.619],
> 2019-01-29 19:24:43,118 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2049.826],
> 2019-01-29 19:24:46,215 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2003.168], CredentialsRequestTime=[2002.836],
> 2019-01-29 19:24:46,216 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2004.182],
> 2019-01-29 19:24:50,384 INFO
>
[jira] [Updated] (FLINK-11459) Presto S3 does not show errors due to missing credentials with minio
[
https://issues.apache.org/jira/browse/FLINK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-11459:
---
Component/s: (was: Connectors / FileSystem)
FileSystems
> Presto S3 does not show errors due to missing credentials with minio
>
>
> Key: FLINK-11459
> URL: https://issues.apache.org/jira/browse/FLINK-11459
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
>Affects Versions: 1.6.2
>Reporter: Nico Kruber
>Priority: Major
>
> It seems that when using minio for S3-like storage and with
> mis-configurations such as missing (maybe also wrong) credentials gets into a
> failing state but with no reason for it:
> {code}
> ...
> 2019-01-29 15:43:27,676 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: taskmanager.heap.mb, 353
> 2019-01-29 15:43:27,738 INFO
> org.apache.flink.configuration.GlobalConfiguration [] - Loading
> configuration property: jobmanager.heap.mb, 429
> 2019-01-29 15:43:27,758 INFO org.apache.flink.api.java.ExecutionEnvironment
> [] - The job has 0 registered types and 0 default Kryo
> serializers
> 2019-01-29 15:43:29,943 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961],
> 2019-01-29 15:43:29,956 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2115.551],
> 2019-01-29 15:43:31,946 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788],
> 2019-01-29 15:43:31,958 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[3610.417],
> 2019-01-29 15:43:33,954 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853],
> 2019-01-29 15:43:33,963 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2917.786],
> 2019-01-29 15:43:36,133 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942],
> 2019-01-29 15:43:36,156 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2029.473],
> 2019-01-29 15:43:38,142 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05],
> 2019-01-29 15:43:38,164 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2092.878],
> 2019-01-29 15:43:42,181 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164],
> 2019-01-29 15:43:42,186 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2011.204],
> 2019-01-29 15:43:44,262 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165],
> 2019-01-29 15:43:44,276 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
> ClientExecuteTime=[2024.312],
> 2019-01-29 15:43:44,585 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - RECEIVED
> SIGNAL 15: SIGTERM. Shutting down as requested.
> 2019-01-29 15:43:44,628 INFO
> org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting
> down BLOB cache
> 2019-01-29 15:43:44,661 INFO org.apache.flink.runtime.blob.BlobServer
> [] - Stopped BLOB server at 0.0.0.0:6124
> {code}
> With AWS S3, it is actually printing an exception instead:
> {code}
> 2019-01-29 19:24:39,968 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: rest.port, 8081
> 2019-01-29 19:24:39,990 INFO org.apache.flink.api.java.ExecutionEnvironment
> - The job has 0 registered types and 0 default Kryo serializers
> 2019-01-29 19:24:43,117 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2047.535], CredentialsRequestTime=[2033.619],
> 2019-01-29 19:24:43,118 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2049.826],
> 2019-01-29 19:24:46,215 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2003.168], CredentialsRequestTime=[2002.836],
> 2019-01-29 19:24:46,216 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ClientExecuteTime=[2004.182],
> 2019-01-29 19:24:50,384 INFO
>
[jira] [Updated] (FLINK-11459) Presto S3 does not show errors due to missing credentials with minio
[
https://issues.apache.org/jira/browse/FLINK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber updated FLINK-11459:
Description:
It seems that when using minio for S3-like storage and with mis-configurations
such as missing (maybe also wrong) credentials gets into a failing state but
with no reason for it:
{code}
...
2019-01-29 15:43:27,676 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.heap.mb, 353
2019-01-29 15:43:27,738 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.heap.mb, 429
2019-01-29 15:43:27,758 INFO org.apache.flink.api.java.ExecutionEnvironment
[] - The job has 0 registered types and 0 default Kryo serializers
2019-01-29 15:43:29,943 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961],
2019-01-29 15:43:29,956 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2115.551],
2019-01-29 15:43:31,946 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788],
2019-01-29 15:43:31,958 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[3610.417],
2019-01-29 15:43:33,954 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853],
2019-01-29 15:43:33,963 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2917.786],
2019-01-29 15:43:36,133 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942],
2019-01-29 15:43:36,156 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2029.473],
2019-01-29 15:43:38,142 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05],
2019-01-29 15:43:38,164 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2092.878],
2019-01-29 15:43:42,181 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164],
2019-01-29 15:43:42,186 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2011.204],
2019-01-29 15:43:44,262 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165],
2019-01-29 15:43:44,276 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency[] -
ClientExecuteTime=[2024.312],
2019-01-29 15:43:44,585 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - RECEIVED
SIGNAL 15: SIGTERM. Shutting down as requested.
2019-01-29 15:43:44,628 INFO org.apache.flink.runtime.blob.TransientBlobCache
[] - Shutting down BLOB cache
2019-01-29 15:43:44,661 INFO org.apache.flink.runtime.blob.BlobServer
[] - Stopped BLOB server at 0.0.0.0:6124
{code}
With AWS S3, it is actually printing an exception instead:
{code}
2019-01-29 19:24:39,968 INFO
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: rest.port, 8081
2019-01-29 19:24:39,990 INFO org.apache.flink.api.java.ExecutionEnvironment
- The job has 0 registered types and 0 default Kryo serializers
2019-01-29 19:24:43,117 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2047.535], CredentialsRequestTime=[2033.619],
2019-01-29 19:24:43,118 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2049.826],
2019-01-29 19:24:46,215 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2003.168], CredentialsRequestTime=[2002.836],
2019-01-29 19:24:46,216 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2004.182],
2019-01-29 19:24:50,384 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2003.15], CredentialsRequestTime=[2002.803],
2019-01-29 19:24:50,384 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2004.308],
2019-01-29 19:24:56,691 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2002.596], CredentialsRequestTime=[2002.45],
2019-01-29 19:24:56,691 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2003.177],
2019-01-29 19:25:07,058 INFO
org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
ClientExecuteTime=[2003.26],
