[jira] [Commented] (FLINK-32241) UnsupportedFileSystemException when using the ABFS Hadoop driver for checkpointing in Flink 1.17

[ https://issues.apache.org/jira/browse/FLINK-32241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17728393#comment-17728393 ] Steve Loughran commented on FLINK-32241: this is odd as the buffer to disk code is derived from

[jira] [Commented] (FLINK-30450) FileSystem supports exporting client-side metrics

[ https://issues.apache.org/jira/browse/FLINK-30450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17649716#comment-17649716 ] Steve Loughran commented on FLINK-30450: if you can grab the IOStatistics from an

[jira] [Commented] (FLINK-26563) HadoopS3RecoverableWriterITCase hang on azure

[ https://issues.apache.org/jira/browse/FLINK-26563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504275#comment-17504275 ] Steve Loughran commented on FLINK-26563: you could tune the s3a retry params to try to recover

[jira] [Commented] (FLINK-23722) S3 Tests fail on AZP: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

[ https://issues.apache.org/jira/browse/FLINK-23722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402184#comment-17402184 ] Steve Loughran commented on FLINK-23722: FWIW, you can fix this by setting fs.s3a.endpoint =

[jira] [Commented] (FLINK-19589) Expose S3 options for tagging and object lifecycle policy for FileSystem

[ https://issues.apache.org/jira/browse/FLINK-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259767#comment-17259767 ] Steve Loughran commented on FLINK-19589: the filesystem createFile(Path path) API Returns a

[jira] [Commented] (FLINK-19595) Flink SQL support S3 select

[ https://issues.apache.org/jira/browse/FLINK-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244004#comment-17244004 ] Steve Loughran commented on FLINK-19595: s3a connector supports s3 select (HADOOP-15229) :

[jira] [Commented] (FLINK-19550) StreamingFileSink can't be interrupted when writing to S3

[ https://issues.apache.org/jira/browse/FLINK-19550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243998#comment-17243998 ] Steve Loughran commented on FLINK-19550: s3a in 3.2.2 and 3.3.1 will ship with 1.11.901, see

[jira] [Commented] (FLINK-15814) Log warning when StreamingFileSink is used with an ambiguous S3 scheme and error when used with s3p

[ https://issues.apache.org/jira/browse/FLINK-15814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045839#comment-17045839 ] Steve Loughran commented on FLINK-15814: Out of curiosity, what is the problem? > Log warning

[jira] [Commented] (FLINK-13602) S3 filesystems effectively do not support credential providers

[ https://issues.apache.org/jira/browse/FLINK-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995629#comment-16995629 ] Steve Loughran commented on FLINK-13602: not much we can do in S3A to help here. Sorry. Could

[jira] [Commented] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3

[ https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819326#comment-16819326 ] Steve Loughran commented on FLINK-8801: --- * s3a in hadoop 3.1+ lets you see etags as the file

[jira] [Commented] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3

[ https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819324#comment-16819324 ] Steve Loughran commented on FLINK-8801: --- we cant set times against objects as they are fixed on

[jira] [Commented] (FLINK-11187) StreamingFileSink with S3 backend transient socket timeout issues

[ https://issues.apache.org/jira/browse/FLINK-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738131#comment-16738131 ] Steve Loughran commented on FLINK-11187: bq. As far as I can tell, this is just a transient s3

[jira] [Commented] (FLINK-11187) StreamingFileSink with S3 backend transient socket timeout issues

[ https://issues.apache.org/jira/browse/FLINK-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725942#comment-16725942 ] Steve Loughran commented on FLINK-11187: There's limits to how well you can buffer...if the post

[jira] [Commented] (FLINK-10817) Upgrade presto dependency to support path-style access

[ https://issues.apache.org/jira/browse/FLINK-10817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679607#comment-16679607 ] Steve Loughran commented on FLINK-10817: If you are working with non-AWS endpoints, you also

[jira] [Commented] (FLINK-10363) S3 FileSystem factory prints secrets into logs

[ https://issues.apache.org/jira/browse/FLINK-10363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623330#comment-16623330 ] Steve Loughran commented on FLINK-10363: see WHIRR-642 for this same issue; it's easy to do. For

[jira] [Commented] (FLINK-10363) S3 FileSystem factory prints secrets into logs

[ https://issues.apache.org/jira/browse/FLINK-10363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622263#comment-16622263 ] Steve Loughran commented on FLINK-10363: Stephan: we went to a lot of effort to not log AWS

[jira] [Commented] (FLINK-9061) add entropy to s3 path for better scalability

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553596#comment-16553596 ] Steve Loughran commented on FLINK-9061: --- It'd still be good to conduct some experiments here about

[jira] [Commented] (FLINK-9525) Missing META-INF/services/*FileSystemFactory in flink-hadoop-fs module

[ https://issues.apache.org/jira/browse/FLINK-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502161#comment-16502161 ] Steve Loughran commented on FLINK-9525: --- we're actually moving Hadoop off that introspection

[jira] [Commented] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422645#comment-16422645 ] Steve Loughran commented on FLINK-9061: --- something less than 8, maybe 5, though it's mostly all

[jira] [Commented] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417964#comment-16417964 ] Steve Loughran commented on FLINK-9061: --- [~greghogan] I cut the link as it was just a duplicate of

[jira] [Comment Edited] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417145#comment-16417145 ] Steve Loughran edited comment on FLINK-9061 at 3/28/18 10:33 AM: - The s3a

[jira] [Commented] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417145#comment-16417145 ] Steve Loughran commented on FLINK-9061: --- The s3a connector will have the same issue, though there we

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417140#comment-16417140 ] Steve Loughran commented on FLINK-8794: --- bq. Enable consistent-view can cause other problems.

[jira] [Commented] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415501#comment-16415501 ] Steve Loughran commented on FLINK-9061: --- [~StephanEwen]: I knew that, but it's still the same AWS

[jira] [Comment Edited] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415491#comment-16415491 ] Steve Loughran edited comment on FLINK-8794 at 3/27/18 12:06 PM: - That's

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415491#comment-16415491 ] Steve Loughran commented on FLINK-8794: --- That's amazon EMR's problem. Switch to their "consistent

[jira] [Commented] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414280#comment-16414280 ] Steve Loughran commented on FLINK-9061: --- you can get it on delete requests too, if you try hard.

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414272#comment-16414272 ] Steve Loughran commented on FLINK-8794: --- {quote} writing to local disks would decrease performance,

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414262#comment-16414262 ] Steve Loughran commented on FLINK-8794: --- it does if you turn s3guard on with Hadoop 3.0+ and its S3A

[jira] [Commented] (FLINK-7589) org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 159764230; received: 64638536)

[ https://issues.apache.org/jira/browse/FLINK-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395479#comment-16395479 ] Steve Loughran commented on FLINK-7589: --- We've been thinking of cutting a shaded version of the

[jira] [Commented] (FLINK-8888) Upgrade AWS SDK in flink-connector-kinesis

[ https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392743#comment-16392743 ] Steve Loughran commented on FLINK-: --- If you are pulling in the shaded SDK, note that it's been

[jira] [Commented] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372701#comment-16372701 ] Steve Loughran commented on FLINK-8543: --- copying a subset of the old file to the new file would seem

[jira] [Commented] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371870#comment-16371870 ] Steve Loughran commented on FLINK-8543: --- There's no truncate operation in the S3 protocol; so not in

[jira] [Commented] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371976#comment-16371976 ] Steve Loughran commented on FLINK-8543: --- bq. So all files being accompanied by .valid-length is

[jira] [Commented] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365520#comment-16365520 ] Steve Loughran commented on FLINK-8543: --- Created HADOOP-15239 ; I'll take a patch with a new test

[jira] [Commented] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365506#comment-16365506 ] Steve Loughran commented on FLINK-8543: --- bq. I am running on a Hortonworks based hadoop environment

[jira] [Commented] (FLINK-7589) org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 159764230; received: 64638536)

[ https://issues.apache.org/jira/browse/FLINK-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166527#comment-16166527 ] Steve Loughran commented on FLINK-7589: --- bq. is there any plan to make Flink use AWS S3 SDK

[jira] [Commented] (FLINK-7589) org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 159764230; received: 64638536)

[ https://issues.apache.org/jira/browse/FLINK-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156763#comment-16156763 ] Steve Loughran commented on FLINK-7589: --- well, you are allowed to file bug reports. However, it's

[jira] [Commented] (FLINK-7266) Don't attempt to delete parent directory on S3

[ https://issues.apache.org/jira/browse/FLINK-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155245#comment-16155245 ] Steve Loughran commented on FLINK-7266: --- if you are using s3a then delete(path, recursive=false)

[jira] [Commented] (FLINK-7266) Don't attempt to delete parent directory on S3

[ https://issues.apache.org/jira/browse/FLINK-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153910#comment-16153910 ] Steve Loughran commented on FLINK-7266: --- FWIW, in s3a we create a single delete request to rm all

[jira] [Commented] (FLINK-7365) excessive warning logs of attempt to override final parameter: fs.s3.buffer.dir

[ https://issues.apache.org/jira/browse/FLINK-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119873#comment-16119873 ] Steve Loughran commented on FLINK-7365: --- There's a special log you can turn off to slience all

[jira] [Commented] (FLINK-5706) Implement Flink's own S3 filesystem

[ https://issues.apache.org/jira/browse/FLINK-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897209#comment-15897209 ] Steve Loughran commented on FLINK-5706: --- If you look at where object stores are most trouble in the

[jira] [Commented] (FLINK-5706) Implement Flink's own S3 filesystem

[ https://issues.apache.org/jira/browse/FLINK-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895808#comment-15895808 ] Steve Loughran commented on FLINK-5706: --- I should add that my current stance with using S3 as a

[jira] [Comment Edited] (FLINK-5706) Implement Flink's own S3 filesystem

[ https://issues.apache.org/jira/browse/FLINK-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895648#comment-15895648 ] Steve Loughran edited comment on FLINK-5706 at 3/4/17 6:24 PM: --- Stefan, I

[jira] [Commented] (FLINK-5706) Implement Flink's own S3 filesystem

[ https://issues.apache.org/jira/browse/FLINK-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895648#comment-15895648 ] Steve Loughran commented on FLINK-5706: --- Stefan, I don't think you appreciate how hard it is to do