Re: IO metrics

o make sure that your custom metrics have different names than the built-in ones as otherwise the will be ignored. On 13.02.2018 20:37, cw7k wrote: > Hi, couple questions on IO metrics listed > here:https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io &

IO metrics

Hi, couple questions on IO metrics listed here:https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io We're trying to get metrics such as throughput to filesystem sinks.  Are the metrics listed on that page automatically recorded, and we just need to retrieve the

build/devops

Hi, a full mvn install takes about 2-2.5 hours.  Does the Flink team run the full testsuite on every dev pull request, or use a main dev branch that gets the full test periodically?  Or a different method?  The 2 hours can be a drag when one team needs another's changes and each attempt at passi

filesystem output partitioning

cw7k : >  Ok, I have the factory working in the WordCount example.  I had to move > the factory code and META-INF into the WordCount project. > For general Flink jobs, I'm assuming that the goal would be to be able to > import the factory from the job itself instead of needing to c

Re: adding a new cloud filesystem

file into each project?  If so, any guidelines on how to do that?On Thursday, January 18, 2018, 10:53:32 AM PST, cw7k wrote: Hi, just a bit more info, I have a test function working using oci://, based on the S3 test: https://github.com/apache/flink/blob/master/flink-filesystems/flink-

Re: adding a new cloud filesystem

entation" error mentioned earlier.   On Thursday, January 18, 2018, 10:22:57 AM PST, cw7k wrote: Thanks.  I now have the 3 requirements fulfilled but the scheme isn't being loaded; I get this error: "Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:

Re: adding a new cloud filesystem

op/S3FileSystemFactory.java#L132 [2] https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-presto/src/main/java/org/apache/flink/fs/s3presto/S3FileSystemFactory.java#L131 2018-01-18 1:24 GMT+01:00 cw7k : >  Thanks. I'm looking at the s3 example and I can only find the &g

Re: adding a new cloud filesystem

Hi, please have a look at this doc page [1]. It describes how to add new file system implementations and also how to configure them. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/filesystems.html#adding-new-file-system-implementations 2018-01-18 0:32 GMT+01:00

adding a new cloud filesystem

16, 2018, 5:27:31 PM PST, cw7k wrote: Hi, question on this page: "You need to point Flink to a valid Hadoop configuration..."https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/aws.html#s3-simple-storage-service How do you point Flink to the Hadoop config?  

Re: difference b/t filesystem and connector-filesystem

[2]. [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/filesystem_sink.html [2] https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/index.html#connectors-in-apache-bahir Cheers, Till On Fri, Jan 12, 2018 at 7:22 PM, cw7k wrote: > Hi, I'm t

Re: difference b/t filesystem and connector-filesystem

r/dev/connectors/index.html#connectors-in-apache-bahir Cheers, Till On Fri, Jan 12, 2018 at 7:22 PM, cw7k wrote: > Hi, I'm trying to understand the difference between the flink-filesystem > and flink-connector-filesystem.  How is each intended to be used? > If adding support for

difference b/t filesystem and connector-filesystem

Hi, I'm trying to understand the difference between the flink-filesystem and flink-connector-filesystem.  How is each intended to be used? If adding support for a different storage provider that supports HDFS, should additions be made to one or the other, or both?  Thanks.