Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6075
@zhangminglei For your interest - there is a new Bucketing Sink in the
Flink master (called `StreamingFileSink`), with a different design: Managing
all state in Flink state (so it is consistent
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6353
Maybe we can solve this simpler? Avoiding concurrency in the execution
graph creation makes the code simpler and more robust - very desirable for an
already fairly complex construct
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/commit/8231b62ff42aae53ca3a7b552980838ccab824ab#commitcomment-29792609
In
flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/scheduler/CoLocationGroup.java:
In
flink-runtime/src
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6300
Nice feature, thanks a lot.
Merged this into the 1.6 and 1.7 branches
---
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6281#discussion_r202556032
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.java
---
@@ -0,0 +1,397
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6326
Thanks for the review and for merging. @NicoK has an end-to-end test for
SSL PR already (#6327) which would be great to rebase on top of this change.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6326
Pushed another commit that rebuilds the generated config docs
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6302
Thanks for this contribution, that's a valuable fix. I have a few thoughts
and suggestions on how we might improve the feature a bit still:
- Can we get id of the `commons-text
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6321
Is the issue addressed here a bug?
If not, and if it seems that the original authors of the code had an
intention of writing the code as it is now, I would suggest to leave it as it
is.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6321
This would be clearly not a hotfix. As per the pull request template,
contributors should use hotfixes mainly for typos and JavaDoc updates.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6326
@EronWright Given our last discussion, I think this should be interesting
to you.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6050
I think makes sense, merging...
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5982
@sihuazhou I got caught up in some other tasks - will try to get back to
this here soon, I would like to have this feature in as a base for "search for
completed checkpoint".
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6116
Is there a workaround for users to disable dependency convergence?
It is actually a problem that we don't control the convergence of some
dependency that is used with varying versions
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6281
@xndai The umbrella issue is
[FLINK-9749](https://issues.apache.org/jira/browse/FLINK-9749) and some parts
on the specifics of block formats (ORC / Parquet) are in
[FLINK-9753](https
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6300
I like the idea of this - should make it much easier to use.
Have you run this code on some heavier data stream to validate that it
works well in practice?
If yes, I would be +1 to this
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6324
Thanks, merging...
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6328
Could you rebase this on top of #6326 ? That PR makes sure SSLEngine
factories are used everywhere, giving a single point to integrate the provider
such that it is available for all SSL
Github user StephanEwen closed the pull request at:
https://github.com/apache/flink/pull/5966
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5966
This PR is subsumed by #6326
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5966
@EronWright Just saw this - I have concurrently reworked this PR into #6326
which does things more cleanly. I would like to get that PR in for 1.6 (got
many users asking for this).
I
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6327
Could we save testing time by just activating SSL for existing test jobs?
Please also check the update of the SSL config keys that may come through
#6326
---
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/6326
Mutual authentication for internal communication
** This is based on #6324 - hence the first commit in this PR should be
discarded from review**
## What is the purpose of the change
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/6324
[FLINK-9424] [security] Set default cipher suite to a more compatible
cipher suite
## What is the purpose of the change
This reverts "[FLINK-9310] [security] Update standard c
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6304#discussion_r201937327
--- Diff: flink-dist/pom.xml ---
@@ -140,6 +140,22 @@ under the License.
+
--- End
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6309
Thanks, merging once CI is green...
---
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/6309
[FLINK-9809] [DataSteam API] Allow setting co-location constraints on
StreamTransformations
## What is the purpose of the change
Flink supports co location constraints for operator
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6305
Please update the description so that reviewers can take a look.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6275
All right, thanks, merging!
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6290
Thanks, merging this...
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6286
+1
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6285
+1
---
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6281#discussion_r201374783
--- Diff:
flink-core/src/test/java/org/apache/flink/core/fs/local/LocalFileSystemResumableWriterTest.java
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6281#discussion_r201400623
--- Diff:
flink-core/src/test/java/org/apache/flink/core/fs/AbstractResumableWriterTest.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6281#discussion_r201369555
--- Diff:
flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java
---
@@ -253,4
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6281#discussion_r201374633
--- Diff:
flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java
---
@@ -253,4
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6281#discussion_r201375290
--- Diff:
flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java
---
@@ -130,7 +130,7 @@ public static boolean
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6281#discussion_r201401033
--- Diff:
flink-core/src/test/java/org/apache/flink/core/fs/AbstractResumableWriterTest.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6281#discussion_r201375170
--- Diff:
flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java
---
@@ -41,6 +44,8
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6275#discussion_r200814568
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ---
@@ -1563,7 +1573,7 @@ public void run
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6075
The dependencies of the `flink-connector-filesystem` are not well set up
already, having an Avro dependency and a Hadoop dependency. I agree that it
would be good to not introduce yet more
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6235
Took a look at this WIP and I think it goes into a good direction.
My most important comment is that I think it would help to move the
"ensureCompatibility" into the confi
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/6275
[FLINK-9776] [runtime] Stop sending periodic interrupts once executing
thread leaves user function / operator code.
## What is the purpose of the change
Upon cancellation, the task
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6075
If it is not a problem that this can lead to poor compression when
checkpoint intervals are short, we could think about merging this as a
temporary solution until
[FLINK-9749](https
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6075
Hi @zhangminglei
Sorry for the late response - I thought about this solution quite a bit and
came to the conclusion that we may need to do a bit more for efficient results:
Please
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6118
Where in the loading of the factories do you see the error?
My suspicion is still an issue with inverted class loading.
To confirm, can we check the following?
- Are you running
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6118
I think this is a misdiagnosis, this should not be merged.
Flink does not need a file system factory for Hadoop, it uses Hadoop's FS
as the general fallback for all schemes that it does
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6103
That all depends why the failure happens in the first place. It seems to
happen if the receiver of a channel starts much faster than the sender. The
longest part of the deployment is library
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5448
There is no problem reusing old keys, if their default unit was "bytes",
because the `MemorySize.parse(...)` interprets a number as bytes, if there is
no unit attached to it.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6108
@kl0u please link the issue once you created it.
This is currently very early, in design discussions between @kl0u, me, and
@aljoscha.
The main points about the rewrite
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5448
Okay, after taking a look, I think we need to add a few changes:
- We need to add an additional `MemoryUnit.parse()` method that takes the
"default" unit, so that we parse th
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6094
Good catch. Apparently the tests don't check all combinations of settings,
otherwise this would have shown up.
Merging this...
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6103
How critical is it to change this setting?
I would assume this should be caught by the regular recovery, so unless
this occurs very often and thus leads to confusing exceptions in the log
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6108
Do you have a Hadoop version older than 2.7?
We are currently attempting to rewrite the Bucketing Sink completely for
better compatibility with S3 and with better support for Parquet
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6111
I have often wondered about this as well. There sure are a lot of log lines.
- The state backends themselves should probably log on `debug` level,
otherwise there is a line per
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5448
Will try and take a look at this soon... Sorry for the delay.
What I would consider very important is that users who don't change their
configuration do not get different behavior all
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5982
I think we need to have a special output stream type
(`AtomicCreatingFsDataOutputStream` or similar) as the return type of
`FileSystem.createAtomic()`. Otherwise, how can a user actually create
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6073
DependencyManagement in the root pom should cover dependencies that we
share and expose across modules.
Enforcing convergence with one module (`flink-table`) for a dependency
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6073#discussion_r190855869
--- Diff: flink-libraries/flink-table/pom.xml ---
@@ -146,6 +147,12 @@ under the License.
flink-test-utils_
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5843
One could add an int to the annotation, as "priority / position" and sort
by that.
Not sure nice, but could be okay.
I think it was nice for users that the most comm
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6043
@cjolif I agree, let's do something here.
@tzulitai what do you think about trying to use the switch to REST to make
a clean cut and start a new connector project (without dependency
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5963
Merging this...
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5954
This looks fine from my side.
Would like to get a +1 from @tillrohrmann before merging this...
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5891
All new dependencies are okay with the Apache License (not for shading,
though!)
Because nothing gets shaded here and we left dependency management to the
user, this upgrade is okay
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5891
To get such dependency upgrade PRs merged, it is useful to always add the
relevant parts of the dependency tree after the update:
Old:
```
[INFO] org.apache.flink:flink-connector
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5948
Thanks. Will merge this, possibly add one more sentence in the process...
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5843
Looks pretty good.
Is there a way we can "sort" the common options? Something like
- host:port (for standalone setups)
- java memory
- default paralleli
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5857
I think this is a nice addition. Basically turns the prometheus "pull
model" into a "push model".
@lamber-ken Can you check that the new dependency is correctly shad
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6015
Similar as to #6016: Should we close this PR until we have consensus
whether we want to change this?
Especially the performance implications in methods/classes on the "hot code
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6016
Should we close this PR until we have consensus whether we want to change
this?
Especially the performance implications in methods/classes on the "hot code
paths" makes thi
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6043
As a high-level comment, I think we may want start making the ElasticSearch
connectors projects independent of each other.
We previously tried to share code between versions, which has
Github user StephanEwen closed the pull request at:
https://github.com/apache/flink/pull/6066
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6066
Closed in e1d1234477c731fe3f398c7f3f12123f73764242
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6066
Thanks for the review.
Addressing the comments and merging this...
---
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6071#discussion_r190554149
--- Diff: pom.xml ---
@@ -308,7 +308,7 @@ under the License.
errors.
[1] https
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6066#discussion_r190543815
--- Diff:
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/OperatorChainTest.java
---
@@ -0,0 +1,142
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/6066
[FLINK-9428] [checkpointing] Allow operators to flush data on checkpoint
pre-barrier
## What is the purpose of the change
Some operators maintain some small transient state that may
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5995
Looks good, thanks!
+1 to merge this
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5996
Async I/O works best with asynchronous clients. For synchronous client, you
need a threadpool or something else to concurrently fire off requests.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5996
I had a quick look at the code example, and it looks like it might not
actually do asynchronous I/O.
It dispatches a synchronous HTTP request on a direct executor
(`onComplete`s in a direct
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5995
Added a few more comment, most importantly around exception wrapping.
Otherwise, looking good...
---
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r189197766
--- Diff:
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroDeserializationSchema.java
---
@@ -0,0 +1,176
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r189197633
--- Diff:
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroDeserializationSchema.java
---
@@ -0,0 +1,176
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r189195014
--- Diff:
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java
---
@@ -0,0 +1,87
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r189185420
--- Diff:
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroDeserializationSchema.java
---
@@ -0,0 +1,176
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r189185186
--- Diff:
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroDeserializationSchema.java
---
@@ -0,0 +1,215
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/commit/c8fa8d025684c2225824c54a7285bbfdec7cfddc#commitcomment-29021995
In
flink-core/src/main/java/org/apache/flink/api/java/typeutils/TypeInfoParser.java:
In
flink-core/src/main/java/org
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6039
Very nice, very helpful for users.
+1 to merge this
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5979
Okay, looks really good from my side.
Would be good if @StefanRRichter or @azagrebin to double check the change,
otherwise good to go.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5966
I would move ahead with this PR as follows:
- Separate internal and external SSL config
- Activate SSL client auth for akka, netty, and blob server (pure
internal communication
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6001
Looks good, thanks, merging this...
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5979
Could you share some micro-benchmark numbers?
When we change something that we know works well to something new, would be
good to understand what benefits we are talking about.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5970
Merging this...
---
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/5977#discussion_r188725700
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/functions/RuntimeContext.java
---
@@ -71,6 +71,17 @@
@PublicEvolving
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/5995
I would actually keep the package name for now. It makes sense, because the
connection to the registry is avro-specific at the moment...
---
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/6001#discussion_r188529928
--- Diff: docs/dev/stream/operators/windows.md ---
@@ -797,7 +797,7 @@ DataStream input = ...;
input
.keyBy()
- .timeWindow
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r188340240
--- Diff:
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroDeserializationSchema.java
---
@@ -0,0 +1,215
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r188316643
--- Diff: flink-formats/flink-avro-confluent-registry/pom.xml ---
@@ -0,0 +1,94 @@
+
+
+http://maven.apache.org/POM/4.0.0
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r188325819
--- Diff:
flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/ConfluentSchemaRegistryCoder.java
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/5995#discussion_r188328236
--- Diff:
flink-formats/flink-avro-confluent-registry/src/test/java/org/apache/flink/formats/avro/registry/confluent/ConfluentSchemaRegistryCoderTest.java
1 - 100 of 4019 matches
Mail list logo
