Thanks for the proposal! @Till Rohrmann
The design looks generally good to me. I have 2 concerns though:
# performance on failover
I can see is that an ExecutionGraph will be built and initialized on each
task failure. The process is possibly to be slow:
1. the topology building can be very slow
Thanks for your thoughts Xintong! What you are writing is very valuable
feedback for me, as I have limited experience with real-world deployments.
It seems that autoscaling support is a really important follow up.
## active resource managers
I guess you can consider reactive mode a special case o
Thanks for the explanation, Robert.
Now I see how these things are expected to be supported in steps.
I think you are right. Demanding a fixed finite amount of resources can be
considered as a special case of `ScalingPolicy`. I'm now good with the
current scope of reactive mode as a first step, a
It is true that we haven't properly discussed whether to support bulk
config changes via a single environment variable or not, Thomas. It is also
true that one could move the logic to the Flink processes (more
specifically to where the config is loaded).
I am somehow assuming that it is easier for
Timo Walther created FLINK-21149:
Summary: Remove deprecated CatalogTable.getProperties
Key: FLINK-21149
URL: https://issues.apache.org/jira/browse/FLINK-21149
Project: Flink
Issue Type: Sub-
Chesnay, thanks for bringing this up. I think it's an alternative that's
worthwhile to discuss, although I do think that we should stick to the current
proposal in the FLIP.
My main concerns about the FLINK_PROPERTIES approach boil down to the
following:
* A single env var per config option is
I'm including Steven and Yang into this discussion (cc'd) who raised some good
points in the initial DISCUSS thread. Do you have any opinion on the current
discussion here?
On Tue, Jan 26, 2021, at 10:59 AM, Ufuk Celebi wrote:
> Chesnay, thanks for bringing this up. I think it's an alternative t
Thanks a lot for all the feedback Steven, Yang Wang and Xintong. I have a
few more comments to add.
# Keep it simple and stupid
As Robert said we would like to keep the new feature initially as simple as
possible in order to quickly implement it. Once we have a basic
implementation, we want to re
I might have joined the discussion a bit late.
My two cents:
- I'm good with EVs overwriting the flink-conf.yaml. However, I'm a bit
unsure about EVs overwriting the dynamic properties.
- Currently, dynamic properties are expected to be the final
effective configuration in many
@Xingtong: The assumption for the mapping was that we only have dots and
hyphens in the keys. Do you have an example for a key which include
underscores? If underscores are common for keys (I couldn't find any existing
options that use it), it would certainly be a blocker for the discussed
appr
Thanks a lot for all your feedback. Let me try to answer it.
# ScaleUpController vs. RescaleController
At the moment, the idea of the declarative scheduler is to run a job with a
parallelism which is as close to the target value as possible but to never
exceed it. Since the target value is curren
@Ufuk,
I also don't find any existing options with underscores in their keys.
However, I do not find any explicit rules forbid using them either. I'm not
saying this should block the FLIP. Just it would be nice to beware of this
issue, and maybe ensure the assumption with test cases if we finally d
I think we have to assume that some user has a custom config option that
uses underscores.
That said, we can probably assume that no one uses other special
characters like question marks and such, which are indeed allowed by the
YAML spec.
These kind of questions are precisely why I prefer t
godfrey he created FLINK-21150:
--
Summary: Introduce ExecEdge to connect two ExecNodes
Key: FLINK-21150
URL: https://issues.apache.org/jira/browse/FLINK-21150
Project: Flink
Issue Type: Sub-task
Hi all,
during a security maintenance PR [1], Chesnay noticed that the
flink-swift-fs-hadoop module is lacking test coverage [2].
Also, there hasn't been any substantial change since 2018, when it was
introduced.
On the user@ ML, I could not find any proof of significant use of the
module (no one
Thank you for starting this discussion Timo!
Maybe we should draft a Wiki page outlining our deprecation and removal
policies for legacy code in Flink. We'll need such a document anyways, and
summarizing the consensus from this discussion into a document would allow
us to move forward with this di
Thanks Till for the explanation and the follow up actions.
That sounds good to me.
Thanks,
Zhu
Till Rohrmann 于2021年1月26日周二 下午7:28写道：
> Thanks a lot for all your feedback. Let me try to answer it.
>
> # ScaleUpController vs. RescaleController
>
> At the moment, the idea of the declarative schedu
@Chesnay
could you explain how underscores in user-defined properties would be
affected with transformation like STATE_BACKEND -> state.backend?
IIUC, this transformation happens in Flink and doesn't alter ENV vars, so
the user app can still parse the original configuration.
OTH, I'm a bit concerne
Hi everyone,
great to see all these responses. I'm happy to drive this effort. I will
try to summarize this discussion and look into how other projects are
handling this topic. A might need a couple days more for this. After
that we can discuss how we would like our guidlines to look like.
T
Thanks Robert and Till for the thorough explanation.
Now I understand key difference between reactive mode and auto scaling mode.
For the latter, we could dynamically adjust the desired value based on
monitoring
the metrics(e.g. cpu, memory, latency, delay, etc.). Since the reactive is
simpler
and
Here's an example: My option key is custom.my_backend_option. With the
current design, the corresponding env variable would be
CUSTOM_MY_BACKEND_OPTION, which would be converted into
custom.my.backend.option .
It is true that users could still parse the original system property as
a fall-back
Mind you that we could of course solve these character issues by first
nailing down which characters we allow in keys (presumably: [a-z0-9-.]).
On 1/26/2021 3:45 PM, Chesnay Schepler wrote:
Here's an example: My option key is custom.my_backend_option. With the
current design, the corresponding
FYI: the StreamConfig actually queries the Configuration where some keys
with underscores. This isn't quite relevant for environment variables
themselves, but where exactly in the API the lookup for environment
variables is supposed to done.
In a similar vein, the metric system does not direct
Aljoscha Krettek created FLINK-21151:
Summary: Extract common full-snapshot writer from RocksDB
full-snapshot strategy
Key: FLINK-21151
URL: https://issues.apache.org/jira/browse/FLINK-21151
Proje
Chesnay Schepler created FLINK-21152:
Summary: Bump flink-shaded to 13.0
Key: FLINK-21152
URL: https://issues.apache.org/jira/browse/FLINK-21152
Project: Flink
Issue Type: Improvement
Hi everyone,
thanks for the livid discussion, it's great to see so many opinions and
ideas!
The point about underscores is a very valid point where the current FLIP,
if we were to stick with it, would have to be improved. I was going to say
that we should exclude that from the discussion about th
Till Rohrmann created FLINK-21153:
-
Summary: yarn-per-job deployment target ignores yarn options
Key: FLINK-21153
URL: https://issues.apache.org/jira/browse/FLINK-21153
Project: Flink
Issue T
Thinking a bit more about the DYNAMIC_PROPERTIES, I have to admit that I
like the fact that it works around the problem of encoding the key names
and that it is more powerful wrt to bulk changes. Also the fact that one
can copy past configuration snippets is quite useful. Given these aspects
and th
Igal Shilman created FLINK-21154:
Summary: Move the shared request-reply protobuf files to a
separate artifact.
Key: FLINK-21154
URL: https://issues.apache.org/jira/browse/FLINK-21154
Project: Flink
Till Rohrmann created FLINK-21155:
-
Summary:
FileSourceTextLinesITCase.testBoundedTextFileSourceWithTaskManagerFailover does
not pass
Key: FLINK-21155
URL: https://issues.apache.org/jira/browse/FLINK-21155
> Here's an example: My option key is custom.my_backend_option. With the
> current design, the corresponding env variable would be
> CUSTOM_MY_BACKEND_OPTION, which would be converted into
> custom.my.backend.option .
I think we don't have to translate CUSTOM_MY_BACKEND_OPTION back. Instead,
we sh
In the end DYNAMIC_PROPERTIES behaves exactly like env.java.opts;
meaning that the existing rules set by the JVM apply.
Here's an example: export DYNAMIC_PROPERTIES='-Drest.port=1234
-Dother.option="iContainAn=Sign"'
This would then be appended as is to the /java/ command.
(
Conceptually
Thanks for the explanations, Till.
Keeping the initial design as simple as possible sounds good to me.
There's no further concern from my side.
Thank you~
Xintong Song
On Tue, Jan 26, 2021 at 9:56 PM Zhu Zhu wrote:
> Thanks Till for the explanation and the follow up actions.
> That sounds go
Hi Jie,
Thanks for the investigation. I think we can first implement pluggable DT
providers, and add renewal abilities incrementally. I'm also curious where
Spark runs its HadoopDelegationTokenManager when renewal is enabled?
Because it seems HadoopDelegationTokenManager needs access to keytab to
Hi all, sorry to butt in.
I am little curious about why do we have to do the overwritten via
environment variables after
loading the configuration from file. Could we support to do the
substitution while loading the
"flink-conf.yaml" file?
For example, I have the following config options in my fl
On Tue, Jan 26, 2021 at 1:36 AM Till Rohrmann wrote:
> I am somehow assuming that it is easier for (human) users to specify a
> single env variable for a config option instead of defining a block of
> Flink configuration values. I wouldn't know how to properly separate the
> entries of the config
Guowei Ma created FLINK-21156:
-
Summary:
org.apache.flink.connector.jdbc.catalog.PostgresCatalogITCase Fail
Key: FLINK-21156
URL: https://issues.apache.org/jira/browse/FLINK-21156
Project: Flink
Guowei Ma created FLINK-21157:
-
Summary: "Running HA per-job cluster (rocks, incremental)
end-to-end test" gets stuck
Key: FLINK-21157
URL: https://issues.apache.org/jira/browse/FLINK-21157
Project: Flink
pengkangjing created FLINK-21158:
Summary: wrong jjvm
Key: FLINK-21158
URL: https://issues.apache.org/jira/browse/FLINK-21158
Project: Flink
Issue Type: Bug
Components: Runtime / We
39 matches
Mail list logo
