[
https://issues.apache.org/jira/browse/FLINK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weihua Hu updated FLINK-31234:
------------------------------
Description:
Flink on Kubernetes does not support redirecting stdout/stderr to files. This
is to allow users to get logs via "kubectl logs".
But for our internal scenario, we use a kubernetes user to submit all jobs to
the k8s cluster and provide a platform for users to submit jobs. Users can't
access kubernetes directly. so we need to display logs/stdout in flink webui.
Because the web ui retrieves the stdout file by filename, which has the same
prefix as \{taskmanager}.log (such as
flink--kubernetes-taskmanager-0-my-first-flink-cluster-taskmanager-1-4.log). We
can't support this with a simple custom image.
IMO, we should add an option to redirect stdout/stderr to files. When this
setting is enabled.
1. flink-console.sh will redirect stdout/err to files.
2. avoid logs twices in the log file and the stdout file. we could do this by
using ThresholdFilter
([log4j|https://logging.apache.org/log4j/2.x/manual/filters.html#ThresholdFilter]
and [logback|https://logback.qos.ch/manual/filters.html#thresholdFilter]) with
a system property.
Of course, this option is false by default.
was:
Flink on Kubernetes does not support redirecting stdout/stderr to files. This
is to allow users to get logs via "kubectl logs".
But for our internal scenario, we use a kubernetes user to submit all jobs to
the k8s cluster and provide a platform for users to submit jobs. Users can't
access kubernetes directly. so we need to display logs/stdout in flink webui.
Because the web ui retrieves the stdout file by filename, which has the same
prefix as \{taskmanager}.log (such as
flink--kubernetes-taskmanager-0-my-first-flink-cluster-taskmanager-1-4.log). We
can't support this with a simple custom image.
IMO, we should add an option to redirect stdout/stderr to files. When this
setting is enabled.
1. flink-console.sh will redirect stdout/err to files.
2. avoid logs twices in the log file and the stdout file.
Of course, this option is false by default.
> Add an option to redirect stdout/stderr for flink on kubernetes
> ---------------------------------------------------------------
>
> Key: FLINK-31234
> URL: https://issues.apache.org/jira/browse/FLINK-31234
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
> Affects Versions: 1.17.0
> Reporter: Weihua Hu
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Flink on Kubernetes does not support redirecting stdout/stderr to files. This
> is to allow users to get logs via "kubectl logs".
> But for our internal scenario, we use a kubernetes user to submit all jobs to
> the k8s cluster and provide a platform for users to submit jobs. Users can't
> access kubernetes directly. so we need to display logs/stdout in flink webui.
> Because the web ui retrieves the stdout file by filename, which has the same
> prefix as \{taskmanager}.log (such as
> flink--kubernetes-taskmanager-0-my-first-flink-cluster-taskmanager-1-4.log).
> We can't support this with a simple custom image.
> IMO, we should add an option to redirect stdout/stderr to files. When this
> setting is enabled.
> 1. flink-console.sh will redirect stdout/err to files.
> 2. avoid logs twices in the log file and the stdout file. we could do this by
> using ThresholdFilter
> ([log4j|https://logging.apache.org/log4j/2.x/manual/filters.html#ThresholdFilter]
> and [logback|https://logback.qos.ch/manual/filters.html#thresholdFilter])
> with a system property.
> Of course, this option is false by default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
