[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-24122:
---
Fix Version/s: 2.0.0
(was: 1.20.0)
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Reporter: zlzhang0122
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-24122:
Fix Version/s: (was: 1.19.0)
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Reporter: zlzhang0122
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-24122:
Fix Version/s: 1.20.0
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Reporter: zlzhang0122
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0, 1.20.0
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Ge updated FLINK-24122:
Fix Version/s: 1.19.0
(was: 1.18.0)
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Reporter: zlzhang0122
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-24122:
-
Fix Version/s: 1.18.0
(was: 1.17.0)
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Reporter: zlzhang0122
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingbo Huang updated FLINK-24122:
-
Fix Version/s: 1.17.0
(was: 1.16.0)
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Reporter: zlzhang0122
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24122:
Fix Version/s: 1.16.0
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Reporter: zlzhang0122
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.16.0
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-24122:
-
Affects Version/s: (was: 1.13.2)
(was: 1.12.3)
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Reporter: zlzhang0122
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.1
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-24122:
-
Priority: Major (was: Minor)
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Affects Versions: 1.12.3, 1.13.2
>Reporter: zlzhang0122
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.1
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-24122:
-
Issue Type: Improvement (was: Bug)
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Affects Versions: 1.12.3, 1.13.2
>Reporter: zlzhang0122
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.14.1
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-24122:
-
Fix Version/s: (was: 1.14.1)
1.15.0
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Reporter: zlzhang0122
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24122:
---
Labels: pull-request-available (was: )
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.12.3, 1.13.2
>Reporter: zlzhang0122
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.14.1
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zlzhang0122 updated FLINK-24122:
Description:
Now, the history server can clean history jobs by two means:
# if users have configured
{code:java}
historyserver.archive.clean-expired-jobs: true{code}
, then compare the files in hdfs over two clean interval and find the delete
and clean the local cache file.
# if users have configured the
{code:java}
historyserver.archive.retained-jobs:{code}
a positive number, then clean the oldest files in hdfs and local.
But the retained-jobs number is difficult to determine.
For example, users may want to check the history jobs yesterday while many jobs
failed today and exceed the retained-jobs number, then the history jobs of
yesterday will be delete. So what if add a configuration which contain a
retained-times that indicate the max time the history job retain?
Also it can't clean the job history files which was no longer in hdfs but still
cached in local filesystem and these files will store forever and can't be
cleaned unless users manually do this. Maybe we can give a option and do this
clean if the option says true.
was:
Now, the history server can clean history jobs by two means:
# if users have configured
{code:java}
historyserver.archive.clean-expired-jobs: true{code}
, then compare the files in hdfs over two clean interval and find the delete
and clean the local cache file.
# if users have configured the
{code:java}
historyserver.archive.retained-jobs:{code}
a positive number, then clean the oldest files in hdfs and local.
But the retained-jobs number is difficult to determine.
For example, users may want to check the history jobs yesterday while many jobs
failed today and exceed the retained-jobs number, then the history jobs of
yesterday will be delete. So what if add a configuration which contain a
retained-times that indicate the max time the history job retain?
Also it can't clean the job history which was no longer in hdfs but still
cached in local filesystem and these files will store forever and can't be
cleaned unless users manually do this. Maybe we can give a option and do this
clean if the option says true.
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.12.3, 1.13.2
>Reporter: zlzhang0122
>Priority: Minor
> Fix For: 1.14.1
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history files which was no longer in hdfs but
> still cached in local filesystem and these files will store forever and can't
> be cleaned unless users manually do this. Maybe we can give a option and do
> this clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zlzhang0122 updated FLINK-24122:
Description:
Now, the history server can clean history jobs by two means:
# if users have configured
{code:java}
historyserver.archive.clean-expired-jobs: true{code}
, then compare the files in hdfs over two clean interval and find the delete
and clean the local cache file.
# if users have configured the
{code:java}
historyserver.archive.retained-jobs:{code}
a positive number, then clean the oldest files in hdfs and local.
But the retained-jobs number is difficult to determine.
For example, users may want to check the history jobs yesterday while many jobs
failed today and exceed the retained-jobs number, then the history jobs of
yesterday will be delete. So what if add a configuration which contain a
retained-times that indicate the max time the history job retain?
Also it can't clean the job history which was no longer in hdfs but still
cached in local filesystem and these files will store forever and can't be
cleaned unless users manually do this. Maybe we can give a option and do this
clean if the option says true.
was:
Now, the history server can clean history jobs by two means:
# if users have configured
{code:java}
historyserver.archive.clean-expired-jobs: true{code}
, then compare the files in hdfs over two clean interval and find the delete
and clean the local cache file.
# if users have configured the
{code:java}
historyserver.archive.retained-jobs:{code}
a positive number, then clean the oldest files in hdfs and local.
But the retained-jobs number is difficult to determine.
For example, users may want to check the history jobs yesterday while many jobs
failed today and exceed the retained-jobs number, then the history jobs of
yesterday will be delete. So what if add a configuration which contain a
retained-times that indicate the max time the history job retain?
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.12.3, 1.13.2
>Reporter: zlzhang0122
>Priority: Minor
> Fix For: 1.14.1
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
> Also it can't clean the job history which was no longer in hdfs but still
> cached in local filesystem and these files will store forever and can't be
> cleaned unless users manually do this. Maybe we can give a option and do this
> clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zlzhang0122 updated FLINK-24122:
Description:
Now, the history server can clean history jobs by two means:
# if users have configured
{code:java}
historyserver.archive.clean-expired-jobs: true{code}
, then compare the files in hdfs over two clean interval and find the delete
and clean the local cache file.
# if users have configured the
{code:java}
historyserver.archive.retained-jobs:{code}
a positive number, then clean the oldest files in hdfs and local.
But the retained-jobs number is difficult to determine.
For example, users may want to check the history jobs yesterday while many jobs
failed today and exceed the retained-jobs number, then the history jobs of
yesterday will be delete. So what if add a configuration which contain a
retained-times that indicate the max time the history job retain?
was:
Now, the history server can clean history jobs by two means:
# if users have configured
{code:java}
historyserver.archive.clean-expired-jobs: true{code}
, then compare the files in hdfs over two clean interval and find the delete
and clean the local cache file.
# if users have configured the
{code:java}
historyserver.archive.retained-jobs{code}
a positive number, then clean the oldest files in hdfs and local.
But it can't clean the job history which was no longer in hdfs but still cached
in local filesystem and these files will store forever and can't be cleaned
unless users manually do this. Maybe we can give a option and do this clean if
the option says true.
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.12.3, 1.13.2
>Reporter: zlzhang0122
>Priority: Minor
> Fix For: 1.14.1
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs:{code}
> a positive number, then clean the oldest files in hdfs and local.
> But the retained-jobs number is difficult to determine.
> For example, users may want to check the history jobs yesterday while many
> jobs failed today and exceed the retained-jobs number, then the history jobs
> of yesterday will be delete. So what if add a configuration which contain a
> retained-times that indicate the max time the history job retain?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zlzhang0122 updated FLINK-24122:
Description:
Now, the history server can clean history jobs by two means:
# if users have configured
{code:java}
historyserver.archive.clean-expired-jobs: true{code}
, then compare the files in hdfs over two clean interval and find the delete
and clean the local cache file.
# if users have configured the
{code:java}
historyserver.archive.retained-jobs{code}
a positive number, then clean the oldest files in hdfs and local.
But it can't clean the job history which was no longer in hdfs but still cached
in local filesystem and these files will store forever and can't be cleaned
unless users manually do this. Maybe we can give a option and do this clean if
the option says true.
was:
Now, the history server can clean history jobs by two means:
# if users have configured
{code:java}
historyserver.web.ssl.enabled: true{code}
, then compare the files in hdfs over two clean interval and find the delete
and clean the local cache file.
# if users have configured the
{code:java}
historyserver.archive.retained-jobs{code}
a positive number, then clean the oldest files in hdfs and local.
But it can't clean the job history which was no longer in hdfs but still cached
in local filesystem and these files will store forever and can't be cleaned
unless users manually do this. Maybe we can give a option and do this clean if
the option says true.
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.12.3, 1.13.2
>Reporter: zlzhang0122
>Priority: Minor
> Fix For: 1.14.1
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.archive.clean-expired-jobs: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs{code}
> a positive number, then clean the oldest files in hdfs and local.
> But it can't clean the job history which was no longer in hdfs but still
> cached in local filesystem and these files will store forever and can't be
> cleaned unless users manually do this. Maybe we can give a option and do this
> clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24122) Add support to do clean in history server
[
https://issues.apache.org/jira/browse/FLINK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zlzhang0122 updated FLINK-24122:
Issue Type: Bug (was: Improvement)
> Add support to do clean in history server
> -
>
> Key: FLINK-24122
> URL: https://issues.apache.org/jira/browse/FLINK-24122
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST
>Affects Versions: 1.12.3, 1.13.2
>Reporter: zlzhang0122
>Priority: Minor
> Fix For: 1.14.1
>
>
> Now, the history server can clean history jobs by two means:
> # if users have configured
> {code:java}
> historyserver.web.ssl.enabled: true{code}
> , then compare the files in hdfs over two clean interval and find the delete
> and clean the local cache file.
> # if users have configured the
> {code:java}
> historyserver.archive.retained-jobs{code}
> a positive number, then clean the oldest files in hdfs and local.
> But it can't clean the job history which was no longer in hdfs but still
> cached in local filesystem and these files will store forever and can't be
> cleaned unless users manually do this. Maybe we can give a option and do this
> clean if the option says true.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
