[jira] [Updated] (HDDS-12581) Multi-threaded Log File Parsing with Batch Updates to DB
[
https://issues.apache.org/jira/browse/HDDS-12581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreeja updated HDDS-12581:
--
Description:
Log files should be parsed using multiple threads, where each thread processes
one file at a time. The number of threads should be dynamically adjusted (can
be given in the command). During the parsing process, batch updates should be
performed on the database. It should be ensured that only one thread performs
the database updates at a time while other threads are blocked during this
operation.
Populates two tables: DatanodeContainerLogTable(acts like a detail log history)
and ContainerLogTable(stores latest state and BCSID for each container-datanode
pair)
CLI command of the form:
{code:java}
ozone debug log container --db= parse --path=
--thread-count={code}
was:
Log files should be parsed using multiple threads, where each thread processes
one file at a time. The number of threads should be dynamically adjusted (can
be given in the command). During the parsing process, batch updates should be
performed on the database. It should be ensured that only one thread performs
the database updates at a time while other threads are blocked during this
operation.
Also create a ozone debug CLI command for the same.
{code:java}
./ozone debug container container_log_parse --parse=
--thread-count={code}
> Multi-threaded Log File Parsing with Batch Updates to DB
>
>
> Key: HDDS-12581
> URL: https://issues.apache.org/jira/browse/HDDS-12581
> Project: Apache Ozone
> Issue Type: Sub-task
>Reporter: Sreeja
>Assignee: Sreeja
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.1.0
>
>
> Log files should be parsed using multiple threads, where each thread
> processes one file at a time. The number of threads should be dynamically
> adjusted (can be given in the command). During the parsing process, batch
> updates should be performed on the database. It should be ensured that only
> one thread performs the database updates at a time while other threads are
> blocked during this operation.
> Populates two tables: DatanodeContainerLogTable(acts like a detail log
> history) and ContainerLogTable(stores latest state and BCSID for each
> container-datanode pair)
> CLI command of the form:
> {code:java}
> ozone debug log container --db= parse --path=
> --thread-count={code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-12581) Multi-threaded Log File Parsing with Batch Updates to DB
[ https://issues.apache.org/jira/browse/HDDS-12581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-12581: -- Labels: pull-request-available (was: ) > Multi-threaded Log File Parsing with Batch Updates to DB > > > Key: HDDS-12581 > URL: https://issues.apache.org/jira/browse/HDDS-12581 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > Labels: pull-request-available > > Log files should be parsed using multiple threads, where each thread > processes one file at a time. The number of threads should be dynamically > adjusted based on the total number of files, with a maximum limit to optimize > resource usage. During the parsing process, batch updates should be performed > on the database. It should be ensured that only one thread performs the > database updates at a time while other threads are blocked during this > operation. > Also create a ozone debug CLI command for the same. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-12581) Multi-threaded Log File Parsing with Batch Updates to DB
[
https://issues.apache.org/jira/browse/HDDS-12581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreeja updated HDDS-12581:
--
Description:
Log files should be parsed using multiple threads, where each thread processes
one file at a time. The number of threads should be dynamically adjusted (can
be given in the command). During the parsing process, batch updates should be
performed on the database. It should be ensured that only one thread performs
the database updates at a time while other threads are blocked during this
operation.
Also create a ozone debug CLI command for the same.
{code:java}
./ozone debug container container_log_parse --parse=
--thread-count={code}
was:
Log files should be parsed using multiple threads, where each thread processes
one file at a time. The number of threads should be dynamically adjusted (can
be given in the command). During the parsing process, batch updates should be
performed on the database. It should be ensured that only one thread performs
the database updates at a time while other threads are blocked during this
operation.
Also create a ozone debug CLI command for the same.
> Multi-threaded Log File Parsing with Batch Updates to DB
>
>
> Key: HDDS-12581
> URL: https://issues.apache.org/jira/browse/HDDS-12581
> Project: Apache Ozone
> Issue Type: Sub-task
>Reporter: Sreeja
>Assignee: Sreeja
>Priority: Major
> Labels: pull-request-available
>
> Log files should be parsed using multiple threads, where each thread
> processes one file at a time. The number of threads should be dynamically
> adjusted (can be given in the command). During the parsing process, batch
> updates should be performed on the database. It should be ensured that only
> one thread performs the database updates at a time while other threads are
> blocked during this operation.
> Also create a ozone debug CLI command for the same.
> {code:java}
> ./ozone debug container container_log_parse --parse=
> --thread-count={code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-12581) Multi-threaded Log File Parsing with Batch Updates to DB
[ https://issues.apache.org/jira/browse/HDDS-12581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-12581: -- Description: Log files should be parsed using multiple threads, where each thread processes one file at a time. The number of threads should be dynamically adjusted (can be given in the command). During the parsing process, batch updates should be performed on the database. It should be ensured that only one thread performs the database updates at a time while other threads are blocked during this operation. Also create a ozone debug CLI command for the same. was: Log files should be parsed using multiple threads, where each thread processes one file at a time. The number of threads should be dynamically adjusted based on the total number of files, with a maximum limit to optimize resource usage. During the parsing process, batch updates should be performed on the database. It should be ensured that only one thread performs the database updates at a time while other threads are blocked during this operation. Also create a ozone debug CLI command for the same. > Multi-threaded Log File Parsing with Batch Updates to DB > > > Key: HDDS-12581 > URL: https://issues.apache.org/jira/browse/HDDS-12581 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > Labels: pull-request-available > > Log files should be parsed using multiple threads, where each thread > processes one file at a time. The number of threads should be dynamically > adjusted (can be given in the command). During the parsing process, batch > updates should be performed on the database. It should be ensured that only > one thread performs the database updates at a time while other threads are > blocked during this operation. > Also create a ozone debug CLI command for the same. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-12581) Multi-threaded Log File Parsing with Batch Updates to DB
[ https://issues.apache.org/jira/browse/HDDS-12581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-12581: -- Description: Log files should be parsed using multiple threads, where each thread processes one file at a time. The number of threads should be dynamically adjusted based on the total number of files, with a maximum limit to optimize resource usage. During the parsing process, batch updates should be performed on the database. It should be ensured that only one thread performs the database updates at a time while other threads are blocked during this operation. Also create a ozone debug CLI command for the same. was:Log files should be parsed using multiple threads, where each thread processes one file at a time. The number of threads should be dynamically adjusted based on the total number of files, with a maximum limit to optimize resource usage. During the parsing process, batch updates should be performed on the database. It should be ensured that only one thread performs the database updates at a time while other threads are blocked during this operation. > Multi-threaded Log File Parsing with Batch Updates to DB > > > Key: HDDS-12581 > URL: https://issues.apache.org/jira/browse/HDDS-12581 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Log files should be parsed using multiple threads, where each thread > processes one file at a time. The number of threads should be dynamically > adjusted based on the total number of files, with a maximum limit to optimize > resource usage. During the parsing process, batch updates should be performed > on the database. It should be ensured that only one thread performs the > database updates at a time while other threads are blocked during this > operation. > Also create a ozone debug CLI command for the same. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
