[jira] [Commented] (HIVE-26898) Split Notification logging so that busy clusters can have better performance
[ https://issues.apache.org/jira/browse/HIVE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682238#comment-17682238 ] Riju Trivedi commented on HIVE-26898: - [~ayushtkn] Thanks for pointing this out. This comment was meant for another internal jira, mistakenly posted here. > Split Notification logging so that busy clusters can have better performance > > > Key: HIVE-26898 > URL: https://issues.apache.org/jira/browse/HIVE-26898 > Project: Hive > Issue Type: New Feature >Reporter: Taraka Rama Rao Lethavadla >Priority: Major > > For DDL & DML events are logged into notifications log table and those get > cleaned as soon as ttl got expired. > In most of the busy clusters, the notification log is growing even though > cleaner is running and kept on cleaning the events. It means the rate of Hive > db operations are very high compared to rate at which cleaning is happening. > So any query on this table is becoming bottle neck at backend DB causing slow > response > The proposal is to split the notification log table in to multiple tables > like > notification_log_dml - for all DML queries > notification_log_insert - for all insert queries > .. > etc. > > So that load on that single table gets reduced improving the performance of > the backend db as well as Hive -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26898) Split Notification logging so that busy clusters can have better performance
[
https://issues.apache.org/jira/browse/HIVE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682236#comment-17682236
]
Ayush Saxena commented on HIVE-26898:
-
{quote}OOM issues In recent versions 717, SP1 and SP2?
{quote}
There is no version like this, these are the only versions that we have here:
[https://hive.apache.org/general/downloads/]
> Split Notification logging so that busy clusters can have better performance
>
>
> Key: HIVE-26898
> URL: https://issues.apache.org/jira/browse/HIVE-26898
> Project: Hive
> Issue Type: New Feature
>Reporter: Taraka Rama Rao Lethavadla
>Priority: Major
>
> For DDL & DML events are logged into notifications log table and those get
> cleaned as soon as ttl got expired.
> In most of the busy clusters, the notification log is growing even though
> cleaner is running and kept on cleaning the events. It means the rate of Hive
> db operations are very high compared to rate at which cleaning is happening.
> So any query on this table is becoming bottle neck at backend DB causing slow
> response
> The proposal is to split the notification log table in to multiple tables
> like
> notification_log_dml - for all DML queries
> notification_log_insert - for all insert queries
> ..
> etc.
>
> So that load on that single table gets reduced improving the performance of
> the backend db as well as Hive
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26898) Split Notification logging so that busy clusters can have better performance
[ https://issues.apache.org/jira/browse/HIVE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682210#comment-17682210 ] Riju Trivedi commented on HIVE-26898: - [~tarak271] Do we see this notification_log table growing and OOM issues In recent versions 717, SP1 and SP2? Improvements like loading notifications in batches (HIVE-19430) and handling batches in different transactions (HIVE-24432) are already included in these versions. > Split Notification logging so that busy clusters can have better performance > > > Key: HIVE-26898 > URL: https://issues.apache.org/jira/browse/HIVE-26898 > Project: Hive > Issue Type: New Feature >Reporter: Taraka Rama Rao Lethavadla >Priority: Major > > For DDL & DML events are logged into notifications log table and those get > cleaned as soon as ttl got expired. > In most of the busy clusters, the notification log is growing even though > cleaner is running and kept on cleaning the events. It means the rate of Hive > db operations are very high compared to rate at which cleaning is happening. > So any query on this table is becoming bottle neck at backend DB causing slow > response > The proposal is to split the notification log table in to multiple tables > like > notification_log_dml - for all DML queries > notification_log_insert - for all insert queries > .. > etc. > > So that load on that single table gets reduced improving the performance of > the backend db as well as Hive -- This message was sent by Atlassian Jira (v8.20.10#820010)
