[ https://issues.apache.org/jira/browse/HIVE-11444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956006#comment-15956006 ]
Eugene Koifman commented on HIVE-11444: --------------------------------------- More generally, raise alert 1. if there are too many open txns 2. if there are too many aborted txns - most likely a misconfigured streaming ingest client. Need to include client info in the alert. 3. if there are a lot of entries in TXN_COMPONENTS - means compactor is not keeping up In extreme cases both can cause the amount of metadata to slow down the metastore operations (TxnHandler/CompactionTxnHandler) a use very large amounts of RAM (ValidTxnList) > ACID Compactor should generate stats/alerts > ------------------------------------------- > > Key: HIVE-11444 > URL: https://issues.apache.org/jira/browse/HIVE-11444 > Project: Hive > Issue Type: Improvement > Components: Transactions > Affects Versions: 1.0.0 > Reporter: Eugene Koifman > Assignee: Eugene Koifman > > Compaction should generate stats about number of files it reads, min/max/avg > size etc. It should also generate alerts if it looks like the system is not > configured correctly. > For example, if there are lots of delta files with very small files, it's a > good sign that Streaming API is configured with batches that are too small. > Simplest idea is to add another periodic task to AcidHouseKeeperService to > //periodically do select count(*), min(txnid),max(txnid), type from > txns group by type. > //1. dump that to log file at info > //2. could also keep counts for last 10min, hour, 6 hours, 24 hours, > etc > //2.2 if a large increase is detected - issue alert (at least to the > log for now) at warn/error > Should also alert if there is ACID activity but no compactions running. > One way to do this is to add logic to TxnHandler to periodically check > contents of COMPACTION_QUEUE table and keep a simple histogram of > compactions over last few hours. > Similarly can run a periodic check of transactions started (or > committed/aborted) and keep a simple histogram. Then the 2 can be used to > detect that there is ACID write activity but no compaction activity. -- This message was sent by Atlassian JIRA (v6.3.15#6346)