[ https://issues.apache.org/jira/browse/PHOENIX-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330919#comment-15330919 ]
Mujtaba Chohan commented on PHOENIX-2995: ----------------------------------------- DDL {code} CREATE TABLE IF NOT EXISTS TEST.METRICS (id INTEGER NOT NULL, id2 INTEGER NOT NULL, val INTEGER, display_name varchar, units varchar CONSTRAINT PK PRIMARY KEY(id,id2)) APPEND_ONLY_SCHEMA = true, UPDATE_CACHE_FREQUENCY = 900000, AUTO_PARTITION_SEQ=TEST_ID_SEQ CREATE VIEW IF NOT EXISTS TEST.<26_char_unique_name> (MYCOL INTEGER) AS SELECT * FROM TEST.METRICS APPEND_ONLY_SCHEMA = true, UPDATE_CACHE_FREQUENCY=900000 {code} >From jvisualvm, it looks like ~65% of 11K is used by stats. I'll get more >details on it. >From talking to Bhinav, millions of view are possible and for acceptable cache >hit rate, say if we are caching 1/10 of those still that would take beyond >100-200k cached metadata views range. > Write performance severely degrades with large number of views > --------------------------------------------------------------- > > Key: PHOENIX-2995 > URL: https://issues.apache.org/jira/browse/PHOENIX-2995 > Project: Phoenix > Issue Type: Bug > Reporter: Mujtaba Chohan > Assignee: James Taylor > Labels: Argus > Attachments: upsert_rate.png > > > Write performance for each 1K batch degrades significantly when there are > *10K* views being written in random with default > {{phoenix.client.maxMetaDataCacheSize}}. With all views created, upsert rate > remains around 25 seconds per 1K batch i.e. ~2K rows/min upsert rate. > When {{phoenix.client.maxMetaDataCacheSize}} is increased to 100MB+ then view > does not need to get re-resolved and upsert rate gets back to normal ~60K > rows/min. > With *100K* views and {{phoenix.client.maxMetaDataCacheSize}} set to 1GB, I > wasn't able create all 100K views as upsert time for each 1K batch keeps on > steadily increasing. > Following graph shows 1K batch upsert rate over time with variation of number > of views. Rows are upserted to random views {{CREATE VIEW IF NOT EXISTS ... > APPEND_ONLY_SCHEMA = true, UPDATE_CACHE_FREQUENCY=900000}} is executed before > upsert statement. > !upsert_rate.png! > Base table is also created with {{APPEND_ONLY_SCHEMA = true, > UPDATE_CACHE_FREQUENCY = 900000, AUTO_PARTITION_SEQ}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)