[ https://issues.apache.org/jira/browse/PHOENIX-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425139#comment-15425139 ]
Hudson commented on PHOENIX-2995: --------------------------------- FAILURE: Integrated in Jenkins build Phoenix-master #1363 (See [https://builds.apache.org/job/Phoenix-master/1363/]) PHOENIX-2995 Write performance severely degrades with large number of (tdsilva: rev 3130fa9919ba937d98853f9be8068a0a8a8a96f5) * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/MetaDataMutated.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/TransactionUtil.java * (edit) phoenix-core/src/test/java/org/apache/phoenix/schema/PMetaDataImplTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java * (add) phoenix-core/src/main/java/org/apache/phoenix/schema/PSynchronizedMetaData.java * (edit) phoenix-core/src/it/java/org/apache/phoenix/end2end/UpsertSelectIT.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/DelegateConnectionQueryServices.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/PMetaData.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/schema/PMetaDataImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/compile/CreateTableCompiler.java > Write performance severely degrades with large number of views > --------------------------------------------------------------- > > Key: PHOENIX-2995 > URL: https://issues.apache.org/jira/browse/PHOENIX-2995 > Project: Phoenix > Issue Type: Bug > Reporter: Mujtaba Chohan > Assignee: Thomas D'Silva > Labels: Argus > Fix For: 4.8.1 > > Attachments: PHOENIX-2995-v2.patch, PHOENIX-2995-v3.patch, > PHOENIX-2995-v4.patch, PHOENIX-2995-v5.patch, PHOENIX-2995-v6.patch, > PHOENIX-2995.patch, create_view_and_upsert.png, image.png, image2.png, > image3.png, upsert_rate.png > > > Write performance for each 1K batch degrades significantly when there are > *10K* views being written in random with default > {{phoenix.client.maxMetaDataCacheSize}}. With all views created, upsert rate > remains around 25 seconds per 1K batch i.e. ~2K rows/min upsert rate. > When {{phoenix.client.maxMetaDataCacheSize}} is increased to 100MB+ then view > does not need to get re-resolved and upsert rate gets back to normal ~60K > rows/min. > With *100K* views and {{phoenix.client.maxMetaDataCacheSize}} set to 1GB, I > wasn't able create all 100K views as upsert time for each 1K batch keeps on > steadily increasing. > Following graph shows 1K batch upsert rate over time with variation of number > of views. Rows are upserted to random views {{CREATE VIEW IF NOT EXISTS ... > APPEND_ONLY_SCHEMA = true, UPDATE_CACHE_FREQUENCY=900000}} is executed before > upsert statement. > !upsert_rate.png! > Base table is also created with {{APPEND_ONLY_SCHEMA = true, > UPDATE_CACHE_FREQUENCY = 900000, AUTO_PARTITION_SEQ}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)