[
https://issues.apache.org/jira/browse/PHOENIX-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15340838#comment-15340838
]
James Taylor commented on PHOENIX-2995:
---------------------------------------
FYI, stats are now cached outside of PTable, so we should be able to cache many
more PTables. I think [~tdsilva] can take a look at this when he's back and we
can optimize things:
- figure out what we think the size of a PTable should be
- potentially have a different cache representation which gets translated into
the full PTable form on the way out
- have a "small memory footprint" option for PTable without any of the maps.
- see above for more ideas (i.e. not storing both byte[] and String
representation, etc.)
- given the particular data set of Argus, there may be a way to cache it much
more efficiently given the commonality across all the metrics. The trick here
would be to do it in a general enough way that it can go into Phoenix.
Also, why does perf drop so dramatically if it's not found in the cache? Is one
additional RPC really account for this?
> Write performance severely degrades with large number of views
> ---------------------------------------------------------------
>
> Key: PHOENIX-2995
> URL: https://issues.apache.org/jira/browse/PHOENIX-2995
> Project: Phoenix
> Issue Type: Bug
> Reporter: Mujtaba Chohan
> Assignee: Thomas D'Silva
> Labels: Argus
> Attachments: upsert_rate.png
>
>
> Write performance for each 1K batch degrades significantly when there are
> *10K* views being written in random with default
> {{phoenix.client.maxMetaDataCacheSize}}. With all views created, upsert rate
> remains around 25 seconds per 1K batch i.e. ~2K rows/min upsert rate.
> When {{phoenix.client.maxMetaDataCacheSize}} is increased to 100MB+ then view
> does not need to get re-resolved and upsert rate gets back to normal ~60K
> rows/min.
> With *100K* views and {{phoenix.client.maxMetaDataCacheSize}} set to 1GB, I
> wasn't able create all 100K views as upsert time for each 1K batch keeps on
> steadily increasing.
> Following graph shows 1K batch upsert rate over time with variation of number
> of views. Rows are upserted to random views {{CREATE VIEW IF NOT EXISTS ...
> APPEND_ONLY_SCHEMA = true, UPDATE_CACHE_FREQUENCY=900000}} is executed before
> upsert statement.
> !upsert_rate.png!
> Base table is also created with {{APPEND_ONLY_SCHEMA = true,
> UPDATE_CACHE_FREQUENCY = 900000, AUTO_PARTITION_SEQ}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
