[
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372424#comment-16372424
]
James Taylor commented on PHOENIX-4625:
---------------------------------------
Thanks for the excellent analysis, [~vik.karma]. How about a patch?
> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -----------------------------------------------------------------------------
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.13.0
> Reporter: Vikas Vishwakarma
> Priority: Major
> Attachments: QS.png
>
>
> We have two different code path
> # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the
> following checks if renew lease feature is supported and if the renew lease
> config is enabled
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) &&
> renewLeaseEnabled
> # In PhoenixConnection for every scan iterator is added to a Queue for lease
> renewal based on just the check if the renew lease feature is supported
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is
> enabled (phoenix.scanner.lease.renew.enabled)
>
> Now consider a situation where Renew lease feature is supported but
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In
> this case PhoenixConnection will keep adding the iterators for every scan
> into the scannerQueue for renewal based on the feature supported check but
> the renewal task is not running because phoenix.scanner.lease.renew.enabled
> is set to false, so the scannerQueue will keep growing as long as the
> PhoenixConnection is alive and multiple scans requests are coming on this
> connection.
>
> We have a use case that uses a single PhoenixConnection that is perpetual and
> does billions of scans on this connection. In this case scannerQueue is
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>
> Add iterators for Lease renewal in PhoenixConnection
> =============================================
> {code:java}
>
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
> if (services.supportsFeature(Feature.RENEW_LEASE))
> {
> checkNotNull(itr); scannerQueue.add(new
> WeakReference<TableResultIterator>(itr));
> }
> }
> {code}
>
> Starting the RenewLeaseTask
> =============================
> checks if Feature.RENEW_LEASE is supported and if
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code:java}
>
> ConnectionQueryServicesImpl {
> ....
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED,
> DEFAULT_RENEW_LEASE_ENABLED);
> .....
> @Override
> public boolean isRenewingLeasesEnabled(){
> return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) &&
> renewLeaseEnabled;
> }
> private void scheduleRenewLeaseTasks() {
> if (isRenewingLeasesEnabled()) {
> renewLeaseExecutor =
> Executors.newScheduledThreadPool(renewLeasePoolSize,
> renewLeaseThreadFactory);
> for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q :
> connectionQueues) {
> renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0,
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS);
> }
> }
> }
> .......
> }
> {code}
>
> To solve this We must add both checks in PhoenixConnection if the feature is
> supported and if the config is enabled before adding the iterators to
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true &&
> phoenix.scanner.lease.renew.enabled is true
> instead of just checking if the feature
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
