Vikas Vishwakarma created PHOENIX-4625:
------------------------------------------
Summary: memory leak in PhoenixConnection if scanner renew lease
thread is not enabled
Key: PHOENIX-4625
URL: https://issues.apache.org/jira/browse/PHOENIX-4625
Project: Phoenix
Issue Type: Bug
Affects Versions: 4.13.0
Reporter: Vikas Vishwakarma
We have two different code path
# In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the
following checks if renew lease feature is supported and if the renew lease
config is enabled supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE)
&& renewLeaseEnabled
# In PhoenixConnection for every scan iterator is added to a Queue for lease
renewal based on just the check if the renew lease feature is supported
services.supportsFeature(Feature.RENEW_LEASE)
In PhoenixConnection we however miss the check whether renew lease config is
enabled (phoenix.scanner.lease.renew.enabled)
Now consider a situation where Renew lease feature is supported but
phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In this
case PhoenixConnection will keep adding the iterators for every scan into the
scannerQueue for renewal based on the feature supported check but the renewal
task is not running because phoenix.scanner.lease.renew.enabled is set to
false, so the scannerQueue will keep growing as long as the PhoenixConnection
is alive and multiple scans requests are coming on this connection.
We have a use case that uses a single PhoenixConnection that is perpetual and
does billions of scans on this connection. In this case scannerQueue is growing
to several GB's and ultimately leading to Consecutive Full GC's/OOM
Add iterators for Lease renewal in PhoenixConnection
=============================================
public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
if (services.supportsFeature(Feature.RENEW_LEASE)) {
checkNotNull(itr);
scannerQueue.add(new WeakReference<TableResultIterator>(itr));
}
}
Starting the RenewLeaseTask
=============================
checks if Feature.RENEW_LEASE is supported and if
phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
ConnectionQueryServicesImpl {
....
this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED,
DEFAULT_RENEW_LEASE_ENABLED);
.....
@Override
public boolean isRenewingLeasesEnabled() {
return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) &&
renewLeaseEnabled;
}
private void scheduleRenewLeaseTasks() {
if (isRenewingLeasesEnabled()) {
renewLeaseExecutor =
Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadFactory);
for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q :
connectionQueues) {
renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0,
renewLeaseTaskFrequency, TimeUnit.MILLISECONDS);
}
}
}
.......
}
To solve this We must add both checks in PhoenixConnection if the feature is
supported and if the config is enabled before adding the iterators to
scannerQueue
ConnectionQueryServices.Feature.RENEW_LEASE is true &&
phoenix.scanner.lease.renew.enabled is true
instead of just checking if the feature
ConnectionQueryServices.Feature.RENEW_LEASE is supported
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
