[
https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Vishwakarma updated PHOENIX-4625:
---------------------------------------
Description:
We have two different code path
# In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the
following checks if renew lease feature is supported and if the renew lease
config is enabled supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE)
&& renewLeaseEnabled
# In PhoenixConnection for every scan iterator is added to a Queue for lease
renewal based on just the check if the renew lease feature is supported
services.supportsFeature(Feature.RENEW_LEASE)
In PhoenixConnection we however miss the check whether renew lease config is
enabled (phoenix.scanner.lease.renew.enabled)
Now consider a situation where Renew lease feature is supported but
phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In this
case PhoenixConnection will keep adding the iterators for every scan into the
scannerQueue for renewal based on the feature supported check but the renewal
task is not running because phoenix.scanner.lease.renew.enabled is set to
false, so the scannerQueue will keep growing as long as the PhoenixConnection
is alive and multiple scans requests are coming on this connection.
We have a use case that uses a single PhoenixConnection that is perpetual and
does billions of scans on this connection. In this case scannerQueue is growing
to several GB's and ultimately leading to Consecutive Full GC's/OOM
Add iterators for Lease renewal in PhoenixConnection
=============================================
{code}
public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
if (services.supportsFeature(Feature.RENEW_LEASE))
{ checkNotNull(itr); scannerQueue.add(new
WeakReference<TableResultIterator>(itr)); }
}
{code}
Starting the RenewLeaseTask
=============================
checks if Feature.RENEW_LEASE is supported and if
phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
{code}
ConnectionQueryServicesImpl {
....
this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED,
DEFAULT_RENEW_LEASE_ENABLED);
.....
@Override
public boolean isRenewingLeasesEnabled()
{ return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) &&
renewLeaseEnabled; }
private void scheduleRenewLeaseTasks() {
if (isRenewingLeasesEnabled()) {
renewLeaseExecutor =
Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadFactory);
for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q :
connectionQueues)
{ renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0,
renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); }
}
}
.......
}
{code}
To solve this We must add both checks in PhoenixConnection if the feature is
supported and if the config is enabled before adding the iterators to
scannerQueue
ConnectionQueryServices.Feature.RENEW_LEASE is true &&
phoenix.scanner.lease.renew.enabled is true
instead of just checking if the feature
ConnectionQueryServices.Feature.RENEW_LEASE is supported
was:
We have two different code path
# In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the
following checks if renew lease feature is supported and if the renew lease
config is enabled supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE)
&& renewLeaseEnabled
# In PhoenixConnection for every scan iterator is added to a Queue for lease
renewal based on just the check if the renew lease feature is supported
services.supportsFeature(Feature.RENEW_LEASE)
In PhoenixConnection we however miss the check whether renew lease config is
enabled (phoenix.scanner.lease.renew.enabled)
Now consider a situation where Renew lease feature is supported but
phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In this
case PhoenixConnection will keep adding the iterators for every scan into the
scannerQueue for renewal based on the feature supported check but the renewal
task is not running because phoenix.scanner.lease.renew.enabled is set to
false, so the scannerQueue will keep growing as long as the PhoenixConnection
is alive and multiple scans requests are coming on this connection.
We have a use case that uses a single PhoenixConnection that is perpetual and
does billions of scans on this connection. In this case scannerQueue is growing
to several GB's and ultimately leading to Consecutive Full GC's/OOM
Add iterators for Lease renewal in PhoenixConnection
=============================================
public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
if (services.supportsFeature(Feature.RENEW_LEASE)) {
checkNotNull(itr);
scannerQueue.add(new WeakReference<TableResultIterator>(itr));
}
}
Starting the RenewLeaseTask
=============================
checks if Feature.RENEW_LEASE is supported and if
phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
ConnectionQueryServicesImpl {
....
this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED,
DEFAULT_RENEW_LEASE_ENABLED);
.....
@Override
public boolean isRenewingLeasesEnabled() {
return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) &&
renewLeaseEnabled;
}
private void scheduleRenewLeaseTasks() {
if (isRenewingLeasesEnabled()) {
renewLeaseExecutor =
Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadFactory);
for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q :
connectionQueues) {
renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0,
renewLeaseTaskFrequency, TimeUnit.MILLISECONDS);
}
}
}
.......
}
To solve this We must add both checks in PhoenixConnection if the feature is
supported and if the config is enabled before adding the iterators to
scannerQueue
ConnectionQueryServices.Feature.RENEW_LEASE is true &&
phoenix.scanner.lease.renew.enabled is true
instead of just checking if the feature
ConnectionQueryServices.Feature.RENEW_LEASE is supported
> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -----------------------------------------------------------------------------
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.13.0
> Reporter: Vikas Vishwakarma
> Priority: Major
> Attachments: QS.png
>
>
> We have two different code path
> # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the
> following checks if renew lease feature is supported and if the renew lease
> config is enabled
> supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) &&
> renewLeaseEnabled
> # In PhoenixConnection for every scan iterator is added to a Queue for lease
> renewal based on just the check if the renew lease feature is supported
> services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is
> enabled (phoenix.scanner.lease.renew.enabled)
>
> Now consider a situation where Renew lease feature is supported but
> phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In
> this case PhoenixConnection will keep adding the iterators for every scan
> into the scannerQueue for renewal based on the feature supported check but
> the renewal task is not running because phoenix.scanner.lease.renew.enabled
> is set to false, so the scannerQueue will keep growing as long as the
> PhoenixConnection is alive and multiple scans requests are coming on this
> connection.
>
> We have a use case that uses a single PhoenixConnection that is perpetual and
> does billions of scans on this connection. In this case scannerQueue is
> growing to several GB's and ultimately leading to Consecutive Full GC's/OOM
>
> Add iterators for Lease renewal in PhoenixConnection
> =============================================
> {code}
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
> if (services.supportsFeature(Feature.RENEW_LEASE))
> { checkNotNull(itr); scannerQueue.add(new
> WeakReference<TableResultIterator>(itr)); }
> }
> {code}
>
> Starting the RenewLeaseTask
> =============================
> checks if Feature.RENEW_LEASE is supported and if
> phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask
> {code}
> ConnectionQueryServicesImpl {
> ....
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED,
> DEFAULT_RENEW_LEASE_ENABLED);
> .....
> @Override
> public boolean isRenewingLeasesEnabled()
> { return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) &&
> renewLeaseEnabled; }
> private void scheduleRenewLeaseTasks() {
> if (isRenewingLeasesEnabled()) {
> renewLeaseExecutor =
> Executors.newScheduledThreadPool(renewLeasePoolSize,
> renewLeaseThreadFactory);
> for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q :
> connectionQueues)
> { renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0,
> renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); }
> }
> }
> .......
> }
> {code}
> To solve this We must add both checks in PhoenixConnection if the feature is
> supported and if the config is enabled before adding the iterators to
> scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true &&
> phoenix.scanner.lease.renew.enabled is true
> instead of just checking if the feature
> ConnectionQueryServices.Feature.RENEW_LEASE is supported
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
