[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373502#comment-16373502 ]
Hudson commented on PHOENIX-4625: --------------------------------- SUCCESS: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #46 (See [https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/46/]) PHOENIX-4625 memory leak in PhoenixConnection if scanner renew lease (tdsilva: rev cb682c9a19695ed33c7e6c3889c20b4071cfa9e7) * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixConnection.java > memory leak in PhoenixConnection if scanner renew lease thread is not enabled > ----------------------------------------------------------------------------- > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug > Affects Versions: 4.14.0 > Reporter: Vikas Vishwakarma > Assignee: Vikas Vishwakarma > Priority: Major > Fix For: 4.14.0, 5.0.0 > > Attachments: GC_After_fix.png, GC_Leak.png, PHOENIX-4625.patch, QS.png > > > We have two different code path > # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the > following checks if renew lease feature is supported and if the renew lease > config is enabled > supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled > # In PhoenixConnection for every scan iterator is added to a Queue for lease > renewal based on just the check if the renew lease feature is supported > services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config is > enabled (phoenix.scanner.lease.renew.enabled) > > Now consider a situation where Renew lease feature is supported but > phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In > this case PhoenixConnection will keep adding the iterators for every scan > into the scannerQueue for renewal based on the feature supported check but > the renewal task is not running because phoenix.scanner.lease.renew.enabled > is set to false, so the scannerQueue will keep growing as long as the > PhoenixConnection is alive and multiple scans requests are coming on this > connection. > > We have a use case that uses a single PhoenixConnection that is perpetual and > does billions of scans on this connection. In this case scannerQueue is > growing to several GB's and ultimately leading to Consecutive Full GC's/OOM > > Add iterators for Lease renewal in PhoenixConnection > ============================================= > {code:java} > > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { > if (services.supportsFeature(Feature.RENEW_LEASE)) > { > checkNotNull(itr); scannerQueue.add(new > WeakReference<TableResultIterator>(itr)); > } > } > {code} > > Starting the RenewLeaseTask > ============================= > checks if Feature.RENEW_LEASE is supported and if > phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask > {code:java} > > ConnectionQueryServicesImpl { > .... > this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, > DEFAULT_RENEW_LEASE_ENABLED); > ..... > @Override > public boolean isRenewingLeasesEnabled(){ > return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && > renewLeaseEnabled; > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { > renewLeaseExecutor = > Executors.newScheduledThreadPool(renewLeasePoolSize, > renewLeaseThreadFactory); > for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q : > connectionQueues) { > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, > renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); > } > } > } > ....... > } > {code} > > To solve this We must add both checks in PhoenixConnection if the feature is > supported and if the config is enabled before adding the iterators to > scannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true && > phoenix.scanner.lease.renew.enabled is true > instead of just checking if the feature > ConnectionQueryServices.Feature.RENEW_LEASE is supported > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)