Re: [pool] Resilience against factory outages (POOL-407)
No need to apologize We all have complex system to deal with all over the place. Gary On Fri, May 31, 2024, 5:55 PM Phil Steitz wrote: > Thx and sorry. > > Phil > > On Fri, May 31, 2024 at 1:45 PM Gary D. Gregory > wrote: > > > Hello Phil, > > > > Before you push, run 'mvn' (buy itself), this runs the default goal, > which > > contains all build checks. > > > > If see the code coverage report, regardless of coverage failures, run: > > > > mvn clean site -Dcommons.jacoco.haltOnFailure=false > > > > I'll update the readme... > > > > TY, > > Gary > > > > On 2024/05/31 20:12:09 Phil Steitz wrote: > > > The build worked locally for me fine. I could not get the site to > build. > > > Is there some sequence that I need to use to get the site to build? I > > did > > > run Checksytle and Findbugs separately. What is the test coverage > plugin > > > and how do I run that? > > > > > > Phil > > > > > > On Fri, May 31, 2024 at 11:53 AM Gary Gregory > > > wrote: > > > > > > > Hi Phil, > > > > > > > > Thank you for the note. I'll try to take a look soon. > > > > > > > > The new code causes the build to fail as it looks like not all of it > is > > > > covered by unit tests. > > > > > > > > Gary > > > > > > > > > > > > On Fri, May 31, 2024, 2:29 PM Phil Steitz > > wrote: > > > > > > > > > I just committed a first attempt at providing the above, intended > > as a > > > > fix > > > > > for POOL-407 and a lot of similar issues reported over the years. > > The > > > > > scenario in POOL-407 is common when resource providers (like > > databases) > > > > go > > > > > down: > > > > > > > > > > 1. makeObject requests start to fail and threads line up waiting on > > the > > > > > deque. > > > > > 2. The provider comes back up so makes will succeed again, but the > > > > clients, > > > > > the pool and the factory are all ignorant of this fact, so no > > clients get > > > > > served. > > > > > > > > > > What I just committed puts the resilience responsibility on the > > factory, > > > > > having it monitor itself. That responsibility could arguably be > put > > > > > instead on the pool. > > > > > > > > > > To use the feature as is, you need to create a > > > > ResilientPooledObjectFactory > > > > > wrapping a PooledObjectFactory, configure it, attach it to its pool > > and > > > > > start its monitor. The formerly disabled GOP test, > > > > > testLivenessOnTransientFactoryFailure, shows how to do it. The > > setup is > > > > a > > > > > little awkward. I would appreciate feedback on the following > > options for > > > > > how to improve it (or any other comments on the code): > > > > > > > > > > 0) Roll it back and come up with something better > > > > > 1) Leave as is > > > > > 2) add a GOP config that results in its factory being wrapped > > > > automatically > > > > > in a RPOF. > > > > > 3) move the functionality into the pool > > > > > > > > > > The other thing that needs to be designed is how to make the > > proactive > > > > make > > > > > attempt strategy configurable. It is hard-coded now in the RPOF > > > > runChecks > > > > > and the Adder inner class. The initial implementation is > primitive: > > > > > Monitor the makeObject log. Any failure triggers start of an Adder > > that > > > > > tries addObject with configurable delay and (hard-coded) max > > failures. > > > > > Once the circular log becomes filled with successes, turn the adder > > off. > > > > > > > > > > Also, RPOF spawns a monitoring thread and, when it detects a > > transient > > > > > failure, an adder thread. Careful review - and improvement - of > the > > > > > management of these threads would be appreciated. I tried to make > > sure, > > > > > and added tests to confirm, that closing the pool kills these > > threads. > > > > > > > > > > Phil > > > > > > > > > > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > >
Re: [pool] Resilience against factory outages (POOL-407)
Thx and sorry. Phil On Fri, May 31, 2024 at 1:45 PM Gary D. Gregory wrote: > Hello Phil, > > Before you push, run 'mvn' (buy itself), this runs the default goal, which > contains all build checks. > > If see the code coverage report, regardless of coverage failures, run: > > mvn clean site -Dcommons.jacoco.haltOnFailure=false > > I'll update the readme... > > TY, > Gary > > On 2024/05/31 20:12:09 Phil Steitz wrote: > > The build worked locally for me fine. I could not get the site to build. > > Is there some sequence that I need to use to get the site to build? I > did > > run Checksytle and Findbugs separately. What is the test coverage plugin > > and how do I run that? > > > > Phil > > > > On Fri, May 31, 2024 at 11:53 AM Gary Gregory > > wrote: > > > > > Hi Phil, > > > > > > Thank you for the note. I'll try to take a look soon. > > > > > > The new code causes the build to fail as it looks like not all of it is > > > covered by unit tests. > > > > > > Gary > > > > > > > > > On Fri, May 31, 2024, 2:29 PM Phil Steitz > wrote: > > > > > > > I just committed a first attempt at providing the above, intended > as a > > > fix > > > > for POOL-407 and a lot of similar issues reported over the years. > The > > > > scenario in POOL-407 is common when resource providers (like > databases) > > > go > > > > down: > > > > > > > > 1. makeObject requests start to fail and threads line up waiting on > the > > > > deque. > > > > 2. The provider comes back up so makes will succeed again, but the > > > clients, > > > > the pool and the factory are all ignorant of this fact, so no > clients get > > > > served. > > > > > > > > What I just committed puts the resilience responsibility on the > factory, > > > > having it monitor itself. That responsibility could arguably be put > > > > instead on the pool. > > > > > > > > To use the feature as is, you need to create a > > > ResilientPooledObjectFactory > > > > wrapping a PooledObjectFactory, configure it, attach it to its pool > and > > > > start its monitor. The formerly disabled GOP test, > > > > testLivenessOnTransientFactoryFailure, shows how to do it. The > setup is > > > a > > > > little awkward. I would appreciate feedback on the following > options for > > > > how to improve it (or any other comments on the code): > > > > > > > > 0) Roll it back and come up with something better > > > > 1) Leave as is > > > > 2) add a GOP config that results in its factory being wrapped > > > automatically > > > > in a RPOF. > > > > 3) move the functionality into the pool > > > > > > > > The other thing that needs to be designed is how to make the > proactive > > > make > > > > attempt strategy configurable. It is hard-coded now in the RPOF > > > runChecks > > > > and the Adder inner class. The initial implementation is primitive: > > > > Monitor the makeObject log. Any failure triggers start of an Adder > that > > > > tries addObject with configurable delay and (hard-coded) max > failures. > > > > Once the circular log becomes filled with successes, turn the adder > off. > > > > > > > > Also, RPOF spawns a monitoring thread and, when it detects a > transient > > > > failure, an adder thread. Careful review - and improvement - of the > > > > management of these threads would be appreciated. I tried to make > sure, > > > > and added tests to confirm, that closing the pool kills these > threads. > > > > > > > > Phil > > > > > > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
Re: [pool] Resilience against factory outages (POOL-407)
I updated the README's "Contributing" sections with: + Before you pushing a PR, run `mvn` (by itself), this runs the default goal, which contains all build checks. + To see the code coverage report, regardless of coverage failures, run `mvn clean site -Dcommons.jacoco.haltOnFailure=false` This change will eventually make it other README files. TY, Gary On 2024/05/31 20:44:51 "Gary D. Gregory" wrote: > Hello Phil, > > Before you push, run 'mvn' (buy itself), this runs the default goal, which > contains all build checks. > > If see the code coverage report, regardless of coverage failures, run: > > mvn clean site -Dcommons.jacoco.haltOnFailure=false > > I'll update the readme... > > TY, > Gary > > On 2024/05/31 20:12:09 Phil Steitz wrote: > > The build worked locally for me fine. I could not get the site to build. > > Is there some sequence that I need to use to get the site to build? I did > > run Checksytle and Findbugs separately. What is the test coverage plugin > > and how do I run that? > > > > Phil > > > > On Fri, May 31, 2024 at 11:53 AM Gary Gregory > > wrote: > > > > > Hi Phil, > > > > > > Thank you for the note. I'll try to take a look soon. > > > > > > The new code causes the build to fail as it looks like not all of it is > > > covered by unit tests. > > > > > > Gary > > > > > > > > > On Fri, May 31, 2024, 2:29 PM Phil Steitz wrote: > > > > > > > I just committed a first attempt at providing the above, intended as a > > > fix > > > > for POOL-407 and a lot of similar issues reported over the years. The > > > > scenario in POOL-407 is common when resource providers (like databases) > > > go > > > > down: > > > > > > > > 1. makeObject requests start to fail and threads line up waiting on the > > > > deque. > > > > 2. The provider comes back up so makes will succeed again, but the > > > clients, > > > > the pool and the factory are all ignorant of this fact, so no clients > > > > get > > > > served. > > > > > > > > What I just committed puts the resilience responsibility on the factory, > > > > having it monitor itself. That responsibility could arguably be put > > > > instead on the pool. > > > > > > > > To use the feature as is, you need to create a > > > ResilientPooledObjectFactory > > > > wrapping a PooledObjectFactory, configure it, attach it to its pool and > > > > start its monitor. The formerly disabled GOP test, > > > > testLivenessOnTransientFactoryFailure, shows how to do it. The setup is > > > a > > > > little awkward. I would appreciate feedback on the following options > > > > for > > > > how to improve it (or any other comments on the code): > > > > > > > > 0) Roll it back and come up with something better > > > > 1) Leave as is > > > > 2) add a GOP config that results in its factory being wrapped > > > automatically > > > > in a RPOF. > > > > 3) move the functionality into the pool > > > > > > > > The other thing that needs to be designed is how to make the proactive > > > make > > > > attempt strategy configurable. It is hard-coded now in the RPOF > > > runChecks > > > > and the Adder inner class. The initial implementation is primitive: > > > > Monitor the makeObject log. Any failure triggers start of an Adder that > > > > tries addObject with configurable delay and (hard-coded) max failures. > > > > Once the circular log becomes filled with successes, turn the adder off. > > > > > > > > Also, RPOF spawns a monitoring thread and, when it detects a transient > > > > failure, an adder thread. Careful review - and improvement - of the > > > > management of these threads would be appreciated. I tried to make sure, > > > > and added tests to confirm, that closing the pool kills these threads. > > > > > > > > Phil > > > > > > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [pool] Resilience against factory outages (POOL-407)
Hello Phil, Before you push, run 'mvn' (buy itself), this runs the default goal, which contains all build checks. If see the code coverage report, regardless of coverage failures, run: mvn clean site -Dcommons.jacoco.haltOnFailure=false I'll update the readme... TY, Gary On 2024/05/31 20:12:09 Phil Steitz wrote: > The build worked locally for me fine. I could not get the site to build. > Is there some sequence that I need to use to get the site to build? I did > run Checksytle and Findbugs separately. What is the test coverage plugin > and how do I run that? > > Phil > > On Fri, May 31, 2024 at 11:53 AM Gary Gregory > wrote: > > > Hi Phil, > > > > Thank you for the note. I'll try to take a look soon. > > > > The new code causes the build to fail as it looks like not all of it is > > covered by unit tests. > > > > Gary > > > > > > On Fri, May 31, 2024, 2:29 PM Phil Steitz wrote: > > > > > I just committed a first attempt at providing the above, intended as a > > fix > > > for POOL-407 and a lot of similar issues reported over the years. The > > > scenario in POOL-407 is common when resource providers (like databases) > > go > > > down: > > > > > > 1. makeObject requests start to fail and threads line up waiting on the > > > deque. > > > 2. The provider comes back up so makes will succeed again, but the > > clients, > > > the pool and the factory are all ignorant of this fact, so no clients get > > > served. > > > > > > What I just committed puts the resilience responsibility on the factory, > > > having it monitor itself. That responsibility could arguably be put > > > instead on the pool. > > > > > > To use the feature as is, you need to create a > > ResilientPooledObjectFactory > > > wrapping a PooledObjectFactory, configure it, attach it to its pool and > > > start its monitor. The formerly disabled GOP test, > > > testLivenessOnTransientFactoryFailure, shows how to do it. The setup is > > a > > > little awkward. I would appreciate feedback on the following options for > > > how to improve it (or any other comments on the code): > > > > > > 0) Roll it back and come up with something better > > > 1) Leave as is > > > 2) add a GOP config that results in its factory being wrapped > > automatically > > > in a RPOF. > > > 3) move the functionality into the pool > > > > > > The other thing that needs to be designed is how to make the proactive > > make > > > attempt strategy configurable. It is hard-coded now in the RPOF > > runChecks > > > and the Adder inner class. The initial implementation is primitive: > > > Monitor the makeObject log. Any failure triggers start of an Adder that > > > tries addObject with configurable delay and (hard-coded) max failures. > > > Once the circular log becomes filled with successes, turn the adder off. > > > > > > Also, RPOF spawns a monitoring thread and, when it detects a transient > > > failure, an adder thread. Careful review - and improvement - of the > > > management of these threads would be appreciated. I tried to make sure, > > > and added tests to confirm, that closing the pool kills these threads. > > > > > > Phil > > > > > > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [pool] Resilience against factory outages (POOL-407)
The build worked locally for me fine. I could not get the site to build. Is there some sequence that I need to use to get the site to build? I did run Checksytle and Findbugs separately. What is the test coverage plugin and how do I run that? Phil On Fri, May 31, 2024 at 11:53 AM Gary Gregory wrote: > Hi Phil, > > Thank you for the note. I'll try to take a look soon. > > The new code causes the build to fail as it looks like not all of it is > covered by unit tests. > > Gary > > > On Fri, May 31, 2024, 2:29 PM Phil Steitz wrote: > > > I just committed a first attempt at providing the above, intended as a > fix > > for POOL-407 and a lot of similar issues reported over the years. The > > scenario in POOL-407 is common when resource providers (like databases) > go > > down: > > > > 1. makeObject requests start to fail and threads line up waiting on the > > deque. > > 2. The provider comes back up so makes will succeed again, but the > clients, > > the pool and the factory are all ignorant of this fact, so no clients get > > served. > > > > What I just committed puts the resilience responsibility on the factory, > > having it monitor itself. That responsibility could arguably be put > > instead on the pool. > > > > To use the feature as is, you need to create a > ResilientPooledObjectFactory > > wrapping a PooledObjectFactory, configure it, attach it to its pool and > > start its monitor. The formerly disabled GOP test, > > testLivenessOnTransientFactoryFailure, shows how to do it. The setup is > a > > little awkward. I would appreciate feedback on the following options for > > how to improve it (or any other comments on the code): > > > > 0) Roll it back and come up with something better > > 1) Leave as is > > 2) add a GOP config that results in its factory being wrapped > automatically > > in a RPOF. > > 3) move the functionality into the pool > > > > The other thing that needs to be designed is how to make the proactive > make > > attempt strategy configurable. It is hard-coded now in the RPOF > runChecks > > and the Adder inner class. The initial implementation is primitive: > > Monitor the makeObject log. Any failure triggers start of an Adder that > > tries addObject with configurable delay and (hard-coded) max failures. > > Once the circular log becomes filled with successes, turn the adder off. > > > > Also, RPOF spawns a monitoring thread and, when it detects a transient > > failure, an adder thread. Careful review - and improvement - of the > > management of these threads would be appreciated. I tried to make sure, > > and added tests to confirm, that closing the pool kills these threads. > > > > Phil > > >
Re: [pool] Resilience against factory outages (POOL-407)
Hi Phil, Thank you for the note. I'll try to take a look soon. The new code causes the build to fail as it looks like not all of it is covered by unit tests. Gary On Fri, May 31, 2024, 2:29 PM Phil Steitz wrote: > I just committed a first attempt at providing the above, intended as a fix > for POOL-407 and a lot of similar issues reported over the years. The > scenario in POOL-407 is common when resource providers (like databases) go > down: > > 1. makeObject requests start to fail and threads line up waiting on the > deque. > 2. The provider comes back up so makes will succeed again, but the clients, > the pool and the factory are all ignorant of this fact, so no clients get > served. > > What I just committed puts the resilience responsibility on the factory, > having it monitor itself. That responsibility could arguably be put > instead on the pool. > > To use the feature as is, you need to create a ResilientPooledObjectFactory > wrapping a PooledObjectFactory, configure it, attach it to its pool and > start its monitor. The formerly disabled GOP test, > testLivenessOnTransientFactoryFailure, shows how to do it. The setup is a > little awkward. I would appreciate feedback on the following options for > how to improve it (or any other comments on the code): > > 0) Roll it back and come up with something better > 1) Leave as is > 2) add a GOP config that results in its factory being wrapped automatically > in a RPOF. > 3) move the functionality into the pool > > The other thing that needs to be designed is how to make the proactive make > attempt strategy configurable. It is hard-coded now in the RPOF runChecks > and the Adder inner class. The initial implementation is primitive: > Monitor the makeObject log. Any failure triggers start of an Adder that > tries addObject with configurable delay and (hard-coded) max failures. > Once the circular log becomes filled with successes, turn the adder off. > > Also, RPOF spawns a monitoring thread and, when it detects a transient > failure, an adder thread. Careful review - and improvement - of the > management of these threads would be appreciated. I tried to make sure, > and added tests to confirm, that closing the pool kills these threads. > > Phil >
[pool] Resilience against factory outages (POOL-407)
I just committed a first attempt at providing the above, intended as a fix for POOL-407 and a lot of similar issues reported over the years. The scenario in POOL-407 is common when resource providers (like databases) go down: 1. makeObject requests start to fail and threads line up waiting on the deque. 2. The provider comes back up so makes will succeed again, but the clients, the pool and the factory are all ignorant of this fact, so no clients get served. What I just committed puts the resilience responsibility on the factory, having it monitor itself. That responsibility could arguably be put instead on the pool. To use the feature as is, you need to create a ResilientPooledObjectFactory wrapping a PooledObjectFactory, configure it, attach it to its pool and start its monitor. The formerly disabled GOP test, testLivenessOnTransientFactoryFailure, shows how to do it. The setup is a little awkward. I would appreciate feedback on the following options for how to improve it (or any other comments on the code): 0) Roll it back and come up with something better 1) Leave as is 2) add a GOP config that results in its factory being wrapped automatically in a RPOF. 3) move the functionality into the pool The other thing that needs to be designed is how to make the proactive make attempt strategy configurable. It is hard-coded now in the RPOF runChecks and the Adder inner class. The initial implementation is primitive: Monitor the makeObject log. Any failure triggers start of an Adder that tries addObject with configurable delay and (hard-coded) max failures. Once the circular log becomes filled with successes, turn the adder off. Also, RPOF spawns a monitoring thread and, when it detects a transient failure, an adder thread. Careful review - and improvement - of the management of these threads would be appreciated. I tried to make sure, and added tests to confirm, that closing the pool kills these threads. Phil
