I can already see that in my attempt at reproducing with sqllite, all queries are using the same connection, despite sett max_connections to 5.
On Friday, July 21, 2023 at 12:42:42 PM UTC-4 Michael Scrivo wrote: > Thank you, that connection logging tip is very helpful! > > On Friday, July 21, 2023 at 12:27:11 PM UTC-4 Jeremy Evans wrote: > >> On Fri, Jul 21, 2023 at 8:58 AM 'Michael Scrivo' via sequel-talk < >> [email protected]> wrote: >> >>> Hey Jeremy, >>> >>> We currently use Sequel with Sharded Connection Pooling (5-10 threads >>> usually) in our codebase, connecting to postgres instances, along with some >>> select usages of Async blocks using the Async gem. All is working great, >>> but recently I wanted to try enabling the fiber_concurrency extension to >>> see if that would squeeze out any additional performance gains. As soon as >>> I enabled it, some of our tests broke, specifically in scenarios where an >>> Async block was used in the context of a model. I started seeing errors >>> like this: >>> >>> 0.0s warn: Async::Task [oid=0x27ee8] [ec=0x27efc] [pid=4388] >>> [2023-07-21 08:48:12 -0700] >>> | Task may have ended with unhandled exception. >>> | NoMethodError: undefined method `foo' for nil:NilClass >>> >>> where foo was a method on some model called within an Async block. The >>> really strange thing, in one case, is that it only threw the error if >>> calling refresh on the model right before hand. ie given this test >>> scernario: >>> >>> models = create models >>> do some updates on the models >>> models.each(&refresh) >>> Model.some_function_with_async_block(models) >>> >>> dies with above exception where foo was called within the async block in >>> some_function_with_async_block. >>> >>> However, if I remove the call to refresh, it all works fine! >>> >>> In another case, there was a a model that had a function with an async >>> block, and in that block it called another model to get some records. With >>> fiber_concurrency on, that dataset was returning nothing when called inside >>> the block. In this case, it didn't need to be in the async block, so moving >>> it outside is a fine solution, but these issues are a bit scary to say the >>> least. >>> >>> Do you have any idea what might be going on here? I've been trying in >>> vain to setup a small reproduction here: >>> https://github.com/mscrivo/sequel-async-bug but have so far been unable >>> to reproduce the exact conditions that trigger the issue in our codebase. >>> I'm not sure if it's pg specific, or some other piece of code in our >>> codebase interacting badly with the fiber_concurrency and async. >>> >> >> The fiber_concurrency extension will result in a separate connection >> per-fiber. If your code is dealing with transactions (and models use >> transactions by default), it's fairly easy to run into situations where the >> connection which ran a query has not committed a transaction, and the fiber >> you are using implicitly via Async uses a different connection and does not >> see the changes. Note that I don't have any experience using Async, so I'm >> not exactly sure what failure conditions are possible with it. >> >> When debugging, I recommend making sure your Database has a logger >> attached and that log_connection_info is true so you can see which >> connection is used for all queries. >> >> If you still need help, please put together a minimal self contained >> example and post it here (not a link to a repository), so I can review. >> >> Thanks, >> Jeremy >> > -- You received this message because you are subscribed to the Google Groups "sequel-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/sequel-talk/fbdf1769-777b-4a64-b44d-7d44b9266388n%40googlegroups.com.
