I've opened https://github.com/sqlalchemy/sqlalchemy/issues/9777 for this.
I have another theory where something might be going wrong earler in the chain.
This change would be "simpler" in that it removes something that's
complicated, prone to race conditions hence seems a good possibility of being
involved here. This would be the patch:
diff --git a/lib/sqlalchemy/orm/strategies.py b/lib/sqlalchemy/orm/strategies.py
index 8e06c4f598..bd0193b905 100644
--- a/lib/sqlalchemy/orm/strategies.py
+++ b/lib/sqlalchemy/orm/strategies.py
@@ -2324,8 +2324,7 @@ class JoinedLoader(AbstractRelationshipLoader):
else:
context.attributes[key] = idx = context.attributes[key] + 1
- if idx >= len(self._aliased_class_pool):
- to_adapt = orm_util.AliasedClass(
+ return orm_util.AliasedClass(
self.mapper,
alias=alt_selectable._anonymous_fromclause(flat=True)
if alt_selectable is not None
@@ -2334,14 +2333,6 @@ class JoinedLoader(AbstractRelationshipLoader):
use_mapper_path=True,
)
- # load up the .columns collection on the Alias() before
- # the object becomes shared among threads. this prevents
- # races for column identities.
- inspect(to_adapt).selectable.c
- self._aliased_class_pool.append(to_adapt)
-
- return self._aliased_class_pool[idx]
-
def _generate_row_adapter(
self,
compile_state,
On Fri, May 12, 2023, at 11:05 PM, Shane Holcombe wrote:
> There seems to be a few moving parts to this so it's been hard to track down,
> I can give that fix a try in a few hours when I'm home.
> I was worried something like that might lose some columns but I'll try and
> see what happens.
> Thanks
> On Saturday, May 13, 2023 at 12:59:33 PM UTC+10 Mike Bayer wrote:
>> __
>> well, not sure yet how a None is coming in there that is sporadic, or even
>> at all, but if that's what's happening then we would think this patch would
>> fix the problem, the Q is does the query still produce the correct results:
>>
>> diff --git a/lib/sqlalchemy/orm/strategies.py
>> b/lib/sqlalchemy/orm/strategies.py
>> index 8e06c4f598..2bac1aad48 100644
>> --- a/lib/sqlalchemy/orm/strategies.py
>> +++ b/lib/sqlalchemy/orm/strategies.py
>> @@ -218,10 +218,10 @@ class ColumnLoader(LoaderStrategy):
>> if adapter:
>> if check_for_adapt:
>> c = adapter.adapt_check_present(c)
>> - if c is None:
>> - return
>> else:
>> c = adapter.columns[c]
>> + if c is None:
>> + return
>>
>> compile_state._append_dedupe_col_collection(c,
>> column_collection)
>>
>>
>>
>> can you try that patch ?
>>
>>
>> On Fri, May 12, 2023, at 10:47 PM, Mike Bayer wrote:
>>> OK, maybe you are onto something with the theory re:
>>> JoinedLoader._generate_row_adapter(). will look at that
>>>
>>> On Fri, May 12, 2023, at 6:16 PM, Shane Holcombe wrote:
>>>> Thanks for linking that github issue, completely missed that one.
>>>>
>>>> Here's the complete stack trace, sorry for not including that at the start
>>>>
>>>> Traceback (most recent call last):
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/werkzeug/serving.py",
>>>> line 333, in run_wsgi
>>>> execute(self.server.app)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/werkzeug/serving.py",
>>>> line 320, in execute
>>>> application_iter = app(environ, start_response)
>>>> File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py",
>>>> line 2551, in __call__
>>>> return self.wsgi_app(environ, start_response)
>>>> File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py",
>>>> line 2531, in wsgi_app
>>>> response = self.handle_exception(e)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_cors/extension.py",
>>>> line 165, in wrapped_function
>>>> return cors_after_request(app.make_response(f(*args, **kwargs)))
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", line
>>>> 673, in error_router
>>>> return original_handler(f)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", line
>>>> 671, in error_router
>>>> return self.handle_error(e)
>>>> File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py",
>>>> line 2528, in wsgi_app
>>>> response = self.full_dispatch_request()
>>>> File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py",
>>>> line 1825, in full_dispatch_request
>>>> rv = self.handle_user_exception(e)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_cors/extension.py",
>>>> line 165, in wrapped_function
>>>> return cors_after_request(app.make_response(f(*args, **kwargs)))
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", line
>>>> 673, in error_router
>>>> return original_handler(f)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", line
>>>> 671, in error_router
>>>> return self.handle_error(e)
>>>> File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py",
>>>> line 1823, in full_dispatch_request
>>>> rv = self.dispatch_request()
>>>> File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/app.py",
>>>> line 1799, in dispatch_request
>>>> return
>>>> self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/api.py", line
>>>> 404, in wrapper
>>>> resp = resource(*args, **kwargs)
>>>> File "/Users/sfh/env/vita/lib/python3.9/site-packages/flask/views.py",
>>>> line 107, in view
>>>> return current_app.ensure_sync(self.dispatch_request)(**kwargs)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_restx/resource.py",
>>>> line 46, in dispatch_request
>>>> resp = meth(*args, **kwargs)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/flask_cors/decorator.py",
>>>> line 128, in wrapped_function
>>>> resp = make_response(f(*args, **kwargs))
>>>> File "/Users/sfh/PycharmProjects/vita/stockholm/geneva/api/auth.py",
>>>> line 151, in decorated
>>>> return f(*args, **kwargs)
>>>> File
>>>> "/Users/sfh/PycharmProjects/vita/stockholm/geneva/api/sales/lead_api.py",
>>>> line 116, in get
>>>> lead = lead_factory.extended_lead_for_id(lead_id)
>>>> File
>>>> "/Users/sfh/PycharmProjects/vita/stockholm/geneva/sales/lead_factory.py",
>>>> line 59, in extended_lead_for_id
>>>> return db.session.execute(stmt).scalar_one_or_none()
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/orm/scoping.py",
>>>> line 724, in execute
>>>> return self._proxied.execute(
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
>>>> line 2232, in execute
>>>> return self._execute_internal(
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
>>>> line 2127, in _execute_internal
>>>> result: Result[Any] = compile_state_cls.orm_execute_statement(
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/orm/context.py",
>>>> line 292, in orm_execute_statement
>>>> result = conn.execute(
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
>>>> line 1413, in execute
>>>> return meth(
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/elements.py",
>>>> line 483, in _execute_on_connection
>>>> return connection._execute_clauseelement(
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
>>>> line 1629, in _execute_clauseelement
>>>> compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/elements.py",
>>>> line 671, in _compile_w_cache
>>>> compiled_sql = self._compiler(
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/elements.py",
>>>> line 288, in _compiler
>>>> return dialect.statement_compiler(dialect, self, **kw)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py",
>>>> line 1426, in __init__
>>>> Compiled.__init__(self, dialect, statement, **kwargs)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py",
>>>> line 867, in __init__
>>>> self.string = self.process(self.statement, **compile_kwargs)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py",
>>>> line 912, in process
>>>> return obj._compiler_dispatch(self, **kwargs)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py",
>>>> line 143, in _compiler_dispatch
>>>> return meth(self, **kw) # type: ignore # noqa: E501
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py",
>>>> line 4593, in visit_select
>>>> compile_state = select_stmt._compile_state_factory(
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/base.py",
>>>> line 678, in create_for_statement
>>>> return klass.create_for_statement(statement, compiler, **kw)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/orm/context.py",
>>>> line 1101, in create_for_statement
>>>> SelectState.__init__(self, self.statement, compiler, **kw)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py",
>>>> line 4501, in __init__
>>>> self.froms = self._get_froms(statement)
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py",
>>>> line 4599, in _get_froms
>>>> [
>>>> File
>>>> "/Users/sfh/env/vita/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py",
>>>> line 4600, in <listcomp>
>>>> element._from_objects
>>>> AttributeError: 'NoneType' object has no attribute '_from_objects'
>>>> In this case the statement we use, extended_lead_for_id, is
>>>>
>>>> from geneva.models import person_factory
>>>> stmt = select(Lead).where(Lead.id == lead_id).options(
>>>> joinedload(Lead.lead_person_client),
>>>> joinedload(Lead.lead_person_primary_contact),
>>>> joinedload(Lead.person),
>>>> joinedload(Lead.person).joinedload(person_factory.Person.contract),
>>>> joinedload(Lead.assigned_to_application_user),
>>>> joinedload(Lead.in_progress_by_application_user),
>>>> joinedload(Lead.closed_by_application_user),
>>>> joinedload(Lead.lu_client_type),
>>>> joinedload(Lead.lu_lead_origin_type),
>>>> joinedload(Lead.lu_lead_status),
>>>> joinedload(Lead.lu_lead_closed_state),
>>>> joinedload(Lead.lu_lead_lost_reason),
>>>> joinedload(Lead.lu_marketing_channel),
>>>> joinedload(Lead.lu_marketing_channel_digital),
>>>> joinedload(Lead.lu_marketing_channel_on_site),
>>>> joinedload(Lead.lu_marketing_channel_out_of_home),
>>>> joinedload(Lead.lu_marketing_channel_print),
>>>> joinedload(Lead.lu_marketing_channel_radio),
>>>> joinedload(Lead.lu_marketing_channel_referral),
>>>> joinedload(Lead.lu_marketing_channel_tv),
>>>> joinedload(Lead.lu_marketing_channel_paid_directory),
>>>> joinedload(Lead.lu_marketing_channel_direct_mail),
>>>> joinedload(Lead.referral_supplier),
>>>> joinedload(Lead.council),
>>>> joinedload(Lead.hospital),
>>>> )
>>>> return db.session.execute(stmt).scalar_one_or_none()
>>>>
>>>> As for the models, we don't really use any default loaders on relationship
>>>> creation, we've tended to avoid that and put the joins in the select
>>>> statements we create
>>>> If you really need to see a particular model from this let me know.
>>>>
>>>> Thanks again for the help
>>>> On Saturday, May 13, 2023 at 1:28:27 AM UTC+10 Mike Bayer wrote:
>>>>> __
>>>>> I really need to see:
>>>>>
>>>>> 1. actual models to the degree they illustrate the default loaders set up
>>>>> on relationships
>>>>> 2. query in use
>>>>> 3. **complete** stack trace, put it on a github gist if it's too long.
>>>>> every line, the error, top to bottom. thanks
>>>>>
>>>>> On Fri, May 12, 2023, at 11:25 AM, Mike Bayer wrote:
>>>>>>
>>>>>>
>>>>>> On Fri, May 12, 2023, at 1:25 AM, Shane Holcombe wrote:
>>>>>>> AttributeError: 'NoneType' object has no attribute '_from_objects'
>>>>>>> (I've shortened this from the entire traceback)
>>>>>>
>>>>>> OK what kind of traceback, is it a recursion overflow kind of traceback
>>>>>> ? (e.g. many hundreds of repeated series of lines)? that would
>>>>>> explain the nonsensical error at least (there's no way for None to be in
>>>>>> the place that this error reports it)
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> SQLAlchemy -
>>>>>> The Python SQL Toolkit and Object Relational Mapper
>>>>>>
>>>>>> http://www.sqlalchemy.org/
>>>>>>
>>>>>> To post example code, please provide an MCVE: Minimal, Complete, and
>>>>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
>>>>>> description.
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "sqlalchemy" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> an email to sqlalchemy+...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/sqlalchemy/9762bf53-8443-4780-9091-136863716290%40app.fastmail.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/sqlalchemy/9762bf53-8443-4780-9091-136863716290%40app.fastmail.com?utm_medium=email&utm_source=footer>.
>>>>>
>>>>
>>>>
>>>> --
>>>> SQLAlchemy -
>>>> The Python SQL Toolkit and Object Relational Mapper
>>>>
>>>> http://www.sqlalchemy.org/
>>>>
>>>> To post example code, please provide an MCVE: Minimal, Complete, and
>>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
>>>> description.
>>>> ---
>>>> You received this message because you are subscribed to the Google Groups
>>>> "sqlalchemy" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to sqlalchemy+...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/sqlalchemy/36d0bc21-98b3-46ef-9085-c786f98234fen%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/sqlalchemy/36d0bc21-98b3-46ef-9085-c786f98234fen%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>>
>>>
>>>
>>> --
>>> SQLAlchemy -
>>> The Python SQL Toolkit and Object Relational Mapper
>>>
>>> http://www.sqlalchemy.org/
>>>
>>> To post example code, please provide an MCVE: Minimal, Complete, and
>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
>>> description.
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "sqlalchemy" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to sqlalchemy+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/sqlalchemy/b84e367a-788e-4034-952d-3bc76c67d71c%40app.fastmail.com
>>>
>>> <https://groups.google.com/d/msgid/sqlalchemy/b84e367a-788e-4034-952d-3bc76c67d71c%40app.fastmail.com?utm_medium=email&utm_source=footer>.
>>
>
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sqlalchemy/d86f6a2d-b2c5-4d06-b9e9-f8e2977858bdn%40googlegroups.com
>
> <https://groups.google.com/d/msgid/sqlalchemy/d86f6a2d-b2c5-4d06-b9e9-f8e2977858bdn%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable
Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/sqlalchemy/8debdf3a-6097-44b6-998e-4011785b15b2%40app.fastmail.com.
