Re: [sqlalchemy] How to inspect filters

[Andrew Martin]

Oh that's really interesting. Thank you for that. I'll definitely tuck that
away in my back pocket. My background is really heavy in raw SQL, and
meta-programming raw SQL is *awful.* Debugging sql that writes sql and
execs it is not fun. I'm not allowed to use sqlalchemy at work because no
one else on the team uses python, and we can't go around implementing stuff
in a way that only one person knows how to work on. But I really want to
get away from the SQL-only approach in my personal/side projects. For some
reason I often find myself really blocked when it comes to sqlalchemy.
Every time I approach my databases I just flip to sql mode and totally
forget that everything in sqla is just plain python, and I can treat it
that way. I see the obvious-level mapping between the two and just kind of
hit a block beyond that. I should probably sit down and read the source
code to try and get past the block, that way I'm not so surprised by answer
that Mike and people like you give me. Anyway, probably TMI. Cheers and
thanks!

On Thu, Jun 27, 2019 at 11:09 AM Jonathan Vanasco 
wrote:

>
>
> On Wednesday, June 26, 2019 at 2:43:44 PM UTC-4, Andrew Martin wrote:
>>
>> That's very interesting, Jonathan. Could you show me a quick example of
>> that approach? I'm not sure I *need* to do that, but I think I would learn
>> about SQLAlchemy from such an example and trying to understand it.
>>
>
> One large project has an 'internal api' that tries to centralize the
> sqlalchemy interface.
>
> Let's say we're searching for a "user".  I would create a dict for the
> data like this:
>
> query_metadata = {'requester': 'site-admin',  # admin interface, user
> interface, user api, etc
>   'filters': {'Username=': 'foo',   # just an
> internal notation
>   },
>'query_data': {},  # information related to what is
> in the query as it is built
>   }
>
>
> This payload is basically the same stuff you'd pass to as queryargs to one
> of your functions above.  We essentially pass it to our version of your
> CRUD service which then acts on it to generate the query.
>
> We don't implement this approach on every query - just on a handful of
> queries that have intense logic with dozens of if/else statements and that
> connect to multiple "filters".  Stashing this in an easily accessible
> manner has just been much easier than constantly trying to examine the
> query to act on it.
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "sqlalchemy" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sqlalchemy/chGVkNwmKyQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> sqlalchemy+unsubscr...@googlegroups.com.
> To post to this group, send email to sqlalchemy@googlegroups.com.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sqlalchemy/39a4a326-c384-4c91-909a-40d9c3acb323%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/CAOVGraLheEcKaoKuwZRUBRQEgGaQ5dG7UuvV2YuW5czhzFLvqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] How to inspect filters

[Jonathan Vanasco]

On Wednesday, June 26, 2019 at 2:43:44 PM UTC-4, Andrew Martin wrote:
>
> That's very interesting, Jonathan. Could you show me a quick example of 
> that approach? I'm not sure I *need* to do that, but I think I would learn 
> about SQLAlchemy from such an example and trying to understand it.
>

One large project has an 'internal api' that tries to centralize the 
sqlalchemy interface.  

Let's say we're searching for a "user".  I would create a dict for the data 
like this:

query_metadata = {'requester': 'site-admin',  # admin interface, user 
interface, user api, etc
  'filters': {'Username=': 'foo',   # just an internal 
notation
  },
   'query_data': {},  # information related to what is 
in the query as it is built
  }


This payload is basically the same stuff you'd pass to as queryargs to one 
of your functions above.  We essentially pass it to our version of your 
CRUD service which then acts on it to generate the query.  

We don't implement this approach on every query - just on a handful of 
queries that have intense logic with dozens of if/else statements and that 
connect to multiple "filters".  Stashing this in an easily accessible 
manner has just been much easier than constantly trying to examine the 
query to act on it.  

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/39a4a326-c384-4c91-909a-40d9c3acb323%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] How to inspect filters

[Andrew Martin]

That's very interesting, Jonathan. Could you show me a quick example of
that approach? I'm not sure I *need* to do that, but I think I would learn
about SQLAlchemy from such an example and trying to understand it.

On Wed, Jun 26, 2019 at 11:00 AM Jonathan Vanasco 
wrote:

> FWIW, I found a better approach to a similar problem was to create a
> dict/object I used to log metadata about the query I wanted... then build
> the query or analyze it based on that metadata.  All the information is in
> the sqlalchemy query, but the execution performance a development time was
> much faster when I stopped analyzing the query and just consulted the dict.
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "sqlalchemy" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sqlalchemy/chGVkNwmKyQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> sqlalchemy+unsubscr...@googlegroups.com.
> To post to this group, send email to sqlalchemy@googlegroups.com.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sqlalchemy/18bed722-c8c3-42a3-97de-a5f58987fdcf%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/CAOVGraJ1QapxWARpnS86pN3ez5fHykA60bRRRgboRXertBXSxA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] How to inspect filters

[Jonathan Vanasco]

FWIW, I found a better approach to a similar problem was to create a 
dict/object I used to log metadata about the query I wanted... then build 
the query or analyze it based on that metadata.  All the information is in 
the sqlalchemy query, but the execution performance a development time was 
much faster when I stopped analyzing the query and just consulted the dict.

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sqlalchemy/18bed722-c8c3-42a3-97de-a5f58987fdcf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] How to inspect filters

[Andrew Martin]

Thank you, as always, for both the specific answer and the general advice. 
Much appreciated!

On Sunday, June 23, 2019 at 5:19:49 PM UTC-5, Mike Bayer wrote:
>
>
>
> On Sun, Jun 23, 2019, at 5:45 PM, Andrew Martin wrote:
>
> I have a generic CRUDService for my web app. It's a pattern that was 
> loosely suggested to me by Mike a while back. I've probably not implemented 
> it the way he intended, but it works pretty well in a pretty small amount 
> of code.
>
>
> The main work, however, that needs to be done is in the read_all method. 
> It needs to be able to handle different filters depending on which view is 
> calling it. Which it can as it is now. But I want to fidget with the 
> defaults a little. Basically, I almost never want to show objects that have 
> been soft deleted. Only admin users/views should be able to fidget with 
> things that we've "deleted". The way the method works now is that if I 
> don't pass any filters in at all, it sets a filter to 
> cls.is_deleted==False. But I'd like to be able to inspect incoming filters. 
> Say the view looks at the resource context and determines that a view 
> should only be allowed to see objects based on some ACL rule. I might 
> create a filter for that view controller that looks like 
> filters = (Project.usergroup==User.usergroup)
>
> (defining the applicable usergroup for a given context is a separate 
> service, but the relationship should be obvious, I think?)
>
> and then have the view query with CRUDService.read_all(Project, 
> request.dbsession, filters). The way I've got it now, that would just 
> replace the default filter, and I would have to add 
> Project.is_deleted==False to the tuple of filters every time I call this. I 
> would like to be able to inspect the filters that are passed into the 
> read_all (which, honestly, I should rename and call it read_many) method 
> and say, if there isn't any reference to the is_deleted column in the 
> filters, then set that as false, otherwise, accept the value in the 
> function parameter. But I find the filter's tuple values quite opaque. Once 
> they are instantiated, they are class-specific, and I'm trying to get at a 
> general way of understanding them. I'll note that this column is guaranteed 
> to be present in the framework. I've modified the declarative base such 
> that every object has the is_deleted. I appreciate any help you might be 
> able to give. It's very likely I'm overthinking this like a numbskull and 
> the answer is painfully obvious. Here's the service class:
>
>
> you can inspect a ColumnElement structure to look for a certain column 
> like this:
>
> from sqlalchemy.sql import visitors
> for elem in visitors.iterate(Project.usergroup == User.usergroup, {}):
>if isinstance(elem, Column) and elem.name == "is_deleted":
>return True
>
> internally, the ORM usually uses visitors.traverse(expression, {}, 
> {"column": visit_column}) but the above is easier to illustrate quickly.
>
> If it were me, I'd likely keep this as an explicit option on read_all(), 
> e.g. read_all(... include_deleted=True), much simpler and won't be called 
> "magic" by anyone, but it can go either way.
>
> also if you know me at all you'd know I would not do the "s.commit()" 
> inside all of the CR/U/D methods.   Your methods that do a "read()" are 
> beginning a transaction and leaving it open implicitly (there's no 
> rollback() or commit() at the end, nor should there be), whereas your CRUD 
> methods are fully committing the transaction for individual 
> INSERT/UPDATE/DELETE statements, and this inconsistency and the difficulty 
> it introduces in what happens when multiple CRUDService methods are called 
> sequentially will eventually lead to architectural problems.
>  
>
>
>
> from pyramid.request import Request
> from crudites.models.meta import Base
> from sqlalchemy.orm.session import Session
> from uuid import UUID
> from typing import List, Tuple, Dict
> from crudites.services.id_service import IDService
>
>
> class CRUDService:
> '''TODO: docstring'''
>
> @staticmethod
> def create(cls: Base, s: Session) -> bool:
> unique_id, public_id = IDService.create_db_ids()
> cls.unique_id=unique_id
> cls.public_id=public_id
> s.add(cls)
> s.commit()
> return True
>
> @staticmethod
> def read_by_pkid(cls: Base, s: Session, id: int) -> Base:
> q = s.query(cls).get(id)
> return q
>
> @staticmethod
> def read_by_unique_id(cls: Base, s: Session, id: UUID) -> Base:
> q = s.query(cls).filter_by(unique_id=id).first()
> return q
>
> @staticmethod
> def read_by_public_id(cls: Base, s: Session, id: str) -> Base:
> q = s.query(cls).filter_by(public_id=id).first()
> return q
>
> @staticmethod
> def read_all(cls: Base, s: Session, filters: Tuple=None) -> List[Base]:
> if not filters:
> filters = (cls.is_deleted==False,)
> q = 

Re: [sqlalchemy] How to inspect filters

[Mike Bayer]

On Sun, Jun 23, 2019, at 5:45 PM, Andrew Martin wrote:
> I have a generic CRUDService for my web app. It's a pattern that was loosely 
> suggested to me by Mike a while back. I've probably not implemented it the 
> way he intended, but it works pretty well in a pretty small amount of code.
> 
> 
> The main work, however, that needs to be done is in the read_all method. It 
> needs to be able to handle different filters depending on which view is 
> calling it. Which it can as it is now. But I want to fidget with the defaults 
> a little. Basically, I almost never want to show objects that have been soft 
> deleted. Only admin users/views should be able to fidget with things that 
> we've "deleted". The way the method works now is that if I don't pass any 
> filters in at all, it sets a filter to cls.is_deleted==False. But I'd like to 
> be able to inspect incoming filters. Say the view looks at the resource 
> context and determines that a view should only be allowed to see objects 
> based on some ACL rule. I might create a filter for that view controller that 
> looks like 
> filters = (Project.usergroup==User.usergroup)
> 
> (defining the applicable usergroup for a given context is a separate service, 
> but the relationship should be obvious, I think?)
> 
> and then have the view query with CRUDService.read_all(Project, 
> request.dbsession, filters). The way I've got it now, that would just replace 
> the default filter, and I would have to add Project.is_deleted==False to the 
> tuple of filters every time I call this. I would like to be able to inspect 
> the filters that are passed into the read_all (which, honestly, I should 
> rename and call it read_many) method and say, if there isn't any reference to 
> the is_deleted column in the filters, then set that as false, otherwise, 
> accept the value in the function parameter. But I find the filter's tuple 
> values quite opaque. Once they are instantiated, they are class-specific, and 
> I'm trying to get at a general way of understanding them. I'll note that this 
> column is guaranteed to be present in the framework. I've modified the 
> declarative base such that every object has the is_deleted. I appreciate any 
> help you might be able to give. It's very likely I'm overthinking this like a 
> numbskull and the answer is painfully obvious. Here's the service class:

you can inspect a ColumnElement structure to look for a certain column like 
this:

from sqlalchemy.sql import visitors
for elem in visitors.iterate(Project.usergroup == User.usergroup, {}):
 if isinstance(elem, Column) and elem.name == "is_deleted":
 return True

internally, the ORM usually uses visitors.traverse(expression, {}, {"column": 
visit_column}) but the above is easier to illustrate quickly.

If it were me, I'd likely keep this as an explicit option on read_all(), e.g. 
read_all(... include_deleted=True), much simpler and won't be called "magic" by 
anyone, but it can go either way.

also if you know me at all you'd know I would not do the "s.commit()" inside 
all of the CR/U/D methods. Your methods that do a "read()" are beginning a 
transaction and leaving it open implicitly (there's no rollback() or commit() 
at the end, nor should there be), whereas your CRUD methods are fully 
committing the transaction for individual INSERT/UPDATE/DELETE statements, and 
this inconsistency and the difficulty it introduces in what happens when 
multiple CRUDService methods are called sequentially will eventually lead to 
architectural problems.



> 
> from pyramid.request import Request
> from crudites.models.meta import Base
> from sqlalchemy.orm.session import Session
> from uuid import UUID
> from typing import List, Tuple, Dict
> from crudites.services.id_service import IDService
> 
> 
> class CRUDService:
>  '''TODO: docstring'''
> 
>  @staticmethod
>  def create(cls: Base, s: Session) -> bool:
>  unique_id, public_id = IDService.create_db_ids()
>  cls.unique_id=unique_id
>  cls.public_id=public_id
>  s.add(cls)
>  s.commit()
>  return True
> 
>  @staticmethod
>  def read_by_pkid(cls: Base, s: Session, id: int) -> Base:
>  q = s.query(cls).get(id)
>  return q
> 
>  @staticmethod
>  def read_by_unique_id(cls: Base, s: Session, id: UUID) -> Base:
>  q = s.query(cls).filter_by(unique_id=id).first()
>  return q
> 
>  @staticmethod
>  def read_by_public_id(cls: Base, s: Session, id: str) -> Base:
>  q = s.query(cls).filter_by(public_id=id).first()
>  return q
> 
>  @staticmethod
>  def read_all(cls: Base, s: Session, filters: Tuple=None) -> List[Base]:
>  if not filters:
>  filters = (cls.is_deleted==False,)
>  q = s.query(cls).filter(*filters).all()
>  return q
> 
>  @staticmethod
>  def update(cls: Base, s: Session) -> bool:
>  s.add(cls)
>  s.commit()
>  return True
> 
>  @staticmethod
>  def delete(cls: Base, s: Session)-> bool:
>  cls.is_deleted=True
>  s.add(cls)
>  s.commit()
>  return True
> 
> 
> 
> 
> Thanks,
> -andrew
> 

> --
>  SQLAlchemy - 
>  The Python SQL Toolkit