I fixed the syntax errors and tried your rewritten query but I got an
error in the definitions:
TypeError Traceback (most recent call
last)<ipython-input-6-97e6a9952682> in <module>() 7 display_name =
Column(String(128, u'utf8_unicode_ci')) 8 ----> 9 class
CivicrmContribution(Base): 10 __tablename__ = 'civicrm_contribution'
11
<ipython-input-6-97e6a9952682> in CivicrmContribution() 11 12 id =
Column(INTEGER, primary_key=True, comment=u'Contribution ID')---> 13
contact_id = Column(ForeignKey(u'civicrm_contact.id'), ondelete=u'CASCADE',
nullable=False, index=True) 14 financial_type_id =
Column(ForeignKey(u'civicrm_financial_type.id'), index=True) 15
contribution_page_id = Column(ForeignKey(u'civicrm_contribution_page.id',
ondelete=u'SET NULL'), index=True)
/Users/ihf/anaconda2/lib/python2.7/site-packages/sqlalchemy/sql/schema.pyc in
__init__(self, *args, **kwargs) 1382 self.info =
kwargs.pop("info") 1383 -> 1384 self._extra_kwargs(**kwargs) 1385
1386 def _extra_kwargs(self, **kwargs):
/Users/ihf/anaconda2/lib/python2.7/site-packages/sqlalchemy/sql/schema.pyc in
_extra_kwargs(self, **kwargs) 1385 1386 def _extra_kwargs(self,
**kwargs):-> 1387 self._validate_dialect_kwargs(kwargs) 1388 1389
# @property
/Users/ihf/anaconda2/lib/python2.7/site-packages/sqlalchemy/sql/base.pyc in
_validate_dialect_kwargs(self, kwargs) 287 raise TypeError(
288 "Additional arguments should be "--> 289
"named <dialectname>_<argument>, got '%s'" % k 290 )
291 dialect_name, arg_name = m.group(1, 2)
TypeError: Additional arguments should be named <dialectname>_<argument>, got
'ondelete'
On Wednesday, August 21, 2019 at 8:55:24 PM UTC-4, Peter Schutt wrote:
>
> A couple of typos found re-reading my post, the Contribution.contact_id
> with a foreign key definition should be `contact_id = Column(Integer,
> ForeignKey('civicrm_contact'), nullable=False)`, I left out the closing
> parenthesis to the ForeignKey constructor.
>
> Also, the very last line, I didn't close the parenthesis for the
> placeholders example, should read: "(e.g. `%(tag_id_2)s`) are passed...".
>
> Couldn't find a way to edit the original.
>
> On Thursday, 22 August 2019 10:43:17 UTC+10, Peter Schutt wrote:
>>
>> Hi Ira,
>>
>> For example Integer(xx) says that Integer cannot have parameters and
>>> Tinyint seems not to exist.
>>
>>
>> I'm aware of sqlacodegen, although never had the need to use it myself.
>> Those issues sound to me like it is using the mysql dialect types, not the
>> standard sqlalchemy types. For example `from sqlalchemy import Integer`
>> will not accept an argument but `from sqlalchemy.dialects.mysql import
>> INTEGER` does accept display_width as a parameter amongst others. Also,
>> `from sqlalchemy import TinyInteger` will raise an import error but `from
>> sqlalchemy.dialects.mysql import TINYINT` works.
>>
>> I presume that this front-end overhead will pay back when I write more
>>> interesting scripts that use data from Civicrm. If this works then I can
>>> map the remainder of the (many) tables.
>>
>>
>> Well, that's certainly the idea. The abstraction allows for writing code
>> that is both easier to write and read. Essentially you write a Python app
>> and let sqlalchemy handle the bridge between your application and the
>> database. It's not a magic bullet, there are plenty of sql paradigms still
>> exposed to you through the ORM and there are limitations to using it,
>> primarily performance relative to doing bulk operations (for example, see
>> here
>> <https://docs.sqlalchemy.org/en/13/faq/performance.html#i-m-inserting-400-000-rows-with-the-orm-and-it-s-really-slow>).
>>
>> Like any tool, you need to weigh up the pros and cons. For example, if you
>> are just using the ORM in order to generate schema or construct raw
>> queries, there are better ways. The benefit of the ORM comes once you start
>> understanding and using the richer feature set. For example, relationships.
>>
>> In the models that you built above, I notice that you haven't defined any
>> foreign keys on columns. I understand that your schema is already created
>> and so the foreign keys are already defined at the database level, but that
>> information is useful to sqlalchemy when you want to create relationships
>> <https://docs.sqlalchemy.org/en/13/orm/tutorial.html#building-a-relationship>
>> and
>> explicit joins in your queries. One example of applying a foreign key to
>> your columns would be on the `Contribution.contact_id` column. Defining
>> that as a foreign key would be as simple as changing the definition to
>> `contact_id = Column(Integer, ForeignKey('civicrm_contact',
>> nullable=False)`. This would then allow you to define a relationship
>> attribute on your Contribution model, e.g. `contact =
>> relationship('Contact')`. That allows you to access the instance that
>> represents the contact associated with a given contribution through
>> instance attribute access, e.g. `contribution_instance.contact` would
>> return an instance of `Contact` and sqlalchemy will issue the necessary
>> queries behind the scenes to make that happen.
>>
>> Now, how to create the equivalent SQL query?
>>
>>
>> Well, I've had to make a couple of guesses to fill in a couple of blanks.
>> I assumed that the column in your query `state_province_id` should be an
>> attribute on the `Address` model, and that the `display_name` column
>> belongs to the `Entity_Tag` model, as neither of those fields are defined
>> elsewhere. We haven't spoken about the Session in any detail yet either,
>> but you can read
>> https://docs.sqlalchemy.org/en/13/orm/tutorial.html#creating-a-session if
>> you need. I've purely tried to emulate your original query as closely as
>> possible.
>>
>> s = Session()
>> subquery = (
>> s.query(Entity_Tag.display_name)
>> .filter(
>> Contribution.receive_date > datetime.date(2005, 7, 1),
>> Contribution.contact_id == Contact.id,
>> Contact.id == Entity_Tag.entity_id,
>> Entity_Tag.tag_id == 6,
>> )
>> .subquery()
>> )
>> result = (
>> s.query(
>> Contact.last_name,
>> Contact.first_name,
>> Address.street_address,
>> Address.city,
>> Address.postalcode,
>> State.name,
>> Country.name.label("country"),
>> )
>> .filter(
>> Contact.id == Entity_Tag.entity_id,
>> Entity_Tag.tag_id == 6,
>> Contact.id == Address.contact_id,
>> Address.state_province_id == State.id,
>> Address.country_id == Country.id,
>> Entity_Tag.display_name.notin_(subquery),
>> )
>> .distinct()
>> .all()
>> )
>>
>> This query issues this sql:
>>
>> SELECT DISTINCT civicrm_contact.last_name AS civicrm_contact_last_name,
>> civicrm_contact.first_name AS civicrm_contact_first_name, civicrm_address
>> .street_address AS civicrm_address_street_address, civicrm_address.city
>> AS civicrm_address_city, civicrm_address.postalcode AS
>> civicrm_address_postalcode, civicrm_state_province.name AS
>> civicrm_state_province_name, civicrm_country.name AS country
>> FROM civicrm_contact, civicrm_address, civicrm_state_province,
>> civicrm_country, civicrm_entity_tag
>> WHERE civicrm_contact.id = civicrm_entity_tag.entity_id AND
>> civicrm_entity_tag.tag_id = %(tag_id_1)s AND civicrm_contact.id =
>> civicrm_address.contact_id AND civicrm_address.state_province_id =
>> civicrm_state_province.id AND civicrm_address.country_id =
>> civicrm_country.id AND civicrm_entity_tag.display_name NOT IN (SELECT
>> civicrm_entity_tag.display_name
>> FROM civicrm_contribution
>> WHERE civicrm_contribution.receive_date > %(receive_date_1)s AND
>> civicrm_contribution.contact_id = civicrm_contact.id AND civicrm_contact.id
>> = civicrm_entity_tag.entity_id AND civicrm_entity_tag.tag_id = %(tag_id_2
>> )s)
>>
>> Notice that the `FROM` clause in the subquery only contains one column,
>> this is becuase the other columns have been automatically correlated with
>> the columns expressed in the enclosing query. See here:
>> https://docs.sqlalchemy.org/en/13/core/tutorial.html#correlated-subqueries
>> .
>>
>> The scalar values in the query that have been substituted with
>> placeholders (e.g. `%(tag_id_2)s` are passed to the db driver in dictionary
>> form along with the query, e.g. `{'tag_id_1': 6, 'receive_date_1':
>> datetime.date(2005, 7, 1), 'tag_id_2': 6}`.
>>
>> On Wednesday, 21 August 2019 10:20:30 UTC+10, Ira Fuchs wrote:
>>>
>>> I noticed that some of the definitions created by sqlacodegen are
>>> resulting in errors in python. For example Integer(xx) says that Integer
>>> cannot have parameters and Tinyint seems not to exist. Perhaps this is a
>>> result of my running a less than current version of mysql?
>>>
>>> On Tuesday, August 20, 2019 at 6:58:03 PM UTC-4, Ira Fuchs wrote:
>>>>
>>>> Just an aside: I discovered the sqlacodegen tool which will create the
>>>> sqlalchemy class definitions automatically. I ran it against the civicrm
>>>> mysql db and it worked. The definitions comprise 5881 lines (428KB).
>>>> Fortunately I don't need much of it for my purposes.
>>>>
>>>> On Tuesday, August 20, 2019 at 6:15:46 PM UTC-4, Ira Fuchs wrote:
>>>>>
>>>>> I think I may have all the definitions:
>>>>>
>>>>> class Contact(Base):
>>>>> __tablename__ = "civicrm_contact"
>>>>> id = Column(Integer, primary_key=True)
>>>>> last_name = Column(String(20), nullable=False)
>>>>> first_name = Column(String(20), nullable=False)
>>>>> class Contribution(Base):
>>>>> __tablename__ = "civicrm_contribution"
>>>>> id = Column(Integer, primary_key=True)
>>>>> contact_id = Column(Integer, nullable=False)
>>>>> receive_date = Column(DateTime,nullable=False)
>>>>> total_amount = Column(DECIMAL(precision=20,scale=2),
>>>>> nullable=False)
>>>>> class Address(Base):
>>>>> __tablename__ = "civicrm_address"
>>>>> id = Column(Integer, primary_key=True)
>>>>> contact_id = Column(Integer, nullable=False)
>>>>> street_address = Column(String(96), nullable=False)
>>>>> city = Column(String(64), nullable=False)
>>>>> postalcode = Column(String(64), nullable=False)
>>>>> country_id = Column(Integer, nullable=False)
>>>>> class Country(Base):
>>>>> __tablename__ = "civicrm_country"
>>>>> id = Column(Integer, primary_key=True)
>>>>> name = Column(String(64))
>>>>> class State(Base):
>>>>> __tablename__ = "civicrm_state_province"
>>>>> id = Column(Integer, primary_key=True)
>>>>> name = Column(String(64), nullable=False)
>>>>> abbreviation = Column(String(4), nullable=False)
>>>>> country_id = Column(Integer, nullable=False)
>>>>> class Entity_Tag(Base):
>>>>> __tablename__ = "civicrm_entity_tag"
>>>>> id = Column(Integer, primary_key=True)
>>>>> entity_id = Column(Integer, nullable=False)
>>>>> tag_id = Column(Integer, nullable=False)
>>>>>
>>>>> Now, how to create the equivalent SQL query? I presume that this
>>>>> front-end overhead will pay back when I write more interesting scripts
>>>>> that
>>>>> use data from Civicrm. If this works then I can map the remainder of the
>>>>> (many) tables.
>>>>>
>>>>> On Tuesday, August 20, 2019 at 5:15:36 PM UTC-4, Ira Fuchs wrote:
>>>>>>
>>>>>> OK, fixed the case:
>>>>>>
>>>>>> class Contact(Base):
>>>>>> __tablename__ = "civicrm_contact"
>>>>>> id = Column(Integer, primary_key=True)
>>>>>> last_name = Column(String(20), nullable=False)
>>>>>> first_name = Column(String(20), nullable=False)
>>>>>> class Contribution(Base):
>>>>>> __tablename__ = "civicrm_contribution"
>>>>>> id = Column(Integer, primary_key=True)
>>>>>> contact_id = Column(Integer, nullable=False)
>>>>>> receive_date = Column(DateTime,nullable=False)
>>>>>> total_amount = Column(DECIMAL(precision=20,scale=2),
>>>>>> nullable=False)
>>>>>>
>>>>>> On Tuesday, August 20, 2019 at 3:59:55 PM UTC-4, Ira Fuchs wrote:
>>>>>>>
>>>>>>> OK, I made some progress (see screenshot). I don't need the full
>>>>>>> model to be represented as I will not need many of the columns in each
>>>>>>> of
>>>>>>> the tables. If I could flesh this out enough so that I can do the query
>>>>>>> posted above I can probably use that as a template to move forward.
>>>>>>>
>>>>>>> On Monday, August 19, 2019 at 7:11:59 AM UTC-4, Peter Schutt wrote:
>>>>>>>>
>>>>>>>> Cool, how about we walk through creating a model for your schema as
>>>>>>>> an example. As we go I can point you to the relevant sections of the
>>>>>>>> tutorials/docs as they are great and will explain the details much
>>>>>>>> better
>>>>>>>> than I can, and I'll try to address any questions you have along the
>>>>>>>> way.
>>>>>>>>
>>>>>>>> The first thing you need is your Base class, all of your ORM models
>>>>>>>> will inherit from this:
>>>>>>>>
>>>>>>>> from sqlalchemy.ext.declarative import declarative_base
>>>>>>>>
>>>>>>>> Base = declarative_base()
>>>>>>>>
>>>>>>>>
>>>>>>>> Your models need to inherit from Base, need a __tablename__ class
>>>>>>>> attribute and at least a primary key column. For example, a class to
>>>>>>>> represent your "db_contact" table might look like this:
>>>>>>>>
>>>>>>>> from sqlalchemy import Column, Integer, String
>>>>>>>>
>>>>>>>> class Contact(Base):
>>>>>>>>
>>>>>>>> __tablename__ = "db_contact"
>>>>>>>>
>>>>>>>> id = Column(Integer, primary_key=True)
>>>>>>>> last_name = Column(String(20), nullable=False)
>>>>>>>> first_name = Column(String(20), nullable=False)
>>>>>>>>
>>>>>>>> Some things to note this far along:
>>>>>>>>
>>>>>>>> - In this example, the name of the Column in the database is
>>>>>>>> inferred from the name of the class attribute that the Column is
>>>>>>>> assigned
>>>>>>>> to.
>>>>>>>> - Column constructors take arguments that define specifics of
>>>>>>>> how the column is defined in the database. E.g. `primary_key=True`,
>>>>>>>> `nullable=False`. By declaring a single integer column as primary
>>>>>>>> key, it
>>>>>>>> will automatically be specified as autoincrement in the schema, as
>>>>>>>> well as
>>>>>>>> the primary key. Setting nullable=False on other columns declares
>>>>>>>> that a
>>>>>>>> NOT NULL constraint should be set on those columns.
>>>>>>>> - As you are using MySQL, String types (which represent VARCHAR
>>>>>>>> type in the database) require a length. If you were creating models
>>>>>>>> for
>>>>>>>> other backends such as SQLite or postgres, then that isn't required.
>>>>>>>>
>>>>>>>> Nothing that I've covered here isn't covered in the early stages of
>>>>>>>> the ORM Tutorial
>>>>>>>> <https://docs.sqlalchemy.org/en/13/orm/tutorial.html#declare-a-mapping>.
>>>>>>>>
>>>>>>>> Also, you can read the Column API
>>>>>>>> <https://docs.sqlalchemy.org/en/13/core/metadata.html#sqlalchemy.schema.Column>
>>>>>>>> to
>>>>>>>> get a feel for the arguments that you can pass to Column in order to
>>>>>>>> define
>>>>>>>> your schema.
>>>>>>>>
>>>>>>>> One more important element is the Foreign Key. I'll use your
>>>>>>>> "db_entity_tag" table to give you an example of that:
>>>>>>>>
>>>>>>>> from sqlalchemy import ForeignKey
>>>>>>>>
>>>>>>>> class EntityTag(Base):
>>>>>>>>
>>>>>>>> __tablename__ = "db_entity_tag"
>>>>>>>>
>>>>>>>> id = Column(Integer, primary_key=True)
>>>>>>>> entity_id = Column(
>>>>>>>> Integer,
>>>>>>>> ForeignKey('db_contact.id', ondelete="CASCADE"),
>>>>>>>> nullable=False,
>>>>>>>> )
>>>>>>>>
>>>>>>>> Pass the name of the related column to the ForeignKey object (note
>>>>>>>> that this isn't "Class.attributename", it is "tablename.columnname")
>>>>>>>> and
>>>>>>>> pass the ForeignKey object as a positional argument to the Column
>>>>>>>> constructor _after_ the column type argument. ForeignKey api is
>>>>>>>> documented
>>>>>>>> here
>>>>>>>> <https://docs.sqlalchemy.org/en/13/core/constraints.html#sqlalchemy.schema.ForeignKey>
>>>>>>>> and
>>>>>>>> touched on in the orm tutorial here
>>>>>>>> <https://docs.sqlalchemy.org/en/13/orm/tutorial.html#building-a-relationship>
>>>>>>>> .
>>>>>>>>
>>>>>>>> Have a go at filling in the columns that I've missed in those
>>>>>>>> models above and defining the rest of the tables in your schema and
>>>>>>>> once
>>>>>>>> you get that done we can move on. Happy to try to answer any questions
>>>>>>>> you
>>>>>>>> might have along the way.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Monday, 19 August 2019 12:36:21 UTC+10, Ira Fuchs wrote:
>>>>>>>>>
>>>>>>>>> Thanks for your reply and offer to help. I am able to create an
>>>>>>>>> Engine and connect to the MySQL db. I can execute simple sql queries
>>>>>>>>> although I wasn't able to get the query I posted to work due to a
>>>>>>>>> syntax
>>>>>>>>> error (probably having to do with the quotes). I have not mapped any
>>>>>>>>> tables
>>>>>>>>> to classes.
>>>>>>>>>
>>>>>>>>> On Sunday, August 18, 2019 at 8:54:57 PM UTC-4, Peter Schutt wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Ira, I'd be happy to help you find your feet with the
>>>>>>>>>> SQLAlchemy ORM.
>>>>>>>>>>
>>>>>>>>>> In general when creating an application that uses the SQLAlchemy
>>>>>>>>>> ORM, you would start with an Engine (for connecting to the db), a
>>>>>>>>>> declarative base class (maps db table to python class) and a Session
>>>>>>>>>> instance (for using a connection to issue queries). Do you have any
>>>>>>>>>> familiarity with those concepts?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Monday, 19 August 2019 02:34:05 UTC+10, Ira Fuchs wrote:
>>>>>>>>>>>
>>>>>>>>>>> I am new to sqlalchemy and I would like to begin using it to
>>>>>>>>>>> create scripts with some sql queries that I have been using but
>>>>>>>>>>> need to
>>>>>>>>>>> change from time to time. It would help a lot if someone could
>>>>>>>>>>> translate
>>>>>>>>>>> one of these queries to python sqlalchemy using an ORM (not simply
>>>>>>>>>>> executing the query as is) so that I could use this as a template
>>>>>>>>>>> for other
>>>>>>>>>>> queries, or at least learn from it.
>>>>>>>>>>>
>>>>>>>>>>> One of my queries looks like this:
>>>>>>>>>>>
>>>>>>>>>>> SELECT DISTINCT last_name, first_name, street_address, city,
>>>>>>>>>>> a.name, postal_code, f.name as country FROM db_contact c,
>>>>>>>>>>> db_entity_tag d , db_address e, db_state_province a, db_country f
>>>>>>>>>>> WHERE
>>>>>>>>>>> c.id = d.entity_id and tag_id = 6 and c.id = e.contact_id AND
>>>>>>>>>>> state_province_id = a.id and e.country_id = f.id and
>>>>>>>>>>> display_name not in ( SELECT display_name FROM db_contribution,
>>>>>>>>>>> db_contact
>>>>>>>>>>> c, db_entity_tag d WHERE receive_date > '2005-07-01' and contact_id
>>>>>>>>>>> =
>>>>>>>>>>> c.id and c.id = entity_id and tag_id = 6 )
>>>>>>>>>>>
>>>>>>>>>>> Thanks for any help.
>>>>>>>>>>>
>>>>>>>>>>>
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable
Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/sqlalchemy/750ff1b0-4d52-40f9-a8a1-efdb170d5408%40googlegroups.com.
