[sqlalchemy] newbie question: best way to factor and group data when only some columns change
I have a tester that controls 30 independent variables. Each variable can have about 40 different values. Periodically, I run a test. I choose 2 of the 30 independent variables, and over the next few minutes, fully iterate those 2 variables relative to each other, visiting each point in a 40*40 space. I want to record the result of the test at each of those 40*40 points. What's the best way to use sqlalchemy+mysql to represent this use case? I'm not even sure of the terminology to use describing this factoring to use for Google searching. I'm inclined to have a separate table for each test run, where that table records {param1 value, param2 value, date, result} and then have a master table, one row per test run, where that row holds the value of all 30 variables, lists the 2 control variables, and then somehow references or names the separate table with the results of the test run. Robert Henry -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
[sqlalchemy] Newbie Question: Aliases not Aliasing?
I posted this questionhttp://stackoverflow.com/questions/16410888/sqlalchemy-aliases-not-aliasingto StackOverflow, but thought this may be a better place to ask. I have the following sqlalchemy code: x = bbc.alias().c w = bbc.alias().cselect([func.distinct(x.region)]).where( select([func.sum(w.population)]).where((w.region == x.region)) 1) I would expect it to produce this SQL: SELECT DISTINCT(x.region)FROM bbc xWHERE 1 ( SELECT SUM(w.population) from bbc w WHERE w.region = x.region) but instead it gives me this: SELECT distinct(bbc_1.region) AS distinct_1 FROM bbc AS bbc_1 WHERE 1 Somehow the WHERE clause is collapsing, but for the life of me I can't see why. Am I doing something wrong with the aliases? -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] Newbie Question: Aliases not Aliasing?
update the SO question too for me...in this case it needs a clue to interpret the select as a scalar: subq = select([func.sum(w.population)]).where((w.region == x.region)) print select([func.distinct(x.region)]).where(subq.as_scalar() 1) the WHERE 1 you're getting is because someselect 1 just comes out to Python True. On May 7, 2013, at 9:58 AM, Haoyi Li haoyi...@gmail.com wrote: I posted this question to StackOverflow, but thought this may be a better place to ask. I have the following sqlalchemy code: x = bbc.alias().c w = bbc.alias().c select([func.distinct(x.region)]).where( select([func.sum(w.population)]).where((w.region == x.region)) 1 ) I would expect it to produce this SQL: SELECT DISTINCT(x.region) FROM bbc x WHERE 1 ( SELECT SUM(w.population) from bbc w WHERE w.region = x.region ) but instead it gives me this: SELECT distinct(bbc_1.region) AS distinct_1 FROM bbc AS bbc_1 WHERE 1 Somehow the WHERE clause is collapsing, but for the life of me I can't see why. Am I doing something wrong with the aliases? -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] Newbie Question: Aliases not Aliasing?
Thanks, that worked! A follow up question to make sure i'm understanding this right: I'm looking at the effect of .as_scalar() on the query, and it seems that any query I call .as_scalar() on simply gets wrapped in parenthesis. I tested it out and it seems that it works even in cases where I don't want a scalar value (e.g. thing._in(myquery.as_scalar()) ), which kind of aligns with my theory that it just wraps stuff in parenthesis. Is that all it does, or does it do something more? It doesn't work on the top level query, but would it be safe to simply wrap all the nested queries with a (...).as_scalar()? Thanks! -Haoyi On Tuesday, May 7, 2013 11:53:52 AM UTC-4, Michael Bayer wrote: update the SO question too for me...in this case it needs a clue to interpret the select as a scalar: subq = select([func.sum(w.population)]).where((w.region == x.region)) print select([func.distinct(x.region)]).where(subq.as_scalar() 1) the WHERE 1 you're getting is because someselect 1 just comes out to Python True. On May 7, 2013, at 9:58 AM, Haoyi Li haoy...@gmail.com javascript: wrote: I posted this questionhttp://stackoverflow.com/questions/16410888/sqlalchemy-aliases-not-aliasingto StackOverflow, but thought this may be a better place to ask. I have the following sqlalchemy code: x = bbc.alias().c w = bbc.alias().cselect([func.distinct(x.region)]).where( select([func.sum(w.population)]).where((w.region == x.region)) 1) I would expect it to produce this SQL: SELECT DISTINCT(x.region)FROM bbc xWHERE 1 ( SELECT SUM(w.population) from bbc w WHERE w.region = x.region) but instead it gives me this: SELECT distinct(bbc_1.region) AS distinct_1 FROM bbc AS bbc_1 WHERE 1 Somehow the WHERE clause is collapsing, but for the life of me I can't see why. Am I doing something wrong with the aliases? -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com javascript:. To post to this group, send email to sqlal...@googlegroups.comjavascript: . Visit this group at http://groups.google.com/group/sqlalchemy?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] Newbie Question: Aliases not Aliasing?
it's a Python thing, it turns the FromClause into a ColumnElement which then has operators like __eq__(), __lt__(), etc. On May 7, 2013, at 2:29 PM, Haoyi Li haoyi...@gmail.com wrote: Thanks, that worked! A follow up question to make sure i'm understanding this right: I'm looking at the effect of .as_scalar() on the query, and it seems that any query I call .as_scalar() on simply gets wrapped in parenthesis. I tested it out and it seems that it works even in cases where I don't want a scalar value (e.g. thing._in(myquery.as_scalar()) ), which kind of aligns with my theory that it just wraps stuff in parenthesis. Is that all it does, or does it do something more? It doesn't work on the top level query, but would it be safe to simply wrap all the nested queries with a (...).as_scalar()? Thanks! -Haoyi On Tuesday, May 7, 2013 11:53:52 AM UTC-4, Michael Bayer wrote: update the SO question too for me...in this case it needs a clue to interpret the select as a scalar: subq = select([func.sum(w.population)]).where((w.region == x.region)) print select([func.distinct(x.region)]).where(subq.as_scalar() 1) the WHERE 1 you're getting is because someselect 1 just comes out to Python True. On May 7, 2013, at 9:58 AM, Haoyi Li haoy...@gmail.com wrote: I posted this question to StackOverflow, but thought this may be a better place to ask. I have the following sqlalchemy code: x = bbc.alias().c w = bbc.alias().c select([func.distinct(x.region)]).where( select([func.sum(w.population)]).where((w.region == x.region)) 1 ) I would expect it to produce this SQL: SELECT DISTINCT(x.region) FROM bbc x WHERE 1 ( SELECT SUM(w.population) from bbc w WHERE w.region = x.region ) but instead it gives me this: SELECT distinct(bbc_1.region) AS distinct_1 FROM bbc AS bbc_1 WHERE 1 Somehow the WHERE clause is collapsing, but for the life of me I can't see why. Am I doing something wrong with the aliases? -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com. To post to this group, send email to sqlal...@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sqlalchemy] Newbie question
On Wed, Jul 27, 2011 at 2:48 PM, Michael Bayer mike...@zzzcomputing.com wrote: On Jul 27, 2011, at 3:21 PM, Kent Tenney wrote: Howdy, I'm aggregating data from several Sqlite files into a Postgres db. The sqlite files are storage for several apps I use: Shotwell, Firefox, Zotero, Banshee ... I just watch and pull from them. I've been using import sqlite3 so far, dumping sql from sqlite, using it to create the Postgres tables. I then add columns to meet my own needs. I now can diff 2 sqlite files, so I know what rows need updating and adding in the Postgres tables. I feel I should be using Sqlalchemy, ok well what problems do you have currently that you'd like to solve ? With some proof-of-concept working, looking ahead to increasing complexity, my sense is that SA can hide some complexity behind a layer which moves data into Postgres. I think this is what you suggest in your final paragraph. but have been intimidated by the wealth of choices SA offers. It offers choices in that you can A. use core only or B. the ORM, as well as choices in that it works with whatever kind of schema you'd like, I'll be studying doc for a better idea of the A/B distinction, as well as what kinds of schemas there are. I have the O'Reilly book, I understand SA has changed considerably since then, am I better off sticking with current doc? which is the same choice you have anyway, and in this case it seems you have already made. Your app sounds like kind of a nuts and bolts table-to-table thing, i.e. is SQL centric, so using constructs like table.select() and table.insert() could perhaps reduce the verbosity of generating those statements by hand, the Table construct itself can turn the equation of what columns am i dealing with here? into a data driven one (the Table is a datastructure, which stores a list of Column objects - a data driven description of a schema). - comfortable in Python, SQL not so much dont sell yourself short, you're moving rows and adding columns and that's a fair degree of knowledge right there. - pull into the Postgres db from other sources - file system content - email - other db's: Mysql, rdf, ... - feed Sphinxsearch from the Postgres db for all of these you'd probably want to figure out some intermediary format that everything goes into, then goes to the database. Depending on how much this format is hardwired to the SQL schema or not, as well as if you're generally dealing with one big table to store a format versus many tables storing a more normalized structure, would determine how well the ORM may or may not be useful. The ORM is good when you have multiple tables in some hierarchical structure that is to be related to an object hierarchy. For a straight up I'd like the columns in this Excel spreadsheet to become columns in a new database table, it might be overkill. I appreciate the advice, I'll be back. Thanks, Kent -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Newbie question
On Jul 28, 2011, at 11:47 AM, Kent Tenney wrote: It offers choices in that you can A. use core only or B. the ORM, as well as choices in that it works with whatever kind of schema you'd like, I'll be studying doc for a better idea of the A/B distinction, as well as what kinds of schemas there are. I have the O'Reilly book, I understand SA has changed considerably since then, am I better off sticking with current doc? Yes unfortunately the OReilly book, written against version 0.3 which was an early developmental stage for SQLAlchemy, will steer you more in the wrong direction today than anything, and we lack the resources to complete a new book which has been under contract for some time now. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Newbie question
Howdy, I'm aggregating data from several Sqlite files into a Postgres db. The sqlite files are storage for several apps I use: Shotwell, Firefox, Zotero, Banshee ... I just watch and pull from them. I've been using import sqlite3 so far, dumping sql from sqlite, using it to create the Postgres tables. I then add columns to meet my own needs. I now can diff 2 sqlite files, so I know what rows need updating and adding in the Postgres tables. I feel I should be using Sqlalchemy, but have been intimidated by the wealth of choices SA offers. I don't want to start down the wrong road. However, as I look towards coding change merging, and the new level of complexity it presents, I think it's time to take the plunge. Data specs: - source data lives in other-owned files - replicate source data tables in Postgres - add columns to Postgres tables - keep Postgres synced with sqlite sources My proclivities: - comfortable in Python, SQL not so much - roadmap - pull into the Postgres db from other sources - file system content - email - other db's: Mysql, rdf, ... - feed Sphinxsearch from the Postgres db I would greatly appreciate any suggestions on how to proceed. Thanks, Kent -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Re: [sqlalchemy] Newbie question
On Jul 27, 2011, at 3:21 PM, Kent Tenney wrote: Howdy, I'm aggregating data from several Sqlite files into a Postgres db. The sqlite files are storage for several apps I use: Shotwell, Firefox, Zotero, Banshee ... I just watch and pull from them. I've been using import sqlite3 so far, dumping sql from sqlite, using it to create the Postgres tables. I then add columns to meet my own needs. I now can diff 2 sqlite files, so I know what rows need updating and adding in the Postgres tables. I feel I should be using Sqlalchemy, ok well what problems do you have currently that you'd like to solve ? but have been intimidated by the wealth of choices SA offers. It offers choices in that you can A. use core only or B. the ORM, as well as choices in that it works with whatever kind of schema you'd like, which is the same choice you have anyway, and in this case it seems you have already made. Your app sounds like kind of a nuts and bolts table-to-table thing, i.e. is SQL centric, so using constructs like table.select() and table.insert() could perhaps reduce the verbosity of generating those statements by hand, the Table construct itself can turn the equation of what columns am i dealing with here? into a data driven one (the Table is a datastructure, which stores a list of Column objects - a data driven description of a schema). - comfortable in Python, SQL not so much dont sell yourself short, you're moving rows and adding columns and that's a fair degree of knowledge right there. - pull into the Postgres db from other sources - file system content - email - other db's: Mysql, rdf, ... - feed Sphinxsearch from the Postgres db for all of these you'd probably want to figure out some intermediary format that everything goes into, then goes to the database. Depending on how much this format is hardwired to the SQL schema or not, as well as if you're generally dealing with one big table to store a format versus many tables storing a more normalized structure, would determine how well the ORM may or may not be useful. The ORM is good when you have multiple tables in some hierarchical structure that is to be related to an object hierarchy. For a straight up I'd like the columns in this Excel spreadsheet to become columns in a new database table, it might be overkill. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Newbie Question - Identity Mapper Issue
Hi, Does SqlAlchemy query call __init__ on the mapped object? i.e. Say I have mapped the object to table as follows: mapper(Obj, Table) Now, table contains: index integer x integer y integer And Obj is defined as Class Obj: def __init__(self, x, y): self.x = x self.y = y self.z = True I have a query later: for query in session.query(Obj).order_by(Obj.index): --- print query.z This fails saying that z is not an element of query. I was under the impression that query will return back initialized objects with data updated with data from DB. But this doesnt seem to be the case. Could someone please help me out? Am I doing something wrong here? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Newbie question
I have just started using Python and TurboGears with SQLAlchemy and I think its great so far. I am working with an already-created database and am wondering what the best way is to represent the relationships between the recurring_task and employee_status table: employee_statuses = Table('employee_status', metadata, #Column('emp_id', Integer, primary_key=True), autoload=True) recurring_tasks = Table('recurring_task', metadata, #Column('rt_id', Integer, primary_key=True), autoload=True) class Employee_status(object): pass class Recurring_task(object): pass employee_status_mapper = mapper(Employee_status, employee_statuses) recurring_task_mapper = mapper(Recurring_task, recurring_tasks) So far, so good - this works in the tg-admin shell. recurring_task has three columns, rt_raiser, rt_owner and rt_status which are all instances of employee_status. I would be grateful if somebody could indicate the best way to represent these relationships in the model as I'm not sure which of the many examples in the documentation is relevant - I am worried because I have one table referencing another table multiple times. Many thanks Jon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Newbie question: sAsync maintained? In use?
'm just getting started with SA, and I'll want to be using it in a back-end server with an existing mssql database. I'm considering using Twisted as the basis for the server, and I've looked a bit into using sAsync. I've had trouble getting the Twisted XML-RPC Server Example to run, and that led me to wonder about the status of the package. Is anyone here using sAsync, or knows enough to comment on its viability for the kind of thing I've described. Thanks for any good words, -- Don Dwiggins Advanced Publishing Technology --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Newbie question: Too many connections
Hi, I keep getting The error: DBAPIError: (Connection failed) (OperationalError) (1040, 'Too many connections'). At no time do I really need more than a few connections so I am doing something basically wrong. From reading the doc I concluded that pooling in the simple cases is automatic. I must have misunderstood. Questions: 1) What is the suggested method to debug such a problem? E.g. retrieve the number of open connections. 2) When is a connection closed? For that matter , when is it open and when taken from the pool? TIA --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Newbie question about transactions
In the following code, I intended that nothing will be inserted into the table because the transaction fails and is rolled back. What actually happens is that the first insert is not rolled back because it s committed by itself. What is the correct way to achieve my goal? TIA from sqlalchemy import * db=create_engine('mysql://user:pw@localhost/db') cn= db.connect() cn.execute(DROP TABLE IF EXISTS test) cn.execute(CREATE TABLE test (`id` bigint(20) NOT NULL , PRIMARY KEY (`id`)) ENGINE=InnoDB) db.echo=True md=BoundMetaData(db) t=Table('test', md, autoload = True) trans = cn.begin() try: insert(t).execute(id=332) insert(t).execute(id=332) trans.commit() except: trans.rollback() raise --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Newbie question about select statement
I everyone. I'm pretty new to SQLAlchemy and never done much sql-related work in the past as well so this could possibly be a silly question. Suppose I've a table with a huge number of records. Now, I'd like to access this table randomly through a list-like interface of a Python class instance, i.e. saying foo[90] would return the 90th row of the table. I know how to accomplish this in play MySQL's SQL, maybe using a LIMIT clause in the select. This way it'd be trivial to write a container class implementing the __getitem__() method with a a SELECT/LIMIT statement. In addition, suppose that the SELECT statement contains a WHERE clause defined at foo instance's creation time (possibly through its constructor). Again, I'd simply plug the WHERE clause in the SELECT statement on every __getitem()__ invocation. I think it's be possible to implement such a beast through SQLAlchemy's API and I thought that Query's select() method of a mapper worked that way, but it seems that it fetches every and each row from the DB just before returning a list of the mapped object instances. Am I wrong? Is there a way/class in the SA's API to have sort of a lazy list of results of a query from which to pick randomly? Thank you, Cristiano --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---