Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread SF Markus Elfring
If the table structure/name is known and expected to be used -- there's not really a good reason to defer creating it . Is the reason good enough to avoid the repeated specification of corresponding meta-data? Is it safer to maintain and manage column attributes for some tables only at a

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread Jonathan Vanasco
On Friday, January 23, 2015 at 12:39:02 PM UTC-5, SF Markus Elfring wrote: Is the reason good enough to avoid the repeated specification of corresponding meta-data? Is it safer to maintain and manage column attributes for some tables only at a single place? How are you currently

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread SF Markus Elfring
How are you currently specifying the meta-data? Should the Python class be sufficient for the definition of a table structure? Will the mapping interface work also without tables that were created by other SQL scripts before? Regards, Markus -- You received this message because you are

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread Michael Bayer
SF Markus Elfring elfr...@users.sourceforge.net wrote: There would still be a race condition within the Postgres internal functions. Are there any chances that this database software implementation will become robust and safe against the discussed race condition? I wonder why

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread Jonathan Vanasco
On Friday, January 23, 2015 at 1:10:37 PM UTC-5, SF Markus Elfring wrote: How often do you need to fill these data structures in a concurrent way? Does parallel table creation become more interesting then? Often and Not at All. -- You received this message because you are subscribed

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread Jonathan Vanasco
On Friday, January 23, 2015 at 11:30:33 AM UTC-5, SF Markus Elfring wrote: Are there any chances that this database software implementation will become robust and safe against the discussed race condition? I would not count on this happening in the near future as it doesn't seem to be

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread Jonathan Vanasco
On Friday, January 23, 2015 at 12:48:46 PM UTC-5, SF Markus Elfring wrote: Should the Python class be sufficient for the definition of a table structure? If you're using the declarative syntax, yes. It's common to have a `models.py` file that simply defines the classes in one place;

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread SF Markus Elfring
Should the Python class be sufficient for the definition of a table structure? If you're using the declarative syntax, yes. Thanks for your acknowledgement. It's common to have a `models.py` file that simply defines the classes in one place; then that is imported and metadata

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread SF Markus Elfring
I wonder why you’re asking of the SQLAlchemy list about a specific developmental goal of the Postgresql project? I hoped that some more corresponding experiences could already be shared here. Wouldn’t you ask them about this? That might follow ... How should I add the parameter IF NOT

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread SF Markus Elfring
I would expect that database implementations will provide functionality for parallel updates including concurrent creation of each table. What would you expect a database to do if it receives 2 CREATE TABLE my_table(...) instructions simultaneously? This depends on the passed parameters.

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread SF Markus Elfring
The parameter IF NOT EXISTS could be passed to the SQL statement CREATE TABLE. * Is this setting already used by the class library SQLAlchemy 0.9.8-78.1? From a quick scan of the docs it appears not. How can parameter additions be achieved for this software? Are you asking if IF NOT

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread Simon King
On Fri, Jan 23, 2015 at 12:17 PM, SF Markus Elfring elfr...@users.sourceforge.net wrote: The parameter IF NOT EXISTS could be passed to the SQL statement CREATE TABLE. * Is this setting already used by the class library SQLAlchemy 0.9.8-78.1? From a quick scan of the docs it appears not.

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread Jonathan Vanasco
Using IF NOT EXISTS would not solve this problem in a high concurrency scenario. There would still be a race condition within the Postgres internal functions. This is because of how Postgres checks for existing tables and creates new ones with its internal bookkeeping. It's explained in

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-23 Thread SF Markus Elfring
Using IF NOT EXISTS would not solve this problem in a high concurrency scenario. Thanks for your feedback. There would still be a race condition within the Postgres internal functions. Are there any chances that this database software implementation will become robust and safe against the

[sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-22 Thread SF Markus Elfring
Hello, I try to write some data from a source code analysis which is performed by a few processor cores simultaneously into a single table. Now I stumble on a message like the following again. … sqlalchemy.exc.IntegrityError: (IntegrityError) duplicate key value violates unique constraint

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-22 Thread SF Markus Elfring
So there is no valid use case for parallel generation of tables except in particular kinds of multi-tenancy situations. I find my use case simple enough (and therefore very valid). I am going to manage two database tables by corresponding Python classes with SQLAlchemy services. I would

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-22 Thread Michael Bayer
SF Markus Elfring elfr...@users.sourceforge.net wrote: So there is no valid use case for parallel generation of tables except in particular kinds of multi-tenancy situations. I find my use case simple enough (and therefore very valid). I am going to manage two database tables by

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-22 Thread Michael Bayer
SF Markus Elfring elfr...@users.sourceforge.net wrote: The concurrent creation of tables is an extremely unusual use case that I would not recommend, but especially when two different threads/processes are contending to create the same table, Should this use case become more known after

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-22 Thread Jonathan Vanasco
On Thursday, January 22, 2015 at 2:40:19 PM UTC-5, SF Markus Elfring wrote: I am surprised that this database software show such (unexpected) behaviour. Aside from what Michael wrote... The reason why you're seeing an IntegrityError like that, is because Postgres is raising an integrity

Re: [sqlalchemy] Difficulties with parallel data insertion into the same table

2015-01-22 Thread SF Markus Elfring
I would appreciate if I can fill these data structures in parallel without a serial database preparation step (table creation with repeated meta-data specification). You’d need to implement checks for this concurrency. I would expect that database implementations will provide functionality