[sqlalchemy] Re: running parallel migrations using sharded/partioned/spaced queries?

[Jonathan Vanasco]

Crap. That's way simpler.  Thanks!

I could even use Postgres as the orchestrator -- just create a table of ids 
and loop through those based on migration status.  (new table would avoid 
the re-write on a full record from Postgres).

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] running parallel migrations using sharded/partioned/spaced queries?

[Jonathan Vanasco]

I have to run a script on 2MM objects to update the database.  Not really a 
schema migration, more like changing the internal data representation in 
the fields.

There's a bit of post-processing and bottlenecks involved, so doing 
everything one-at-a-time will take a few days.

I'd like to split this out into 5-10 'task runners' that are each 
responsible for a a section of the database (ie, every 5th record).  That 
should considerably drop the runtime.

I thought I had seen a recipe for this somewhere, but checked and couldn't 
find anything.  That leads me to question if this is a good idea or not. 
 Anyone have thoughts/pointers?

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] Re: Exception in logger while using query options

[Ofir Herzas]

BTW: sqlalchemy 1.0.8, python 3.3.4

On Wednesday, October 14, 2015 at 12:34:04 PM UTC+3, Ofir Herzas wrote:
>
> An exception pertaining to the logger is raised when using query.options
>
> Here is the relevant part of the exception:
>
>
> query = query.options(sqlalchemy.orm.load_only(*properties))
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/query.py",
>  line 1100, in options
> return self._options(False, *args)
>   File "", line 2, in _options
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/base.py",
>  line 201, in generate
> fn(self, *args[1:], **kw)
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/query.py",
>  line 1117, in _options
> opt.process_query(self)
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
>  line 95, in process_query
> self._process(query, True)
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
>  line 305, in _process
> val._bind_loader(query, query._attributes, raiseerr)
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
>  line 422, in _bind_loader
> effective_path.set(context, "loader", loader)
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/path_registry.py",
>  line 63, in set
> log.debug("set '%s' on path '%s' to '%s'", key, self, value)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 1225, in debug
> self._log(DEBUG, msg, args, **kwargs)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 1372, in _log
> self.handle(record)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 1382, in handle
> self.callHandlers(record)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 1444, in 
> callHandlers
> hdlr.handle(record)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 835, in handle
> self.emit(record)
>   File "/usr/local/lib/python3.3/logging/handlers.py", line 869, in emit
> msg = self.format(record)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 810, in format
> return fmt.format(record)
>   File "/opt/enigmai/6.0-test/server/handlers/LogFormatter.py", line 77, in 
> format
> return super(LogFormatter, self).format(record)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 548, in format
> record.message = record.getMessage()
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 311, in getMessage
> msg = msg % self.args
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
>  line 183, in __str__
> return "Load(strategy=%r)" % self.strategy
> TypeError: not all arguments converted during string formatting
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] Exception in logger while using query options

[Ofir Herzas]

An exception pertaining to the logger is raised when using query.options

Here is the relevant part of the exception:


query = query.options(sqlalchemy.orm.load_only(*properties))
  File 
"/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/query.py",
 line 1100, in options
return self._options(False, *args)
  File "", line 2, in _options
  File 
"/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/base.py",
 line 201, in generate
fn(self, *args[1:], **kw)
  File 
"/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/query.py",
 line 1117, in _options
opt.process_query(self)
  File 
"/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
 line 95, in process_query
self._process(query, True)
  File 
"/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
 line 305, in _process
val._bind_loader(query, query._attributes, raiseerr)
  File 
"/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
 line 422, in _bind_loader
effective_path.set(context, "loader", loader)
  File 
"/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/path_registry.py",
 line 63, in set
log.debug("set '%s' on path '%s' to '%s'", key, self, value)
  File "/usr/local/lib/python3.3/logging/__init__.py", line 1225, in debug
self._log(DEBUG, msg, args, **kwargs)
  File "/usr/local/lib/python3.3/logging/__init__.py", line 1372, in _log
self.handle(record)
  File "/usr/local/lib/python3.3/logging/__init__.py", line 1382, in handle
self.callHandlers(record)
  File "/usr/local/lib/python3.3/logging/__init__.py", line 1444, in 
callHandlers
hdlr.handle(record)
  File "/usr/local/lib/python3.3/logging/__init__.py", line 835, in handle
self.emit(record)
  File "/usr/local/lib/python3.3/logging/handlers.py", line 869, in emit
msg = self.format(record)
  File "/usr/local/lib/python3.3/logging/__init__.py", line 810, in format
return fmt.format(record)
  File "/opt/enigmai/6.0-test/server/handlers/LogFormatter.py", line 77, in 
format
return super(LogFormatter, self).format(record)
  File "/usr/local/lib/python3.3/logging/__init__.py", line 548, in format
record.message = record.getMessage()
  File "/usr/local/lib/python3.3/logging/__init__.py", line 311, in getMessage
msg = msg % self.args
  File 
"/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
 line 183, in __str__
return "Load(strategy=%r)" % self.strategy
TypeError: not all arguments converted during string formatting

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] Problem with hybrid_property

[Simon King]

Works for me:

Python 2.7.10 (default, Aug 22 2015, 20:33:39)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from modelosCaja import *
>>> c = se.query(Caja).get(1)
>>> c

>>> c.imp
40.0
>>> sa.__version__
‘1.0.8'


Have you tried printing out the things I suggested earlier?

Simon

> On 14 Oct 2015, at 08:58, Cecilio Ruiz  wrote:
> 
> 
> 
> Inimport sqlalchemy as sa
> from sqlalchemy.orm import mapper, sessionmaker
> from sqlalchemy.ext.hybrid import hybrid_property, hybrid_method
> from sqlalchemy.ext.declarative import declarative_base
> 
> db_engine = sa.create_engine('sqlite:///caja.sqlite')
> metadata = sa.MetaData()
> Base = declarative_base()
> 
> # Table definition - Conceptos
> # 
> Conceptos_table = sa.Table("Conceptos", metadata,
> sa.Column('id', sa.Integer, nullable=True, autoincrement=True, 
> primary_key=True),
> sa.Column('Nombre', sa.String))
> 
> # Table definition - Caja
> # 
> Caja_table = sa.Table("Caja", metadata,
> sa.Column('id', sa.Integer, nullable=True, autoincrement=True, 
> primary_key=True),
> sa.Column('id_Conceptos', sa.Integer, sa.ForeignKey("Conceptos.id"), 
> nullable=True),
> sa.Column('Entidad', sa.String, nullable=True),
> sa.Column('Fecha', sa.Date, nullable=True),
> sa.Column('Grupo', sa.Integer, nullable=True),
> sa.Column('Entrada', sa.Float, default=0, nullable=True),
> sa.Column('Salida', sa.Float, default=0, nullable=True))
> 
> 
> # Mapping Objects
> class Conceptos(object):
> def __init__(self, Nombre):
> #self.id = id
> self.Nombre = Nombre
> 
> def __repr__(self):
> return "" % (self.id, self.Nombre)
> 
> class Caja(object):
> def __init__(self, id_Conceptos, Entidad, Fecha, Grupo, Entrada, Salida): 
>
> self.id_Conceptos = id_Conceptos
> self.Entidad = Entidad
> self.Fecha = Fecha
> self.Grupo = Grupo
> self.Entrada = Entrada
> self.Salida = Salida
> 
> @hybrid_property
> def imp(self):
> return float(self.Entrada - self.Salida)
> 
> def __repr__(self):
> return " Imp:%.2f)>" % (self.id, self.id_Conceptos, self.Entidad, self.Fecha, 
> self.Grupo, self.Entrada, self.Salida, self.imp)
> 
> 
> # Declare mappings
> mapper(Conceptos, Conceptos_table)
> mapper(Caja, Caja_table)
> 
> # Create a session
> session = sessionmaker(bind=db_engine)
> se = session()
> metadata.create_all(db_engine)
> 
> 
> 
> This is modelosCaja.py
> Now I open the Python in terminal (CMD)
> ---in python terminal--
> from modeloCaja import *
> c = se.query(Caja).imp
> >> the herror.
> 
> 
> 
> 
> 
> El miércoles, 14 de octubre de 2015, 0:05:45 (UTC+2), Simon King escribió:
> Do you perhaps have more than one “Caja” class in your application? Try 
> printing out some of the following values: 
> 
> your_instance.__dict__ 
> type(your_instance) 
> type(your_instance).__dict__ 
> sys.modules[type(your_instance).__module__] 
> 
> Can you drop into pdb at the point where the exception occurs and poke around 
> at the object interactively? 
> 
> Otherwise, you’re going to have to send us a complete script that 
> demonstrates the problem. 
> 
> Simon 
> 
> > On 13 Oct 2015, at 22:37, Cecilio Ruiz  wrote: 
> > 
> > The error message says: : "AttributeError: 'Caja' object has no attribute 
> > 'imp' 
> > 
> > Yes, is typo error en email. The imp attibute is lower case. 
> > 
> > El martes, 13 de octubre de 2015, 23:01:34 (UTC+2), Simon King escribió: 
> >   return " > Imp:%.2f)>" % (self.id, self.id_Conceptos, self.Entidad, self.Fecha, 
> > self.Grupo, self.Entrada, self.Salida, self.imp) 
> > > 
> > 
> > Can you share the full traceback? Caja.Imp doesn’t exist, but Caja.imp 
> > (note lower case “i”) should, so assuming that was a typo in your email, 
> > there must be some other problem. 
> > 
> > Simon 
> > 
> > -- 
> > You received this message because you are subscribed to the Google Groups 
> > "sqlalchemy" group. 
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to sqlalchemy+...@googlegroups.com. 
> > To post to this group, send email to sqlal...@googlegroups.com. 
> > Visit this group at http://groups.google.com/group/sqlalchemy. 
> > For more options, visit https://groups.google.com/d/optout. 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To post to this group, send email to 

Re: [sqlalchemy] running parallel migrations using sharded/partioned/spaced queries?

[Mike Bayer]

On 10/14/15 12:55 PM, jason kirtland wrote:
> If you can partition the rows numerically, this is trivially easily to
> implement using redis as the orchestrator.
> 
> For example if you have integer PKs, you might have a loop like:
> 
> offset = 0
> while offset < tablesize:
> for row in query[offset:batchsize]:
> migrate(row)
> commit()
> offset += batchsize
> 
> With redis orchestrating, you use a key in redis and INCRBY to reliably
> distribute batches to an arbitrary number of workers on an arbitrary
> number of hosts.
> 
>while True:
>offset = redis.incrby('migration-offset', batchsize)
>rows = query[offset:batchsize]
>if not rows:
>break
>for row in rows:
>migrate(row)
>commit()
> 
> INCRBY is atomic and returns the adjusted value, so every invocation of
> this script that calls into redis and INCRBYs by, say, 1000, has its own
> chunk of 1000 to work on. For a starting value of -1000 and four
> invocations, you'd see 0, 1000, 2000 and 3000.
> 
> I'll typically do this on one invocation, see that it's running well and
> that I chose a performant batch size, and then spin up additional
> workers on more cores until the migration hits the overall throughput
> required.

What am I missing that one wouldn't use say multiprocessing.Pool() to do
this kind of thing in the general sense?   If we're only talking about
5-10 runners they could just as well be local forked processes.



> 
> 
> 
> On Wed, Oct 14, 2015 at 9:32 AM, Jonathan Vanasco  > wrote:
> 
> I have to run a script on 2MM objects to update the database.  Not
> really a schema migration, more like changing the internal data
> representation in the fields.
> 
> There's a bit of post-processing and bottlenecks involved, so doing
> everything one-at-a-time will take a few days.
> 
> I'd like to split this out into 5-10 'task runners' that are each
> responsible for a a section of the database (ie, every 5th record). 
> That should considerably drop the runtime.
> 
> I thought I had seen a recipe for this somewhere, but checked and
> couldn't find anything.  That leads me to question if this is a good
> idea or not.  Anyone have thoughts/pointers?
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to sqlalchemy+unsubscr...@googlegroups.com
> .
> To post to this group, send email to sqlalchemy@googlegroups.com
> .
> Visit this group at http://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlalchemy+unsubscr...@googlegroups.com
> .
> To post to this group, send email to sqlalchemy@googlegroups.com
> .
> Visit this group at http://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: [sqlalchemy] Problem with hybrid_property

[Cecilio Ruiz]

Yes this is the result of printing:

c = se.query(Caja).get(1)
c.__dict__
{'Entidad': u'Eprueba', '_sa_instance_state': 
, 'Salida': 10.0, 
'Fecha': datetime.date(2015, 10, 12), 'id_Conceptos': 1, 'Grupo': u'grp', 
'id': 1, 'Entrada': 50.0}

type(c)


type(c).__dict__
dict_proxy({'Entidad': , '__module__': 'modeloCaja', '_sa_class_manager': 
 at 2fc7ae0>, 'Salida': 
, 
'Fecha': , 'id_Conceptos': 
, 
'id': , '__repr__': , '__dict__': 
, 'Grupo': 
, 
'__weakref__': , '__doc__': 
None, '__init__': , 'Entrada': 
})

sys.modules[type(c).__module__]


My sa.vesion is *0.7.10*
Can it be that the version is out of date?


I appreciate all your efforts to resolve.


El miércoles, 14 de octubre de 2015, 21:52:05 (UTC+2), Simon King escribió:
>
> Works for me: 
>
> Python 2.7.10 (default, Aug 22 2015, 20:33:39) 
> [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.1)] on darwin 
> Type "help", "copyright", "credits" or "license" for more information. 
> >>> from modelosCaja import * 
> >>> c = se.query(Caja).get(1) 
> >>> c 
>  Imp:40.00)> 
> >>> c.imp 
> 40.0 
> >>> sa.__version__ 
> ‘1.0.8' 
>
>
> Have you tried printing out the things I suggested earlier? 
>
> Simon 
>
> > On 14 Oct 2015, at 08:58, Cecilio Ruiz  
> wrote: 
> > 
> > 
> > 
> > Inimport sqlalchemy as sa 
> > from sqlalchemy.orm import mapper, sessionmaker 
> > from sqlalchemy.ext.hybrid import hybrid_property, hybrid_method 
> > from sqlalchemy.ext.declarative import declarative_base 
> > 
> > db_engine = sa.create_engine('sqlite:///caja.sqlite') 
> > metadata = sa.MetaData() 
> > Base = declarative_base() 
> > 
> > # Table definition - Conceptos 
> > # 
> > Conceptos_table = sa.Table("Conceptos", metadata, 
> > sa.Column('id', sa.Integer, nullable=True, autoincrement=True, 
> primary_key=True), 
> > sa.Column('Nombre', sa.String)) 
> > 
> > # Table definition - Caja 
> > # 
> > Caja_table = sa.Table("Caja", metadata, 
> > sa.Column('id', sa.Integer, nullable=True, autoincrement=True, 
> primary_key=True), 
> > sa.Column('id_Conceptos', sa.Integer, sa.ForeignKey("Conceptos.id"), 
> nullable=True), 
> > sa.Column('Entidad', sa.String, nullable=True), 
> > sa.Column('Fecha', sa.Date, nullable=True), 
> > sa.Column('Grupo', sa.Integer, nullable=True), 
> > sa.Column('Entrada', sa.Float, default=0, nullable=True), 
> > sa.Column('Salida', sa.Float, default=0, nullable=True)) 
> > 
> > 
> > # Mapping Objects 
> > class Conceptos(object): 
> > def __init__(self, Nombre): 
> > #self.id = id 
> > self.Nombre = Nombre 
> > 
> > def __repr__(self): 
> > return "" % (self.id, self.Nombre) 
> > 
> > class Caja(object): 
> > def __init__(self, id_Conceptos, Entidad, Fecha, Grupo, Entrada, 
> Salida): 
> > self.id_Conceptos = id_Conceptos 
> > self.Entidad = Entidad 
> > self.Fecha = Fecha 
> > self.Grupo = Grupo 
> > self.Entrada = Entrada 
> > self.Salida = Salida 
> > 
> > @hybrid_property 
> > def imp(self): 
> > return float(self.Entrada - self.Salida) 
> > 
> > def __repr__(self): 
> > return " Sal: %i Imp:%.2f)>" % (self.id, self.id_Conceptos, self.Entidad, 
> self.Fecha, self.Grupo, self.Entrada, self.Salida, self.imp) 
> > 
> > 
> > # Declare mappings 
> > mapper(Conceptos, Conceptos_table) 
> > mapper(Caja, Caja_table) 
> > 
> > # Create a session 
> > session = sessionmaker(bind=db_engine) 
> > se = session() 
> > metadata.create_all(db_engine) 
> > 
> > 
> > 
> > This is modelosCaja.py 
> > Now I open the Python in terminal (CMD) 
> > ---in python terminal-- 
> > from modeloCaja import * 
> > c = se.query(Caja).imp 
> > >> the herror. 
> > 
> > 
> > 
> > 
> > 
> > El miércoles, 14 de octubre de 2015, 0:05:45 (UTC+2), Simon King 
> escribió: 
> > Do you perhaps have more than one “Caja” class in your application? Try 
> printing out some of the following values: 
> > 
> > your_instance.__dict__ 
> > type(your_instance) 
> > type(your_instance).__dict__ 
> > sys.modules[type(your_instance).__module__] 
> > 
> > Can you drop into pdb at the point where the exception occurs and poke 
> around at the object interactively? 
> > 
> > Otherwise, you’re going to have to send us a complete script that 
> demonstrates the problem. 
> > 
> > Simon 
> > 
> > > On 13 Oct 2015, at 22:37, Cecilio Ruiz  wrote: 
> > > 
> > > The error message says: : "AttributeError: 'Caja' object has no 
> attribute 'imp' 
> > > 
> > > Yes, is typo error en email. The imp attibute is lower case. 
> > > 
> > > El martes, 13 de octubre de 2015, 23:01:34 (UTC+2), Simon King 
> escribió: 
> > >   return " %i Imp:%.2f)>" % 

Re: [sqlalchemy] Exception in logger while using query options

[Mike Bayer]

On 10/14/15 5:34 AM, Ofir Herzas wrote:
> An exception pertaining to the logger is raised when using query.options
> 
> Here is the relevant part of the exception:

issue 3539 fixed for 1.0.9:

https://bitbucket.org/zzzeek/sqlalchemy/issues/3539



> 
> 
> query = query.options(sqlalchemy.orm.load_only(*properties))
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/query.py",
>  line 1100, in options
> return self._options(False, *args)
>   File "", line 2, in _options
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/base.py",
>  line 201, in generate
> fn(self, *args[1:], **kw)
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/query.py",
>  line 1117, in _options
> opt.process_query(self)
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
>  line 95, in process_query
> self._process(query, True)
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
>  line 305, in _process
> val._bind_loader(query, query._attributes, raiseerr)
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
>  line 422, in _bind_loader
> effective_path.set(context, "loader", loader)
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/path_registry.py",
>  line 63, in set
> log.debug("set '%s' on path '%s' to '%s'", key, self, value)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 1225, in debug
> self._log(DEBUG, msg, args, **kwargs)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 1372, in _log
> self.handle(record)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 1382, in handle
> self.callHandlers(record)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 1444, in 
> callHandlers
> hdlr.handle(record)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 835, in handle
> self.emit(record)
>   File "/usr/local/lib/python3.3/logging/handlers.py", line 869, in emit
> msg = self.format(record)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 810, in format
> return fmt.format(record)
>   File "/opt/enigmai/6.0-test/server/handlers/LogFormatter.py", line 77, in 
> format
> return super(LogFormatter, self).format(record)
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 548, in format
> record.message = record.getMessage()
>   File "/usr/local/lib/python3.3/logging/__init__.py", line 311, in getMessage
> msg = msg % self.args
>   File 
> "/opt/enigmai/ve/python-3.3.4/lib/python3.3/site-packages/sqlalchemy/orm/strategy_options.py",
>  line 183, in __str__
> return "Load(strategy=%r)" % self.strategy
> TypeError: not all arguments converted during string formatting
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlalchemy+unsubscr...@googlegroups.com
> .
> To post to this group, send email to sqlalchemy@googlegroups.com
> .
> Visit this group at http://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: 'AutogenContext' object has no attribute '__getitem__'

[Fayaz useful]

Does it no longer support the autogen_context['imports'] list, like here?
http://alembic.readthedocs.org/en/latest/autogenerate.html#affecting-the-rendering-of-types-themselves

On Wed, Oct 14, 2015 at 6:57 PM, Mike Bayer 
wrote:

>
>
> On 10/14/15 12:41 AM, Fayaz Yusuf Khan wrote:
> > Hi,
> > Looks like the autogen_context parameter has changed type recently.
>
> indeed it has:
>
>
> http://alembic.readthedocs.org/en/latest/api/autogenerate.html#alembic.autogenerate.api.AutogenContext
>
>
>
>
> >
> > I'm getting this exception:
> >
> > Traceback (most recent call last):
> >   File "/home/fayaz/Programming/weaver-env/bin/weaver", line 9, in
> 
> > load_entry_point('weaver==2.12.1', 'console_scripts', 'weaver')()
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> > line 700, in __call__
> > return self.main(*args, **kwargs)
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> > line 680, in main
> > rv = self.invoke(ctx)
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> > line 1027, in invoke
> > return _process_result(sub_ctx.command.invoke(sub_ctx))
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> > line 1027, in invoke
> > return _process_result(sub_ctx.command.invoke(sub_ctx))
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> > line 873, in invoke
> > return ctx.invoke(self.callback, **ctx.params)
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> > line 508, in invoke
> > return callback(*args, **kwargs)
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/decorators.py",
> > line 16, in new_func
> > return f(get_current_context(), *args, **kwargs)
> >   File "/home/fayaz/Programming/weaver-backend/weaver/cli.py", line 44,
> > in script
> > weaver.db.generate_migrate_script(url=ctx.obj, message=message)
> >   File
> > "/home/fayaz/Programming/weaver-backend/weaver/db/sql/__init__.py", line
> > 132, in generate_migrate_script
> > make_config(url), message, autogenerate=True)
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/command.py",
> > line 121, in revision
> > revision_context.generate_scripts()
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/api.py",
> > line 412, in generate_scripts
> > yield self._to_script(generated_revision)
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/api.py",
> > line 336, in _to_script
> > autogen_context, migration_script, template_args
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> > line 40, in _render_python_into_templatevars
> > _render_cmd_body(upgrade_ops, autogen_context))
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> > line 63, in _render_cmd_body
> > lines = render_op(autogen_context, op)
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> > line 75, in render_op
> > lines = util.to_list(renderer(autogen_context, op))
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> > line 102, in _render_modify_table
> > t_lines = render_op(autogen_context, t_op)
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> > line 75, in render_op
> > lines = util.to_list(renderer(autogen_context, op))
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> > line 299, in _add_column
> > "column": _render_column(column, autogen_context),
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> > line 542, in _render_column
> > 'type': _repr_type(column.type, autogen_context),
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> > line 565, in _repr_type
> > rendered = _user_defined_render("type", type_, autogen_context)
> >   File
> >
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> > line 513, in _user_defined_render
> > rendered = render(type_, object_, autogen_context)
> >   File
> > "/home/fayaz/Programming/weaver-backend/weaver/db/migrations/env.py",
> > line 52, in render_item
> > autogen_context['imports'].add(
> >
> >
> >
> > 

Re: [sqlalchemy] running parallel migrations using sharded/partioned/spaced queries?

[jason kirtland]

If you can partition the rows numerically, this is trivially easily to
implement using redis as the orchestrator.

For example if you have integer PKs, you might have a loop like:

offset = 0
while offset < tablesize:
for row in query[offset:batchsize]:
migrate(row)
commit()
offset += batchsize

With redis orchestrating, you use a key in redis and INCRBY to reliably
distribute batches to an arbitrary number of workers on an arbitrary number
of hosts.

   while True:
   offset = redis.incrby('migration-offset', batchsize)
   rows = query[offset:batchsize]
   if not rows:
   break
   for row in rows:
   migrate(row)
   commit()

INCRBY is atomic and returns the adjusted value, so every invocation of
this script that calls into redis and INCRBYs by, say, 1000, has its own
chunk of 1000 to work on. For a starting value of -1000 and four
invocations, you'd see 0, 1000, 2000 and 3000.

I'll typically do this on one invocation, see that it's running well and
that I chose a performant batch size, and then spin up additional workers
on more cores until the migration hits the overall throughput required.



On Wed, Oct 14, 2015 at 9:32 AM, Jonathan Vanasco 
wrote:

> I have to run a script on 2MM objects to update the database.  Not really
> a schema migration, more like changing the internal data representation in
> the fields.
>
> There's a bit of post-processing and bottlenecks involved, so doing
> everything one-at-a-time will take a few days.
>
> I'd like to split this out into 5-10 'task runners' that are each
> responsible for a a section of the database (ie, every 5th record).  That
> should considerably drop the runtime.
>
> I thought I had seen a recipe for this somewhere, but checked and couldn't
> find anything.  That leads me to question if this is a good idea or not.
> Anyone have thoughts/pointers?
>
> --
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sqlalchemy+unsubscr...@googlegroups.com.
> To post to this group, send email to sqlalchemy@googlegroups.com.
> Visit this group at http://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

Re: 'AutogenContext' object has no attribute '__getitem__'

[Mike Bayer]

On 10/14/15 12:41 AM, Fayaz Yusuf Khan wrote:
> Hi,
> Looks like the autogen_context parameter has changed type recently.

indeed it has:

http://alembic.readthedocs.org/en/latest/api/autogenerate.html#alembic.autogenerate.api.AutogenContext




> 
> I'm getting this exception:
> 
> Traceback (most recent call last):
>   File "/home/fayaz/Programming/weaver-env/bin/weaver", line 9, in 
> load_entry_point('weaver==2.12.1', 'console_scripts', 'weaver')()
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> line 700, in __call__
> return self.main(*args, **kwargs)
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> line 680, in main
> rv = self.invoke(ctx)
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> line 1027, in invoke
> return _process_result(sub_ctx.command.invoke(sub_ctx))
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> line 1027, in invoke
> return _process_result(sub_ctx.command.invoke(sub_ctx))
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> line 873, in invoke
> return ctx.invoke(self.callback, **ctx.params)
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/core.py",
> line 508, in invoke
> return callback(*args, **kwargs)
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/click/decorators.py",
> line 16, in new_func
> return f(get_current_context(), *args, **kwargs)
>   File "/home/fayaz/Programming/weaver-backend/weaver/cli.py", line 44,
> in script
> weaver.db.generate_migrate_script(url=ctx.obj, message=message)
>   File
> "/home/fayaz/Programming/weaver-backend/weaver/db/sql/__init__.py", line
> 132, in generate_migrate_script
> make_config(url), message, autogenerate=True)
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/command.py",
> line 121, in revision
> revision_context.generate_scripts()
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/api.py",
> line 412, in generate_scripts
> yield self._to_script(generated_revision)
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/api.py",
> line 336, in _to_script
> autogen_context, migration_script, template_args
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> line 40, in _render_python_into_templatevars
> _render_cmd_body(upgrade_ops, autogen_context))
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> line 63, in _render_cmd_body
> lines = render_op(autogen_context, op)
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> line 75, in render_op
> lines = util.to_list(renderer(autogen_context, op))
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> line 102, in _render_modify_table
> t_lines = render_op(autogen_context, t_op)
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> line 75, in render_op
> lines = util.to_list(renderer(autogen_context, op))
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> line 299, in _add_column
> "column": _render_column(column, autogen_context),
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> line 542, in _render_column
> 'type': _repr_type(column.type, autogen_context),
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> line 565, in _repr_type
> rendered = _user_defined_render("type", type_, autogen_context)
>   File
> "/home/fayaz/Programming/weaver-env/local/lib/python2.7/site-packages/alembic/autogenerate/render.py",
> line 513, in _user_defined_render
> rendered = render(type_, object_, autogen_context)
>   File
> "/home/fayaz/Programming/weaver-backend/weaver/db/migrations/env.py",
> line 52, in render_item
> autogen_context['imports'].add(
> 
> 
> 
> Here's the code:
> 
> def render_item(type_, obj, autogen_context):
> from weaver.core.model import PriceType
> if type_ == 'type':
> if isinstance(obj, PriceType):
> print autogen_context
> autogen_context['imports'].add(
> 'from weaver.core.model import PriceType')
> return '%r' % obj
> return False
> 
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy-alembic" group.
> To unsubscribe