Thanks, this is in fact what I implemented now as a method in the Sensor class, exploiting also the bulk_insert_mappings since the number of readings are quite a lot (400k each time):
def store_readings(self, session): if not(self.id): session.add(self) session.flush() sensor_id = self.id times, voltages = self.get_values_from_somewhere() n = len(times) chunk_size = 100000 for i in six.moves.range(0, n, chunk_size): begin = i end = i+chunk_size time_chunk = times[begin:end] voltage_chunk = voltages[begin:end] session.bulk_insert_mappings(PassageData, [ { 'time': t, 'voltage': v, 'sensor_id': sensor_id } for t, v in zip(time_chunk, voltave_chunk)] ) session.commit() The problem with the custom collection is that maybe we lose performances that are gained while using the numpy powerful indexing. I need, still, to study a bit the suggestion given by Mike. Thanks, On Friday, July 28, 2017 at 9:28:19 PM UTC+2, Jonathan Vanasco wrote: > > Unless you need to use all the readings immediately, have you considered > just making a custom def under the Sensor model, and then inserting all the > readings via sqlalchemy core? That would allow you to insert them without > creating ORM objects, which people using numpy and a lot of data often like > to avoid. > > Then you could do... > > s = Sensor() > session.add(s) > session.add_readings(dates, voltages, values) > > it would look something like this... > > class Sensor(Base): > def add_readings(self, dates, voltages, values): > if not self.id: > # flush this to the session > session = object_session(self) > session.flush(objects=[self]) > # insert via core > > -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.