I was not able to find anything generating new instances. In facts the culprit was that with the `prop.schedule()` method I'm using multiprocessing. So the instances of the `Passage` objects that are in different processes, gain a different `id` (and moreover they're referenced instances of related objects as GroundStation or Satellite) and when I "join" the result from all the processes, the Passage objects contain references to the objects that became "new" since they were run in a separate process.
In facts I removed the parallel multiprocessing part, and everything is working as expected. How should I cope with multiprocessing (I'm already using `scoped session`)? Merging objects when joining the results in the different processes (tried on the fly, so far getting other errors... did not investigate enough)? Are there best practices? On Mon, Jan 15, 2018 at 11:53 AM, Simon King <si...@simonking.org.uk> wrote: > Yes, if you can't find where you are creating new Satellite instances, > I'd probably stick an assert statement in Satellite.__init__ and see > where it gets triggered. > > Simon > > On Mon, Jan 15, 2018 at 10:34 AM, Ruben Di Battista > <rubendibatti...@gmail.com> wrote: > > Dear Simon, > > > > thanks again for your kind help. > > > > Actually the creation of new instances is not intended. But I'm not > getting > > where they are created... > > > > I give you more insight: > > > > This is the scheduler object with the associated propagate() method > > > > class Scheduler(six.with_metaclass(abc.ABCMeta)): > > """ This class gets a list of GroundStation objects and a list of > > Satellites > > objects and compute all the passages of the Satellites over the > > GroundStations > > > > Args: > > sat_list(list): List of Satellites objects > > gs_list(list): List of GroundStation objects > > start_day(datetime): The datetime object representing the day > from > > which to start the propagation > > time_of_propagation(int): Number of hours to propagate > > [default:24] > > deltaT(float): Time step to use for angles retrieval, in seconds > > """ > > > > def __init__(self, sat_list, gs_list, start_day, > time_of_propagation=24, > > deltaT=0.05): > > # Monkey patch the Satellite class with the cost_function > specific > > # of the scheduler algorithm. > > sat_class = type(sat_list[0]) > > sat_class.cost_function = self.cost_function > > > > self.sat_list = sat_list > > self.gs_list = gs_list > > self.start_day = start_day > > self.time_of_propagation = time_of_propagation > > self.deltaT = deltaT > > > > def propagate(self): > > """ This method computes all the passages of the Satellites over > the > > GroundStations > > > > Args: > > > > Returns: > > all_passages(PassageList): A list ordered from the earliest > > passage\ > > of all passages > > > > Raises: > > ModelNotAvailable: When a satellite is too far from Earth and > > the > > models available in Orbital are not good, a > > ModelNotAvailable is > > raised > > """ > > > > all_passages = PassageList() > > > > # Loop Over the gs list > > for gs in self.gs_list: > > # Loop over the satellites list > > for sat in self.sat_list: > > # Compute all the passages in the specified period > > passages = \ > > sat.get_next_passes(gs, self.start_day, > > self.time_of_propagation, > > deltaT=self.deltaT) > > > > # Unfolding the list of passages in a flat list > > all_passages = all_passages + passages > > > > return all_passages > > > > > > It just basically loops over all the ground station and satellites and > > generates all the passages. Maybe the fact that I monkey patch the > satellite > > class induces the creation of a new instance of `Satellite`? > > > > The details of the `get_next_passes` method of the `Satellite` class for > > what concerns the `Passage` instance creation, skipping the algorithmic > > part, are: > > > > def _generate_passage(self, next_pass, ground_station, deltaT): > > """ This method returns a Passage Object from the data returned > from > > the original Orbital.get_next_passes method. > > > > """ > > > > aos, los, tca = next_pass > > > > return Passage(satellite=self, > > ground_station=ground_station, > > aos=aos, los=los, tca=tca, > > deltaT=deltaT) > > > > > > `self` should be a reference to the instance of `Satellite` already > loaded > > from DB. I will try to dive more into the code... > > > > > > Thanks a lot for the kind help of all of you, > > > > On Monday, January 15, 2018 at 10:06:24 AM UTC+1, Simon King wrote: > >> > >> On Sat, Jan 13, 2018 at 3:31 PM, Ruben Di Battista > >> <rubendi...@gmail.com> wrote: > >> > > >> > > >> > On Friday, January 12, 2018 at 10:54:49 AM UTC+1, Simon King wrote: > >> >> > >> >> If I understand your code correctly, scheduler.propagate() creates a > >> >> large number of Passage instances, and you only want a small subset > of > >> >> them to be added to the database. Is that correct? > >> > > >> > > >> > Correct! > >> > > >> >> > >> >> I would guess that the passages are getting added to the session > >> >> because you are setting their 'satellite' property to point to a > >> >> Satellite which is already in the database. This then causes the > >> >> passages to be added to the session due to the default cascade rules > >> >> on the relationship > >> >> (http://docs.sqlalchemy.org/en/latest/orm/cascades.html). > >> >> > >> >> If that really is the case, you can change the cascade rules for that > >> >> relationship, and then you'll probably need to explicitly add the > >> >> passages you want to *keep* to the session instead. > >> >> > >> >> Hope that helps, > >> >> > >> >> Simon > >> > > >> > > >> > Dear Simon, > >> > thank you. That was the case. Modifying the cascade disabling the > >> > backref > >> > cascade does not load in the DB the passages at propagation time. > >> > > >> > But now, when I manually add the subset of passages after the > >> > optimization, > >> > I get a: > >> > > >> > InvalidRequestError: Can't attach instance <Satellite at 0x108c4dfd0>; > >> > another instance with key [...] is already present in this section. > >> > > >> > So, I suppose that disabling the backref cascade now SQLAlchemy is not > >> > capable anymore to recognize the already loaded Satellite objects... > >> > > >> > Should I maybe merge somewhere? > >> > > >> > >> It sounds like your propagate() function is creating new Satellite > >> instances with the same primary key as instances that have been loaded > >> from the database. Merging is one way to avoid that, and the > >> UniqueObject pattern is another. It's difficult to say which is more > >> appropriate for your usage without seeing more code. But if > >> propagate() is only supposed to be calculating passages, I don't > >> understand why new Satellites are being created at all - shouldn't all > >> the Satellites already be loaded? > >> > >> Simon > > > > -- > > SQLAlchemy - > > The Python SQL Toolkit and Object Relational Mapper > > > > http://www.sqlalchemy.org/ > > > > To post example code, please provide an MCVE: Minimal, Complete, and > > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > > description. > > --- > > You received this message because you are subscribed to the Google Groups > > "sqlalchemy" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to sqlalchemy+unsubscr...@googlegroups.com. > > To post to this group, send email to sqlalchemy@googlegroups.com. > > Visit this group at https://groups.google.com/group/sqlalchemy. > > For more options, visit https://groups.google.com/d/optout. > > -- > SQLAlchemy - > The Python SQL Toolkit and Object Relational Mapper > > http://www.sqlalchemy.org/ > > To post example code, please provide an MCVE: Minimal, Complete, and > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > description. > --- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com. > To post to this group, send email to sqlalchemy@googlegroups.com. > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. > -- _ -. .´ |∞∞∞∞ ', ; |∞∞∞∞∞∞ ˜˜ |∞∞∞∞∞∞∞∞∞ RdB ,., |∞∞∞∞∞∞ .' '. |∞∞∞∞ -' `' http://rdb.is -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
