Hi,
I was just wondering at first whether there was a known issue with
ARRAY types, but here is the query:
spectra =
session.query(Spectrum).join(SpectrumHeader).filter(SpectrumHeader.filename
== spPlateFilename).all()
It should return ~1000 objects equaling about 120MB of data. In
Python, this
On Jul 19, 2010, at 9:52 AM, Adrian Price-Whelan wrote:
Hi,
I was just wondering at first whether there was a known issue with
ARRAY types, but here is the query:
spectra =
session.query(Spectrum).join(SpectrumHeader).filter(SpectrumHeader.filename
== spPlateFilename).all()
It should
does it take a few seconds to fully fetch all the results and it only gets
1000 rows ? or is that just to get the initial result?
I'm not sure what you mean by this - the query does return 1000 rows.
also if any of the individual columns are very large BLOBs or perhaps very
large PG
On Jul 19, 2010, at 11:46 AM, Adrian Price-Whelan wrote:
does it take a few seconds to fully fetch all the results and it only gets
1000 rows ? or is that just to get the initial result?
I'm not sure what you mean by this - the query does return 1000 rows.
if you run a query from a SQL
Try running that query directly against the database see how long that
takes.
Also try running explain on that query make sure it is using your
indexes properly.
Since you are only using a single filter make sure that the
spectrum_header.filename
has an index, and make sure your foreign key
Here is some more detailed information trying the query multiple ways:
Piping the command into psql and writing to a tmp file takes 12 seconds (tmp
file is 241MB):
time echo SELECT spectrum.pk AS spectrum_pk, spectrum.ra AS spectrum_ra,
spectrum.dec AS spectrum_dec, spectrum.values AS
Also, as a follow-up - inserting these ARRAYs into the database is
very slow as well, slower than expected that is. Granted, it is
looping over 1000 objects and inserting a few 4000 element arrays for
each object, but doing one big SQL query takes considerably less time
than session.add(object),
On Jul 19, 2010, at 1:53 PM, Adrian Price-Whelan wrote:
Here is some more detailed information trying the query multiple ways:
Piping the command into psql and writing to a tmp file takes 12 seconds (tmp
file is 241MB):
time echo SELECT spectrum.pk AS spectrum_pk, spectrum.ra AS
First off, thanks for your quick replies!
I will look into this, but I can tell you that the arrays are strictly numbers
and the array columns are type numeric[]
Thanks again,
Adrian
On Jul 19, 2010, at 3:47 PM, Michael Bayer wrote:
On Jul 19, 2010, at 1:53 PM, Adrian Price-Whelan wrote:
On Jul 19, 2010, at 3:55 PM, Adrian Price-Whelan wrote:
First off, thanks for your quick replies!
I will look into this, but I can tell you that the arrays are strictly
numbers and the array columns are type numeric[]
so its going to be doing that somewhat inefficient isinstance(list)
On Jul 19, 2010, at 4:08 PM, Michael Bayer wrote:
so its going to be doing that somewhat inefficient isinstance(list) thing
you see below, this appears to be how it handles arrays of arbitrary numbers
of dimensions. This could be optimized if the ARRAY type accepted some clues
as to how
On Jul 19, 2010, at 4:38 PM, thatsanicehatyouh...@mac.com wrote:
On Jul 19, 2010, at 4:08 PM, Michael Bayer wrote:
so its going to be doing that somewhat inefficient isinstance(list) thing
you see below, this appears to be how it handles arrays of arbitrary numbers
of dimensions. This
Hi Michael,
Assuming I understood you correctly, I tried the code below. The result was the
same (the query took 486 seconds). Since I autoload everything, I first adjust
the column types to the class you defined. Did I misunderstand something?
Thanks again for your help.
Cheers,
Demitri
---
On Jul 19, 2010, at 5:46 PM, thatsanicehatyouh...@mac.com wrote:
Hi Michael,
Assuming I understood you correctly, I tried the code below. The result was
the same (the query took 486 seconds). Since I autoload everything, I first
adjust the column types to the class you defined. Did I
On Jul 19, 2010, at 5:46 PM, thatsanicehatyouh...@mac.com wrote:
Hi Michael,
Assuming I understood you correctly, I tried the code below. The result was
the same (the query took 486 seconds). Since I autoload everything, I first
adjust the column types to the class you defined. Did I
On 07/19/2010 02:47 PM, Michael Bayer wrote:
On Jul 19, 2010, at 1:53 PM, Adrian Price-Whelan wrote
Here is some more detailed information trying the query multiple ways:
Piping the command into psql and writing to a tmp file takes 12 seconds (tmp
file is 241MB):
time echo SELECT
Hi,
Pasted below is a profile of the earlier code posted. I did update it with your
new definition of ARRAY Michael, but that only shaved off 18 seconds (down to
468s total) when run without the profiler.
The large number of __new__ calls roughly tracks with the number of numeric
values
On Jul 19, 2010, at 8:33 PM, thatsanicehatyouh...@mac.com wrote:
Hi,
Pasted below is a profile of the earlier code posted. I did update it with
your new definition of ARRAY Michael, but that only shaved off 18 seconds
(down to 468s total) when run without the profiler.
The large
18 matches
Mail list logo