Re: [GENERAL] full text search on hstore or json with materialized view?

2017-04-18 Thread George Neuner
On Tue, 18 Apr 2017 14:38:15 -0700, Rj Ewing wrote: >I am evaluating postgres for as a datastore for our webapp. We are moving >away from a triple store db due to performance issues. > >Our data model consists of sets of user defined attributes. Approx 10% of >the attributes tend to be 100% fille

Re: [GENERAL] Large data and slow queries

2017-04-18 Thread Samuel Williams
Thanks John. Yes, you are absolutely right, you want the index to be bottom heavy so you can cull as much as possible at the top. I'm familiar with that, once implementing a brute-force sudoku solver, it has the same principle. I've been working on this all afternoon. By reducing the longitude, la

Re: [GENERAL] Large data and slow queries

2017-04-18 Thread John R Pierce
On 4/18/2017 9:01 PM, Samuel Williams wrote: We want the following kinds of query to be fast: SELECT ... AND (latitude > -37.03079375089291 AND latitude < -36.67086424910709 AND longitude > 174.6307139779924 AND longitude < 175.0805140220076); I wonder if GIST would work better if you use the

Re: [GENERAL] Large data and slow queries

2017-04-18 Thread Samuel Williams
Oh, I've also tried earth distance and ll_to_earth in a GIST index... it was slower that the BTREE index on a small subset of data in my tests. On 19 April 2017 at 16:01, Samuel Williams wrote: > Hi. > > We have 400,000,000 records in a table (soon to be 800,000,000), here > is the schema (\d+) >

[GENERAL] Large data and slow queries

2017-04-18 Thread Samuel Williams
Hi. We have 400,000,000 records in a table (soon to be 800,000,000), here is the schema (\d+) https://gist.github.com/ioquatix/bddda36d9e4ffaceb7a62d7b62259121 We want the following kinds of query to be fast: SELECT DISTINCT "user_event"."user_id" FROM "user_event" WHERE "user_event"."what" IN

[GENERAL] Experiencing "compressed data is corrupted" error

2017-04-18 Thread Sar
Hello, I have a massive table in my database, about 1.5TB and around 3.5 billion rows. (that is one partition of massive table). When I try to do backup, just on this one big table, after a while I get following error:  pg_dump: Dumping the contents of table "ta_low" failed: PQgetResult() fa

Re: [GENERAL] streaming replication and archive_status

2017-04-18 Thread Jeff Janes
On Tue, Apr 18, 2017 at 5:20 AM, Luciano Mittmann wrote: > > > Hi Jeff, > > **Does each file in pg_xlog/archive_status/ have a corresponding file one > directory up? > > no corresponding file on pg_xlog directory. That is the question.. for > some reason or some parameter that I do not know, the

Re: [GENERAL] full text search on hstore or json with materialized view?

2017-04-18 Thread Rj Ewing
A step in the right direction for me, however it doesn't appear to support per field full text searching. It is exciting though! On Tue, Apr 18, 2017 at 3:00 PM, Bruce Momjian wrote: > On Tue, Apr 18, 2017 at 02:38:15PM -0700, Rj Ewing wrote: > > I am evaluating postgres for as a datastore for o

Re: [GENERAL] full text search on hstore or json with materialized view?

2017-04-18 Thread Bruce Momjian
On Tue, Apr 18, 2017 at 02:38:15PM -0700, Rj Ewing wrote: > I am evaluating postgres for as a datastore for our webapp. We are moving away > from a triple store db due to performance issues. > > Our data model consists of sets of user defined attributes. Approx 10% of the > attributes tend to be 1

[GENERAL] full text search on hstore or json with materialized view?

2017-04-18 Thread Rj Ewing
I am evaluating postgres for as a datastore for our webapp. We are moving away from a triple store db due to performance issues. Our data model consists of sets of user defined attributes. Approx 10% of the attributes tend to be 100% filled with 50% of the attributes having approx 25% filled. This

Re: [GENERAL] tuple statistics update

2017-04-18 Thread Tom Lane
Tom DalPozzo writes: > Hi, I'm using libpq to insert tuples in my table and keep looking at > statistics through psql instead. > I noticed that sometimes n_tuple_ins is not updated even after 1 min that > my transaction committed. If your session is just sitting, that's not surprising. I think s

Re: [GENERAL] tuple statistics update

2017-04-18 Thread Adrian Klaver
On 04/17/2017 09:18 AM, Tom DalPozzo wrote: Hi, I'm using libpq to insert tuples in my table and keep looking at statistics through psql instead. I noticed that sometimes n_tuple_ins is not updated even after 1 min that my transaction committed. My libpq connection is kept alive. If I close the c

[GENERAL] tuple statistics update

2017-04-18 Thread Tom DalPozzo
Hi, I'm using libpq to insert tuples in my table and keep looking at statistics through psql instead. I noticed that sometimes n_tuple_ins is not updated even after 1 min that my transaction committed. My libpq connection is kept alive. If I close the connection then the stats get updated. I know t

[GENERAL] [OT] Help: stories of database security and privacy

2017-04-18 Thread Lifepillar
Hi folks, in a few weeks I'll start a short course on the basics of database security for a group of high-school students with a background in elementary relational theory and SQL. I plan to explain the usage of grant/revoke, RBAC, DAC, and inference in statistical databases. I'd like to take the

Re: [GENERAL] # of connections and architecture design

2017-04-18 Thread Jeff Janes
On Tue, Apr 18, 2017 at 2:42 AM, Moreno Andreo wrote: > Hi all, > As many of you has read last Friday (and many has tired to help, too, > and I still thank you very much), I had a bad service outage. > I was pointed to reduce number of maximum connections using a pooler, and > that's what I'm

[GENERAL] With the password stored a another database, how to securely connecting to server

2017-04-18 Thread DrakoRod
Hi folks I've a questions about the secure conections. I'm designed a local app with a micro database with h2 (is the java app), this app connect to PostgreSQL with many users. I want store password encrypted with md5 in the microdb. But my question, with this encrypted password. How to securely

Re: [GENERAL] QGIS Loads Black Screen For PostGIS Out-Db Raster Data

2017-04-18 Thread Adrian Klaver
On 04/18/2017 07:37 AM, Osahon Oduware wrote: I forgot to mention that I have put the question on the qgis-user mailing list. Thanks for your help. Great, because I don't have a clue where to go from here. -- Adrian Klaver adrian.kla...@aklaver.com -- Sent via pgsql-general mailing list (p

Re: [GENERAL] QGIS Loads Black Screen For PostGIS Out-Db Raster Data

2017-04-18 Thread Osahon Oduware
I forgot to mention that I have put the question on the qgis-user mailing list. Thanks for your help. On Tue, Apr 18, 2017 at 3:31 PM, Osahon Oduware wrote: > Hi Adrian, > > I think it is a problem with QGIS as things seems fine from the PostGIS > end (raster data) going by the following queries

Re: [GENERAL] CANNOT USE ANY INDEX ON UPSERT (INSERT.....ON CONFLICT)

2017-04-18 Thread agharta
Hi, Thank you for your suggestion, i'll try to implement it. Many thanks, Cheers, Agharta Il 18/04/2017 12:38, Alban Hertroys ha scritto: On 18 Apr 2017, at 10:13, agharta wrote: Hi all, I have a problem with INSERT ... ON CONFLICT sql command. Reading 9.6 documentation i see that ON CON

Re: [GENERAL] QGIS Loads Black Screen For PostGIS Out-Db Raster Data

2017-04-18 Thread Osahon Oduware
Hi Adrian, I think it is a problem with QGIS as things seems fine from the PostGIS end (raster data) going by the following queries: *SELECT r_table_name, r_raster_column, out_db FROM raster_columns;* returned the following output: "test_rast","rast","[True, True, True]" *SELECT t.rid, (md).isou

Re: [GENERAL] QGIS Loads Black Screen For PostGIS Out-Db Raster Data

2017-04-18 Thread Adrian Klaver
On 04/18/2017 06:41 AM, Osahon Oduware wrote: Hi All, I have an out-db raster on PostGIS which I tried to load in QGIS (v 2.14.12-Essen) utilizing the DB Manager to establish a connection. I could view the raster table, but when I try to "Add to canvas" it displays a black screen instead of the

Re: [GENERAL] UDP buffer drops / statistics collector

2017-04-18 Thread Tim Kane
Okay, so I've run an strace on the collector process during a buffer drop event. I can see evidence of a recvfrom loop pulling in a *maximum* of 142kb. While I've had already increased rmem_max, it would appear this is not being observed by the kernel. rmem_default is set to 124kb, which would exp

Re: [GENERAL] Clone PostgreSQL schema

2017-04-18 Thread Melvin Davidson
On Tue, Apr 18, 2017 at 3:48 AM, R. Reiterer wrote: > Hi Melvin, > > after a first test, the function seems to work perfect! MANY THX!!! > > Regards, > > Reinhard > > Am 17.04.2017 17:21 schrieb Melvin Davidson: > >> On Mon, Apr 17, 2017 at 11:02 AM, Melvin Davidson >> wrote: >> >> On Mon, Apr 1

[GENERAL] QGIS Loads Black Screen For PostGIS Out-Db Raster Data

2017-04-18 Thread Osahon Oduware
Hi All, I have an out-db raster on PostGIS which I tried to load in QGIS (v 2.14.12-Essen) utilizing the DB Manager to establish a connection. I could view the raster table, but when I try to "Add to canvas" it displays a black screen instead of the raster image. I would be glad if someone could

Re: [GENERAL] Error During PostGIS Build From Source on Linux

2017-04-18 Thread Osahon Oduware
Hi Adrian, This suggestion helped to resolve the problem. Thanks a lot for your help. On Sat, Apr 15, 2017 at 8:02 PM, Adrian Klaver wrote: > On 04/15/2017 11:25 AM, John R Pierce wrote: > >> On 4/15/2017 10:49 AM, Adrian Klaver wrote: >> >>> >>> Probably going to involve dealing with ldconfig.

Fwd: [GENERAL] Error During PostGIS Build From Source on Linux

2017-04-18 Thread Osahon Oduware
Noted Something I meant to refer to earlier, it is considered bad etiquette to > cross post to multiple lists. It means in this case I have to remember to > drop the postgis list from my replies as I am not a member of that list and > I get a rejection notice if I forget.

Re: [GENERAL] streaming replication and archive_status

2017-04-18 Thread Luciano Mittmann
2017-04-17 20:04 GMT-03:00 Jeff Janes : > 2017-04-17 17:08 GMT-03:00 Jeff Janes : > >> On Mon, Apr 17, 2017 at 12:22 PM, Luciano Mittmann >> wrote: >> >>> Hi All, >>> >>> anyone knows why there are so many files in the directory >>> pg_xlog/archive_status/ in replication server? >>> >>> # pg_xlog

Re: [GENERAL] CANNOT USE ANY INDEX ON UPSERT (INSERT.....ON CONFLICT)

2017-04-18 Thread Alban Hertroys
> On 18 Apr 2017, at 10:13, agharta wrote: > > Hi all, > > I have a problem with INSERT ... ON CONFLICT sql command. > > Reading 9.6 documentation i see that ON CONFLICT command will accpets only > index_column_name or index_expression (unique composite/primary indexes are > valid too). >

[GENERAL] # of connections and architecture design

2017-04-18 Thread Moreno Andreo
Hi all, As many of you has read last Friday (and many has tired to help, too, and I still thank you very much), I had a bad service outage. I was pointed to reduce number of maximum connections using a pooler, and that's what I'm building in test lab, but I'm wondering if there's something

[GENERAL] CANNOT USE ANY INDEX ON UPSERT (INSERT.....ON CONFLICT)

2017-04-18 Thread agharta
Hi all, I have a problem with INSERT ... ON CONFLICT sql command. Reading 9.6 documentation i see that ON CONFLICT command will accpets only index_column_name or index_expression (unique composite/primary indexes are valid too). So, my problem is that i can't create any type of upsert-valid

[GENERAL] UDP buffer drops / statistics collector

2017-04-18 Thread Tim Kane
Hi all, I'm seeing sporadic (but frequent) UDP buffer drops on a host that so far I've not been able to resolve. The drops are originating from postgres processes, and from what I know - the only UDP traffic generated by postgres should be consumed by the statistics collector - but for whatever r