Re: [GENERAL] dynamic schema modeling and performance

2017-04-12 Thread Dorian Hoxha
related > entities) and then creating a table for each schema. I don't expect having > more then a few thousand projects anytime soon. We have a relatively > targeted audience. > > Or would it be better to use jsonb data types and create a denormalized > index elsewhere? >

Re: [GENERAL] dynamic schema modeling and performance

2017-04-11 Thread Dorian Hoxha
If you are asking if you should go nosql, 99% you should not. On Tue, Apr 11, 2017 at 10:06 PM, Poul Kristensen wrote: > dataverse.org uses Postgresql and is well documented + it is completely > user driven. Maybe the concept could be usefull for you. I have installed > and configuration a few t

Re: [GENERAL] Free seminar in Malmo: PostgreSQL at 10TB and Beyond

2017-01-21 Thread Dorian Hoxha
Just share the slides/video in this thread friend. On Sat, Jan 21, 2017 at 10:57 AM, Seref Arikan wrote: > Any chance this will be recorded? The content looks great and would be of > interest to many. > > Cheers > Seref > > > On Sat, Jan 21, 2017 at 8:55 AM, Chris Travers > wrote: > >> Hi; >> >

[GENERAL] When updating row that has TOAST column, is the TOAST column also reinserted ? Or just the oid to the value?

2016-12-13 Thread Dorian Hoxha
Hello friends, When updating row that has TOAST column, is the TOAST column also inserted ? Or just the oid? Say I have a 1MB value in the TOAST column, and I update the row by changing another column, and since every update is an insert, will it also reinsert the toast-column ? The column that I

Re: [GENERAL] ZSON, PostgreSQL extension for compressing JSONB

2016-10-04 Thread Dorian Hoxha
@Aleksander ~everyone wants lower data storage and wants some kind of compression. Can this be made to automatically retrain when analyzing (makes sense?)? And create a new dictionary only if it changes compared to the last one. On Tue, Oct 4, 2016 at 5:34 PM, Aleksander Alekseev < a.aleks...@post

Re: [GENERAL] Multiple inserts

2016-09-23 Thread Dorian Hoxha
If the connection is in autocommit, then each statement will also incur a commit (write to the commit log on disk). On Fri, Sep 23, 2016 at 2:01 PM, Rakesh Kumar wrote: > Hi > > I am noticing that if I do this > > insert into table values(1,a) > insert into table values(2,b) > > insert into tabl

Re: [GENERAL] What limits Postgres performance when the whole database lives in cache?

2016-09-02 Thread Dorian Hoxha
Check out Voltdb (or Scylladb which is more different) for the changes in architecture required to achieve those performance increases. On Fri, Sep 2, 2016 at 7:32 PM, Andres Freund wrote: > On 2016-09-02 11:10:35 -0600, Scott Marlowe wrote: > > On Fri, Sep 2, 2016 at 4:49 AM, dandl wrote: > >

Re: [GENERAL] Uber migrated from Postgres to MySQL

2016-07-26 Thread Dorian Hoxha
Many comments: https://news.ycombinator.com/item?id=12166585 https://www.reddit.com/r/programming/comments/4uph84/why_uber_engineering_switched_from_postgres_to/ On Tue, Jul 26, 2016 at 7:39 PM, Guyren Howe wrote: > Honestly, I've never heard of anyone doing that. But it sounds like they > had

Re: [GENERAL] Multimaster

2016-04-10 Thread Dorian Hoxha
@Konstantin 1. It's ok in my cases. 2. Not required in my cases. 3. Just require users to use different servers for now I think. Sometimes(always?) users can be greedy with feature requests. 4. I want magically consistency + failover (I can instruct the client to retry all masters). Good-cluster i

Re: [GENERAL] JSONB performance enhancement for 9.6

2016-01-20 Thread Dorian Hoxha
Is there any database that actually supports what the original poster wanted ? The only thing that I know that's similar is bigtable/hbase/hypertable wide column store. The way it works is: break the lexicographically sorted rows into blocks of compressed XXKB, and then keeps an index on the start

Re: [GENERAL] Happy New Year

2016-01-01 Thread Dorian Hoxha
Happy Holidays! Let's have automatic sharding and distributed transactions! On Fri, Jan 1, 2016 at 3:51 PM, Melvin Davidson wrote: > Happy New Year to all! > > On Fri, Jan 1, 2016 at 2:40 AM, Michael Paquier > wrote: > >> On Fri, Jan 1, 2016 at 11:36 AM, Joshua D. Drake >> wrote: >> > Welcome

Re: [GENERAL] Is BDR support distributed table on slave nodes with ACID and join support.

2015-07-17 Thread Dorian Hoxha
1,2,3: You can't shard with BDR. It's only for multimaster (at least for now). Please read the docs. On Fri, Jul 17, 2015 at 9:02 AM, Amit Bondwal wrote: > Hello everyone, > > We ae working on a application in which we are using posgresql as a > database. We are sure that in future it will exten

Re: [GENERAL] Database designpattern - product feature

2015-06-02 Thread Dorian Hoxha
to specific groups of > products. > > Freundliche Grüsse > > Adrian Stern > unchained - web solutions > > adrian.st...@unchained.ch > +41 79 292 83 47 > > On Tue, Jun 2, 2015 at 12:58 PM, Dorian Hoxha > wrote: > >> Please do reply-all so you also reply to

Re: [GENERAL] Database designpattern - product feature

2015-06-02 Thread Dorian Hoxha
Please do reply-all so you also reply to the list. It's not ~good to develop with sqlite and deploy on posgresql. You should have your 'dev' as close to 'prod' as possible. Product_feature is another table in this case ? On Tue, Jun 2, 2015 at 11:44 AM, Adrian Stern wrote: > Database changeabi

Re: [GENERAL] Database designpattern - product feature

2015-06-01 Thread Dorian Hoxha
What about keeping all the dynamic columns of each product in a json(b) column ? Maybe you can make constraints that check the product_type and json->field->type ? On Mon, Jun 1, 2015 at 4:35 PM, Adrian Stern wrote: > Hi, I'm new > > I've been working as the sole administrator of various postgre

Re: [GENERAL] Stellar Phoenix File Recovery Software

2015-05-01 Thread Dorian Hoxha
That's spam. Can an admin ban this user/email ? On Fri, May 1, 2015 at 8:22 AM, recoverdata wrote: > When a file is deleted from your computer, its contents aren't immediately > destroyed. Windows simply marks the hard drive space as being available for > use by changing one character in the f

Re: [GENERAL] Streaming-SQL Database PipelineDB (Based on PostgreSQL 9.4) - Available in Beta

2015-04-25 Thread Dorian Hoxha
Hi Jeff, Looks good. Some questions: - how are data stored? btree, lsm etc? - indexes on the views ex doing a view of counting daily views for each url I may need to query by day or by url (table scans?) - distribution / sharding? how will stuff be consistent if the view's dat

Re: [GENERAL] PostgreSQL-related legal question

2015-03-11 Thread Dorian Hoxha
I don't see how it could have negative impact on the postgresql project? It's not like your job will be to find vulnerabilities and not disclose them ? On Wed, Mar 11, 2015 at 1:28 PM, Bill Moran wrote: > > I've been asked to sign a legal document related to a PostgreSQL- > related job opening.

Re: [GENERAL] select top_countries and for each country, select top_cities in that country, in 1 query

2014-08-28 Thread Dorian Hoxha
Thanks John. On Thu, Aug 28, 2014 at 2:35 PM, John McKown wrote: > On Mon, Aug 18, 2014 at 10:52 AM, John McKown > wrote: > > > > SELECT avg(b.countcountry)::int as "CountryCount", b.country, a.city, > > count(a.city) as "CityCount" > > FROM t AS a > > INNER JOIN > > (SELECT COUNT(country) AS

[GENERAL] select top_countries and for each country, select top_cities in that country, in 1 query

2014-08-18 Thread Dorian Hoxha
I have CREATE TABLE t (country text, city text); I want to get with 1 query, select count(country),country GROUP BY country ORDER BY count(country) DESC And for each country, to get the same for cities. Is it possible ? Thanks

Re: [GENERAL] Psycopg2 : error message.

2014-05-16 Thread Dorian Hoxha
Since I can't understand(french?) the language, what does it mean? Probably wrong authentication(password?). On Fri, May 16, 2014 at 1:19 PM, image wrote: > Thanks for your help. > > So i remove the first "try" + i remove the space before "conn=" > > Indeed i have a new error message in my DOS i

Re: [GENERAL] Psycopg2 : error message.

2014-05-16 Thread Dorian Hoxha
Also remove the first "try" + remove the space before "conn=" so you have this: #!/Python27/python.exe import psycopg2 # Try to connect conn=psycopg2.connect("dbname='busard_test' user='laurent' host='localhost' password='cactus'") cur = conn.cursor() On Fri, May 16, 2014 at 12:41 P

Re: [GENERAL] Psycopg2 : error message.

2014-05-16 Thread Dorian Hoxha
Try to use: dont catch the exception when you make the connection, to see the right error because: "i am unable to connect" may mean different things: 1.wrong user 2.wrong pass 3.server down etc On Fri, May 16, 2014 at 12:12 PM, image wrote: > Dear all, > > I'm writing a python script for a w

Re: [GENERAL] are analyze statistics synced with replication?

2014-05-15 Thread Dorian Hoxha
If you don't do read queries on the slave than it will not have hot data/pages/rows/tables/indexes in ram like the primary ? (it smoked weed and was happy doing nothing so it was happy, but when responsibility came (being promoted to master) it failed hard) On Thu, May 15, 2014 at 6:46 AM, Kevin

Re: [GENERAL] Log Data Analytics : Confused about the choice of Database

2014-05-14 Thread Dorian Hoxha
c/functions-json.html for expression indexes (also make partial indexes, when fields don't exist). (more functions will come with jsonb in 9.4). > > Peeyush Agarwal > > > On Tue, May 13, 2014 at 3:13 PM, Dorian Hoxha wrote: > >> Why not store session as integer? >> &

Re: [GENERAL] Full-Text Search question

2014-05-14 Thread Dorian Hoxha
Search for fulltext tutorial + json functions http://www.postgresql.org/docs/9.3/static/functions-json.html On Wed, May 14, 2014 at 1:00 AM, Jesus Rafael Sanchez Medrano < jesusraf...@gmail.com> wrote: > thanks... could you please be so kind to post some snippet/code for this? > > Att. > == > J

Re: [GENERAL] Log Data Analytics : Confused about the choice of Database

2014-05-13 Thread Dorian Hoxha
Why not store session as integer? And timestamp as timesamp(z?) ? If you know the types of events, also store them as integer , and save a map of them in the app or on another table ? And save the parameters as a json column, so you have more data-types? Hstore only has strings. Be carefull wit

Re: [GENERAL] Altering array(composite-types) without breaking code when inserting them and similar questions

2014-04-29 Thread Dorian Hoxha
So : 1. drop function 2. alter type: add column 3. create again function with new default argument in a transaction ? On Tue, Apr 29, 2014 at 4:22 PM, Merlin Moncure wrote: > On Sun, Apr 27, 2014 at 4:57 PM, Dorian Hoxha > wrote: > > Since my alternative is using j

Re: [GENERAL] Altering array(composite-types) without breaking code when inserting them and similar questions

2014-04-27 Thread Dorian Hoxha
composite_types by not specifying all of the columns for each composite_type ? So if i later add other columns to the composite_type, the insert query doesn't break ? Thanks On Mon, Apr 21, 2014 at 1:46 PM, Dorian Hoxha wrote: > Maybe the char array link is wrong ? I don't think an ar

Re: [GENERAL] Re: Postgresql the right tool (queue using advisory_locks + long transactions)

2014-04-27 Thread Dorian Hoxha
I'll probably ask @pgbouncer mailing list if i can use it with advisory_locks per session. If not, even raw sessions will be enough. Some comments inline. Thanks On Sun, Apr 27, 2014 at 10:07 PM, David G Johnston < david.g.johns...@gmail.com> wrote: > Dorian Hoxha wrote > > H

[GENERAL] Postgresql the right tool (queue using advisory_locks + long transactions)

2014-04-27 Thread Dorian Hoxha
Hi list, I am trying to use postgresql as a queue for long-jobs (max ~4 hours) using advisory_locks. I can't separate the long-job into sub-jobs. 1. At ultimate-best-case scenario there will be ~100 workers, so no web-scale performance required. Is there a problem with 100 open sessions

Re: [GENERAL] hstore binary representation of keys

2014-04-22 Thread Dorian Hoxha
Currently hstore is mongodb. It writes the keys everytime (and values as strings!, its mostly for dynamic keys or very sparse keys in my opinion). You can shorten keys,or put them in dedicated columns. I haven't read that there is a plan to compress the strings. On Tue, Apr 22, 2014 at 2:01 PM,

Re: [GENERAL] Altering array(composite-types) without breaking code when inserting them and similar questions

2014-04-21 Thread Dorian Hoxha
e top-posted (Dang iPhone). Continued below: > > On 04/20/2014 05:54 PM, Dorian Hoxha wrote: > > Because i always query the whole row, and in the other way(many tables) i > will always join + have other indexes. > > > On Sun, Apr 20, 2014 at 8:56 PM, Rob Sargent wrote: > >

Re: [GENERAL] Altering array(composite-types) without breaking code when inserting them and similar questions

2014-04-20 Thread Dorian Hoxha
o most future type changess. > > Sent from my iPhone > > On Apr 20, 2014, at 11:57 AM, Dorian Hoxha wrote: > > Was just curious about the overhead. > > I know the columns, but i may need to add other columns in the future. > Yeah, json is the alternative if this doesn't w

Re: [GENERAL] Altering array(composite-types) without breaking code when inserting them and similar questions

2014-04-20 Thread Dorian Hoxha
sider > using json or hstore if the data is unstructured. > El 20/04/2014 14:04, "Dorian Hoxha" escribió: > > Hi list, >> >> I have a >> create type thetype(width integer, height integer); >> create table mytable(thetype thetype[]); >> >

[GENERAL] Altering array(composite-types) without breaking code when inserting them and similar questions

2014-04-20 Thread Dorian Hoxha
Hi list, I have a create type thetype(width integer, height integer); create table mytable(thetype thetype[]); How can i make an insert statement so if i later add fields to the composite type, the code/query doesn't break ? Maybe by specifying the fields of the composite type in the query ? Thi

Re: [GENERAL] Could use some advice on search architecture

2014-04-19 Thread Dorian Hoxha
Postgresql has 2 column store, 1-in memory(cant remember the name) and http://www.citusdata.com/blog/76-postgresql-columnar-store-for-analytics On Sat, Apr 19, 2014 at 2:10 PM, Robin wrote: > bottom post > On 19/04/2014 12:46, R. Pasch wrote: > > On 19-4-2014 9:38, Robin wrote: > > > Well, giv

[GENERAL] New OpenSource columnstore from CitusData for Postgresql

2014-04-03 Thread Dorian Hoxha
Link to hackernews which also has some comments from the devs https://news.ycombinator.com/item?id=7523950 Very interesting: They use foreign data tables as an abstraction to separate the storage layer from the rest of the database.

Re: [GENERAL] Pagination count strategies

2014-04-03 Thread Dorian Hoxha
Cache the total ? On Thu, Apr 3, 2014 at 3:34 PM, Leonardo M. Ramé wrote: > Hi, in one of our systems, we added a kind of pagination feature, that > shows N records of Total records. > > To do this, we added a "count(*) over() as Total" field in our queries > in replacement of doing two queries

Re: [GENERAL] Complex sql, limit-for-each group by, arrays, updates

2013-12-03 Thread Dorian Hoxha
I'll probably go by using 3 queries and putting them in a transaction. Thanks On Wed, Nov 27, 2013 at 5:38 PM, David Johnston wrote: > Dorian Hoxha wrote > > Hi, > > > > So i have (table where data will be read) : > > CREATE TABLE data (vid,cid,pid,number); &

[GENERAL] Complex sql, limit-for-each group by, arrays, updates

2013-11-27 Thread Dorian Hoxha
Hi, So i have (table where data will be read) : CREATE TABLE data (vid,cid,pid,number); Tables where data will be writen/updated: CREATE TABLE pid_top_vids (pid, vid[]) CREATE TABLE pid_top_cids (pid, cid[]) CREATE TABLE cid_top_vids (cid, vid[]) I need to , possibly in 1 query, this will run o

[GENERAL] Multiple aggs,sums in 1 query

2013-11-22 Thread Dorian Hoxha
I have: create table tbl (a,b,c,d,e,f,g,h); And i need to select in 1 query ,or the most performant way: top 5(a) top 5(b) top 5(c): for each top5(c): top 5(d) count(f) GROUP BY f I can make these in separate queries but that means that postgresql would read the table multiple-times? Is it pos

[GENERAL] Return only non-null columns

2013-11-13 Thread Dorian Hoxha
Is it possible to: SELECT * FROM table But to return only non-null columns ? Since i use psycopg2 with DictCursor (a hashtable) it's better for me when i don't have the column that to have it as NULL. Thanks