Re: [HACKERS] proposal: schema variables

2017-10-30 Thread Hannu Krosing
but you can always do with a (id, value) as ( values (1, 'foo'), (2, 'bar'), (3, 'baz') ) select set_config('custom.value',(select value from a where id = 2),true); if you are worried about the evaluation order On 29 October 2017 at 09:51, Chris Travers wrote: > > > On Sat, Oct 28, 2017 at 4

Re: [HACKERS] Why we lost Uber as a user

2016-08-01 Thread Hannu Krosing
und that, like doing pg_dump > on a slave. Is there any theoretical obstacle which would make it impossible to teach VACUUM not to hold back the whole vacuum horizon, but just to leave a single transaction alone in case of a long-running REPEATABLE READ transaction ? -- Hannu Krosing PostgreSQL Con

Re: [HACKERS] Design for In-Core Logical Replication

2016-07-25 Thread Hannu Krosing
host=bar > user=repuser PUBLICATION mypub; > For the pgq-like version which consider a PUBLICATION just as list of tables to subscribe, I would add CREATE SUBSCRIPTION mysub WITH CONNECTION 'dbname=foo host=bar user=repuser' PUBLICATION mypub, mypub1; ALTER SUBSCRIPTION mysub DRO

Re: [HACKERS] Column Redaction

2014-10-10 Thread Hannu Krosing
gested above is to write a special type, which redacts data on output. You can even make the type output function dependent on backup role. Just make sure that users are aware that it is not really a security feature which protects against attackers. Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Anonymous code block with parameters

2014-09-18 Thread Hannu Krosing
On 09/19/2014 12:14 AM, Hannu Krosing wrote: > On 09/18/2014 10:40 PM, Marko Tiikkaja wrote: >> On 2014-09-18 10:29 PM, Vik Fearing wrote: >>> On 09/18/2014 10:16 PM, Hannu Krosing wrote: >>>> I guess it proves (a little) that WITH is the right place to do these &g

Re: [HACKERS] Anonymous code block with parameters

2014-09-18 Thread Hannu Krosing
On 09/18/2014 10:40 PM, Marko Tiikkaja wrote: > On 2014-09-18 10:29 PM, Vik Fearing wrote: >> On 09/18/2014 10:16 PM, Hannu Krosing wrote: >>> I guess it proves (a little) that WITH is the right place to do these >>> kind of things ... >> >> I've been w

Re: [HACKERS] Anonymous code block with parameters

2014-09-18 Thread Hannu Krosing
ralize WITH clause, so there SEQENCES, VIEWS, .. can be > defined for "single usage" +2 I just proposed the same thing in another branch of this discussion before reading this :) I guess it proves (a little) that WITH is the right place to do these kind of things ... Cheers -- H

Re: [HACKERS] Anonymous code block with parameters

2014-09-18 Thread Hannu Krosing
ul. But it's simply a different >> feature. >> > > > +1 > > If my memory isn't failing, when we implemented DO there were > arguments for this additional feature, but we decided that it wouldn't > be done at least on the first round. But we've had DO

Re: [HACKERS] Anonymous code block with parameters

2014-09-16 Thread Hannu Krosing
ut then we would still miss anonymous/in-line/on-the-spot functions > > Any new useful syntax should be clean, simple, natural and shorter > than create function ... This is not how SQL works, nor ADA nor pl/pgsql ;) > > and without risks a conflicts with ANSI SQL > > I prefer a typed session variables, where is not risk of SQL injection > or some performance lost. The benefit of typed server side variables > can be for wide group of users. Agreed but this would be a much bigger project, as Heikki already mentioned re. temp things on replicas. especially if typed session variables could hold temporary functions . DECLARE FUNCTION mytempfucntion () ... Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ

Re: [HACKERS] Anonymous code block with parameters

2014-09-16 Thread Hannu Krosing
ot returning anything. DO you have a better syntax for "temporary / in-line functions" ? What I would like to to is to make DO blocks equal to any other data source, so you could do WITH mydoblock(col1, col2)(DO $$ ... $$ LANGUAGE plpgsql RETURNS TABLE (col1 text, col2 int4)) SELECT *

Re: [HACKERS] PL/pgSQL 1.2

2014-09-04 Thread Hannu Krosing
tatements* in pl/pgsql when we really want scalars. My understanding is that one main drivers of starting this thread was wanting also guaranteed SCALAR versions of these. And wanting them in a way that is easy to use. Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ

Re: [HACKERS] PL/pgSQL 1.2

2014-09-03 Thread Hannu Krosing
On 09/04/2014 12:17 AM, Marko Tiikkaja wrote: > On 2014-09-03 23:19, Hannu Krosing wrote: >> 1. Conditions for number of rows returned by SELECT or touched by UPDATE >

Re: [HACKERS] PL/pgSQL 1.2

2014-09-03 Thread Hannu Krosing
of "" around `` forced use of quote_ident() 3. A way to tell pl/pggsql not to cache plans fro normal queries --- This could be done using a #pragma or special /* NOPLANCACHE */ comment as suggested by Pavel Or we

Re: [HACKERS] PL/pgSQL 2

2014-09-02 Thread Hannu Krosing
pletely unsubstantiated claims about *all* NoSQL scaling *easily* and *transparently* beyond one node :) -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PL/pgSQL 2

2014-09-02 Thread Hannu Krosing
de the > latter very clearly... Have you ever tried any of the "real" NoSQL products version of "infinite scalability" ? We are no worse than most if you use just the unstructured part (which is what the NoSQL crowd provides) and something like pl/proxy for scaling. Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ

Re: [HACKERS] On partitioning

2014-09-01 Thread Hannu Krosing
On 09/01/2014 05:52 PM, Andres Freund wrote: > On 2014-08-29 20:12:16 +0200, Hannu Krosing wrote: >> It would need to replace plain tid (pagenr, tupnr) with triple of (partid, >> pagenr, tupnr). >> >> Cross-partition indexes are especially needed if we want to allow pu

Re: [HACKERS] PL/pgSQL 2

2014-09-01 Thread Hannu Krosing
edding. > >> From the top of my head, these are Things I personally would want to see >> in plpgsql2: > - Accept RECORD input, dynamic access to fields of records without > resorting to hstore hacks. This is certainly my #1. > Also, an easy way to tell pl/pgsql to *not* cache

Re: [HACKERS] PL/pgSQL 2

2014-09-01 Thread Hannu Krosing
On 09/01/2014 12:55 PM, Andres Freund wrote: > On 2014-09-01 12:49:22 +0200, Marko Tiikkaja wrote: >> On 9/1/14 12:12 PM, Andres Freund wrote: >>> On 2014-09-01 12:00:48 +0200, Marko Tiikkaja wrote: >>>> On 9/1/14 11:53 AM, Hannu Krosing wrote: >>>>>

Re: [HACKERS] PL/pgSQL 2

2014-09-01 Thread Hannu Krosing
On 09/01/2014 12:00 PM, Marko Tiikkaja wrote: > On 9/1/14 11:53 AM, Hannu Krosing wrote: >> On 09/01/2014 11:24 AM, Andres Freund wrote: >>> Look at the *disaster* the few minor changes in python3 were. It's now, >>> years after, only starting to get used again.

Re: [HACKERS] PL/pgSQL 2

2014-09-01 Thread Hannu Krosing
fter, only starting to get used again. > > You're going to have to find a more gradual way of doing this. Probably a better way (and there has been some talk of it) is having some kind of PRAGMA functionality, or pl/pgsql specific LOCAL SET to affect "just this function" and not

Re: [HACKERS] On partitioning

2014-08-31 Thread Hannu Krosing
LE FOR PARTITIONMASTER WITH (ALL_INDEXES_SAME=ON, SAME_STRUCTURE_ALWAYS=ON, SINGLE_INHERITANCE_ONLY=ON, NESTED_INHERITS=OFF, PARTITION_FUNCTION=default_range_partitioning(int) ); and then force these when adding inherited tables (in this case partition tables) either

Re: [HACKERS] On partitioning

2014-08-29 Thread Hannu Krosing
t;TABLE PARTITION ", but for example if in >the future we add hash partitioning, we might need some non-integer >addressing (OTOH assigning sequential numbers to hash partitions doesn't >seem so bad). Discussing with users of other DBMSs partitioning feature, >

Re: [HACKERS] On partitioning

2014-08-29 Thread Hannu Krosing
Cross-partition indexes are especially needed if we want to allow putting UNIQUE constraints on non-partition-key columns. Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-08 Thread Hannu Krosing
sists mostly of a strictly-increasing series of integer offsets. How hard and how expensive would it be to teach pg_lzcompress to apply a delta filter on suitable data ? So that instead of integers their deltas will be fed to the "real" compressor -- Hannu Krosing PostgreSQL Consultant P

Re: [HACKERS] UPDATE SET (a,b,c) = (SELECT ...) versus rules

2014-06-17 Thread Hannu Krosing
On 06/17/2014 11:22 AM, Vik Fearing wrote: > On 06/17/2014 09:43 AM, Hannu Krosing wrote: >> On 06/14/2014 09:35 PM, Tom Lane wrote: >>>> As I mentioned awhile ago, I'm thinking about implementing the >>>> SQL-standard construct >>>> &

Re: [HACKERS] UPDATE SET (a,b,c) = (SELECT ...) versus rules

2014-06-17 Thread Hannu Krosing
, > it doesn't seem appropriate to work harder than this; and frankly > I don't see how we could avoid multiple sub-select evaluations anyway, > if the NEW references are in WHERE or other odd places. > > Another possible answer is to just throw a "not implemented

Re: [HACKERS] "RETURNING PRIMARY KEY" syntax extension

2014-06-10 Thread Hannu Krosing
On 06/10/2014 11:02 AM, Tom Dunstan wrote: > On 10 June 2014 17:49, Hannu Krosing <mailto:ha...@2ndquadrant.com>> wrote: > > RETURNING GENERATED KEYS perhaps, but then how do we determine that? > What about RETURNING CHANGED FIELDS ? > > Might be quite c

Re: [HACKERS] "RETURNING PRIMARY KEY" syntax extension

2014-06-10 Thread Hannu Krosing
t; > [4] > http://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#execute(java.lang.String,%20java.lang.String[]) > <http://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#execute%28java.lang.String,%20java.lang.String[]%29> -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ

Re: [HACKERS] "RETURNING PRIMARY KEY" syntax extension

2014-06-09 Thread Hannu Krosing
mary keys, and not for example generated alternate keys ? Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://

Re: [HACKERS] json casts

2014-05-27 Thread Hannu Krosing
rrect cast. > > I'd be inclined to think a more useful answer to this issue would be to > make json.c special-case timestamps, as it already does for numerics. > > regards, tom lane But I agree that special-casing the code to use the de-facto json standa

Re: [HACKERS] Priority table or Cache table

2014-05-26 Thread Hannu Krosing
On 05/26/2014 04:16 PM, Fujii Masao wrote: > On Sun, May 25, 2014 at 6:52 PM, Hannu Krosing wrote: >> On 05/20/2014 01:46 PM, Fujii Masao wrote: >>> On Mon, Mar 17, 2014 at 1:16 PM, Haribabu Kommi >>> wrote: >>>> ... >>>> I Implemented a pr

Re: [HACKERS] Priority table or Cache table

2014-05-25 Thread Hannu Krosing
a possibility to override this in postgresql.conf for optimising what you described but for most uses it is best to be in the database, at least to get started. Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent vi

Re: [HACKERS] "Considerer Harmful Considered Harmful" categorized as Mostly Harmless

2014-04-30 Thread Hannu Krosing
t;> >> I wasn't sure if the whole article was a parody. >> >> > > > The rebuttal would be: "'Considered Harmful' Considered Harmful" > Considered Harmful. > > Don't you just love recursion? Nah, I'd categorize it as "Most

Re: [HACKERS] 9.4 Proposal: Initdb creates a single table

2014-04-24 Thread Hannu Krosing
ntifier and just warn about the dangers in documentation ? we could expose it something like next_uuid(); As the article points out " Since the identifiers have a finite size, it is possible for two differing items to share the same identifier." so it is a known limitation of UUID and

Re: [HACKERS] 9.4 Proposal: Initdb creates a single table

2014-04-23 Thread Hannu Krosing
mber when we didn't have a database called Postgres? Remember how > much simpler life is now? Remember that now. > > We can also insert a single row, Id = 0 with "Postgres sample data" in > it, but that seems a step too far. > -- Hannu Krosing PostgreSQL Consulta

Re: [HACKERS] AXLE Plans for 9.5 and 9.6

2014-04-22 Thread Hannu Krosing
On 04/22/2014 02:04 PM, Simon Riggs wrote: > On 22 April 2014 00:24, Josh Berkus wrote: >> On 04/21/2014 03:41 PM, Simon Riggs wrote: >>> Storage Efficiency >>> * Compression >>> * Column Orientation >> You might look at turning this: >> >> http://citusdata.github.io/cstore_fdw/ >> >> ... into a m

Re: [HACKERS] AXLE Plans for 9.5 and 9.6

2014-04-22 Thread Hannu Krosing
t of Postgres. What would be of more general usefulness is probably better planning and better performance of FDW interface. So instead of integrating one specific FDW it would make sense to improve postgresql so that it can use (properly written) FDWs at native speeds Regards -- Hannu Krosi

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-22 Thread Hannu Krosing
On 04/17/2014 10:39 PM, Andres Freund wrote: > On 2014-04-17 13:33:27 -0700, Peter Geoghegan wrote: >> Just over 99.6% of pages (leaving aside the meta page) in the big 10 >> GB pgbench_accounts_pkey index are leaf pages. What is the depth of b-tree at this percentage ? Cheers Hannu -- Sent vi

Re: [HACKERS] DISCARD ALL (Again)

2014-04-21 Thread Hannu Krosing
register a clean-up function and not trying to second-guess what global cached state *might* exists. Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] DISCARD ALL (Again)

2014-04-21 Thread Hannu Krosing
On 04/18/2014 01:38 AM, Fabrízio de Royes Mello wrote: > > > > On Thu, Apr 17, 2014 at 6:51 PM, Alvaro Herrera > mailto:alvhe...@2ndquadrant.com>> wrote: > > > > It does sounds a legitimate feature request to me. I don't remember if > > we honored the request to add resetting of cached sequences,

Re: [HACKERS] Question about optimising (Postgres_)FDW

2014-04-16 Thread Hannu Krosing
On 04/16/2014 03:16 PM, Hannu Krosing wrote: > On 04/16/2014 01:35 PM, Etsuro Fujita wrote: >> (2014/04/16 6:55), Hannu Krosing wrote: > ... >> Maybe I'm missing something, but I think that you can do what I think >> you'd like to do by the following procedure: &g

Re: [HACKERS] Question about optimising (Postgres_)FDW

2014-04-16 Thread Hannu Krosing
On 04/16/2014 01:35 PM, Etsuro Fujita wrote: > (2014/04/16 6:55), Hannu Krosing wrote: ... > > Maybe I'm missing something, but I think that you can do what I think > you'd like to do by the following procedure: No, what I'd like PostgreSQL to do is to 1. select th

Re: [HACKERS] Question about optimising (Postgres_)FDW

2014-04-15 Thread Hannu Krosing
On 04/16/2014 06:12 AM, Hannu Krosing wrote: > On 04/16/2014 01:25 AM, Tom Lane wrote: >> Hannu Krosing writes: >>> Is there a way to force it to prefer a plan where the results of (select >>> id from onemillion where data > '0.9' limit 100) >

Re: [HACKERS] Question about optimising (Postgres_)FDW

2014-04-15 Thread Hannu Krosing
On 04/16/2014 01:25 AM, Tom Lane wrote: > Hannu Krosing writes: >> Is there a way to force it to prefer a plan where the results of (select >> id from onemillion where data > '0.9' limit 100) >> are passed to FDW as a single IN ( = ANY(...)) query and are retr

[HACKERS] Question about optimising (Postgres_)FDW

2014-04-15 Thread Hannu Krosing
a > '0.9' limit 100) are passed to FDW as a single IN ( = ANY(...)) query and are retrieved all at once ? If not, how hord would it be to add this feature ? -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ --

Re: [HACKERS] Autonomous Transaction (WIP)

2014-04-08 Thread Hannu Krosing
On 04/09/2014 08:44 AM, Pavan Deolasee wrote: > On Wed, Apr 9, 2014 at 11:03 AM, Rajeev rastogi > mailto:rajeev.rast...@huawei.com>> wrote: > > > Though autonomous transaction uses mixed approach of > sub-transaction as well as main > transaction, transaction state of autonomous transac

Re: [HACKERS] json(b) equality rules

2014-04-03 Thread Hannu Krosing
On 04/03/2014 04:32 AM, Oleg Bartunov wrote: > Hi there, > > I'm wondering if we should follow all js equility rules as > nicely visualized in > http://strilanc.com/visualization/2014/03/27/Better-JS-Equality-Table.html Probably not as JSON is general interchange format. If somebody wants JavaScri

Re: [HACKERS] GSoC proposal - "make an unlogged table logged"

2014-03-03 Thread Hannu Krosing
n you'd have to do that is to make the table contents > appear on slave servers. If you don't consider replication then it might > seem easier. So switch on logging and then perform CLUSTER/VACUUM FULL ? Should this work, or is something extra needed ? Cheers -- Hannu Krosing PostgreS

Re: [HACKERS] jsonb and nested hstore

2014-02-27 Thread Hannu Krosing
On 02/26/2014 09:17 AM, Christophe Pettus wrote: > On Feb 25, 2014, at 1:57 PM, Hannu Krosing wrote: > >> It is not in any specs, but nevertheless all major imlementations do it and >> some code depends on it. > I have no doubt that some code depends on it, but "all ma

Re: [HACKERS] jsonb and nested hstore

2014-02-26 Thread Hannu Krosing
On 02/26/2014 07:41 PM, Josh Berkus wrote: > On 02/26/2014 07:02 AM, Merlin Moncure wrote: >> On Tue, Feb 25, 2014 at 3:57 PM, Hannu Krosing wrote: >>> It is not in any specs, but nevertheless all major imlementations do it and >>> some code depends on it. >>&g

Re: [HACKERS] jsonb and nested hstore

2014-02-25 Thread Hannu Krosing
aviour is currently also met only by json and not by jsonb. > Merlin: We should present them side-by-side with a complex comparison. > Robert: Josh wants to junk all relational data and use only jsonb! I > mean, really, WTF? Cheers -- Hannu Krosing PostgreSQL Consultant Performance,

Re: [HACKERS] WAL Rate Limiting

2014-02-20 Thread Hannu Krosing
t I don't think these should be controlled by the same GUC though you may want to have some checking for sane values between the two Maybe call this one `maintenance_wal_rate_limit_delay` same way as we have `maintenance_work_mem` Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalabil

Re: [HACKERS] Ctrl+C from sh can shut down daemonized PostgreSQL cluster

2014-02-15 Thread Hannu Krosing
be attached to > the terminal and die when I C-c it but if it's started by pg_ctl I > would have thought it was running independently of my terminal and shell. > In this case maybe it is pg_ctl which should do the deamoinizing ? Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ

Re: [HACKERS] jsonb and nested hstore

2014-02-11 Thread Hannu Krosing
thing that we've never attached to any other type before. > For example, why didn't we attach a version code to the json type send > function? JSON is supposed to be a *standard* way of encoding data in strings. If the ever changes, it will not be JSON type anymore. Cheers -- Ha

Re: [HACKERS] jsonb and nested hstore

2014-02-10 Thread Hannu Krosing
out, strlen(out)); >> +PG_RETURN_BYTEA_P(pq_endtypsend(&buf)); >> +} > I'd suggest making the format discernible from possible different future > formats, to allow introducing a proper binary at some later time. Maybe > just send a int8 first, containing the format. +

Re: [HACKERS] jsonb and nested hstore

2014-02-10 Thread Hannu Krosing
On 02/05/2014 06:48 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 02/05/2014 11:40 AM, Tom Lane wrote: >>> switching to "binary is the same as text" may well be the most prudent >>> path here. >> If we do that we're going to have to live with that forever, aren't we? > Yeah, but the other si

Re: [HACKERS] jsonb and nested hstore

2014-01-30 Thread Hannu Krosing
argument is the main >> reason why I think this should be handled now (not forcing hstore >> dependency to handle complex json is gravy). > > > We can't reference any hstore code in jsonb. There is no guarantee > that hstore will even be loaded. > > We'd

Re: [HACKERS] Standalone synchronous master

2014-01-26 Thread Hannu Krosing
ncourage the submitter to resumbit and improved version of this patch > (one with more monitorability) for 9.5 CF1. That'll give us a whole > dev cycle to argue about it. > Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadra

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-17 Thread Hannu Krosing
ne actively degrades the > performance of the other. Right, got it now. Yes. A step in right solutions would be some way to tune this on per-device basis, but as large part of this in linux seems to be driven from the keeping-vm-clean side it guess it will be far from simple. > > Cheers,

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Hannu Krosing
On 01/15/2014 02:01 PM, Jan Kara wrote: > On Wed 15-01-14 12:16:50, Hannu Krosing wrote: >> On 01/14/2014 06:12 PM, Robert Haas wrote: >>> This would be pretty similar to copy-on-write, except >>> without the copying. It would just be >>> forget-from-the-

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Hannu Krosing
On 01/15/2014 12:16 PM, Hannu Krosing wrote: > On 01/14/2014 06:12 PM, Robert Haas wrote: >> This would be pretty similar to copy-on-write, except >> without the copying. It would just be >> forget-from-the-buffer-pool-on-write. > +1 > > A version of this could proba

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-15 Thread Hannu Krosing
ffers are dirty inside postgresql. Maybe in background writer. Question to LKM folks - will kernel react well to frequent changes to /proc/sys/vm/dirty_* ? How frequent can they be (every few second? every second? 100Hz ?) Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Hannu Krosing
you want the kernel to make that choice for you? The short answer is "crash safety". A database system worth its name must make sure that all data reported as stored to clients is there even after crash. Write ahead log is the means for that. And writing wal files and data pages has to be in certain order to guarantee consistent recovery after crash. -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Hannu Krosing
On 01/14/2014 09:39 AM, Claudio Freire wrote: > On Tue, Jan 14, 2014 at 5:08 AM, Hannu Krosing wrote: >> Again, as said above the linux file system is doing fine. What we >> want is a few ways to interact with it to let it do even better when >> working with postgresql by tel

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Hannu Krosing
file handling has been much better for most of active development of postgresql we have been able to avoid it and still have reasonable performance. What was been pointed out above are some (allegedly desktop/mobile influenced) decisions which broke good performance. Cheers -- Hannu Krosing Po

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Hannu Krosing
pied away for potential modifying but ended up clean in the end. And let the linux kernel decide if and how long to keep these pages in its cache using its superior knowledge of disk subsystem and about what else is going on in the system in general. Just food for thought We want to hav

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-13 Thread Hannu Krosing
> Let's just add user defined operator for '[]' (weirdly-positioned but 2 argument, almost infix :) ) and add that to JSON arrays to get 0-based ones into poastgresq ;) Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-13 Thread Hannu Krosing
eally be written - or marked dirty to be written later - more levels than just to the linux cache, as they *already* are on the disk. It is probably ok to put them in the LRU position as they are "written" out from postgresql, though it may be better if we get some more control over where

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Hannu Krosing
r which became disconnected from the world and slave is available) or c) do something else depending on circumstances/policy :) NB! Note that in case of b) 'syncrep_taking_too_long_command' will very likely also not reach the monitor daemon, so it can not relay on this as main trigger!

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Hannu Krosing
s broken. People here are telling you that it would not be safe, use at least RAID-1 if you want availability Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Hannu Krosing
the driver that one connection is bound as "the slave" of another and driver can manage using the right LSNs. That is the last LSN received from master is always attached to queries on slaves. Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availabil

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
On 01/09/2014 04:15 PM, MauMau wrote: > From: "Hannu Krosing" >> On 01/09/2014 01:57 PM, MauMau wrote: >>> Let me ask a (probably) stupid question. How is the sync rep >>> different from RAID-1? >>> >>> When I first saw sync rep, I expected

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
gonna do it. > > Yeah, and I think that the logging command that was suggested allows > for that *if configured correctly*. *But* for relying on this, we would also need to make logging *synchronous*, which would probably not go down well with many people, as it makes things even more fragile from availability viewpoint (and slower as well). Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
pid question. How is the sync rep > different from RAID-1? > > When I first saw sync rep, I expected that it would provide the same > guarantees as RAID-1 in terms of durability (data is always mirrored > on two servers) and availability (if one server goes down, another > se

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
actions unless > we can guarantee the data's been replicated. For single host scenario this would be similar to asking for a mode which turns fsync=off in case of disk failure :) Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant N

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
connecting with that name Also, I can attest that I have had clients who want exactly that - a system stop until admin intervention in case of a designated sync standby failing. And they actually run more than one standby, they just want to make sure that sync rep to 2nd data center always happens

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
that the leader fails. But there is no guarantee that the synchronous replica actually is ahead of async ones. > In this scenario > though, if the "successor" fails, you actually probably want to keep > accepting writes; since you weren't using synchronous for durability &

Re: [HACKERS] nested hstore patch

2013-12-23 Thread Hannu Krosing
ertain elegance to this, it also > sorta looks like punctuation soup. I can't help wondering whether > we'd be better off sticking to function names. > Has anybody looked into how hard it would be to add "method" notation to postgreSQL, so that instead of calling get

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2013-12-16 Thread Hannu Krosing
ess > that would make it "inverse aggregate transition function". > Would that make the CREATE AGGREGATE option be INVFUNC ? > > Any other ideas or +1's for any of the existing ones? +1, inverse good :) > > Regards > > David Rowley > > -- > Ants Aasma > > -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ

Re: [HACKERS] [PATCH] Add transforms feature

2013-12-11 Thread Hannu Krosing
WITH (attribute, ...) available in CREATE FUNCTION syntax for passing function-specific flags ? Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ANALYZE sampling is too good

2013-12-10 Thread Hannu Krosing
read that way that > doesn't introduce bias? > Initially/experimentally we could just compare it to our current approach :) That is, implement *some* block sampling and then check it against what we currently have. Then figure out the bad differences. Rinse. Repeat. Cheers -- Hannu

Re: [HACKERS] [PATCH] Add transforms feature

2013-12-06 Thread Hannu Krosing
uages (standard JavaScript, CoffeeScript, LiveScript) from the same codebase. Taking the plv8 ideas further we could also create a JavaScript-based "sandboxed python" using thins like skulpt and pyjamas which compile python source code to JavaScript VM and inherit all the sandboxing of v8. C

Re: [HACKERS] [PATCH] Add transforms feature

2013-11-27 Thread Hannu Krosing
nk we should look at allowing the conversion functions to be written in the target pl language in addition to C. I'll see if I can resurrect my patch for support of "cstring" and "internal" types in pl/python function defs for this. -- Hannu Krosing PostgreSQL Consulta

Re: [HACKERS] [PATCH] Add transforms feature

2013-11-26 Thread Hannu Krosing
ction takes a transformed hstore, and this other one > takes a non-transformed hstore. Now life is good. But that's not > what is being proposed. You mean something like CREATE FUNCTION f(i int, h1 hstore USING TRANSFORM x, h2 hstore) ... where h1 would go through transform x and 1 a

Re: [HACKERS] [PATCH] Add transforms feature

2013-11-26 Thread Hannu Krosing
uld be calling a pl/v8 function from pl/python and converting directly between integers in both, without going through PostgreSQL type. Another and maybe even more interesting would be automatic null-transforms between two pl/python functions. Cheers -- Hannu Krosing PostgreSQL Consultant Perfo

Re: [HACKERS] additional json functionality

2013-11-20 Thread Hannu Krosing
On 11/18/2013 06:49 PM, Josh Berkus wrote: > On 11/18/2013 06:13 AM, Peter Eisentraut wrote: >> On 11/15/13, 6:15 PM, Josh Berkus wrote: >>> Thing is, I'm not particularly concerned about *Merlin's* specific use >>> case, which there are ways around. What I am concerned about is that we >>> may hav

Re: [HACKERS] additional json functionality

2013-11-18 Thread Hannu Krosing
l reasonably 'json' but not too easy to confuse with existing json when typing And it perhaps hints better at the main difference from string-json, namely that it is an object and not textual source code / notation / processing info . Cheers -- Hannu Krosing PostgreSQL Co

Re: [HACKERS] additional json functionality

2013-11-17 Thread Hannu Krosing
ts/Dictionaries as per JSON standard) > as in “we actually know how to store your json documents”? > > Regards, Cheers -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] additional json functionality

2013-11-17 Thread Hannu Krosing
On 11/17/2013 09:02 PM, David E. Wheeler wrote: > On Nov 16, 2013, at 2:04 PM, Hannu Krosing wrote: > >>> It’s still input and output as JSON, though. >> Yes, because JavaScript Object Notation *is* a serialization format >> (aka Notation) for converting JavaScript Ob

Re: [HACKERS] pre-commit triggers

2013-11-17 Thread Hannu Krosing
On 11/17/2013 07:31 PM, Andres Freund wrote: > On 2013-11-17 09:39:26 +0100, Hannu Krosing wrote: >>> Besides, exactly what would you do in such a trigger? >> The use case would be telling another system about the rollback. >> >> Basically sending a "ignore w

Re: [HACKERS] pre-commit triggers

2013-11-17 Thread Hannu Krosing
On 11/17/2013 04:20 PM, Alvaro Herrera wrote: > Hannu Krosing wrote: > >> So it would send a network message, a signal or writing something to >> external file. > If you're OK with a C function, you could try registering a callback, > see RegisterXactCallback(). >

Re: [HACKERS] pre-commit triggers

2013-11-17 Thread Hannu Krosing
On 11/17/2013 01:42 AM, Tom Lane wrote: > Hannu Krosing writes: >> I have not looked at the patch, but does it also run pre-rollback ? > error in trigger -> instant infinite loop. Means this needs to have some kind of recursion depth limit, like python >>> def x(): ...

Re: [HACKERS] additional json functionality

2013-11-16 Thread Hannu Krosing
On 11/16/2013 10:30 PM, David E. Wheeler wrote: > On Nov 16, 2013, at 12:04 PM, Hannu Krosing wrote: > >> Then perhaps name the "new binary json" as jsob (JavaScript Object Binary) >> or just jsobj (JavaScript Object) and keep current json for what it is, >> na

Re: [HACKERS] additional json functionality

2013-11-16 Thread Hannu Krosing
On 11/16/2013 12:15 AM, Josh Berkus wrote: > On 11/15/2013 02:59 PM, Merlin Moncure wrote: >> On Fri, Nov 15, 2013 at 4:31 PM, Hannu Krosing >> wrote: >> I think you may be on to something here. This might also be a way >> opt-in to fast(er) serialization (up

Re: [HACKERS] pre-commit triggers

2013-11-16 Thread Hannu Krosing
demand came from a client migrating > from that system to Postgres. > > cheers > > andrew > > -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ

Re: [HACKERS] additional json functionality

2013-11-15 Thread Hannu Krosing
s to binary json object if de-duplication and fast access to internals is needed I do not think we need anything else for this As far as I understand merlin is mostly ok with stored json being normalised and the problem is just with constructing "extended" json (a.k.a. "processi

Re: [HACKERS] additional json functionality

2013-11-14 Thread Hannu Krosing
which output "json" to output "text" and have new "json" type which stores proper JavaScript Object - like structured data. I would like to go a step further and have it automatically support not only the json data types as data but all postgresql data types by

Re: [HACKERS] additional json functionality

2013-11-14 Thread Hannu Krosing
On 11/14/2013 05:06 PM, Merlin Moncure wrote: > On Thu, Nov 14, 2013 at 9:42 AM, Hannu Krosing wrote: >> This is supported by the fact that current functions on json-source >> treat it as json-object (for example key lookup gives you the value >> of latest key and not a lis

Re: [HACKERS] nested hstore patch

2013-11-14 Thread Hannu Krosing
On 11/14/2013 01:47 PM, Andrew Dunstan wrote: > > On 11/14/2013 03:21 AM, Hannu Krosing wrote: >> On 11/14/2013 01:32 AM, David E. Wheeler wrote: >>> On Nov 13, 2013, at 3:59 PM, Hannu Krosing >>> wrote: >>> >>>> I remember strong voices in sup

Re: [HACKERS] additional json functionality

2013-11-14 Thread Hannu Krosing
On 11/14/2013 04:07 PM, Merlin Moncure wrote: > On Wed, Nov 13, 2013 at 6:01 PM, Hannu Krosing wrote: >> >> I guess we should not replace current JSON type with hstore based >> one, but add something json-like based on nested hstore instead. >> >> Maybe call it j

  1   2   3   4   5   6   7   8   9   10   >