Re: [HACKERS] bug: fuzzystrmatch levenshtein is wrong

[marcin mank]

On Tue, Dec 8, 2009 at 4:10 AM, Robert Haas  wrote:
>> The current behavior of levenshtein(text,text,int,int,int) is wrong. 
>> Consider:
>
> Is this the same problem as bug #5098?

Yes. Exact same change, plus the shortcut evaluation (for when one of
the inputs is empty) was also wrong. I fixed that too.

Greetings
Maricn Mańk

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML Was: CommitFest status/management

[Florian Weimer]

* Alvaro Herrera:

> Florian Weimer escribió:
>> * Dimitri Fontaine:
>> 
>> > Well we have JSON and agreed it was a good idea to have it. Now JSON is
>> > a subset of YAML and some would prefer another YAML style (me included).
>> 
>> YAML is rather difficult to parse, and most parsers assume a trusted
>> document source because they support arbitrary class instantiations
>> (and it's not possible to switch this off, or it's rather involved to
>> do so).
>
> That's irrelevant because EXPLAIN YAML is already a trusted document
> source.

Uhm, really?  Do I have to expect code execution on the client when I
connect to a database?  I hope not.

-- 
Florian Weimer
BFK edv-consulting GmbH   http://www.bfk.de/
Kriegsstraße 100  tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL format

[Fujii Masao]

On Tue, Dec 8, 2009 at 10:28 AM, Simon Riggs  wrote:
> If this was earlier in the release cycle, I'd feel happier.
>
> 2.5 months before beta is the wrong time to re-design the crash recovery
> data format, especially because its only "a bit awkward". We're bound to
> break something unforeseen and not have time to fix it. If you were
> telling me "impossible", I'd be all ears.

To avoid the harmful effect on the existing feature, how about
introducing new function which reads WAL records in byte level
for Streaming Replication? ISTM that making one function
ReadRecord cover several cases (a crash recovery and replication)
would increase complexity.

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A sniffer for the buffer

[Greg Smith]

Jonas J wrote:
I took a look in the code again and made some changes. For the 
readBuffer im doing now:
ReadBuffer(Relation reln, BlockNumber blockNum)   
fprintf(fp,"r%u\n",(unsigned int) blockNum); //as defined in 
header, typedef uint32 BlockNumber;

and from the write pages:
write_buffer(Buffer buffer, bool unpin){   
fprintf(fp,"w%d\n",BufferGetBlockNumber(buffer));  //get the 
blockNumber of this buffer
It's better to keep this discussion going on the list so you can get 
alternate suggestions besides mine.  The above is better.  You're still 
missing the relation number, which you're really going to want 
eventually.  And I think you're still vulnerable to printing the 
information out before the block is locked properly in the write_buffer 
case.


So, I had never used DTrace, where can I find some good paper to start 
studying it ?? Also, do it work with linux, or only solaris ??
You can get it to work on Linux for PostgreSQL use if you use systemtap; 
there's some people who've posted suggestions about that around. 

P.S.: I'm using PostgreSQL 8.1.4 to run with TPCC-UVA benchmarks tests...
Ah.  PostgreSQL 8.1 is significantly slower than the current versions, 
and you're not going to get as much help with issues related to the 
source code as if you're using a newer one.  For example, I don't know 
the 8.1 buffer implementation code nearly as well because I didn't 
really start tinkering with it until 8.2, so some of the things you're 
asking about I don't have easy access or recollection of.  Also, the 
DTrace stuff is only really going to be helpful if you're starting with 8.4.


There may be some work to get TPCC-UVA working with 8.4 though, 
particularly due to some changes made in 8.3 related to casting data to 
other types.  There is a TPC-C implementation that's used pretty often 
by developers here named dbt-2:  http://wiki.postgresql.org/wiki/DBT-2 
that will work with a newer version.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL format

[Heikki Linnakangas]

Tom Lane wrote:
> Heikki Linnakangas  writes:
>> In particular I wonder why we bother with the page headers. A much
>> simpler format would be:
> 
>> - get rid of page headers, except for the header at the beginning of
>> each WAL segment
>> - get rid of continuation records
>> - at the end of WAL segment, when there's not enough space to write the
>> next WAL record, always write an XLOG SWITCH record to fill the rest of
>> the segment.
> 
> What do you do with a WAL record that doesn't fit in a segment?  (They
> do exist.)  I don't think you can eliminate continuation records.
> You could maybe use them only at segment boundaries but I doubt that
> makes things any simpler than they are now.

Hmm, yeah, it doesn't make it that much simpler then.

-- 
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] EXPLAIN BUFFERS

[Euler Taveira de Oliveira]

Itagaki Takahiro escreveu:
> Here is an updated patch per discussion.
> 
>   * Counters are accumulative. They contain I/Os by child nodes.
>   * Text format shows all counters.
>   * Add "shared_" prefix to variables representing shared buffers/blocks.
> 
Nice. Despite of the other opinions, I'm satisfied with your text format
sentence. It is: (i) clear (I don't have to think or check the docs to know
what information is that.) and (ii) not so verbose (There are nodes that are
even longer than that.).

The only thing that needs some fix is:

+ Include information on the buffers. Specifically, include the number of
+ hits/reads/writes of shared blocks and local blocks, and number of reads
+ and writes of temp blocks. Shared blocks contain global tables and
+ indexes, local blocks contain temporary tables and indexes, and temp
+ blocks contain disk blocks used in sort and materialized plans.
+ The value of a parent node contains values of its children.
+ This parameter should be used with ANALYZE parameter.
+ This parameter defaults to FALSE.

That could be:

Include information on the buffers. Specifically, include the number of shared
blocks hits, reads, and writes, the number of local blocks hits, reads, and
writes, and the number of temp blocks reads and writes. Shared blocks, local
blocks, and temp blocks contain tables and indexes, temporary tables and
temporary indexes, and disk blocks used in sort and materialized plans,
respectively. The number of blocks of an upper-level node includes the blocks
of all its child nodes. It should be used with ANALYZE
parameter. This parameter defaults to FALSE.

If there is no more objections, I'll flag the patch 'ready for committer' (you 
:).


-- 
  Euler Taveira de Oliveira
  http://www.timbira.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL format

[Heikki Linnakangas]

Tom Lane wrote:
> "Kevin Grittner"  writes:
>> Heikki Linnakangas  wrote:
>>> In particular I wonder why we bother with the page headers.
>  
>> Since we re-use the file for a new segment, without overwriting the
>> old contents, it seems like we would need to do *something* to
>> reliably determine when we've hit the end of a segment and have
>> moved into old data from a previous use of the file.  Would your
>> proposed changes cover that adequately?
> 
> AFAICT the proposal would make us 100% dependent on the record CRC
> to detect when a record has been torn (ie, only the first few sectors
> made it to disk).  I'm a bit nervous about that from a reliability
> standpoint --- with a 32-bit CRC you've got a 1-in-4-billion chance
> of accepting bad data.  Checking the page headers too gives us many
> more bits that have to be as-expected to consider the data good.

We also check the prev-link, and some weak checks on rmid, and the
length fields.

> Since the records are fed to XLogInsert as units, it seems like the
> actual problem might be addressable by hooking in the sync-rep data
> sending at that level, rather than looking at the WAL page buffers
> as I gather it must be doing now.

No, walsender reads from disk. The sending side actually looks OK to me,
it's the code in ReadRecord that reads partial pages at the receiving
end that I'd like to simplify. It works as it is, but we have to re-read
the most recent page when it wasn't received as whole yet, and add some
state to track that. I think it's already relying on the fact that
walsender always sends full records (it can stop at a page boundary, at
a continuation record).

-- 
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] questions about concurrency control in Postgresql

[Daniel Farina]

2009/12/7 黄晓骋 :
> Hello,
>
> I think in Postgresql, concurrency control acts like this:
>
> tuple's infomask shows if it is being updated. If it is being updated now,
> the latter transaction should reread the tuple and get the newer tuple.
> During the progress of getting the newer tuple, it must use transaction
> lock, I mean XactLockTableWait(...).

That is a table lock...depending on the lock, other backends may be
allowed to update tuples in the relation still.  Fine-grained tuple
locks are used to prevent unnecessary contention for a table-wide
lock.

See the documentation at the manual page:

http://www.postgresql.org/docs/8.4/static/explicit-locking.html

It gives a thorough treatment of table and row locking levels and
conflicts, as well as what gets what locks.

fdr

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] EXPLAIN BUFFERS

[Greg Smith]

Robert Haas wrote:

I could live with the equals signs, but the use of parentheses seems
weird and inconsistent with normal english usage (which permits
parentheses as a means of making parenthetical comments).

But it is consistent with people seeing:

Seq Scan on foo (cost=0.00..155.00 rows=1 width=4)

Which seems to be what was being emulated here.  I though that was 
pretty reasonable given this is a related feature.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] EXPLAIN BUFFERS

[Takahiro Itagaki]

Robert Haas  wrote:

> On Mon, Dec 7, 2009 at 11:09 PM, Greg Smith  wrote:
> > (1) Blocks Shared: (hit=2 read=1641 written=0) Local: (hit=0 read=0
> > written=0) Temp: (read=1443 written=1443)

> I could live with the equals signs, but the use of parentheses seems
> weird and inconsistent with normal english usage (which permits
> parentheses as a means of making parenthetical comments).  (You can
> also put an entire sentence in parentheses.)  But you can't: (do
> this).

+1 for (1) personally, if we could think it is not in English but just
a symbol. I have another idea to make it alike with ANALYZE output.

(4) Blocks (shared hit=2 read=1641 ...) (local hit=0 ...) (temp read=0 ...)

Which is the best? I think it's a matter of preference.
  (0) 109 characters - Shared Blocks: hit 2 read 1641 wrote 0, ...
  (1) 105 characters - Blocks Shared: (hit=2 ...
  (2)  96 characters - Blocks Shared:hit=2 ...
  (3)  82 characters - Blocks Shared:(hit=2 ...
  (4)  82 characters - Blocks (shared hit=2 ...


BTW, I found text is a *bad* format because it requires much discussion ;)
We have a little choice in XML, JSON and YAML formats.

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] questions about concurrency control in Postgresql

[黄晓骋]

Hello,

I think in Postgresql, concurrency control acts like this:

tuple's infomask shows if it is being updated. If it is being updated now,
the latter transaction should reread the tuple and get the newer tuple.
During the progress of getting the newer tuple, it must use transaction
lock, I mean XactLockTableWait(...).

>From the above, I think the tuple lock is unnecessary, because it uses
transaction lock.

Besides, tuple lock is unlocked after the tuple is updated but not after the
transaction commits. I mean it's not 2PL.

So, may you tell me why there is tuple lock in Postgresql ? Is the tuple
lock necessary?

 

Thanks,

 

--Huang Xiaocheng

--Database & Information System Lab, Nankai University

 

Re: [HACKERS] EXPLAIN BUFFERS

[Robert Haas]

On Mon, Dec 7, 2009 at 11:09 PM, Greg Smith  wrote:
> Robert Haas wrote:
>
> On Mon, Dec 7, 2009 at 9:58 PM, Itagaki Takahiro
>  wrote:
>
>
> Obviously I should not hide any information only in the text format.
> The new output will be: (in one line)
>  Shared Blocks: (hit=2 read=1641 written=0) Local Blocks: (hit=0 read=0
> written=0) Temp Blocks: (read=1443 written=1443)
>
>
> Hmm, that's a little awkward.  I think we could drop some of the
> punctuation.
>
> Shared Blocks: hit 2 read 1641 wrote 0, Local Blocks: hit 0 read 0
> wrote 0, Temp Blocks: read 1443 wrote 1443
>
>
> Having written more things to parse log files that should have been saved in
> a better format after the fact than I'd like, I'd say that replacing the "="
> signs used as a delimiter with a space is a step backwards.  That doesn't
> save any space anyway, and drifts too far from what one gets out of regular
> EXPLAIN I think.
>
> I was perfectly happy with proposed text format as being a reasonable
> trade-off between *possible* to parse if all you have is the text format,
> while still being readable.  If you want to compress horizontally, get rid
> of "Blocks" after the first usage is the first thing to do:
>
> (1) Blocks Shared: (hit=2 read=1641 written=0) Local: (hit=0 read=0
> written=0) Temp: (read=1443 written=1443)
>
> That's already at about the same length as what you suggested at 105
> characters, without losing any useful formatting.
>
> If further compression is required, you could just remove all the
> parentheses:
>
> (2) Blocks Shared:hit=2 read=1641 written=0 Local:hit=0 read=0 written=0
> Temp:read=1443 written=1443
>
> I don't really like this though.  Instead you could abbreviate the rest of
> the repeated text and reduce the number of spaces:
>
> (3) Blocks Shared:(hit=2 read=1641 written=0) Local:(h=0 r=0 w=0)
> Temp:(r=1443 w=1443)
>
> And now we're at the smallest result yet without any real loss in
> readability--I'd argue it's faster to read in fact.  This has a good balance
> of fitting on a reasonably wide console (the above is down to 82
> characters), still being readable to anyone, and being possible to machine
> parse in a pinch if all you have are text logs around (split on : then =).
> It might be too compressed down for some tastes though.

I could live with the equals signs, but the use of parentheses seems
weird and inconsistent with normal english usage (which permits
parentheses as a means of making parenthetical comments).  (You can
also put an entire sentence in parentheses.)  But you can't: (do
this).

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] EXPLAIN BUFFERS

[Greg Smith]

Robert Haas wrote:

On Mon, Dec 7, 2009 at 9:58 PM, Itagaki Takahiro
 wrote:
  

Obviously I should not hide any information only in the text format.
The new output will be: (in one line)
 Shared Blocks: (hit=2 read=1641 written=0) Local Blocks: (hit=0 read=0 
written=0) Temp Blocks: (read=1443 written=1443)



Hmm, that's a little awkward.  I think we could drop some of the punctuation.

Shared Blocks: hit 2 read 1641 wrote 0, Local Blocks: hit 0 read 0
wrote 0, Temp Blocks: read 1443 wrote 1443
  
Having written more things to parse log files that should have been 
saved in a better format after the fact than I'd like, I'd say that 
replacing the "=" signs used as a delimiter with a space is a step 
backwards.  That doesn't save any space anyway, and drifts too far from 
what one gets out of regular EXPLAIN I think.


I was perfectly happy with proposed text format as being a reasonable 
trade-off between *possible* to parse if all you have is the text 
format, while still being readable.  If you want to compress 
horizontally, get rid of "Blocks" after the first usage is the first 
thing to do:


(1) Blocks Shared: (hit=2 read=1641 written=0) Local: (hit=0 read=0 
written=0) Temp: (read=1443 written=1443)


That's already at about the same length as what you suggested at 105 
characters, without losing any useful formatting. 

If further compression is required, you could just remove all the 
parentheses:


(2) Blocks Shared:hit=2 read=1641 written=0 Local:hit=0 read=0 written=0 
Temp:read=1443 written=1443


I don't really like this though.  Instead you could abbreviate the rest 
of the repeated text and reduce the number of spaces:


(3) Blocks Shared:(hit=2 read=1641 written=0) Local:(h=0 r=0 w=0) 
Temp:(r=1443 w=1443)


And now we're at the smallest result yet without any real loss in 
readability--I'd argue it's faster to read in fact.  This has a good 
balance of fitting on a reasonably wide console (the above is down to 82 
characters), still being readable to anyone, and being possible to 
machine parse in a pinch if all you have are text logs around (split on 
: then =).  It might be too compressed down for some tastes though.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com

Re: [HACKERS] A sniffer for the buffer

[Daniel Farina]

On Sun, Dec 6, 2009 at 9:04 AM, Greg Smith  wrote:
> And that won't work at all.  "Buffer" is a structure, not an integer.  You
> need to wait until it's been locked, then save the same data as on the read
> side (relation and block number) from inside the structure.  You probably
> want to hook FlushBuffer to do that job.  In fact, there's already a DTrace
> probe in there you could easily use to collect the data you want without
> even touching the source code if you can get a DTrace capable system.  Note
> the following code in bufmgr.c FlushBuffer:

Apropos DTrace:  I have just today compiled Postgres 8.3 with
--enable-dtrace on an Ubuntu 9.10 machine with no issues or
workarounds.  I just had to install the package systemtap-sdt-dev.

I haven't tried to do anything with it just yet though...

fdr

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Greg Smith]

David P. Quigley wrote:
> Not to start a flame war here about access control models but you gave 3
> different examples one of which I don't think has any means to do
> anything productive here.
You won't be starting a flame war for the same reason some of the
community members are so concerned about this patch. There aren't enough
people familiar with this part of the security field within our database
developer community to even be able to answer fairly basic questions
like the one you just clarified. If you can help bring more qualified
reviewers to bear on that, it would be extremely helpful. I even tried
to organize a meetup between PostgreSQL hackers working in this area and
the security people I knew around here (Baltimore/DC) last year, but
just couldn't find any interested enough to show. Other than a brief
visit on this list from some of the Tresys guys, we haven't seen much
input here beyond that offered by the patch author, who's obviously
qualified but at the end of the day is still only one opinion. He's also
not in a good position to tell other people their ideas are misinformed
either.

-- 
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.

[Robert Haas]

On Mon, Dec 7, 2009 at 8:04 PM, Ashish  wrote:
> Hi Robert,
>
> Thanks. If I may, what encompasses your area of expertise...
>
> BTW Congratulation on becoming a committer!

Thanks.  As others have said, it's probably best to pick a project
first, or at least an area.  It's more important to find something
you're interested in working on than to think about working with some
particular person.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] bug: fuzzystrmatch levenshtein is wrong

[Robert Haas]

On Mon, Dec 7, 2009 at 8:33 AM, marcin mank  wrote:
> The current behavior of levenshtein(text,text,int,int,int) is wrong. Consider:

Is this the same problem as bug #5098?

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] some questions in postgresql developping

[Robert Haas]

2009/12/7 黄晓骋 :
> I’m a student from Nankai University in China. Now I and my team do a
> project which aims to integrate XML to Postgresql. What I do is to complete
> the function of XML Update.
>
> Now I’m researching in concurrency control. I have read the code about the
> concurrency control for a long time and I’m confident that I know it much.
> But I am puzzled that why we need to lock tuple. I think locking transaction
> is sufficient. I don’t think the tuple lock is good at improving executing
> rate or anything.

The purpose of locking is to preserve correct semantics in the face of
concurrent activity, not to improve execution speed.  The reasons why
tuple locking is necessary should be covered in any introductory
database textbook.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML

[Andrew Dunstan]

Tom Lane wrote:

Andrew Dunstan  writes:
  
I was in fact prepared to commit this patch, despite some significant 
misgivings about its wisdom, mainly because it does have such a low 
impact. But then other people raised objections. I'm not sure how strong 
those objections are, though.



The "lite" version posted by Itagaki-san on 11/30 seems short enough
that maybe we should just stop arguing and apply it.  There were some
other versions that fooled around with existing logic, which I was a lot
less happy about because of the difficulty of being sure that nothing
was broken.
  


Well, I guess he can commit it himself now ;-)


I definitely don't think we should get involved with trying to create
support for plugin formatters or anything like that --- the amount of
effort required seems far out of proportion to the benefit.

  


right.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] EXPLAIN BUFFERS

[Robert Haas]

On Mon, Dec 7, 2009 at 9:58 PM, Itagaki Takahiro
 wrote:
> Here is an updated patch per discussion.
>
>  * Counters are accumulative. They contain I/Os by child nodes.
>  * Text format shows all counters.
>  * Add "shared_" prefix to variables representing shared buffers/blocks.
>
> Euler Taveira de Oliveira  wrote:
>
>> Itagaki Takahiro escreveu:
>> > I think the current output is enough and useful in normal use.
>> > We can use XML or JSON format for more details.
>> >
>> I don't think it is a good idea to have different information in different
>> formats. I'm with Robert; *don't* do that.
>
> I'm afraid of the human-unreadability of the text format, that is discussed
> in the YAML format thread. ...but I found we say the following in the docs.
>
>  XML or JSON output contains the same information as the text output format
>  http://developer.postgresql.org/pgdocs/postgres/sql-explain.html
>
> Obviously I should not hide any information only in the text format.
> The new output will be: (in one line)
>  Shared Blocks: (hit=2 read=1641 written=0) Local Blocks: (hit=0 read=0 
> written=0) Temp Blocks: (read=1443 written=1443)

Hmm, that's a little awkward.  I think we could drop some of the punctuation.

Shared Blocks: hit 2 read 1641 wrote 0, Local Blocks: hit 0 read 0
wrote 0, Temp Blocks: read 1443 wrote 1443

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Robert Haas]

On Mon, Dec 7, 2009 at 9:12 PM, Alvaro Herrera
 wrote:
> Tom Lane wrote:
>> Simon Riggs  writes:
>> > If we do need to do this, perhaps we should change the older parameter
>> > to be partition_exclusion.
>>
>> Yeah, if we do want to do something about this then changing the name of
>> the existing GUC would be a lot less work.  However, partition_exclusion
>> seems to imply that it *only* applies to partitioned tables, which is
>> not the case...
>
> Perhaps
> table_exclusion = {on, off, partition}
>
> Of course, constraint_exclusion should continue to work as a synonym for
> backwards compatibility, but it wouldn't be documented.

This seems pretty horrible to me.  I'm not sure whether the current
code supports excluding appendrel members that are not base tables,
but even if it doesn't it certainly might some day.  Furthermore, the
term "constraint exclusion" is a well-established.  We're not going to
reduce confusing by adding a new feature with a similar name and
simultaneously renaming the old feature to something different.  If
we're going to rename anything, it should be the new feature, though
I'm not entirely convinced that's really necessary either.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] EXPLAIN BUFFERS

[Itagaki Takahiro]

Here is an updated patch per discussion.

  * Counters are accumulative. They contain I/Os by child nodes.
  * Text format shows all counters.
  * Add "shared_" prefix to variables representing shared buffers/blocks.

Euler Taveira de Oliveira  wrote:

> Itagaki Takahiro escreveu:
> > I think the current output is enough and useful in normal use.
> > We can use XML or JSON format for more details.
> > 
> I don't think it is a good idea to have different information in different
> formats. I'm with Robert; *don't* do that.

I'm afraid of the human-unreadability of the text format, that is discussed
in the YAML format thread. ...but I found we say the following in the docs.

  XML or JSON output contains the same information as the text output format
  http://developer.postgresql.org/pgdocs/postgres/sql-explain.html

Obviously I should not hide any information only in the text format.
The new output will be: (in one line)
  Shared Blocks: (hit=2 read=1641 written=0) Local Blocks: (hit=0 read=0 
written=0) Temp Blocks: (read=1443 written=1443)

> There are nodes that don't read or write blocks.

This will be impossible now because we re-defined the meaning of counters
as "accumulated number of I/O". Even if the node never read or write blocks,
it might contain some child nodes with I/O.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center



explain_buffers_20091208.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Alvaro Herrera]

KaiGai Kohei escribió:
> I could not find the message from David P. Quigley in the list,
> although pgsql-hackers@postgresql.org was Cc:'ed.
> (something troubled?)

Weird.  It didn't even made it to the moderator queue for some reason.
Perhaps the system dropped it as spam.

> So, I'll send it again for your information.

Good move.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[KaiGai Kohei]

I could not find the message from David P. Quigley in the list,
although pgsql-hackers@postgresql.org was Cc:'ed.
(something troubled?)

So, I'll send it again for your information.

 Original Message 
Subject: Re: [HACKERS] Adding support for SE-Linux security
Date: Mon, 07 Dec 2009 14:53:03 -0500
From: David P. Quigley 
Organization: National Security Agency
To: Tom Lane 
CC: Bruce Momjian ,Robert Haas ,   
 Josh Berkus ,KaiGai Kohei ,
j...@commandprompt.com,David Fetter ,Itagaki Takahiro
,KaiGai Kohei ,
pgsql-hackers@postgresql.org
References: <200912071800.nb7i0kb01...@momjian.us>   
<21873.1260209...@sss.pgh.pa.us>

On Mon, 2009-12-07 at 13:17 -0500, Tom Lane wrote:
> Bruce Momjian  writes:
> > Robert Haas wrote:
> >> Yes, I think that's the right way to think about it.  At a guess, it's
> >> two man-months of work to get it in, and ripping it out is likely
> >> technically fairly simple but will probably be politically impossible.
>
> > I figure if there is sufficient usage, we will not need to remove it,
> > and if there isn't, we will have no objections to removing it.
>
> That leaves a wide gray area where there are a few people using it but
> not really enough to justify the support effort.  Even if there are
> demonstrably no users (which can never be demonstrated in practice),
> politically it's very hard to rip out a "major feature" --- it makes the
> project look bad.  So I think the above is Pollyanna-ish nonsense.
> Once we ship a release with SEPostgres in it, we're committed.
>
> > As Alvaro mentioned, the original patch used ACE but it added too much
> > code so the community requested its removal from the patch.  It could be
> > re-added if we have a need.
>
> The main problem I saw with ACE was that it didn't appear to actually
> add any flexibility --- it was just an extra layer of function calls
> in an entirely SELinux-centric design.  In order to have a "pluggable
> interface" layer that is worth the electrons it's written on, you need
> to start out with more than one target system in mind to be plugged in.
> So that would mean, at minimum, investigating something like AppArmor or
> TrustedSolaris to see what its needs are before we sit down to design
> the plugin layer.  (Which, of course, nobody here is actually interested
> enough to do.  But without that research there is no point in demanding
> a plugin layer.)
>
>   regards, tom lane
>

Not to start a flame war here about access control models but you gave 3
different examples one of which I don't think has any means to do
anything productive here. Looking at the 3 examples you gave and what
SEPostgres is trying to accomplish here is what I see.

The point of SEPostgres is to do object labeling in a database. Two of
the three examples you gave are label based security mechanisms.
AppArmor while it might be able to have a scheme to mediate access to
database/table doesn't seem to have a reasonable way to write policy to
individual records. Since AppArmor is path name based you need an
identifiable path to whatever you want to write access control rules
for. With something like a database where data dynamically comes and
goes and changes doing rules on individual records seems difficult. I
also can't find any information about AppArmor being used as a
user-space object manager. If someone wants to prove me wrong I'm
willing to listen to their argument.

Lets look at the remaining two. Do you really mean TrustedSolaris(TSOL)
or do you mean Solaris with Trusted Extensions(TX). There is a big
difference here. From my understanding Sun no longer supports TSOL and
moved on to TX. The difference here is that they moved from fine grained
object labeling to zone based labeling. There is a project currently in
development (although moving a bit slow) called FMAC which will bring
SELinux style MAC to Solaris which will reintroduce fine grained
labeling. You will still be able to use zones however process and object
labels will not be based on zone in FMAC.

I understand the concerns with a generic framework introducing a lot of
code. The Linux Kernel LSM Framework seems to grow as more models are
introduced but if you want to support various models you have to live
with the fact that there is no common interface for all of these
systems. It has been tried numerous times in the past and has failed.
>From what I've seen I think that will be less of a concern with Postgres
since the objects that Kaigai wants to mediate is a much smaller set
than the kernel. If you want to see another instance of where work
similar to this was done you should look at the X-Access Control
eXtensions (X-ACE) found in the X server. Eamon Walsh designed this
framework based not on anything SELinux specific but through an analysis
of the X Server which yielded a list of object he though needed
mediating. There are two modules for X-ACE now the FLASK module
(SELinux) and a module put forth by Cas

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Jeff Davis]

On Mon, 2009-12-07 at 17:41 -0800, Josh Berkus wrote:
> Actually, he has a very good point; we're going to get a *lot* of
> confusion from the users on this one.  I just wish I'd noticed the issue
> before.

The issue has been mentioned several times, but must have gotten lost
among the other emails.

I put one such mention on the commitfest page so that it wouldn't be
lost, but I suppose it was anyway:

http://archives.postgresql.org/message-id/1258227283.708.108.ca...@jdavis

Regards,
Jeff Davis


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Jeff Davis]

On Mon, 2009-12-07 at 23:12 -0300, Alvaro Herrera wrote:
> Perhaps
> table_exclusion = {on, off, partition}

Sounds good to me.

> Of course, constraint_exclusion should continue to work as a synonym for
> backwards compatibility, but it wouldn't be documented.

+1.

Regards,
Jeff Davis


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Alvaro Herrera]

Tom Lane wrote:
> Simon Riggs  writes:
> > If we do need to do this, perhaps we should change the older parameter
> > to be partition_exclusion.
> 
> Yeah, if we do want to do something about this then changing the name of
> the existing GUC would be a lot less work.  However, partition_exclusion
> seems to imply that it *only* applies to partitioned tables, which is
> not the case...

Perhaps
table_exclusion = {on, off, partition}

Of course, constraint_exclusion should continue to work as a synonym for
backwards compatibility, but it wouldn't be documented.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Andreas 'ads' Scherbaum]

On Mon, 07 Dec 2009 20:20:45 -0500 Tom Lane wrote:

> Simon Riggs  writes:
> > If we do need to do this, perhaps we should change the older parameter
> > to be partition_exclusion.
> 
> Yeah, if we do want to do something about this then changing the name of
> the existing GUC would be a lot less work.  However, partition_exclusion
> seems to imply that it *only* applies to partitioned tables, which is
> not the case...

It is less coding work - but it will for sure confuse the users.


Bye

-- 
Andreas 'ads' Scherbaum
German PostgreSQL User Group
European PostgreSQL User Group - Board of Directors
Volunteer Regional Contact, Germany - PostgreSQL Project

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[KaiGai Kohei]

Robert Haas wrote:
> On Mon, Dec 7, 2009 at 1:00 PM, Bruce Momjian  wrote:
>> As Alvaro mentioned, the original patch used ACE but it added too much
>> code so the community requested its removal from the patch.  It could be
>> re-added if we have a need.
> 
> Well, there's no point in putting that framework back in unless we can
> make it sufficiently general that it could be used to serve the needs
> of more than one security model.  And so far, the signs have not been
> promising.  David Quigley suggests downthread that making a truly
> general model isn't really possible, and he may be right, or not.  I
> was just mentioning that it's an angle I have been thinking about
> investigating, but it may be a dead end.

I also agree that the common framework just increases complexity of
the patch at the moment.

> The real issue is making the code committable, and then maintaining
> it, as Tom rightly says, forever.  We've got to make sure that we're
> willing to take that on before we do it, and I don't think it's a
> small task.  It isn't so much whether we want the feature as whether
> the level of effort is proportionate to the benefit.

Needless to say, we can provide development resource to maintain this
feature. If we escape to anywhere just after commit it, it will be removed
as Bruce pointed out. But it shall be incorrect.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei 

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Josh Berkus]

On 12/7/09 5:17 PM, Tom Lane wrote:
> David Fetter  writes:
>> It's not work you personally would have to do, and the confusion we've
>> already bought with this naming scheme is already evident.
> 
> What confusion?  The only person complaining is you.

Actually, he has a very good point; we're going to get a *lot* of
confusion from the users on this one.  I just wish I'd noticed the issue
before.

--Josh Berkus


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL format

[Simon Riggs]

On Mon, 2009-12-07 at 21:28 +0200, Heikki Linnakangas wrote:

> The changes to ReadRecord in the streaming replication patch feel a
> bit awkward, because it has to work around the fact that WAL is
> streamed as a stream of bytes, but ReadRecord works one page at a
> time. I'd like to replace ReadRecord with a simpler ring buffer
> approach, but handling the continuation records makes it a bit hard.

If this was earlier in the release cycle, I'd feel happier.

2.5 months before beta is the wrong time to re-design the crash recovery
data format, especially because its only "a bit awkward". We're bound to
break something unforeseen and not have time to fix it. If you were
telling me "impossible", I'd be all ears.

I feel your pain, but less drastic solutions are always best in such an
important area, at least while we lack automated test harnesses there.

-- 
 Simon Riggs   www.2ndQuadrant.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[KaiGai Kohei]

Bruce Momjian wrote:
> Tom Lane wrote:
>> Bruce Momjian  writes:
>>> Robert Haas wrote:
 Yes, I think that's the right way to think about it.  At a guess, it's
 two man-months of work to get it in, and ripping it out is likely
 technically fairly simple but will probably be politically impossible.
>>> I figure if there is sufficient usage, we will not need to remove it,
>>> and if there isn't, we will have no objections to removing it.
>> That leaves a wide gray area where there are a few people using it but
>> not really enough to justify the support effort.  Even if there are
>> demonstrably no users (which can never be demonstrated in practice),
>> politically it's very hard to rip out a "major feature" --- it makes the
>> project look bad.  So I think the above is Pollyanna-ish nonsense.
> 
> I don't even know what "Pollyanna-ish nonsense" means, and it would be
> better if you used less flowery/inflamitory prose.

Apart from standpoint of the discussion, idiomatic phrases are not
oftern friendly for non-native English speakers.

>> Once we ship a release with SEPostgres in it, we're committed.
> 
> The MS Windows port took 1-2 years to solidify and during the
> solidification period we accepted problems and didn't treat it as a
> major platform.  I think if SE-Linux support is added, there would be a
> similar period where the features is not treated as major while we work
> out any problems.  We might even label it that way.

It also seems to me an realistic attitude.
The first guy needs courage independently from the class of features.
Thus, anybody attend to see case examples in conferences. I don't think
here is no fundamental differences.

> Labeling SE-Postgres as such might minimize the political problems of
> removing it in the future, if that becomes necessary.

For us, the name is not an important issue.
And, I believe our continued contributions in the future shall make it
unnecessary to remove it later.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei 

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Tom Lane]

Simon Riggs  writes:
> If we do need to do this, perhaps we should change the older parameter
> to be partition_exclusion.

Yeah, if we do want to do something about this then changing the name of
the existing GUC would be a lot less work.  However, partition_exclusion
seems to imply that it *only* applies to partitioned tables, which is
not the case...

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Tom Lane]

David Fetter  writes:
> It's not work you personally would have to do, and the confusion we've
> already bought with this naming scheme is already evident.

What confusion?  The only person complaining is you.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[KaiGai Kohei]

Tom Lane wrote:
> Robert Haas  writes:
>> On Mon, Dec 7, 2009 at 9:48 AM, Bruce Momjian  wrote:
>>> I wonder if we should rephrase this as, "How hard will this feature be
>>> to add, and how hard will it be to remove in a few years if we decide we
>>> don't want it?"
> 
>> Yes, I think that's the right way to think about it.  At a guess, it's
>> two man-months of work to get it in,
> 
> It's not the "get it in" part that scares me.  The problem I have with
> it is that I see it as a huge time sink for future maintenance problems,
> most of which will be classifiable as security breaches which increases
> the pain of dealing with them immeasurably.

We can clearly say that acception of this feature is equivalent to
getting a new developer to maintain this feature into the community.

It is preferable to change my role in this community; I'd like to perform
as a maintainer of this feature rather than a person who send a large
patch for each commit-fest.


> If I had more confidence that the basic design was right or useful
> I might not be so worried about the maintenance prospects, but frankly
> I have almost no confidence in it.  This comes back to the lack of
> involvement of any potential user community.

We should not ignore a fact several commercial database software provides
advanced security options that are partially similar to SE-PgSQL. It allows
them to reach a region where PgSQL has not reached yet, and these features
are supported by an amount of users.

Anyway, it seems to me it is counterproductive to discuss whether the
potential users are larger or smaller, because it is a difficult job
to estimate it correctly, even if we would be experienced marketers.

-- 
OSS Platform Development Division, NEC
KaiGai Kohei 

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.

[Ashish]

Hi Robert,

Thanks. If I may, what encompasses your area of expertise...

BTW Congratulation on becoming a committer!

Regards
Ashish

On Mon, 7 Dec 2009, Robert Haas wrote:


On Sun, Dec 6, 2009 at 9:24 PM,   wrote:

2. Would someone be willing to be a mentor? It would be nice to be able to get 
some guidance on a one-to-one basis.


I might be willing to do this, but if you pick a project that is
outside my area of knowledge then I might not be able to help as much.

...Robert





--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Simon Riggs]

On Mon, 2009-12-07 at 13:53 -0800, David Fetter wrote:

> We have a very unfortunate naming situation with Jeff Davis's new
> feature, namely the shorter name, which is one permutation away from
> an existing and entirely unrelated feature: Constraint Exclusion,
> which has to do with queries over partitioned tables and like
> entities.

I agree, though with partitioning patch about to become reality the name
confusion will be less noticeable.

> Renaming it, which I believe we should do Really Soon(TM), to Operator
> [Exclusion] Constraints would fix this problem.

If we do need to do this, perhaps we should change the older parameter
to be partition_exclusion.

-- 
 Simon Riggs   www.2ndQuadrant.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PostgreSQL Committers

[Koichi Suzuki]

2009/12/8 Kevin Grittner :
> Jaime Casanova  wrote:
>> Dave Page  wrote:
>>>
>>> The new committers are:
>>>
>>> Robert Haas
>>> Simon Riggs
>>> Greg Stark
>>> ITAGAKI Takahiro
>>>
>>> Congratulations!
>>>
>>
>> +1
>
> Outstanding!  Congratulations, all!

+1
Appreciate for their contributions.   Congratulations all!

Koichi Suzuki

>
> -Kevin
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] bug: json format and auto_explain

[Tom Lane]

Robert Haas  writes:
> There's an awful lot of layers of function calls happening in
> explain.c for no really discernable purpose that I can see.
> ExplainOneQuery() calls either ExplainOneUtility() if it has a utility
> command or ExplainOneQuery_hook() if that's defined or else it plans
> the query and then calls ExplainOnePlan().  ExplainOneUtility() in
> turn calls ExplainExecuteQuery for execute statements and handles
> everything else internally.  The placement of the hook seems pretty
> dubious to me (does anyone actually use it?), and the whole structure
> seems like it could probably be flattened a bit.

IIRC, a fair amount of the messiness has to do with handling prepared
statements.  Maybe refactoring the division of labor between prepare.c
and explain.c would help.  Whether it's worth the trouble is
questionable though.  The immediate issue seems to be insufficient
thought about how to manage the initialization conditions for the
various formatters, and I don't think that any flattening or refactoring
of the rest of the logic would have made any difference there.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.

[Josh Berkus]

On 12/7/09 4:41 PM, Ashish wrote:
> Albe & Joshua, thanks for the advice. I am in the process of deciding
> what to work on and am looking at the TODO list. I definitely do not
> intend to work in a vacuum :-) I am really excited about this and look
> forward to being challenged and learning a lot.

When you decide what you want to work on, let us know and we'll try to
find you an appropriate mentor.

--Josh Berkus

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.

[Ashish]

Albe & Joshua, thanks for the advice. I am in the process of deciding what to 
work on and am looking at the TODO list. I definitely do not intend to work in a 
vacuum :-) I am really excited about this and look forward to being challenged and 
learning a lot.

Regards
Ashish


On Mon, 7 Dec 2009, Joshua Tolley wrote:


On Mon, Dec 07, 2009 at 09:53:32AM +0100, Albe Laurenz wrote:

abindra wrote:

Next quarter I am planning to do an Independent Study course
where the main objective would be to allow me to get familiar
with the internals of Postgres by working on a project(s). I
would like to work on something that could possibly be
accepted as a patch.

This is (I think) somewhat similar to what students do during
google summer and I was hoping to get some help here in terms of:
1. A good project to work on for a newbie.
2. Would someone be willing to be a mentor? It would be nice
to be able to get some guidance on a one-to-one basis.


I would start with the TODO list: http://wiki.postgresql.org/wiki/Todo
These are things for which there is a consensus that it would be
a good idea to implement them. Pick things that look interesting to
you, and try to read the discussions in the archives that lead
to the TODO items.


I agree the TODO list is a good place to start. Other good sources include the
-hackers list and comments in the code. I was surprised when I began taking an
interest in PostgreSQL how rarely interesting projects mentioned on -hackers
made it into the TODO list; I've come to realize that the TODO contains, in
general, very non-controversial items everyone is pretty sure we could use,
whereas -hackers ranges freely over other topics which are still very
interesting but often more controversial or less obviously necessary.
Committed patches both large and small address TODO list items fairly rarely,
so don't get too hung up on finding something from the TODO list alone.


Bring the topic up in the hackers list, say that you would like
to work on this or that TODO item, present your ideas of how you
want to do it. Ask about things where you feel insecure.
If you get some support, proceed to write a patch. Ask for
directions, post half-baked patches and ask for comments.

That is because you will probably receive a good amount of
critizism and maybe rejection, and if you invest a couple of
months into working on something that nobody knows about *and*
your work gets rejected, that is much worse than drawing fire
right away.


+1. Especially when developing a complex patch, and especially when you're new
to the community, you need to avoid working in a vacuum, for social as well as
technical reasons. The more complex a patch, the more consensus you'll
eventually need to achieve before getting it committed, in general, and it
helps to gain that consensus early on, rather than after you've written a lot
of code. The keyword "proposal" might be a useful search term when digging in
the -hackers archives for historical examples.

--
Joshua Tolley / eggyknap
End Point Corporation
http://www.endpoint.com





--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML

[Tom Lane]

Andrew Dunstan  writes:
> I was in fact prepared to commit this patch, despite some significant 
> misgivings about its wisdom, mainly because it does have such a low 
> impact. But then other people raised objections. I'm not sure how strong 
> those objections are, though.

The "lite" version posted by Itagaki-san on 11/30 seems short enough
that maybe we should just stop arguing and apply it.  There were some
other versions that fooled around with existing logic, which I was a lot
less happy about because of the difficulty of being sure that nothing
was broken.

I definitely don't think we should get involved with trying to create
support for plugin formatters or anything like that --- the amount of
effort required seems far out of proportion to the benefit.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[David Fetter]

On Mon, Dec 07, 2009 at 07:11:56PM -0500, Tom Lane wrote:
> David Fetter  writes:
> > We have a very unfortunate naming situation with Jeff Davis's new
> > feature, namely the shorter name, which is one permutation away
> > from an existing and entirely unrelated feature: Constraint
> > Exclusion, which has to do with queries over partitioned tables
> > and like entities.
> 
> > Renaming it, which I believe we should do Really Soon(TM), to
> > Operator [Exclusion] Constraints would fix this problem.
> 
> Too late.  I just spent about two days making that patch follow the
> "exclusion constraints" naming, and I'm not undoing that work.

It's not work you personally would have to do, and the confusion we've
already bought with this naming scheme is already evident.

Cheers,
David.
-- 
David Fetter  http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david.fet...@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Reading recovery.conf earlier

[Simon Riggs]

On Mon, 2009-12-07 at 19:21 -0500, Tom Lane wrote:
> Simon Riggs  writes:
> > On Mon, 2009-12-07 at 19:07 +0200, Heikki Linnakangas wrote:
> >> Why not just follow the example of postresql.conf?
> 
> > Much better idea.
> 
> Rather than reinventing all the infrastructure associated with GUCs,
> maybe we should just make the recovery parameters *be* GUCs.  At least
> for all the ones that could be of interest outside the recovery
> subprocess itself.
> 
> As an example of the kind of thing you'll find yourself coding if you
> make an independent facility: how will people find out the active
> values?

You're right, I was literally just writing that code.

Also, currently I have two parameters: wal_standby_info and
recovery_connections. If this was a GUC, then I could just have one
parameter: recovery_connections.

So, much agreed.

-- 
 Simon Riggs   www.2ndQuadrant.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Reading recovery.conf earlier

[Tom Lane]

Simon Riggs  writes:
> On Mon, 2009-12-07 at 19:07 +0200, Heikki Linnakangas wrote:
>> Why not just follow the example of postresql.conf?

> Much better idea.

Rather than reinventing all the infrastructure associated with GUCs,
maybe we should just make the recovery parameters *be* GUCs.  At least
for all the ones that could be of interest outside the recovery
subprocess itself.

As an example of the kind of thing you'll find yourself coding if you
make an independent facility: how will people find out the active
values?

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Tom Lane]

David Fetter  writes:
> We have a very unfortunate naming situation with Jeff Davis's new
> feature, namely the shorter name, which is one permutation away from
> an existing and entirely unrelated feature: Constraint Exclusion,
> which has to do with queries over partitioned tables and like
> entities.

> Renaming it, which I believe we should do Really Soon(TM), to Operator
> [Exclusion] Constraints would fix this problem.

Too late.  I just spent about two days making that patch follow the
"exclusion constraints" naming, and I'm not undoing that work.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML

[Andrew Dunstan]

Josh Berkus wrote:

Not everything is sanely convertible into some sort of plugin. A plugin
mechanism for this would be FAR more trouble that it is worth, IMNSHO.

We are massively over-egging this pudding (as a culinary blogger you
should appreciate this analogy).



OK, then let's just accept it.  It's small, has a maintainer, is useful
to some people, and doesn't create any wierd complications.  I think,
given the knowledge that YAML is now a subdialect of JSON it could
potentially be made smaller, but I can't say how at the moment.


  


Actually, it's the other way, JSON is a subset of YAML.

I was in fact prepared to commit this patch, despite some significant 
misgivings about its wisdom, mainly because it does have such a low 
impact. But then other people raised objections. I'm not sure how strong 
those objections are, though.


I must say that while the YAML output might look a bit nicer than the 
JSON output, the difference strikes me as mostly marginal. But I guess 
it's like beauty and obscenity, something in the eye of the beholder. De 
gustibus non est disputandum.


cheers

andrew




--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] DTrace compiler warnings

[Bernd Helmle]

--On 14. November 2009 15:25:25 +0100 Zdenek Kotala  
wrote:



Hmm, const is also problem on solaris. dtrace generates probe.h file and
eats const. It generates following noise on solaris build:

"postgres.c", line 554: warning: argument #1 is incompatible with
prototype:
prototype: pointer to char : "../../../src/include/utils/probes.h",
line 582
argument : pointer to const char

IIRC, it was discussed. I cc Robert. He should know answer why const is
ignored.


Have you dug into this further?

--
Thanks

Bernd

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [patch] executor and slru dtrace probes

[Bernd Helmle]

--On 13. November 2009 23:29:41 +0100 Zdenek Kotala  
wrote:



t contains two DTrace probe groups. One is related to monitoring SLRU
and second is about executor nodes.

I merged it with the head.

Original end of mail thread is here:

http://archives.postgresql.org/pgsql-hackers/2009-04/msg00148.php


I've started to review this.

It seems to me the attached patch wasn't adjusted or discussed again to 
address Tom's complaints? At least the executor probes contained here hold 
still the same issues mentioned by Tom in the discussion linked here.



--
Thanks

Bernd

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Reading recovery.conf earlier

[Simon Riggs]

On Mon, 2009-12-07 at 19:07 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > How does this sound?
> > 
> > At startup we will delete recovery.conf.running, if it exists.
> > At startup recovery.conf will be copied to recovery.conf.running, which
> > will be the file read by any additional processes that read
> > recovery.conf during this execution. The permissions on the file will
> > then be set to read-only.
> 
> Why not just follow the example of postresql.conf?

Much better idea.

-- 
 Simon Riggs   www.2ndQuadrant.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] bug: json format and auto_explain

[Robert Haas]

On Mon, Dec 7, 2009 at 11:07 AM, Tom Lane  wrote:
> Itagaki Takahiro  writes:
>> Tom Lane  wrote:
>>> Looks like auto_explain is under the illusion that it need not call
>>> ExplainBeginOutput/ExplainEndOutput.
>
>> Explain{Begin/End}Output are static functions, so we cannot call them
>> from an external contrib module. Instead, I'll suggest to call them
>> automatically from ExplainPrintPlan. The original codes in ExplainPrintPlan
>> was moved into ExplainOneResult, that name might be debatable.
>
> This isn't an adequate solution I'm afraid --- what about the other
> functions that are exported by explain.h?
>
> I too am not totally thrilled with the idea of exporting
> Explain{Begin/End}Output, but it might be the best solution.
> We might also need to think about refactoring those functions:
> there seem to be two different things going on there, one being
> format-specific initialization which will certainly be necessary,
> and one being output of a wrapper structure which might or might
> not be appropriate for what auto_explain or other callers want.
>
> In any case you need to expend more effort on the comments for the
> functions.  Inadequate specification of ExplainPrintPlan's call
> requirements is what got us into this problem in the first place,
> and the proposed patch makes that worse not better.

There's an awful lot of layers of function calls happening in
explain.c for no really discernable purpose that I can see.
ExplainOneQuery() calls either ExplainOneUtility() if it has a utility
command or ExplainOneQuery_hook() if that's defined or else it plans
the query and then calls ExplainOnePlan().  ExplainOneUtility() in
turn calls ExplainExecuteQuery for execute statements and handles
everything else internally.  The placement of the hook seems pretty
dubious to me (does anyone actually use it?), and the whole structure
seems like it could probably be flattened a bit.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Robert Haas]

On Mon, Dec 7, 2009 at 1:00 PM, Bruce Momjian  wrote:
> As Alvaro mentioned, the original patch used ACE but it added too much
> code so the community requested its removal from the patch.  It could be
> re-added if we have a need.

Well, there's no point in putting that framework back in unless we can
make it sufficiently general that it could be used to serve the needs
of more than one security model.  And so far, the signs have not been
promising.  David Quigley suggests downthread that making a truly
general model isn't really possible, and he may be right, or not.  I
was just mentioning that it's an angle I have been thinking about
investigating, but it may be a dead end.

The real issue is making the code committable, and then maintaining
it, as Tom rightly says, forever.  We've got to make sure that we're
willing to take that on before we do it, and I don't think it's a
small task.  It isn't so much whether we want the feature as whether
the level of effort is proportionate to the benefit.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Exclusion Constraint vs. Constraint Exclusion

[Stephen Cook]

David Fetter wrote:

Folks,

We have a very unfortunate naming situation with Jeff Davis's new
feature, namely the shorter name, which is one permutation away from
an existing and entirely unrelated feature: Constraint Exclusion,
which has to do with queries over partitioned tables and like
entities.

Renaming it, which I believe we should do Really Soon(TM), to Operator
[Exclusion] Constraints would fix this problem.

Cheers,
David.



+1

I thought I was just too dumb to follow the conversation.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Writeable CTE patch

[Alvaro Herrera]

Marko Tiikkaja escribió:
> Tom Lane wrote:
> >The only thing that I'd be comfortable with is
> >copying the snap and modifying the copy.
> 
> I don't see an easy way to do that with the current code;
> CopySnapshot() is static and PushUpdatedSnapshot() seems to be a bit
> of a pain since it messes up some of the existing code which uses
> the active snapshot stack.  Any ideas?

That API is rather new.  Maybe we need a new entry point, say
GetActiveSnapshotCopy or some such.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Exclusion Constraint vs. Constraint Exclusion

[David Fetter]

Folks,

We have a very unfortunate naming situation with Jeff Davis's new
feature, namely the shorter name, which is one permutation away from
an existing and entirely unrelated feature: Constraint Exclusion,
which has to do with queries over partitioned tables and like
entities.

Renaming it, which I believe we should do Really Soon(TM), to Operator
[Exclusion] Constraints would fix this problem.

Cheers,
David.
-- 
David Fetter  http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david.fet...@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML

[Josh Berkus]

> Not everything is sanely convertible into some sort of plugin. A plugin
> mechanism for this would be FAR more trouble that it is worth, IMNSHO.
> 
> We are massively over-egging this pudding (as a culinary blogger you
> should appreciate this analogy).

OK, then let's just accept it.  It's small, has a maintainer, is useful
to some people, and doesn't create any wierd complications.  I think,
given the knowledge that YAML is now a subdialect of JSON it could
potentially be made smaller, but I can't say how at the moment.

--Josh Berkus

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Install chapter broken link

[Greg Smith]

Magnus Hagander wrote:

Our instalation chapter (15.2) has a link to
http://developer.postgresql.org/~petere/bsd-gettext/ to get gettext on
"other systems". this link is broken. Can somebody provide a proper
one, or should we remove it?
  
Goals of his version described at 
http://archives.postgresql.org/pgsql-hackers/2001-05/msg01114.php lest 
anyone wonder why there was a personal port here (I know I did).  The 
version most like that available now would be the ones listed at 
ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/devel/gettext/README.html


The only popular platform I'm aware of that doesn't include it nowadays 
is OS X:  
http://blog.doughellmann.com/2009/06/installing-gnu-gettext-for-use-with.html

which does mean the need to install your own hasn't completely gone away.

Looks like all the various free BSDs are using GNU gettext now; that's 
what I found in the NetBSD link above, and at 
http://www.freebsd.org/cgi/cvsweb.cgi/ports/devel/gettext/  Even Solaris 
aims to be compatible with the GNU version as of 2004.


I think http://www.gnu.org/software/gettext/ is the appropriate new link 
destination.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL format

[Tom Lane]

Alvaro Herrera  writes:
> Heikki Linnakangas wrote:
>> - at the end of WAL segment, when there's not enough space to write the
>> next WAL record, always write an XLOG SWITCH record to fill the rest of
>> the segment.

> What happens if a record is larger than a WAL segment?  For example,
> what if I insert a 16 MB+ datum into a varlena field?

That case doesn't pose a problem --- the datum would be toasted into
individual tuples that are certainly no larger than a page.  However
we do have cases where a WAL record can get arbitrarily large; in
particular a commit record with many subtransactions and/or many
disk files to delete.  These cases do get exercised in the field
too --- I can recall at least one related bug report.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL format

[Andres Freund]

On Monday 07 December 2009 21:44:37 Tom Lane wrote:
> "Kevin Grittner"  writes:
> > Heikki Linnakangas  wrote:
> >> In particular I wonder why we bother with the page headers.
> >
> > Since we re-use the file for a new segment, without overwriting the
> > old contents, it seems like we would need to do *something* to
> > reliably determine when we've hit the end of a segment and have
> > moved into old data from a previous use of the file.  Would your
> > proposed changes cover that adequately?
> AFAICT the proposal would make us 100% dependent on the record CRC
> to detect when a record has been torn (ie, only the first few sectors
> made it to disk).  I'm a bit nervous about that from a reliability
> standpoint --- with a 32-bit CRC you've got a 1-in-4-billion chance
> of accepting bad data.  Checking the page headers too gives us many
> more bits that have to be as-expected to consider the data good.
One could argue that thats a good argument to go back to 64bit CRCs. 
Considering that they are more seldomly computed with such a change and that 
CPUs got more modern...

Andres

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL format

[Tom Lane]

"Kevin Grittner"  writes:
> Heikki Linnakangas  wrote:
>> In particular I wonder why we bother with the page headers.
 
> Since we re-use the file for a new segment, without overwriting the
> old contents, it seems like we would need to do *something* to
> reliably determine when we've hit the end of a segment and have
> moved into old data from a previous use of the file.  Would your
> proposed changes cover that adequately?

AFAICT the proposal would make us 100% dependent on the record CRC
to detect when a record has been torn (ie, only the first few sectors
made it to disk).  I'm a bit nervous about that from a reliability
standpoint --- with a 32-bit CRC you've got a 1-in-4-billion chance
of accepting bad data.  Checking the page headers too gives us many
more bits that have to be as-expected to consider the data good.

Since the records are fed to XLogInsert as units, it seems like the
actual problem might be addressable by hooking in the sync-rep data
sending at that level, rather than looking at the WAL page buffers
as I gather it must be doing now.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Build sizes vs docs

[Tom Lane]

Magnus Hagander  writes:
> Came cross this when updating the cvs fix. We declare size requirements as:
>Also check that you have sufficient disk space. You will need about
>65 MB for the source tree during compilation and about  MB for
>the installation directory. An empty database cluster takes about
>25 MB; databases take about five times the amount of space that a
>flat text file with the same data would take. If you are going to
>run the regression tests you will temporarily need up to an extra
>90 MB. Use the df command to check free disk

> My source *without* compile is 82 Mb, and with a build in it (linux
> i686) is 110Mb during compilations, rather than 65.
> An empty cluster takes about 33Mb.
> The regression database takes about 132Mb.
> (this is 8.4)

> Should we fix these numbers, or just remove them? They're clearly
> platform dependent, but perhaps there's still a point in including
> them - mainly as hints?

Maybe round them off to an order of magnitude.  I think it's useful
to have some idea of the size requirements, even if they change over
time.  It wouldn't be a bad idea to say "as of 8.4" or some such, too.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL format

[Tom Lane]

Heikki Linnakangas  writes:
> In particular I wonder why we bother with the page headers. A much
> simpler format would be:

> - get rid of page headers, except for the header at the beginning of
> each WAL segment
> - get rid of continuation records
> - at the end of WAL segment, when there's not enough space to write the
> next WAL record, always write an XLOG SWITCH record to fill the rest of
> the segment.

What do you do with a WAL record that doesn't fit in a segment?  (They
do exist.)  I don't think you can eliminate continuation records.
You could maybe use them only at segment boundaries but I doubt that
makes things any simpler than they are now.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL format

[Alvaro Herrera]

Heikki Linnakangas wrote:

> - at the end of WAL segment, when there's not enough space to write the
> next WAL record, always write an XLOG SWITCH record to fill the rest of
> the segment.

What happens if a record is larger than a WAL segment?  For example,
what if I insert a 16 MB+ datum into a varlena field?

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL format

[Kevin Grittner]

Heikki Linnakangas  wrote:
 
> In particular I wonder why we bother with the page headers.
 
Since we re-use the file for a new segment, without overwriting the
old contents, it seems like we would need to do *something* to
reliably determine when we've hit the end of a segment and have
moved into old data from a previous use of the file.  Would your
proposed changes cover that adequately?  (I'm not sure I understood
your proposal well enough to be comfortable about that.)
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Install chapter broken link

[Magnus Hagander]

Our instalation chapter (15.2) has a link to
http://developer.postgresql.org/~petere/bsd-gettext/ to get gettext on
"other systems". this link is broken. Can somebody provide a proper
one, or should we remove it?

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Build sizes vs docs

[Magnus Hagander]

Came cross this when updating the cvs fix. We declare size requirements as:

   Also check that you have sufficient disk space. You will need about
   65 MB for the source tree during compilation and about  MB for
   the installation directory. An empty database cluster takes about
   25 MB; databases take about five times the amount of space that a
   flat text file with the same data would take. If you are going to
   run the regression tests you will temporarily need up to an extra
   90 MB. Use the df command to check free disk


My source *without* compile is 82 Mb, and with a build in it (linux
i686) is 110Mb during compilations, rather than 65.
An empty cluster takes about 33Mb.
The regression database takes about 132Mb.
(this is 8.4)


Should we fix these numbers, or just remove them? They're clearly
platform dependent, but perhaps there's still a point in including
them - mainly as hints?


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] WAL format

[Heikki Linnakangas]

While looking at the streaming replication patch, I can't help but
wonder why our WAL format is so complicated.

WAL is divided into WAL segments, each 16 MB by default. Each WAL
segment is divided into pages, 8k by default. At the beginning of each
WAL page, there's a page header, but the header at the first page of
each WAL segment contains a few extra fields.

If a WAL record crosses a page boundary, we write as much of it as fits
onto the first page, and so-called continuation records with the rest of
the data on subsequent pages.

In particular I wonder why we bother with the page headers. A much
simpler format would be:

- get rid of page headers, except for the header at the beginning of
each WAL segment
- get rid of continuation records
- at the end of WAL segment, when there's not enough space to write the
next WAL record, always write an XLOG SWITCH record to fill the rest of
the segment.

The page addr stored in the WAL page header gives some extra protection
for detecting end of valid WAL correctly, but we rely on the prev-links
and CRC within page for that anyway, so I wouldn't mind losing that.

The changes to ReadRecord in the streaming replication patch feel a bit
awkward, because it has to work around the fact that WAL is streamed as
a stream of bytes, but ReadRecord works one page at a time. I'd like to
replace ReadRecord with a simpler ring buffer approach, but handling the
continuation records makes it a bit hard.

-- 
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML

[Greg Smith]

Josh Berkus wrote:

What it's sounding like is that we ought to have a plug-in (both for
postmaster and for psql) which allows the calling of an external library
to parse explain output.  Then people could covert XML/JSON into
whatever they wanted.
  
It would be kinder to just reject the YAML patch than to make it wait 
for this.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML

[Andrew Dunstan]

Josh Berkus wrote:

All,

What it's sounding like is that we ought to have a plug-in (both for
postmaster and for psql) which allows the calling of an external library
to parse explain output.  Then people could covert XML/JSON into
whatever they wanted.


  


Not everything is sanely convertible into some sort of plugin. A plugin 
mechanism for this would be FAR more trouble that it is worth, IMNSHO.


We are massively over-egging this pudding (as a culinary blogger you 
should appreciate this analogy).


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] What happened to pl/proxy and FDW?

[Josh Berkus]

Marko, Asko, hackers:

I thought the idea was that we were going to add PL/proxy to 8.5, with
support for the foriegn data wrapper syntax?  What happened to that?

--Josh Berkus

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML

[Josh Berkus]

All,

What it's sounding like is that we ought to have a plug-in (both for
postmaster and for psql) which allows the calling of an external library
to parse explain output.  Then people could covert XML/JSON into
whatever they wanted.

--Josh Berkus


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML Was: CommitFest status/management

[David E. Wheeler]

On Dec 7, 2009, at 9:52 AM, Alvaro Herrera wrote:

>> Tallying up the votes on this patch, we have Tom Lane, Andrew Dunstan,
>> Josh Drake, and myself arguing against including this in core, and
>> Josh Berkus and Itagaki Takahiro arguing for it.  Ron Mayer seemed
>> somewhat in favor of it in his message on-list but sent a message
>> off-list taking the opposite position.  Brendan Jurd cast aspersions
>> on the human-readability of the text format but didn't explicitly take
>> a position on the core issue,
> 
> Please add my vote for YAML, because of the human-unreadability of the
> default text format.  In fact I'd argue we could rip out the default
> text format and replace it with this one.

+1

David

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Bruce Momjian]

Tom Lane wrote:
> Bruce Momjian  writes:
> > Robert Haas wrote:
> >> Yes, I think that's the right way to think about it.  At a guess, it's
> >> two man-months of work to get it in, and ripping it out is likely
> >> technically fairly simple but will probably be politically impossible.
> 
> > I figure if there is sufficient usage, we will not need to remove it,
> > and if there isn't, we will have no objections to removing it.
> 
> That leaves a wide gray area where there are a few people using it but
> not really enough to justify the support effort.  Even if there are
> demonstrably no users (which can never be demonstrated in practice),
> politically it's very hard to rip out a "major feature" --- it makes the
> project look bad.  So I think the above is Pollyanna-ish nonsense.

I don't even know what "Pollyanna-ish nonsense" means, and it would be
better if you used less flowery/inflamitory prose.

> Once we ship a release with SEPostgres in it, we're committed.

The MS Windows port took 1-2 years to solidify and during the
solidification period we accepted problems and didn't treat it as a
major platform.  I think if SE-Linux support is added, there would be a
similar period where the features is not treated as major while we work
out any problems.  We might even label it that way.

Labeling SE-Postgres as such might minimize the political problems of
removing it in the future, if that becomes necessary.

I know there has been complaints about the lack of SE-PostgreSQL
developers, but given the number of developers we had for the Win32 port
vs. the installed base, I think having one dedicated SE-PostgreSQL
developer is much more percentage-wise than we had for MS Windows.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Tom Lane]

Bruce Momjian  writes:
> Robert Haas wrote:
>> Yes, I think that's the right way to think about it.  At a guess, it's
>> two man-months of work to get it in, and ripping it out is likely
>> technically fairly simple but will probably be politically impossible.

> I figure if there is sufficient usage, we will not need to remove it,
> and if there isn't, we will have no objections to removing it.

That leaves a wide gray area where there are a few people using it but
not really enough to justify the support effort.  Even if there are
demonstrably no users (which can never be demonstrated in practice),
politically it's very hard to rip out a "major feature" --- it makes the
project look bad.  So I think the above is Pollyanna-ish nonsense.
Once we ship a release with SEPostgres in it, we're committed.

> As Alvaro mentioned, the original patch used ACE but it added too much
> code so the community requested its removal from the patch.  It could be
> re-added if we have a need.

The main problem I saw with ACE was that it didn't appear to actually
add any flexibility --- it was just an extra layer of function calls
in an entirely SELinux-centric design.  In order to have a "pluggable
interface" layer that is worth the electrons it's written on, you need
to start out with more than one target system in mind to be plugged in.
So that would mean, at minimum, investigating something like AppArmor or
TrustedSolaris to see what its needs are before we sit down to design
the plugin layer.  (Which, of course, nobody here is actually interested
enough to do.  But without that research there is no point in demanding
a plugin layer.)

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Alvaro Herrera]

Martijn van Oosterhout escribió:
> On Mon, Dec 07, 2009 at 01:09:59PM -0300, Alvaro Herrera wrote:

> > This is how the code was developed initially -- the patch was called
> > PGACE and SELinux was but the first implementation on top of it.
> 
> I find it astonishing that after SE-PgSQL was implemented on top of a
> pluggable system (PGACE) and this system was removed at request of the
> "community" [1] that at this late phase people are suggesting it needs
> to be added back again. Havn't the goalposts been moved enough times?

Yeah.  I think the idle discussions here have created more work
themselves than the hypothetical maintenance work that would be spent on
this (undoubtely useful) feature.

> (It seems we've gone from a patch that had been around for years
> solving actual people's problems to a patch which does barely anything
> and we don't know whether it solves anybodies problem).

Agreed :-(

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML Was: CommitFest status/management

[Joshua D. Drake]

On Mon, 2009-12-07 at 14:52 -0300, Alvaro Herrera wrote:
> Robert Haas escribió:
> 
> > Tallying up the votes on this patch, we have Tom Lane, Andrew Dunstan,
> > Josh Drake, and myself arguing against including this in core, and
> > Josh Berkus and Itagaki Takahiro arguing for it.  Ron Mayer seemed
> > somewhat in favor of it in his message on-list but sent a message
> > off-list taking the opposite position.  Brendan Jurd cast aspersions
> > on the human-readability of the text format but didn't explicitly take
> > a position on the core issue,
> 
> Please add my vote for YAML, because of the human-unreadability of the
> default text format.  In fact I'd argue we could rip out the default
> text format and replace it with this one.

That would almost correct. My standing is we should have one format that
is parsed. The current "text" explain output certainly does not qualify.
The XML one or YAML one does.

Pick one, then parse it into whatever you want from the client.

Sincerely,

Joshua D. Drake



-- 
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
If the world pushes look it in the eye and GRR. Then push back harder. - 
Salamander


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Bruce Momjian]

Robert Haas wrote:
> > Agreed. ?SE-Linux support might expand our user base and give us
> > additional credibility, or it might be a feature that few people use ---
> > and I don't think anyone knows the outcome.
> >
> > I wonder if we should rephrase this as, "How hard will this feature be
> > to add, and how hard will it be to remove in a few years if we decide we
> > don't want it?" ?SE-Linux support would certainly put Postgres in a
> > unique security category, and it builds on our existing good security
> > reputation.
> 
> Yes, I think that's the right way to think about it.  At a guess, it's
> two man-months of work to get it in, and ripping it out is likely
> technically fairly simple but will probably be politically impossible.

I figure if there is sufficient usage, we will not need to remove it,
and if there isn't, we will have no objections to removing it.

> > but I am not advocating AppArmor support. ?I think the whole issue is
> > whether support for external integrated security systems is appropriate
> > for Postgres.
> 
> It's not something I've run into a need for in my own work, but I
> think there are definitely people out there who do need it, and I'd
> like to see us be able to support it.  One of the things that I think
> would be worth looking into is whether there is a way to make this
> pluggable, so that selinux and apparmor and trusted solaris and so on
> could make use of the same framework, but that requires understanding
> all of them well enough to design a framework that can meet all of
> those needs.  Every framework effort we've seen from KaiGai so far has
> seemed extremely SE-Linux-specific and therefore pointless.  But
> really doing this right is a big development project, and not
> something I can do in my free time.

As Alvaro mentioned, the original patch used ACE but it added too much
code so the community requested its removal from the patch.  It could be
re-added if we have a need.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Block-level CRC checks

[Greg Stark]

On Fri, Dec 4, 2009 at 10:47 PM, Chuck McDevitt  wrote:
> A curiosity question regarding torn pages:  How does this work on file 
> systems that don't write in-place, but instead always do copy-on-write?
>
> My example would be Sun's ZFS file system (In Solaris & BSD).  Because of its 
> "snapshot & rollback" functionality, it never writes a page in-place, but 
> instead always copies it to another place on disk.  How does this affect the 
> corruption caused by a torn write?
>
> Can we end up with horrible corruption on this type of filesystem where we 
> wouldn't on normal file systems, where we are writing to a previously zeroed 
> area on disk?
>
> Sorry if this is a stupid question... Hopefully somebody can reassure me that 
> this isn't an issue.

It's not a stupid question, we're not 100% sure but we believe ZFS
doesn't need full page writes because it's immune to torn pages.

I think the idea of ZFS is that the new partially written page isn't
visible because it's not linked into the tree until it's been
completely written. To me it appears this would depend on the drive
system ordering writes very strictly which seems hard to be sure is
happening. Perhaps this is tied to the tricks they do to avoid
contention on the root, if they do a write barrier before every root
update that seems like it should be sufficient to me, but I don't know
at that level of detail.

-- 
greg

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML Was: CommitFest status/management

[Alvaro Herrera]

Robert Haas escribió:

> Tallying up the votes on this patch, we have Tom Lane, Andrew Dunstan,
> Josh Drake, and myself arguing against including this in core, and
> Josh Berkus and Itagaki Takahiro arguing for it.  Ron Mayer seemed
> somewhat in favor of it in his message on-list but sent a message
> off-list taking the opposite position.  Brendan Jurd cast aspersions
> on the human-readability of the text format but didn't explicitly take
> a position on the core issue,

Please add my vote for YAML, because of the human-unreadability of the
default text format.  In fact I'd argue we could rip out the default
text format and replace it with this one.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] operator exclusion constraints

[Jeff Davis]

On Mon, 2009-12-07 at 00:26 -0500, Tom Lane wrote:
> Jeff Davis  writes:
> > [ exclusion constraint patch ]
> 
> Applied after quite a lot of editorialization.  For future reference,
> here is a summary of what I did:

Thank you for the suggestions, and the other constructive criticism
during development.

Regards,
Jeff Davis


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] YAML Was: CommitFest status/management

[Alvaro Herrera]

Florian Weimer escribió:
> * Dimitri Fontaine:
> 
> > Well we have JSON and agreed it was a good idea to have it. Now JSON is
> > a subset of YAML and some would prefer another YAML style (me included).
> 
> YAML is rather difficult to parse, and most parsers assume a trusted
> document source because they support arbitrary class instantiations
> (and it's not possible to switch this off, or it's rather involved to
> do so).

That's irrelevant because EXPLAIN YAML is already a trusted document
source.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Tom Lane]

Chris Browne  writes:
> I feel about the same way about this as I did about the adding of
> "native Windows" support; I'm a bit concerned that this could be a
> destabilizing influence.  I was wrong back then; the Windows support
> hasn't had the ill effects I was concerned it might have.

That's an interesting analogy.  I too am not entirely convinced that
it's a good comparison, but if it is, consider these points:

* The goal of the Windows port was pretty well-defined and easily
tested: "make it work on Windows".  The goalposts didn't move except
perhaps when MS came out with new Windows versions.

* We had a *lot* of users available to help flush out problems.

* There wasn't any need to treat bugs as security sensitive, which is
problematic not least because it restricts the free flow of information.

Any one of those points would be good reason to think that getting
SEPostgres to stability will be lots more drawn-out and painful than
getting the Windows port to stability was.  With all three pointing in
the same direction, the tea leaves don't look good.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Reading recovery.conf earlier

[Heikki Linnakangas]

Simon Riggs wrote:
> How does this sound?
> 
> At startup we will delete recovery.conf.running, if it exists.
> At startup recovery.conf will be copied to recovery.conf.running, which
> will be the file read by any additional processes that read
> recovery.conf during this execution. The permissions on the file will
> then be set to read-only.

Why not just follow the example of postresql.conf?

-- 
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Chris Browne]

t...@sss.pgh.pa.us (Tom Lane) writes:
> Robert Haas  writes:
>> On Mon, Dec 7, 2009 at 9:48 AM, Bruce Momjian  wrote:
>>> I wonder if we should rephrase this as, "How hard will this feature be
>>> to add, and how hard will it be to remove in a few years if we decide we
>>> don't want it?"
>
>> Yes, I think that's the right way to think about it.  At a guess, it's
>> two man-months of work to get it in,
>
> It's not the "get it in" part that scares me.  The problem I have with
> it is that I see it as a huge time sink for future maintenance problems,
> most of which will be classifiable as security breaches which increases
> the pain of dealing with them immeasurably.

Ah, yes, the importance of this is not to be underestimated...

Once "SE-Pg" is added in, *any* bug found in it is likely to be
considered a security bug, and hence a candidate for being a CERT
Advisory.

Some bad things are liable to happen:

  a) Such problems turn into a "hue and cry" situation requiring
 dropping everything else to "fix the security problem."

  b) If everyone isn't using "SE-Pg", then people won't be particularly
 looking for bugs, and hence bugs are likely to linger somewhat,
 with the consequence that a) occurs with some frequency.

  c) Having a series of CERT advisories issued is not going to be
 considered a good thing, reputation-wise!

I feel about the same way about this as I did about the adding of
"native Windows" support; I'm a bit concerned that this could be a
destabilizing influence.  I was wrong back then; the Windows support
hasn't had the ill effects I was concerned it might have.

I'd hope that my concerns about "SE-Pg" are just as wrong as my concerns
about native Windows support.  Hope doesn't make it so, alas...
-- 
select 'cbbrowne' || '@' || 'gmail.com';
http://www3.sympatico.ca/cbbrowne/languages.html
"Just because it's free doesn't mean you can afford it."  -- Unknown

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PostgreSQL Committers

[Chris Browne]

dp...@pgadmin.org (Dave Page) writes:
> Congratulations!

+1

Congratulations, indeed, to this worthy set of developers!
-- 
output = reverse("moc.liamg" "@" "enworbbc")
http://linuxfinances.info/info/multiplexor.html
"Power tends  to corrupt and absolute power  corrupts absolutely."  
-- First Baron Acton, 1834 - 1902

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Tom Lane]

Martijn van Oosterhout  writes:
> I find it astonishing that after SE-PgSQL was implemented on top of a
> pluggable system (PGACE) and this system was removed at request of the
> "community" [1] that at this late phase people are suggesting it needs
> to be added back again. Havn't the goalposts been moved enough times?

The reason the goalposts keep moving is that nobody has a very clear
handle on what the requirements are, which stems from the lack of a
clear target community with definable needs.  We have had a couple of
apparently-knowledgeable people pop up and say "you should do this",
but then they disappear again without sticking around for any detailed
discussion of features (let alone code).

> (It seems we've gone from a patch that had been around for years
> solving actual people's problems to a patch which does barely anything
> and we don't know whether it solves anybodies problem).

Do we know that any version of this patch has solved any actual people's
problems?  I know KaiGai-san has been putting it out as a Fedora package
but there's little if any evidence that anyone's actually using that.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] strange segfault with 8.3.8

[Greg Smith]

Filip Rembiałkowski wrote:
Dec  7 07:24:45 nsXX kernel: postgres[22481]: segfault at 
7fda5e1d5000 ip 7fda604553c3 sp 7fffe41faf28 error 4 in 
libc-2.9.so [7fda603d1000+168000]
Dec  7 07:24:45 nsXX kernel: grsec: From XX.YY.ZZ.51: Segmentation 
fault occurred at 7fda5e1d5000 in 
/usr/lib/postgresql/8.3/bin/postgres[postgres:22481] uid/euid:103/103 
gid/egid:114/114, parent 
/usr/lib/postgresql/8.3/bin/postgres[postgres:29857] uid/euid:103/103 
gid/egid:114/114


This is on 8.3.8, latest Ubuntu distribution, server under quite heavy 
load.


afterwards we had some corrupted pages ("//ERROR/: /compressed data/ 
is corrupt/").
Random segfaults only under heavy load with corrupted pages afterwards 
normally means bad RAM.  I'd suggest a round of memtest86+ on the system 
if it's PC hardware.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com

Re: [HACKERS] YAML Was: CommitFest status/management

[Ron Mayer]

Greg Smith wrote:
> That's a step backwards.  By providing JSON format, we've also satisfied
> people who want YAML.  Ripping out JSON would mean we *only* support
> YAML.  There are far many more JSON parsers than YAML parsers, which is
> why I thought the current code committed was good enough.

XML parsers are common enough IMHO the other computer readable formats
can't be that important from a computer-readability perspective, leaving
their main benefit as being human friendly.

I like YAML output; but I think the most compelling arguments against the
patch are that if so many people want different formats it may be a good
use case for external modules.  And far more than yaml output, I'd like
to see a flexible module system with an equivalent of "cpan install yaml"
or "gem install yaml".


I suppose one could argue that instead of YAML we design a different
human-oriented format for loosely structured data; but that seems
even harder.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] new CommitFest states (was: YAML)

[Greg Smith]

Robert Haas wrote:

On a related note, Greg Smith requested a state called "Discussing
Review", which would logically follow "Needs Review" and precede
"Waiting for Author"/"Ready for Committer"/"Returned with Feedback".
I'm not altogether convinced of the value of that state, but I'm not
altogether opposed to it either.  If we're going to have a discussion
of CommitFest states, we probably ought to talk about that one, too.
  
Don't know what it is about YAML that it encourages slipping into CF 
management..."Discussing Review" is a state patches seem to fall into 
for a time.  I'd like to see it added mainly because it simplifies work 
for a lazier (than Robert) CF manager like myself, which I think is a 
more appropriate target for this process.  Some of the process that 
works for him I don't think can be replicated by other personalities 
very well.


If a patch is being actively discussed on the list, often the author is 
at the mercy of said discussion ending before they can make another 
forward step; this is why "Waiting for Author" isn't appropriate.  
Having the patch sitting in "Needs Review" instead is unfair to the 
reviewer, who would like to be able to mark it as "I'm done" and move 
on.  That's also why sitting in that state takes up time for the CF 
manager.  They need to scan all "waiting for [author|review]" patches to 
get a list of people to nudge, and in this case there is no one to 
nudge--we're all at the mercy of the list to reach some sort of conclusion.


The obvious concern here is "who has the action item them?"  In this 
case, that's the CF manager's job--to help wrap the discussion up once 
it's died down and figure out what state the patch should go into next.  
Reviewers would mark a patch "Discussing Review" once they're done and 
have sent their review to the list when it doesn't fit any of the 
existing next states well, and they're not sure what happens next.  
Basically it allows them to formally push making a hard decision about a 
patch upwards, which is effectively what's happening now.  Then the CF 
manager or committer can mark it "returned with feedback" or flat-out 
"rejected" if the resulting discussion isn't favorable, rather than 
making the reviewer responsible for that, once discussion has wrapped 
up.  Or the author/CF manager can eventually mark it "waiting for 
author" once one of them has decided that's the logical next step.  I 
plan to turn the whole thing into a fairly easy to follow state diagram 
as documentation on the process, there's just this one common state I 
don't have a label for now:  when things are trapped on the list, and 
nobody has an obvious next move until that settles down.


There is no need or want for a "Needs discussion" before review or code 
submission.  That just encourages abusing the CF app and process for 
something you can do quite nicely with the mailing list.  If you believe 
in a feature but don't want to get community buy-in on the list first, 
write a patch to prove it works.  If the reviewer torpedos your patch, 
don't expect you'll get a free round of discussing whether everyone 
wants that feature or not out of the deal.  For this YAML example, had 
the code in the patch been junk we wouldn't even have gotten to 
discussion of its utility--the reviewer would have rejected it and that 
would have been it.  And that IMHO is how it should be.


--
Greg Smith2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
g...@2ndquadrant.com  www.2ndQuadrant.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Martijn van Oosterhout]

On Mon, Dec 07, 2009 at 01:09:59PM -0300, Alvaro Herrera wrote:
> > Given the extreme patience and diligence exhibited by KaiGai, I
> > hesitate to say this, but it seems to me that this would be
> > critically important for the long term success of this feature.  I
> > have no idea how much work it would be to make the interface to the
> > external security system pluggable, but if it's at all feasible, I
> > think it should be done.
> 
> This is how the code was developed initially -- the patch was called
> PGACE and SELinux was but the first implementation on top of it.

I find it astonishing that after SE-PgSQL was implemented on top of a
pluggable system (PGACE) and this system was removed at request of the
"community" [1] that at this late phase people are suggesting it needs
to be added back again. Havn't the goalposts been moved enough times?

[1] http://archives.postgresql.org/pgsql-hackers/2009-01/msg02295.php

(It seems we've gone from a patch that had been around for years
solving actual people's problems to a patch which does barely anything
and we don't know whether it solves anybodies problem).

Have a nice day,
-- 
Martijn van Oosterhout  http://svana.org/kleptog/
> Please line up in a tree and maintain the heap invariant while 
> boarding. Thank you for flying nlogn airlines.


signature.asc
Description: Digital signature

Re: [HACKERS] New PostgreSQL Committers

[Kevin Grittner]

Jaime Casanova  wrote:
> Dave Page  wrote:
>>
>> The new committers are:
>>
>> Robert Haas
>> Simon Riggs
>> Greg Stark
>> ITAGAKI Takahiro
>>
>> Congratulations!
>>
> 
> +1
 
Outstanding!  Congratulations, all!
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [patch] pg_ctl init extension

[Zdenek Kotala]

Greg,

thanks for your useful comments. I attached  new doc patch version. I
removed example changes and add link to create database cluster (I hope)
everywhere. Please, look on it and let me know if there is still
something what should be changed.

Thanks Zdenek


Greg Smith píše v ne 06. 12. 2009 v 22:29 -0500:
> I just noticed that there was an updated patch here that never made its 
> way onto the CommitFest app listing.  I just fixed that and took a quick 
> look at it.  I was in favor of this code change, but I have to say even 
> I don't really like how it ended up getting documented--and I'm sure 
> there are others would be downright hostile toward it.
> 
> The biggest problem is that all of the places that used to say 
> "" when talking about creating a cluster now just say 
> "database cluster initialization"--with no link to a section covering 
> that topic.  That's not a good forward step.  The part I'm more 
> favorable toward that I expect other people to really cringe at is that 
> the examples showing how to manually run initdb have all been switched 
> to use "pg_ctl initdb" too.
> 
> If we're going to have a smooth transition toward supporting both styles 
> of working in the next release, I think what needs to happen to the 
> documentation here is adding a very clear section saying that "initdb" 
> and "pg_ctl initdb" are the same thing, and noting why both forms 
> exist.  Maybe a short note in both pg_ctl and initdb pointing at each 
> other; not sure yet what's best.  Then update all the current places 
> that say "initdb" that have been rewritten in this doc patch to 
> "database cluster initialization" to reference things appropriate still. 
> 
> Going as far as making all the examples exclusively use the pg_ctl form 
> right now is probably more than the community at large wants to handle 
> just yet I suspect.  At best, maybe we could make some or all of those 
> either use both forms, or link them to the new discussion of 
> alternatives section. 
> 
> I'm glad we made some progress (and are basically at code complete now) 
> on this patch this round.  Given that this patch doesn't have a large 
> amount of support, I think that the remaining work here is fine-tuning 
> the documentation to cover the new option available without introducing 
> and abrupt change people won't like.  I'm going to mark this "returned 
> with feedback" for now since I think that's going to take a bit more 
> work than we really have time for right now, particularly given the 
> limited number of people who care about this change.  Zdenek, once this 
> CommitFest clears out, I can help out with getting the documentation 
> parts here smoothed over so this is in proper shape to commit during the 
> next one; I don't think there's anything left you need to do.
> 

diff -r ab32ed8164e7 doc/src/sgml/config.sgml
--- a/doc/src/sgml/config.sgml	Mon Dec 07 15:10:09 2009 +0100
+++ b/doc/src/sgml/config.sgml	Mon Dec 07 17:12:32 2009 +0100
@@ -54,9 +54,10 @@

 One way to set these parameters is to edit the file
 postgresql.confpostgresql.conf,
-which is normally kept in the data directory. (initdb
-installs a default copy there.) An example of what this file might look
-like is:
+which is normally kept in the data directory.
+(database cluster initialization
+process installs a default copy there.)
+An example of what this file might look like is:
 
 # This is a comment
 log_connections = yes
@@ -365,8 +366,8 @@
 Determines the maximum number of concurrent connections to the
 database server. The default is typically 100 connections, but
 might be less if your kernel settings will not support it (as
-determined during initdb).  This parameter can
-only be set at server start.
+determined during database cluster
+initialization). This parameter can only be set at server start.

 

@@ -747,7 +748,8 @@
 Sets the amount of memory the database server uses for shared
 memory buffers.  The default is typically 32 megabytes
 (32MB), but might be less if your kernel settings will
-not support it (as determined during initdb).
+not support it (as determined during database
+cluster initialization).
 This setting must be at least 128 kilobytes.  (Non-default
 values of BLCKSZ change the minimum.)  However,
 settings significantly higher than the minimum are usually needed
@@ -4267,10 +4269,10 @@
 keywords US, NonEuro, and
 NonEuropean are synonyms for MDY. See
  for more information. The
-built-in default is ISO, MDY, but
-initdb will initialize the
-configuration file with a setting that corresponds to the
-behavior of the chosen lc_time locale.
+built-in default is ISO, MDY, but 
+database cluster initialization
+will initialize the configuration file w

Re: [HACKERS] [PATCH] Largeobject Access Controls (r2460)

[Kevin Grittner]

KaiGai Kohei  wrote:
 
> Apart from SELinux, it is quite natural to apply any access
> controls on binary data. If we could not have any valid access
> controls, users will not want to store their sensitive
> information, such as confidential PDF files, as a large object.
 
Absolutely.  The historical security issues for large objects
immediately eliminated them as a possible place to store PDF files.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] EXPLAIN BUFFERS

[Euler Taveira de Oliveira]

Itagaki Takahiro escreveu:
> I think the current output is enough and useful in normal use.
> We can use XML or JSON format for more details.
> 
I don't think it is a good idea to have different information in different
formats. I'm with Robert; *don't* do that. If you want to suppress the other
ones in text format, do it in the others too. One idea is to add them only if
we prefer the VERBOSE output.

> I think
> Blocks Hit: 1641  Read: 0  Temp Read: 1443
> means
> Blocks (Hit: 1641  Read: 0  Temp Read: 1443)
> i.e., Blocks of hit, blocks of reads, and Blocks of temp reads.
> 
But the latter is more clear than the former.

> I cannot understand what you mean -- should I suppress the lines when they
> have all-zero values?
> 
There are nodes that don't read or write blocks. If we go this way, we need to
document this behavior.


-- 
  Euler Taveira de Oliveira
  http://www.timbira.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] strange segfault with 8.3.8

[Filip Rembiałkowski]

hello,

from kernel log:

Dec  7 07:24:45 nsXX kernel: postgres[22481]: segfault at 7fda5e1d5000
ip 7fda604553c3 sp 7fffe41faf28 error 4 in libc-2.9.so
[7fda603d1000+168000]
Dec  7 07:24:45 nsXX kernel: grsec: From XX.YY.ZZ.51: Segmentation fault
occurred at 7fda5e1d5000 in
/usr/lib/postgresql/8.3/bin/postgres[postgres:22481] uid/euid:103/103
gid/egid:114/114, parent
/usr/lib/postgresql/8.3/bin/postgres[postgres:29857] uid/euid:103/103
gid/egid:114/114

This is on 8.3.8, latest Ubuntu distribution, server under quite heavy load.

afterwards we had some corrupted pages ("*ERROR: compressed data is corrupt*
").

any clues / directions welcome.



-- 
Filip Rembiałkowski
JID,mailto:filip.rembialkow...@gmail.com
http://filip.rembialkowski.net/

Re: [HACKERS] bug: json format and auto_explain

[Robert Haas]

On Mon, Dec 7, 2009 at 10:42 AM, Alvaro Herrera
 wrote:
> Tom Lane wrote:
>> Euler Taveira de Oliveira  writes:
>> > While testing the EXPLAIN BUFFERS patch I found a bug. I'm too tired to
>> > provide a fix right now but if someone can take it I will appreciate. It 
>> > seems
>> > ExplainJSONLineEnding() doesn't expect es->grouping_stack list as a null 
>> > pointer.
>>
>> Looks like auto_explain is under the illusion that it need not call
>> ExplainBeginOutput/ExplainEndOutput.
>
> Fortunately the author of auto_explain can now commit the fix by himself ...
> Kudos, BTW :-)

I just tested and this has been broken since it was committed (by
Tom).  However, I believe I did test it on the last version of the
patch that I submitted, and I think it worked then.  So we have lots
of choices for who should fix this:

- me, since it's my feature
- Tom, since it appears that he broke it
- Itagaki-san, since he's the auto_explain maintainer

:-)

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Adding support for SE-Linux security

[Alvaro Herrera]

Kevin Grittner escribió:

> > I'd like to see us be able to support it.  One of the things that
> > I think would be worth looking into is whether there is a way to
> > make this pluggable, so that selinux and apparmor and trusted
> > solaris and so on could make use of the same framework
>  
> Given the extreme patience and diligence exhibited by KaiGai, I
> hesitate to say this, but it seems to me that this would be
> critically important for the long term success of this feature.  I
> have no idea how much work it would be to make the interface to the
> external security system pluggable, but if it's at all feasible, I
> think it should be done.

This is how the code was developed initially -- the patch was called
PGACE and SELinux was but the first implementation on top of it.

-- 
Alvaro Herrerahttp://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] bug: json format and auto_explain

[Tom Lane]

Itagaki Takahiro  writes:
> Tom Lane  wrote:
>> Looks like auto_explain is under the illusion that it need not call
>> ExplainBeginOutput/ExplainEndOutput.

> Explain{Begin/End}Output are static functions, so we cannot call them
> from an external contrib module. Instead, I'll suggest to call them
> automatically from ExplainPrintPlan. The original codes in ExplainPrintPlan
> was moved into ExplainOneResult, that name might be debatable.

This isn't an adequate solution I'm afraid --- what about the other
functions that are exported by explain.h?

I too am not totally thrilled with the idea of exporting
Explain{Begin/End}Output, but it might be the best solution.
We might also need to think about refactoring those functions:
there seem to be two different things going on there, one being
format-specific initialization which will certainly be necessary,
and one being output of a wrapper structure which might or might
not be appropriate for what auto_explain or other callers want.

In any case you need to expend more effort on the comments for the
functions.  Inadequate specification of ExplainPrintPlan's call
requirements is what got us into this problem in the first place,
and the proposed patch makes that worse not better.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] bug: json format and auto_explain

[Euler Taveira de Oliveira]

Alvaro Herrera escreveu:
> Fortunately the author of auto_explain can now commit the fix by himself ...
> Kudos, BTW :-)
> 
Congratulations!


-- 
  Euler Taveira de Oliveira
  http://www.timbira.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Need a mentor, and a project.

[Robert Haas]

On Sun, Dec 6, 2009 at 9:24 PM,   wrote:
> 2. Would someone be willing to be a mentor? It would be nice to be able to 
> get some guidance on a one-to-one basis.

I might be willing to do this, but if you pick a project that is
outside my area of knowledge then I might not be able to help as much.

...Robert

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers