Re: [HACKERS] Tackling JsonPath support
On Fri, Dec 2, 2016 at 1:32 PM, Nico Williams wrote: ... On Fri, Dec 02, 2016 at 08:53:33AM -0500, Robert Haas wrote: > > On Tue, Nov 29, 2016 at 11:50 AM, Christian Convey > > wrote: > > > I think I can satisfy (3) with a PG extension which provides a > function that > > > approximately implements JSONPath. My short-term plans are to submit > such a > > > patch. > > > > FWIW, I think that's a fine plan. I don't really know whether > > JSONPath is the right standard to pick for the task of extracting bits > > It's not even a standard. Are there particular proposals that the ANSI > SQL working group is considering? > Hi Nico, it seems to be something in the works with the standards committee. We were discussing it earlier in the thread: [1] Kind regards, Christian [1] http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg297732.html
Re: [HACKERS] Tackling JsonPath support
On Tue, Nov 29, 2016 at 8:18 AM, Petr Jelinek wrote: ... > Just to add to this, the SQL/JSON proposals I've seen so far, and what > Oracle, MSSQL and Teradata chose to implement already is basically > subset of jsonpath (some proposals/implementations also include > lax/strict prefix keyword on top of that). I think that should give us > some hint on what the base functionality should look like. I agree. My guess is that PG users would benefit most from: (1) Conformance to whatever ISO standard regarding JSON operators eventually makes it out of the working group. (2) Compatibility with other widely-used DBMS's. (3) Compatibility with the JSONPath functionality used by web developers. (Although I don't currently have a grasp on which frameworks / libraries this entails.) I *think* that (1), (2), and (3) are in approximate agreement about the syntax and semantics of the path-expression language: the language proposed by Stefan Groessner, plus the strict vs. lax distinction. I think I can satisfy (3) with a PG extension which provides a function that approximately implements JSONPath. My short-term plans are to submit such a patch. Hopefully that patch's function will be a helpful starting point for satisfying (1) and (2) as well. But that can be decided later. Nico Williams has argued for using "jq". I don't think jq satisfies any of (1), (2), or (3), so I don't see a good case for incorporating it in my short-term plans. There *may* be a case for using jq internally to my implementation; I'll try to look into that.
Re: [HACKERS] Tackling JsonPath support
On Mon, Nov 28, 2016 at 10:37 PM, Pavel Stehule wrote: > > > 2016-11-29 7:34 GMT+01:00 Christian Convey : > >> On Mon, Nov 28, 2016 at 9:28 PM, Pavel Stehule >> wrote: >> >>> >>> We now support XPath function - JSONPath is similar to XPath - >>> >>> it is better for user, because have to learn only one language. >>> >> >> I'm not sure I understand. >> >> Are you suggesting that we use XPath, not JSONPath, as our language for >> json-path expressions? >> > > surely not. > > follow ANSI/SQL :) > I see. Then I'm afraid I still don't understand what you're main point was when you wrote: > We now support XPath function - JSONPath is similar to XPath - > > it is better for user, because have to learn only one language. > - C
Re: [HACKERS] Tackling JsonPath support
On Mon, Nov 28, 2016 at 9:28 PM, Pavel Stehule wrote: > We now support XPath function - JSONPath is similar to XPath - it is > better for user, because have to learn only one language. > I'm not sure I understand. Are you suggesting that we use XPath, not JSONPath, as our language for json-path expressions? - C
Re: [HACKERS] Tackling JsonPath support
On Mon, Nov 28, 2016 at 6:26 PM, Nico Williams wrote: > On Mon, Nov 28, 2016 at 05:50:40PM -0800, Christian Convey wrote: > > On Mon, Nov 28, 2016 at 11:23 AM, Nico Williams > > wrote: > > ... > > > JSON Path is not expressive enough (last I looked) and can be mapped > > > onto jq if need be anyways. > > > > Hi Nico, > > > > Could you please clarify what you mean by "not expressive enough"? > > jq is a functional language that has these and other features: > > - recursion > - generators > - lazy evaluation (of sorts) > - path expressions > - math functionality (libm, basically) > - reduction > - functions > - and other things > > (jq does not have higher-order functions in that functions cannot return > functions and functions are not values, though it does have closures.) > > jq is and feels a lot like a SQL, but for JSON. > > > I ask because I've been struggling to identify clear requirements for the > > json-path functionality I'm trying to provide. It sounds like perhaps > you > > have something concrete in mind. > > SQL imposes structure on data. Recursion makes SQL structure looser in > the sense that it may not be easy or possible to express certain > desirable schema constraints in SQL terms without resorting to triggers, > say. Storing documents in XML, JSON, or other such recursion-friendly > formats (perhaps in semantically equivalent but query-optimized forms) > is also a way to avoid strict structure (thus one needs schema > validators for XML, for example). > > Less rigid schema constraints do not and should not preclude powerful > query languages. > > One could convert such documents to a SQL EAV schema, if one has an > RDBMS with an ANY type (e.g., something like SQLite3's duck typing), and > then use SQL to query them. But that may be more difficult to use than > a SQL with support for XML/JSON/... and query sub-languages for those. > > SQL is very powerful. One might like to have similarly powerful, > format-specific query languages for documents stored in XML, JSON, > etcetera, in a SQL RDBMS. jq is such a language, for JSON documents. > Ditto XPath/XSLT, for XML. While XPath is expressive and compact, XSLT > is rather verbose; jq is as expressive as XSLT, but with the compact > verbosity of XPath. > > > Since I myself have no need currently for this functionality, I'm left > > guessing about hypothetical users of it. My current mental model is: > > That's a bit like asking what is the use for SQL :^) The point is that > SQL is a powerful query language, and so is jq. Each is appropriate to > its own domain; both could be used together. > Thanks for the explanation. It sounds like your original point was NOT that json-path isn't sufficient for "${specific use X}". Instead, your point was that jq seems to have many advantages over json-path in general, and therefore PG should offer jq instead or, or in addition to, json-path. Is that what you're saying? ... > > (c) Pavel. (I'm still somewhat unclear on what has him interested in > this, > > and what his specific constraints are.) > > Hmm? > Context: The reason I'm trying to work on a json-path implementation is that Pavel Stehule suggested it as a good first PG-hacking project for me. At the time, it sounded like he had a use for the feature. - C
Re: [HACKERS] Tackling JsonPath support
On Mon, Nov 28, 2016 at 11:23 AM, Nico Williams wrote: ... > JSON Path is not expressive enough (last I looked) and can be mapped > onto jq if need be anyways. > Hi Nico, Could you please clarify what you mean by "not expressive enough"? I ask because I've been struggling to identify clear requirements for the json-path functionality I'm trying to provide. It sounds like perhaps you have something concrete in mind. Since I myself have no need currently for this functionality, I'm left guessing about hypothetical users of it. My current mental model is: (a) Backend web developers. AFAICT, their community has mostly settled on the syntax/semantics proposed by Stefan Groessner. It would probably be unkind for PG's implementation to deviate from that without a good reason. (b) PG hackers who will eventually implement the ISO SQL standard operators. In the standards-committee meeting notes I've seen, it seemed to me that they were planning to define some operators in terms of json-path expression. So it would probably be good if whatever json-path function I implement turns out to comply with that standard, so that the PG-hackers can use it as a building block for their work. (c) Pavel. (I'm still somewhat unclear on what has him interested in this, and what his specific constraints are.) - Christian
Re: [HACKERS] Tackling JsonPath support
On Mon, Nov 28, 2016 at 9:47 AM, Pavel Stehule wrote > > > I thought by adding my first implementation to "contrib", we could make > this functionality available to end-users, even before there was a > consensus about what PG's "official" JSON-related operators should have for > syntax and semantics. > > > > this time the commiters dislike the contrib dir. It is hard to push there > anything :-(. You can try it, but it can be lost time. > Thanks for the warning. I'm okay with my patch adding the "json_path" function to the core PG code. I would still suggest that we hold off on having my first patch implement an official JSON-related operator such as "JSON_TABLE". I would prefer to have my "json_path" function available to users even before we know how "JSON_TABLE", etc. should behave. Does that sound reasonable?
Re: [HACKERS] Tackling JsonPath support
On Mon, Nov 28, 2016 at 9:40 AM, Pavel Stehule wrote: ... > > Hi Pavel, > > > > Can you clarify what you meant? I *think* you're saying: > > > > * It's not important for me to match the syntax/semantics of the > json-path implementations found in MySQL / Oracle / DB2 / MS SQL Server, > and > > > > oh no. the syntax is important. But for start we can have a subset. For > json table function .. json to relation mapping is important path > expression. some other features like predicates > are nice, but can be implemented later. > > Im sorry. My English is bad. > Hi Pavel, You're English is very good, actually. I think the confusion arises from me speaking in vague terms. I apologize for that. Allow me to be more specific about what I'm proposing to do. I propose adding to "contrib" a function with the following characteristics: * Its signature is "json_path( jsonb from_json, string json_path_expression) --> jsonb". * The function will hopefully be a useful building block for PG's implementation of "official" JSON operators such as "JSON_TABLE". Once the PG community agrees on what those operators' syntax/semantics should be. * The function will hopefully be immediately useful to PG users who want JSONPath -like operations on their "jsonb" objects. - C
Re: [HACKERS] Tackling JsonPath support
On Mon, Nov 28, 2016 at 5:20 AM, Pavel Stehule wrote: ... > Incremental work is great idea - I like this this style. Instead contrib, > you can use public repository on github. Minimally for first stage is > better to live outside core - you are not restricted by PostgreSQL > development process. When your code will be stabilized, then you can go to > commitfest. I believe so we need good JSON support. The XML support helps > to PostgreSQL lot of, JSON will be great too. > Hi Pavel, Thanks for the suggestion. I am planning to use one of my own public github repos as the location for my work. I thought by adding my first implementation to "contrib", we could make this functionality available to end-users, even before there was a consensus about what PG's "official" JSON-related operators should have for syntax and semantics. Does my reasoning make sense? - C
Re: [HACKERS] Tackling JsonPath support
On Mon, Nov 28, 2016 at 5:20 AM, Pavel Stehule wrote: ... > Con: "JSON path expression" is a recurring them in the *grammars* of >> user-facing operators in [1], [2], [3], and [4]. But it doesn't >> necessarily follow that the function implemented in Step 2 will provide >> useful infrastructure for PG's eventual implementations of "JSON_TABLE", >> etc. >> > > We can implement subset only - our XPath based on libxml2 does it too. The > good target is support of usual examples on the net. > Hi Pavel, Can you clarify what you meant? I *think* you're saying: * It's not important for me to match the syntax/semantics of the json-path implementations found in MySQL / Oracle / DB2 / MS SQL Server, and * Instead, I should just use examples / explanations on the web as my guidance. Thanks, Christian
Re: [HACKERS] Tackling JsonPath support
>From looking at other databases' docs, it seems like the behavior of various JSON-related operators / functions are described partially in terms of a "json path expression": * In Oracle, "JSON_TABLE", "JSON_exists_column", "JSON_value_column": [1] * In MySQL: [2] * In DB2: [3] * In MS SQL Server: [4] * (Whatever the Standards committee will end up producing.) If I'm correctly understanding the situation, It sounds like we have two big unknowns: (a) The exact syntax/semantics of JSON path searching, especially w.r.t. corner cases and error handling, and (b) The syntax/semantics of whatever SQL operators / functions are currently defined in terms of (a). E.g., "JSON_TABLE". If that's correct, then what do you guys think about us taking the following incremental approach? Step 1: I'll dig into the implementations described above, to see what's similar and different between the JSON-path-expression syntax and semantics offered by each. I then report my findings here, and we can hopefully reach a consensus about the syntax/semantics of PG's json-path-expression handling. Step 2: I submit a patch for adding a new function to "contrib", which implements the JSON-path-expression semantics chosen in Step 1. The function will be named such that people won't confuse it with any (eventual) SQL-standard equivalent. Step 3: PG developers can, if they choose, start defining new JSON operator / functions, and/or port existing JSON-related functions, in terms of the function created in Step 2. I see the following pros / cons to this approach: Pro: It gives us a concrete start on this functionality, even though we're not sure what's happening with the SQL standard. Pro: The risk of painting ourselves into a corner is relatively low, because we're putting the functionality in "contrib", and avoid function names which conflict with likely upcoming standards. Pro: It might permit us to give PG users access to JSONPath -like functionality sooner than if we wait until we're clear on the ideal long-term interface. Con: "JSON path expression" is a recurring them in the *grammars* of user-facing operators in [1], [2], [3], and [4]. But it doesn't necessarily follow that the function implemented in Step 2 will provide useful infrastructure for PG's eventual implementations of "JSON_TABLE", etc. - Christian [1] https://docs.oracle.com/database/121/SQLRF/functions092.htm#SQLRF56973 [2] https://dev.mysql.com/doc/refman/5.7/en/json-path-syntax.html [3] http://www.ibm.com/support/knowledgecenter/ssw_ibm_i_72/db2/rbafzjsonpath.htm [4] https://msdn.microsoft.com/en-us/library/mt577087.aspx
Re: [HACKERS] Tackling JsonPath support
Hi Pavel, Can I check a few assumptions about what you're suggesting for this task? * Our ultimate goal is to give Postgres an implementation of the functions "JSON_EXISTS", "JSON_VALUE", and "JSON_QUERY" which fully comply with the SQL standards. * The best representation of those standards is found here: [1]. * When [1] mentions a "JSON path expression" or "JSON path language", it's referring to the query language described here: [2]. * Even if other popular DBMS's deviate from [1], or other popular JSONPath implementations deviate from [2], we remain committed to a faithful implementation of [1]. * It's okay for my first commit to implement just two things: (a) a PG-internal implementation of JsonPath, and (b) a user-visible implementation of "JSON_QUERY" based on (a). Later commits could add implementations of "JSON_VALUE", "JSON_EXISTS", etc. in terms of (a). Thanks, Christian [1] http://jtc1sc32.org/doc/N2501-2550/32N2528-WG3-Tutorial-Opening-Plenary.pdf [2] http://goessner.net/articles/JsonPath On Fri, Sep 16, 2016 at 2:28 AM, Pavel Stehule wrote: > Hi > > 2016-09-15 18:05 GMT+02:00 Christian Convey : > >> On Mon, Sep 5, 2016 at 1:44 PM, Pavel Stehule >> wrote: >> ... >> >> > I wrote XMLTABLE function, and I am thinking about JSON_TABLE function. >> But >> > there is one blocker - missing JsonPath support in our JSON >> implementation. >> > >> > So one idea - implement JsonPath support and related JSON query >> functions. >> > This can help with better standard conformance. >> >> Hi Pavel, >> >> Are you still looking for someone to add the JsonPath support to the >> JSON implementation? And if so, how urgently are people waiting for >> it? >> > > yes - JsonPath support should be great. I hope so this or next commitfest > the XMLTABLE patch will be committed, and with JsonPath I can start to work > on JSON_TABLE function. > > But the JsonPath can be merged separately without dependency to > JSON_TABLE. There are more JSON searching functions, and these functions > should to support JsonPath be ANSI SQL compliant. > > >> >> I'd be happy to start working on a patch, but since I'm new to PG >> development, I'm probably not the fastest person to get it done. >> > > It is not problem. Probably you should to do this work without deep > knowledges about PostgreSQL internals. The work with data types (and > functions for data types) is well isolated from PostgreSQL engine. > > You can learn from current searching on JSON - > postgresql/src/backend/utils/adt/json.c > > And it is good start to be PostgreSQL's hacker - I started with > implementation of own data type and related functions. > > Regards > > Pavel > > >> Kind regards, >> Christian >> > >
[HACKERS] Tackling JsonPath support
On Mon, Sep 5, 2016 at 1:44 PM, Pavel Stehule wrote: ... > I wrote XMLTABLE function, and I am thinking about JSON_TABLE function. But > there is one blocker - missing JsonPath support in our JSON implementation. > > So one idea - implement JsonPath support and related JSON query functions. > This can help with better standard conformance. Hi Pavel, Are you still looking for someone to add the JsonPath support to the JSON implementation? And if so, how urgently are people waiting for it? I'd be happy to start working on a patch, but since I'm new to PG development, I'm probably not the fastest person to get it done. Kind regards, Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] C++ port of Postgres
Some of your patches look useful, but unrelated to C++: 7, 8, 15, 16(?), 20. I applied that subset to 9.6 and got a clean "make check". Would it make sense to add them to the next commitfest, regardless of the C++ effort? On Wed, Aug 31, 2016 at 9:41 AM, Peter Eisentraut wrote: > [trimmed cc list because of big attachments] > > On 8/16/16 4:22 PM, Jim Nasby wrote: >> Joy, do you have an idea what a *minimally invasive* patch for C++ >> support would look like? That's certainly the first step here. > > I developed a minimally invasive patch for C++ support a few years ago > shortly after I wrote that blog post. Since there appears to have been > some interest here now, I have updated that and split it up into logical > chunks. > > So here you go. > > To build this, you need to configure with g++ <= version 5. (4.x works, > too.) g++ version 6 does not work yet because of the issues described > in patch 0013. > > Then you also need to edit src/Makefile.custom and set > > COPT = -fpermissive -Wno-sign-compare -Wno-write-strings > > The -W options are optional just to reduce some noise. Cleaning up > those warnings can be a separate project that might also have some > benefit under C. > > The -fpermissive option is a g++ specific option that reduces some > errors to warnings. (So this won't work with clang or other compilers > at all at this point.) In particular, C++ does not allow casting from > or to void pointers without a cast, but -fpermissive allows that. The > step from this to "real" C++ would be adding a bunch of casts around > things like malloc and palloc and other places. That would be mostly > busy work, so I have excluded that here. > > The patches are numbered approximately in increasing order of dubiosity. > So 0001 is probably a straight bug fix, 0002 and 0003 are arguably > minor bug fixes as well. The patches through 0012 can probably be > considered for committing in some form. After that it gets a bit hackish. > > -- > Peter Eisentraut http://www.2ndQuadrant.com/ > PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers > -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] C++ port of Postgres
> P.S. I'm asking because I was planning to review that patch. But I >> can't tell if any more review by a non-committer is still required by >> the commitfest workflow. > > > I think this has gotten enough attention, for the commitfest workflow. The > workflow is flexible, depending on the nature of patch. But of course, if > you're interested, feel free to review and comment anyway! Hi Heikki, Thanks for the help. If you think nothing more is needed for this patch set w.r.t. the 2016-09 commitfest, I'll move on to other things. - Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] C++ port of Postgres
Hi Heikki, Could I ask you a newbie-reviewer question about something I'm seeing here? https://commitfest.postgresql.org/10/776/ >From some reading I've done (e.g., Stephen Frost's PGCon 2011 slides), I got the impression that a successful patch would always have this sequence of states in commitfest: 1. patch-record created ... 2. Needs Review ... 3. Ready for Committer But if I'm reading the patch's activity log correctly, it looks like you marked the patch as "Ready for Committer" (2016-09-06 18:59:02) without any record of it having been reviewed. Was that intentional? Thanks very much, Christian P.S. I'm asking because I was planning to review that patch. But I can't tell if any more review by a non-committer is still required by the commitfest workflow. Kind regards, Christian On Tue, Sep 6, 2016 at 3:15 PM, Christian Convey wrote: > On Tue, Sep 6, 2016 at 3:12 PM, Tom Lane wrote: >>> (2) It seems like there are still a few big questions about this commit: >>>- Is it wanted at the moment? It didn't seem like there's a >>> consensus about whether or not this enhancement should be >>> merged, even if the patch is pretty minimal. >>>- It seems like there are two competing patch >>> sets in play for this enhancement: Joy's and >>> Peter's. Presumably at most one of them would >>> be merged. >> >> These are things that reviews should be helping to decide. It's probably >> a squishier topic than some patches, but if you're interested, feel free >> to read code and weigh in. > > Thanks. It sounds like worst-case scenario, I perform an unneeded > review. I'll give it a shot. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] C++ port of Postgres
> Thanks. It sounds like worst-case scenario, I perform an unneeded > review. I'll give it a shot. Hi guys, Apologies for more boring process-related questions, but any pointers would be greatly appreciated... I'm a bit confused about how PG's code-review process is meant to handle this C++ port. My confusion may stem from the combination of my inexperience with the process, and there being two competing patch sets. Here's some background: * My intention was to review Joy's patch. * On "commitfest.postgresql.org" (for 2016-09), the only C++ -related patch I found was Peter's: [1] * I wrongly assumed that the commitfest entry would be for Joy's patch, not Peter's, so I signed up as its reviewer. (That's fine - I don't mind reviewing both authors' patch sets.) But here are my questions: Q1) My understanding of PG's code-review process is that it's a pipeline: Step 1. The discussion starts on the pgsql-hackers mailing list, where the author posts a patch. He/she may also post revised patches based on the discussion. Step 2. A subset of those discussions are modeled by new entries in the commitfest website. Step 3. A subset of those commitfest items get merged. If that's correct, then it sounds like the only way Joy's commit has a chance of acceptance is if Peter's commit is rejected. Because Peter's commit might be merged as part of the 2016-09 commitfest, but Joy's can show up until 2016-11 at the earliest. Is my understanding correct? There seems to be a little ambiguity regarding the exact version of the code to be reviewed. This is true for both Joy's and Peter's submissions: * Joy's email provides a link to a Github repo, but does not specify a particular commit (or even branch) in that repo: [2] * In the email thread, Peter did provide a patch set: [3] but the corresponding commitfest entry references a github branch: [4] So I have a few questions here: Q2) Are authors expected to submit an unambiguous patch to frame the discussion? (I.e,. a specific patch file, or a specific git commit hash, as opposed to a github repo or a github branch.) Q3) Are authors expected to submit a single patch/commit, or is it acceptable / desirable for a large patch to be broken up as Peter has done? Q4) Do we require that any submitted patches appear as attachments on the pgsql-hackers email list, or is a github URL good enough? Q5) (This question is more generic.) I'm accustomed to using Github's pull-request system, where I can engage in dialog regarding specifc lines of a patch. I haven't noticed anything similar being used for PG code reviews, but perhaps I'm just looking in the wrong places. Are all PG code reviews basically just back-and-forth email conversations on the pgsql-hackers list? Thanks, Christian [1] https://commitfest.postgresql.org/10/776/ [2] https://www.postgresql.org/message-id/CABgyVxDBd3EvRdo-Rd6eo8QPEqV8%3DShaU2SJfo16wfE0R-hXTA%40mail.gmail.com [3] https://www.postgresql.org/message-id/bf9de63c-b669-4b8c-d33b-4a5ed11cd5d4%402ndquadrant.com [4] https://github.com/petere/postgresql/tree/commitfest/c%2B%2B -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suggestions for first contribution?
On Wed, Sep 7, 2016 at 10:44 AM, Stas Kelvich wrote: > There is also a list of projects for google summer of code: > https://wiki.postgresql.org/wiki/GSoC_2016 > > That topics expected to be doable by a newcomer during several months. It is > also slightly > outdated, but you always can check current state of things by searching this > mail list on interesting topic. > > Also postgres have several internal API’s like fdw[1] or gist[2] that allows > you to implements something > useful without digging too much into a postgres internals. > > [1] https://www.postgresql.org/docs/current/static/fdwhandler.html > [2] https://www.postgresql.org/docs/current/static/gist-extensibility.html Thanks Stas, I'll have a look at those. I appreciate the links. - Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suggestions for first contribution?
On Wed, Sep 7, 2016 at 10:34 AM, Yury Zhuravlev wrote: > Christian Convey wrote: >> >> Yury doesn't seem to need help >> with CMake > > Hello. > I am sorry that the only answer is now. > I need help but with write CMake code: > 1. Make ecpg tests > 2. Make MinGW/Msys support > 3. many many ... > all targets and discussion here: > https://github.com/stalkerg/postgres_cmake/issues > > If you can only test CMake for Ubuntu x86_64 that it is not necessary. > If I not fully understand you - sorry. Hi Yury, no problem. It sounds like most of the platform testing you want requires hardware I don't have, unfortunately. - Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suggestions for first contribution?
On Wed, Sep 7, 2016 at 10:14 AM, Aleksander Alekseev wrote: > Here is another idea for a contribution - refactoring. > > Currently there are a lot of procedures in PostgreSQL code that > definitely don't fit on one screen (i.e. ~50 lines). Also many files are > larger than say 1000 lines of code. For instance, psql_completion > procedure is more than 2000 lines long! I think patches that would make > such code easier to read would have a good chance to be accepted. Thanks for the suggestion! I can see how refactoring could make a lot of sense for some functions. That's probably a task I'd want to take on after I had more experience with PG's code base. I think being familiar with most/all of PG's code would make that task go more smoothly. - Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] C++ port of Postgres
On Tue, Sep 6, 2016 at 3:12 PM, Tom Lane wrote: >> (2) It seems like there are still a few big questions about this commit: >>- Is it wanted at the moment? It didn't seem like there's a >> consensus about whether or not this enhancement should be >> merged, even if the patch is pretty minimal. >>- It seems like there are two competing patch >> sets in play for this enhancement: Joy's and >> Peter's. Presumably at most one of them would >> be merged. > > These are things that reviews should be helping to decide. It's probably > a squishier topic than some patches, but if you're interested, feel free > to read code and weigh in. Thanks. It sounds like worst-case scenario, I perform an unneeded review. I'll give it a shot. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] C++ port of Postgres
On Wed, Aug 31, 2016 at 9:41 AM, Peter Eisentraut wrote: >> Joy, do you have an idea what a *minimally invasive* patch for C++ >> support would look like? That's certainly the first step here. > > I developed a minimally invasive patch for C++ support a few years ago > shortly after I wrote that blog post. Since there appears to have been > some interest here now, I have updated that and split it up into logical > chunks. > > So here you go. Hi folks, I'm ready for my first-ever PG code review, and I was considering tackling this one. I.e., https://commitfest.postgresql.org/10/776/ Could someone help me with a few procedural questions? (1) This page: https://wiki.postgresql.org/wiki/Reviewing_a_Patch lists the current commitfest's manager as "(vacant)". But this page: https://commitfest.postgresql.org/ seems to indicate that a commitfest is currently in progress. Is there a commitfest manager at the moment? (2) It seems like there are still a few big questions about this commit: - Is it wanted at the moment? It didn't seem like there's a consensus about whether or not this enhancement should be merged, even if the patch is pretty minimal. - It seems like there are two competing patch sets in play for this enhancement: Joy's and Peter's. Presumably at most one of them would be merged. Are these signs that I should find a different commit to review for now, so that this one can be further discussed? Thanks very much, Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suggestions for first contribution?
Thanks everyone for the suggestions. It sounds like the most useful thing I can do at the moment is perform code reviews. I assumed I'd need more experience with the PG code base, but I keep on reading that newcomers' reviews are welcome. Unless someone has a better idea, I'll start with that. The projects that Pavel and David mentioned sound very interesting. Hopefully I can tackle one of them after I've done a few code reviews. Thanks again, Christian On Tue, Sep 6, 2016 at 4:20 AM, Aleksander Alekseev wrote: > Hello, Christian > >> Can anyone suggest a project for my first PG contribution? > > In my experience good starting points are: > > * Doing code review. It's really important for PostgreSQL project. > Everyone likes to write code, no one likes to read or test it :) > * Finding typos, there are a lot of them. > * Finding bugs: static code analysis, sanitizers, etc. Also subscribe to > pgsql-bugs@ mailing list. > * Fixing bottlenecks. Tricky part here is to find one. Could be > difficult unless you are working for a company with many clients or > maintain a project that uses PostgreSQL. > > Also please note that such activity as answering questions on IRC or in > pgsql-general@, highlighting project news in your blog, writing articles > (or maybe even a book) about PostgreSQL internals, organizing local user > group, etc - all of this are very significant contributions. > > Good luck! > > -- > Best regards, > Aleksander Alekseev -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Suggestions for first contribution?
Hi guys, Can anyone suggest a project for my first PG contribution? My first two ideas didn't pan out: Yury doesn't seem to need help with CMake, and the TODO list's "-Wcast-align" project (now deleted) appeared impractical. I can continue trying things from the TODO list, but if someone knows of a project that's definitely worth doing, all the better. I'd prefer doing back-end coding, but I'm flexible. The only two caveats are: (1) I can't work on speedup-focused changes without my employer's permission, and (2) I only have access to Linux boxes. Any advice would be greatly appreciated. Kind regards, Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Obsolete TODO item "-Wcast-align" ?
On Sun, Sep 4, 2016 at 5:56 PM, Tom Lane wrote: > Christian Convey writes: >> I chose this item from the TODO page: "[E] Remove warnings created by >> -Wcast-align". It didn't have a check-mark after the "[E]", which I >> took to mean it's an outstanding issue. >> However, I'm starting to wonder if it's already been fixed: > > No, but you only see it on some platforms/compilers. On my OS X laptop > (clang-based not gcc-based compiler), turning that on generates just a > hair short of 13000 warnings :-( > > I think that TODO item is indeed obsolete, but more in the direction > of "we're never gonna do that". There are too many places where we > do need to cast up from generic pointer to specific structure pointer, > and there doesn't seem to be any practical way to tell a compiler which > instances are useful to warn about. Thanks for the response. I'm unclear about how the TODO list is curated. Is there someone whose attention I should direct to this thread? - Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Obsolete TODO item "-Wcast-align" ?
Hi guys, I'm trying to get my feet wet in PG development, but I ran into a snag with the TODO item I picked. Could someone please tell me if I'm making an obvious error? I chose this item from the TODO page: "[E] Remove warnings created by -Wcast-align". It didn't have a check-mark after the "[E]", which I took to mean it's an outstanding issue. However, I'm starting to wonder if it's already been fixed: (1) I wasn't able to reproduce it. I built PG using: * PG source commit 6591f4... (a recent master/HEAD) * Ubuntu's patched version of GCC 4.8.4 * "-Wcast-align" but I didn't see anything in the compiler output indicating that the warning got triggered. (2) It looks like someone else tried to tackle this TODO item about two years ago, but he also was unable to reproduce it: https://www.postgresql.org/message-id/flat/CALgh3eseTDT9RspDi0AfhE0j0A3e%2B5XiS3ykQyhK7Z5RgPQTLA%40mail.gmail.com#calgh3esetdt9rspdi0afhe0j0a3e+5xis3ykqyhk7z5rgpq...@mail.gmail.com I didn't find any other discussions about that line-item via Google, or by looking at the official/unofficial TODO-detail pages. Thanks very much, Christian Convey -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: About CMake v2
Hi Yury, >> I glad to hear it. I suppose you can just try build postgres and send all >> problems to github tracker. >> https://github.com/stalkerg/postgres_cmake/issues FYI, I had success using your "postgres_cmake" repo. I tested it up through "make check" and "make install". Here are the details: * postgres_cmake commit e7e75160d4533cd8caa9f3f0dd7b485dbd4e7bdf * compiler = cc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4 * cmake version = 3.5.3 Kind regards, Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: About CMake v2
Hi Yury, On Fri, Aug 19, 2016 at 9:46 AM, Yury Zhuravlev wrote: > Christian Convey wrote: >> >> I'm interested in helping with your CMake effort. I don't have any >> experience contributing to PG, but I do have some free time at the >> moment. Please let me know if I can help. > > I glad to hear it. I suppose you can just try build postgres and send all > problems to github tracker. > https://github.com/stalkerg/postgres_cmake/issues Thanks, I'll be happy to do that. There's been a lot of discussion on this thread regarding the minimum required CMake version. If you'd like me to test with a particular version (or versions) of CMake, please let me know. - Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: About CMake v2
Hi Stefan, >> I ask because I'm curious if/how someone in Yury's situation could >> predict which minimum version of CMake must be supported in order for >> his patch to be accepted. (And if he accepts my offer to pitch in, >> I'll actually need that particular detail.) > > well I personally think the level to meet would be that all the systems > on the buildfarm that can build -HEAD at the time the patch is proposed > for a commit should be able to build using the new system with whatever > cmake version is available in those by default (if it is at all). What standard would you suggest for those platforms which don't have an obvious default version of CMake? I assume this includes all versions of Windows and of OS X, but perhaps I'm misinformed. Thanks, Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: About CMake v2
> > I don't think we're interested in maintaining more build systems than we > already are. If cmake can replace the current MSVC tooling and > autoconf, all in one, my impression is that we're in. If we're > replacing two tools we've hammered pretty well over the years with two > tools that we haven't, I doubt we're interested. Good to know, thanks. - Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: About CMake v2
Hi Tom, >> I ask because I'm curious if/how someone in Yury's situation could >> predict which minimum version of CMake must be supported in order for >> his patch to be accepted. (And if he accepts my offer to pitch in, >> I'll actually need that particular detail.) > > well I personally think the level to meet would be that all the systems > on the buildfarm that can build -HEAD at the time the patch is proposed > for a commit should be able to build using the new system with whatever > cmake version is available in those by default (if it is at all). I see. In other projects I've worked on, the configuration of a build farm has been driven by some list of platforms that were considered important to support. Is that the case here as well? I.e., is the build-farm population just a convenient proxy for some other source of information regarding what platforms are important? Apologies if my questions are so basic that I can find the answers elsewhere. I'll happily follow any RTFM links. Thanks again, Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: About CMake v2
Hi Stefan, >> Yury: Would it make sense to add a call to "cmake_minimum_required" in >> one or more of your CMakeLists.txt files? > > it would make sense nevertheless but I dont think that 2.8.11 is old > enough - looking at the release information and the feature compatibily > matrix it would seems we should more aim at something like 2.8.0 or 2.8.3... I'm new to PG development, so I don't know what ideas the community is open to. But I wonder if there's any merit to the following approach... * Allow the CMake-based build system to assume a fairly modern version of CMake. (Maybe 2.8.12, or 3.0.) * For systems where the minimum CMake version isn't readily available, have an alternative build system which is just a simplistic Bash script that naively performs a full build every time it's invoked. The idea being that PG contributors are mostly the people who want efficient rebuilds, and most/all of them could easily install that minimal CMake version. *IF* it proved possible to write a clear, maintainable Bash script for that, perhaps that would eliminate most concerns about CMake not being well-supported on uncommon platforms / platform versions. - Christian -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: About CMake v2
Hi Tom, Thanks for that information. Is there some document I can read that explains which platform versions (e.g., OpenBSD 5.3) are considered strongly supported? I ask because I'm curious if/how someone in Yury's situation could predict which minimum version of CMake must be supported in order for his patch to be accepted. (And if he accepts my offer to pitch in, I'll actually need that particular detail.) Kind regards, Christian > As an additional comment, I don't see us accepting a build system > that doesn't run on pretty old cmakes. We have never had a policy > of "latest and greatest is required" for *any* build support tool, > and cmake isn't likely to be given an exception. > > BTW, I just noticed that cmake doesn't seem to be supplied as part of > Apple's dev tools, at least not up to current (El Capitan) releases. > That's going to be a rather large minus to be taken into account > whenever we make the go/no-go decision on this. > > regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: About CMake v2
Hi Karl, I'll need to let Yury answer your original question regarding the best way to report CMake-related bugs. Regarding the errors you're getting... I just looked at CMake's online documentation regarding your "target_compile_definitions" error. >From what I can tell, the "target_compile_definition" property was introduced in CMake 2.8.11. It sounds like your version of CMake is just a little too old. Regarding how one can know the required CMake version: My modus operandi for CMake projects in general is (1) read the project's how-to-build docs, and if that's not heplful, (2) hope that the project's CMake files contain a "cmake_minimum_required" call to give me a clear error message. I didn't find any such indication in Yuri's files, although perhaps I missed it. Yury: Would it make sense to add a call to "cmake_minimum_required" in one or more of your CMakeLists.txt files? Kind regards, Christian On Thu, Aug 18, 2016 at 3:08 PM, Stefan Kaltenbrunner wrote: > On 08/18/2016 08:57 PM, Christian Convey wrote: >> Hi Stefan, >> >> I think I've seen similar errors when a project's CMake files assumed >> a newer version of CMake than the one being run. >> >> Which version of CMake gave you those errors? (Sorry if you provided >> that detail and I'm just missing it.) > > > % cmake --version > cmake version 2.8.10.2 > > a quick look in the docs does not seem to reveal any kind of "minimum" > cmake version required - and the above is what is packaged as part of > openbsd 5.3 (which is outdated and unsupported upstream but it is > currently perfectly fine building all postgresql versions including > -HEAD and serving as a buildfarm member for years) > > > > Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: About CMake v2
Hi Stefan, I think I've seen similar errors when a project's CMake files assumed a newer version of CMake than the one being run. Which version of CMake gave you those errors? (Sorry if you provided that detail and I'm just missing it.) Kind regards, Christian On Thu, Aug 18, 2016 at 2:45 PM, Stefan Kaltenbrunner wrote: > On 06/29/2016 06:23 PM, Yury Zhuravlev wrote: >> Hello Hackers. >> >> I decided to talk about the current state of the project: >> 1. Merge with 9.6 master. 2. plpython2, plpython3, plperl, pltcl, plsql >> all work correctly (all tests pass). >> 3. Works done for all contrib modules. 4. You can use gettext, .po->.mo >> will have converted by CMake. 5. All test pass under some Linux, >> FreeBSD, Solaris10 (on Sparc), Windows MSVC 2015. MacOSX I think not big >> trouble too. 6. Prototype for PGXS (with MSVC support) done. >> I think is very big progress but I came across one problem. >> I not have access to many OS and platforms. For each platform need tests >> and small fixes. I can't develop and give guarantee without it. >> >> I think this is very important work which makes it easier further >> support Postgres but I can not do everything himself. It's physically >> impossible. >> >> I think without community support I can't do significantly more. >> >> Current version you can get here: >> https://github.com/stalkerg/postgres_cmake > > hmm how do you actually want reports on how it works? > > I just played with it on spoonbill (OpenBSD 5.3/sparc64) and got this: > > CMake Error at CMakeLists.txt:1250 (file): > file does not recognize sub-command GENERATE > > > CMake Error at src/port/CMakeLists.txt:156 (target_compile_definitions): > Unknown CMake command "target_compile_definitions". > > > -- Configuring incomplete, errors occurred! > > > there is also a ton of stuff like: > > > Make Error: Internal CMake error, TryCompile generation of cmake failed > -- Looking for opterr - not found > -- Looking for optreset > CMake Error at CMakeLists.txt:10 (ADD_EXECUTABLE): > Target "cmTryCompileExec3458204847" links to item " m" which has > leading or > trailing whitespace. This is now an error according to policy CMP0004. > > > CMake Error: Internal CMake error, TryCompile generation of cmake failed > -- Looking for optreset - not found > -- Looking for fseeko > CMake Error at CMakeLists.txt:10 (ADD_EXECUTABLE): > Target "cmTryCompileExec2628321539" links to item " m" which has > leading or > trailing whitespace. This is now an error according to policy CMP0004. > > > CMake Error: Internal CMake error, TryCompile generation of cmake failed > > > which I have no idea whether they are an actual problem or just > "configure" noise > > > > > > Stefan > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: About CMake v2
Hi Yury, I'm interested in helping with your CMake effort. I don't have any experience contributing to PG, but I do have some free time at the moment. Please let me know if I can help. I have an intermediate level of experience with CMake, Python, and Bash scripting. My native environment is Mint 17.3 (i.e., Ubuntu 15.10), but I'd be happy to create additional VM's as needed. I'm also happy to look into bugs on other systems (VMS, etc.) if I can get SSH access. Kind regards, Christian Convey On Fri, Jul 1, 2016 at 5:26 AM, Dmitry Maslyuk wrote: > Hi! > > I think, we need simple configure script generator for backward > compatibility and easy using this build system. > Try playing with cmake build system under Win2008+MinGW. I plan to write > perl script for automatic build this > with depends. > > On 29.06.2016 19:23, Yury Zhuravlev wrote: >> >> Hello Hackers. >> >> I decided to talk about the current state of the project: >> 1. Merge with 9.6 master. 2. plpython2, plpython3, plperl, pltcl, plsql >> all work correctly (all tests pass). >> 3. Works done for all contrib modules. 4. You can use gettext, .po->.mo >> will have converted by CMake. 5. All test pass under some Linux, FreeBSD, >> Solaris10 (on Sparc), Windows MSVC 2015. MacOSX I think not big trouble too. >> 6. Prototype for PGXS (with MSVC support) done. >> I think is very big progress but I came across one problem. >> I not have access to many OS and platforms. For each platform need tests >> and small fixes. I can't develop and give guarantee without it. >> >> I think this is very important work which makes it easier further support >> Postgres but I can not do everything himself. It's physically impossible. >> >> I think without community support I can't do significantly more. >> >> Current version you can get here: >> https://github.com/stalkerg/postgres_cmake >> >> Thanks! > > > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Breaking compile-time dependency cycles of Postgres subdirs?
On Mon, Feb 10, 2014 at 10:28 AM, Tom Lane wrote: > I think if it had been a clear, enforced goal all along, it might've been > possible to build the system with such a restriction (for the most part at > least). At this point though, the amount of work and code churn involved > seems like it'd far exceed the benefits. > > That makes sense to me. I certainly didn't think it was a slam-dunk that what I was proposing would be an improvement. It just seemed like a question worth asking. Thanks for your thoughts. - Christian
Re: [HACKERS] Breaking compile-time dependency cycles of Postgres subdirs?
On Sun, Feb 9, 2014 at 8:06 PM, Robert Haas wrote: > On Fri, Feb 7, 2014 at 7:39 AM, Christian Convey > wrote: > > This question is mostly just curiosity... > > As someone very new to this code base, I think these cycles make it a > little > > harder to figure out the runtime and compile-time dependencies between > the > > subsystems these directories seem to represent. I wonder if that's a > > problem others face as well? > > There are probably some cases that could be improved, but I have my > doubts about whether eliminating cycles is a reasonable goal. > Sometimes, two modules really do depend on each other. And, you're > talking about this not just on the level of individual files but > entire subtrees. There are 90,000 lines of code in src/backend/access > (whose headers are in src/include/access) and more than 38,000 in > src/backend/storage (whose headers are in src/include/storage); > expecting all dependencies between those modules to go in one > direction doesn't feel terribly reasonable. If it could be done at > all, you'd probably end up separating code into lots of little tiny > directories, splitting apart modules with logically related > functionality into chunks living in entirely different parts of the > source tree - and I don't think that would be an improvement. > > Thanks Robert. IMHO, whether or not it would be beneficial depends on which files (or definitions within files) had to be broken out into additional subdirectories in order to break the cycles. If it could be accomplished with at most a few additional subdirectories that were also intuitively meaningful groupings of files/definitions, it could be a win. But if not, I agree it would be a step backwards. Still, I'm thinking this might be a problem we need to partially solve if we're going to support a pluggable storage manager, particularly if we allow a pluggable storage manager to use the system's buffer system and/or block I/O system. I guess it depends on exactly what we want from a pluggable storage manager. - Christian
[HACKERS] Breaking compile-time dependency cycles of Postgres subdirs?
This question is mostly just curiosity... There are build-time dependency cycles between some of Postgres' code subdirectories. For example, "storage" and "access" have such a cycle: storage/buffpage.h #includes access/xlogdefs.h access/visibilitymap.h #includes storage/block.h Has there been any discussion about reorganizing these directories so that no such cycles exist? As someone very new to this code base, I think these cycles make it a little harder to figure out the runtime and compile-time dependencies between the subsystems these directories seem to represent. I wonder if that's a problem others face as well?
Re: [HACKERS] Changeset Extraction v7.3
It seems to me that the terms "physical", "logical", and "binary" are always relative to the perspective of the component being worked on. "Physical" often means "one level of abstraction below mine, and upon which my work builds". "Logical" means "my work's level of abstraction". And "Binary" means "data which I'm not going to pretend I know or care how to interpret." So I'd suggest "block" and "tuple", perhaps. On Wed, Jan 29, 2014 at 4:25 AM, Albe Laurenz wrote: > Andreas Karlsson wrote: > > On 01/28/2014 10:56 PM, Andres Freund wrote: > >> On 2014-01-28 21:48:09 +, Thom Brown wrote: > >>> On 28 January 2014 21:37, Robert Haas wrote: > On Tue, Jan 28, 2014 at 11:53 AM, Robert Haas > wrote: > The point of Andres's patch set is to introduce a new technology > called logical decoding; that is, the ability to get a replication > stream that is based on changes to tuples rather than changes to > blocks. It could also be called logical replication. In these > patches, our existing replication is referred to as "physical" > replication, which sounds kind of funny to me. Anyone have another > suggestion? > >>> > >>> Logical and Binary replication? > >> > >> Unfortunately changeset extraction output's can be binary data... > > > > I think "physical" and "logical" are fine and they seem to be well known > > terminology. Oracle uses those words and I have also seen many places > > use "physical backup" and "logical backup", for example on Barman's > > homepage. > > +1 > > I think it also fits well with the well-known terms "physical [database] > design" and "logical design". Not that it is the same thing, but I > believe that every database person, when faced with the distiction > "physical" versus "logical", will conclude that the former refers to > data placement or block structure, while the latter refers to the > semantics of the data being stored. > > Yours, > Laurenz Albe > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers >
Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)
On Mon, Jan 27, 2014 at 7:14 PM, Kouhei Kaigai wrote: > FDW's join pushing down is one of the valuable use-cases of this interface, > but not all. As you might know, my motivation is to implement GPU > acceleration > feature on top of this interface, that offers alternative way to scan or > join > relations or potentially sort or aggregate. > I'm curious how this relates to the pluggable storage idea discussed here https://wiki.postgresql.org/wiki/PgCon_2013_Developer_Meeting and here http://www.databasesoup.com/2013/05/postgresql-new-development-priorities-2.html I haven't seen a specific proposal about how much functionality should be encapsulated by a pluggable storage system. But I wonder if that would be the best place for specialized table-scan code to end up?
Re: [HACKERS] alternative back-end block formats
On Tue, Jan 28, 2014 at 12:39 PM, Tom Lane wrote: > TBH, I'd rather we waited till the commitfest is over. This is certainly > material for 9.5, if not even further out, so there's no pressing need for > a debate right now; and we have plenty of stuff we do need to deal with > right now. Works for me. I'll just lurk in the meantime, and see what I can figure out. Thanks. - Christian
Re: [HACKERS] alternative back-end block formats
> > There are a couple of really huge issues that would have to be argued out > before any progress could be made. > Is this something that people want to spend time on right now? As I mentioned earlier, I'm game. But it doesn't sound like I'll get very far without adult supervision.
Re: [HACKERS] alternative back-end block formats
On Tue, Jan 28, 2014 at 5:42 AM, Cédric Villemain wrote: ... > As written in the meeting notes, Tom Lane revealed an interest in > pluggable storage. So it might be interesting to check that. > > https://wiki.postgresql.org/wiki/PgCon_2013_Developer_Meeting > Thanks. I just read those meeting notes, and also Josh Berkus' blog on the topic: http://www.databasesoup.com/2013/05/postgresql-new-development-priorities-2.html I was only thinking to enable pluggable operations on a single, specified heap page, probably as a function of which table owned the page. Josh's blog seems to describe something a little broader in scope, although I can't tell from that post exactly what functionality that entails. Either way, this sounds like something I'd enjoy pitching in on, to whatever extent I could be useful. Has anyone started work on this yet?
Re: [HACKERS] alternative back-end block formats
Hi Craig, On Sun, Jan 26, 2014 at 5:47 AM, Craig Ringer wrote: > On 01/21/2014 07:43 PM, Christian Convey wrote: > > Hi all, > > > > I'm playing around with Postgres, and I thought it might be fun to > > experiment with alternative formats for relation blocks, to see if I can > > get smaller files and/or faster server performance. > > It's not clear how you'd do this without massively rewriting the guts of > Pg. > > Per the docs on internal structure, Pg has a block header, then tuples > within the blocks, each with a tuple header and list of Datum values for > the tuple. Each Datum has a generic Datum header (handling varlena vs > fixed length values etc) then a type-specific on-disk representation > controlled by the type output function for that type. > I'm still in the process of getting familiar with the pg backend code, so I don't have a concrete plan yet. However, I'm working on the assumption that some set of macros and functions encapsulates the page layout. If/when I tackle this, I expect to add a layer of indirection somewhere around that boundary, so that some non-catalog tables, whose schemas meet certain simplifying assumptions, are read and modified using specialized code. I don't want to get into the specific optimizations I'd like to try, only because I haven't fully studied the code yet, so I don't want to put my foot in my mouth. What concrete problem do you mean to tackle? What idea do you want to > explore or implement? > My real motivation is that I'd like to get more familiar with the pg backend codebase, and tilting at this windmill seemed like an interesting way to accomplish that. If I was focused on really solving a real-world problem, I'd say that this lays the groundwork for table-schema-specific storage optimizations and optimized record-filtering code. But I'd only make that argument if I planned to (a) perform a careful study with statistically significant benchmarks, and/or (b) produce a merge-worthy patch. At this point I have no intentions of doing so. My main goal really is just to have fun with the code. > > Does anyone know if this has been done before with Postgres? I would > > have assumed yes, but I'm not finding anything in Google about people > > having done this. > > AFAIK (and I don't know much in this area) the storage manager isn't > very pluggable compared to the rest of Pg. > Thanks for the warning. Duly noted. Kind regards, Christian
[HACKERS] alternative back-end block formats
Hi all, I'm playing around with Postgres, and I thought it might be fun to experiment with alternative formats for relation blocks, to see if I can get smaller files and/or faster server performance. Does anyone know if this has been done before with Postgres? I would have assumed yes, but I'm not finding anything in Google about people having done this. Thanks, Christian
Re: [HACKERS] Question about Lockhart's book
Thanks very much Josh. Those sound like great ideas - I'll try to give them a shot. On Fri, Dec 27, 2013 at 1:52 PM, Josh Berkus wrote: > On 12/27/2013 08:14 AM, Christian Convey wrote: > > Hi Andrew, > > > > Thanks for your response. Sometimes overall software architectures stay > > (mostly) unchanged for a long time, and so I figured that's possibly the > > case for Postgresql as well. But I didn't know, which is why I asked. > > Some things in that book will still be accurate and informative. The > problem is that you, as a beginner, won't know which things are still > good and which are obsolete. > > I'd suggest: > > - Developer documentation in our primary docs > - Developer FAQ on the wiki > - Bruce's presentations on various internals > - Tom's presentations on how the query planner works > - Various other people's presentations on other aspects, such as foreign > data wrappers, event triggers, etc. > > Unfortunately, there's no central index of presentations. > > I'm a big fan of "learn by doing", and here's a program which would > bring you up on a LOT of PostgreSQL: > > 1. Write a few of your own C functions, including trigger functions and > an operator. > > 2. Write your own foreign data wrapper for something. > > 3. Write your own Type, including input/output functions, stats > estimation and custom indexing. > > -- > Josh Berkus > PostgreSQL Experts Inc. > http://pgexperts.com > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers >
Re: [HACKERS] Question about Lockhart's book
Hi Andrew, Thanks for your response. Sometimes overall software architectures stay (mostly) unchanged for a long time, and so I figured that's possibly the case for Postgresql as well. But I didn't know, which is why I asked. Kind regards, Christian On Fri, Dec 27, 2013 at 11:11 AM, Andrew Dunstan wrote: > > On 12/27/2013 10:55 AM, Christian Convey wrote: > >> Hi guys, >> >> I'm starting to poke around the internals of Postgres. Does anyone know >> the extent to which Thomas Lockhart's book, "PostgresSQL Programmer's >> Guide" is accurate with respect to the current state of the code base? >> >> >> > Umm, that book was published in 2000, from what I can see on Amazon. Would > you use a book published 13 years ago to educate yourself on, say, the > Linux code base? 13 years is an eternity in this business. > > cheers > > andrew > >
[HACKERS] Question about Lockhart's book
Hi guys, I'm starting to poke around the internals of Postgres. Does anyone know the extent to which Thomas Lockhart's book, "PostgresSQL Programmer's Guide" is accurate with respect to the current state of the code base? Thanks, Christian