Re: assertion failure at cost_memoize_rescan()

2024-06-17 Thread Kohei KaiGai
2024年6月17日(月) 8:27 David Rowley : > > On Mon, 17 Jun 2024 at 10:23, Tomas Vondra > wrote: > > Interesting. Seems like a bug due to the two places clamping the values > > inconsistently. It probably does not matter in other contexts because we > > don't subtract the values like this, but here it tr

assertion failure at cost_memoize_rescan()

2024-06-14 Thread Kohei KaiGai
Hello, I met an assertion failure, and identified the root of the problem, but no idea how to fix it. The location of the problematic Assert() is at cost_memoize_rescan() to check 'hit_ratio' is between 0.0 and 1.0. The 'calls' is provided by the caller, and 'ndistinct' is the result of estimate_

Re: Using non-grouping-keys at HAVING clause

2023-09-08 Thread Kohei KaiGai
2023年9月8日(金) 19:07 Vik Fearing : > > On 9/8/23 09:42, Kohei KaiGai wrote: > > Hello, > > > > I got a trouble report here: > > https://github.com/heterodb/pg-strom/issues/636 > > > > It says that PG-Strom raised an error when the HAVING clause used > >

Using non-grouping-keys at HAVING clause

2023-09-08 Thread Kohei KaiGai
Hello, I got a trouble report here: https://github.com/heterodb/pg-strom/issues/636 It says that PG-Strom raised an error when the HAVING clause used non-grouping-keys, even though the vanilla PostgreSQL successfully processed the query. SELECT MAX(c0) FROM t0 GROUP BY t0.c1 HAVING t0.c0

Re: Asynchronous execution support for Custom Scan

2022-12-01 Thread Kohei KaiGai
> > IIUC, we already can use ctid in the where clause on the latest > > PostgreSQL, can't we? > > Oh, sorry, I missed the TidRangeScan. My apologies for the noise. > I made the ctidscan extension when we developed CustomScan API towards v9.5 or v9.6, IIRC. It would make sense just an example of Cus

Re: Asynchronous execution support for Custom Scan

2022-08-23 Thread Kohei KaiGai
I internally suggested him to expand the ctidscan module for the PoC purpose. https://github.com/kaigai/ctidscan Even though it does not have asynchronous capability actually, but suitable to ensure API works and small enough for reviewing. Best regards, 2022年8月22日(月) 17:55 Etsuro Fujita : > > H

Re: TRUNCATE on foreign table

2021-04-13 Thread Kohei KaiGai
2021年4月14日(水) 0:00 Fujii Masao : > > On 2021/04/13 23:25, Kohei KaiGai wrote: > > 2021年4月13日(火) 21:03 Bharath Rupireddy > > : > >> Yeah, ONLY clause is not pushed to the remote server in case of SELECT > >> commands. This is also true for DELETE and UP

Re: TRUNCATE on foreign table

2021-04-13 Thread Kohei KaiGai
2021年4月13日(火) 21:03 Bharath Rupireddy : > > On Tue, Apr 13, 2021 at 2:37 PM Kohei KaiGai wrote: > > Here are two points to discuss. > > > > Regarding to the FDW-APIs, yes, nobody can deny someone want to implement > > their own FDW module that adds special handling

Re: TRUNCATE on foreign table

2021-04-13 Thread Kohei KaiGai
2021年4月13日(火) 16:17 Fujii Masao : > > On 2021/04/13 14:22, Kohei KaiGai wrote: > > Let me remind the discussion at the design level. > > > > If postgres_fdw (and other FDW drivers) needs to consider whether > > ONLY-clause is given > > on the foreign table

Re: TRUNCATE on foreign table

2021-04-12 Thread Kohei KaiGai
2021年4月9日(金) 23:49 Kohei KaiGai : > > 2021年4月9日(金) 22:51 Fujii Masao : > > > > On 2021/04/09 12:33, Kohei KaiGai wrote: > > > 2021年4月8日(木) 22:14 Fujii Masao : > > >> > > >> On 2021/04/08 22:02, Kohei KaiGai wrote: > > >>>>

Re: TRUNCATE on foreign table

2021-04-09 Thread Kohei KaiGai
2021年4月9日(金) 22:51 Fujii Masao : > > On 2021/04/09 12:33, Kohei KaiGai wrote: > > 2021年4月8日(木) 22:14 Fujii Masao : > >> > >> On 2021/04/08 22:02, Kohei KaiGai wrote: > >>>> Anyway, attached is the updated version of the patch. This is still > >&g

Re: TRUNCATE on foreign table

2021-04-08 Thread Kohei KaiGai
2021年4月8日(木) 22:14 Fujii Masao : > > On 2021/04/08 22:02, Kohei KaiGai wrote: > >> Anyway, attached is the updated version of the patch. This is still based > >> on the latest Kazutaka-san's patch. That is, extra list for ONLY is still > >> passed to FDW.

Re: TRUNCATE on foreign table

2021-04-08 Thread Kohei KaiGai
2021年4月8日(木) 18:25 Fujii Masao : > > On 2021/04/08 15:48, Kohei KaiGai wrote: > > 2021年4月8日(木) 15:04 Fujii Masao : > >> > >> On 2021/04/08 13:43, Kohei KaiGai wrote: > >>> In case when a local table (with no children) has same contents, > >>>

Re: TRUNCATE on foreign table

2021-04-07 Thread Kohei KaiGai
2021年4月8日(木) 15:04 Fujii Masao : > > On 2021/04/08 13:43, Kohei KaiGai wrote: > > In case when a local table (with no children) has same contents, > > TRUNCATE command > > witll remove the entire table contents. > > But if there are local child tables that inher

Re: TRUNCATE on foreign table

2021-04-07 Thread Kohei KaiGai
2021年4月8日(木) 11:44 Fujii Masao : > > On 2021/04/08 10:56, Kohei KaiGai wrote: > > 2021年4月8日(木) 4:19 Fujii Masao : > >> > >> On 2021/04/06 21:06, Kazutaka Onishi wrote: > >>> Thank you for checking v13, and here is v14 patch. > >>> > &g

Re: TRUNCATE on foreign table

2021-04-07 Thread Kohei KaiGai
2021年4月8日(木) 4:19 Fujii Masao : > > On 2021/04/06 21:06, Kazutaka Onishi wrote: > > Thank you for checking v13, and here is v14 patch. > > > >> 1) Are we using all of these macros? I see that we are setting them > >> but we only use TRUNCATE_REL_CONTEXT_ONLY. If not used, can we remove > >> them? >

Re: TRUNCATE on foreign table

2021-04-03 Thread Kohei KaiGai
2021年4月4日(日) 13:07 Bharath Rupireddy : > > On Sat, Apr 3, 2021 at 8:31 PM Zhihong Yu wrote: > > w.r.t. Bharath's question on using hash table, I think the reason is that > > the search would be more efficient: > > Generally, sequential search would be slower if there are many entries > in a list.

Re: TRUNCATE on foreign table

2021-04-01 Thread Kohei KaiGai
2021年4月1日(木) 18:53 Fujii Masao : > > On 2021/04/01 0:09, Kohei KaiGai wrote: > > What does the "permission checks" mean in this context? > > The permission checks on the foreign tables involved are already checked > > at truncate_check_rel(), by PostgreSQL'

Re: TRUNCATE on foreign table

2021-03-31 Thread Kohei KaiGai
2021年3月30日(火) 2:53 Fujii Masao : > > On 2021/03/28 2:37, Kazutaka Onishi wrote: > > Fujii-san, > > > > Thank you for your review! > > Now I prepare v5 patch and I'll answer to your each comment. please > > check this again. > > Thanks a lot! > > > 5. For example, we can easily do that by truncate f

Re: TRUNCATE on foreign table

2021-03-29 Thread Kohei KaiGai
2021年3月30日(火) 3:45 Fujii Masao : > > On 2021/03/28 2:37, Kazutaka Onishi wrote: > > Fujii-san, > > > > Thank you for your review! > > Now I prepare v5 patch and I'll answer to your each comment. please > > check this again. > > m(_ _)m > > > > 1. In postgres-fdw.sgml, "and truncatable" should be ap

Re: TRUNCATE on foreign table

2021-03-29 Thread Kohei KaiGai
2021年3月30日(火) 2:54 Fujii Masao : > > On 2021/03/29 13:55, Michael Paquier wrote: > > On Mon, Mar 29, 2021 at 10:53:14AM +0900, Fujii Masao wrote: > >> I understand the motivation of this. But the other DMLs like UPDATE also > >> do the same thing for foreign tables? That is, when those DML commands

Re: TRUNCATE on foreign table

2021-03-28 Thread Kohei KaiGai
Fujii-san, > XLOG_HEAP_TRUNCATE record is written even for the truncation of > a foreign table. Why is this necessary? > Foreign-tables are often used to access local data structure, like columnar data files on filesystem, not only remote accesses like postgres_fdw. In case when we want to impleme

Re: TRUNCATE on foreign table

2021-03-28 Thread Kohei KaiGai
Onishi-san, The v5 patch contains full-contents of "src/backend/commands/tablecmds.c.orig". Please check it. 2021年3月28日(日) 2:37 Kazutaka Onishi : > > Fujii-san, > > Thank you for your review! > Now I prepare v5 patch and I'll answer to your each comment. please > check this again. > m(_ _)m > > 1

Re: How to retain lesser paths at add_path()?

2021-03-06 Thread Kohei KaiGai
2020年11月6日(金) 0:40 Dmitry Dolgov <9erthali...@gmail.com>: > > > On Tue, Jan 14, 2020 at 12:46:02AM +0900, Kohei KaiGai wrote: > > The v2 patch is attached. > > > > This adds two dedicated lists on the RelOptInfo to preserve lesser paths > > if extension

Re: contrib/cube - binary input/output handlers

2021-03-05 Thread Kohei KaiGai
2021年3月6日(土) 11:21 Tom Lane : > > Kohei KaiGai writes: > > 2021年3月6日(土) 1:41 Tom Lane : > >> Scanning the code, I have a couple of gripes. I'm not sure it's > >> a good plan to just send the "header" field raw like that --- > >> w

Re: contrib/cube - binary input/output handlers

2021-03-05 Thread Kohei KaiGai
2021年3月6日(土) 1:41 Tom Lane : > > Kohei KaiGai writes: > > Thanks, the attached patch add cube--1.5 for binary send/recv functions and > > modification of cube type using the new ALTER TYPE. > > Hm, that was already superseded by events (112d411fb). > As long as we ge

Re: contrib/cube - binary input/output handlers

2021-03-05 Thread Kohei KaiGai
Thanks, the attached patch add cube--1.5 for binary send/recv functions and modification of cube type using the new ALTER TYPE. Best regards, 2021年3月4日(木) 0:45 Tom Lane : > > I wrote: > > You can add the I/O functions with ALTER TYPE nowadays. > > To be concrete, see 949a9f043eb70a4986041b4751357

Re: contrib/cube - binary input/output handlers

2021-03-03 Thread Kohei KaiGai
2021年3月3日(水) 23:33 Peter Eisentraut : > > On 24.02.21 04:18, Kohei KaiGai wrote: > > This patch adds cube_send / cube_recv handlers on the contrib/cube data > > type. > > Once this patch was applied to, the libpq client can obtain the table > > data using binary mo

contrib/cube - binary input/output handlers

2021-02-23 Thread Kohei KaiGai
Hello, I noticed that contrib/cube data type does not support binary input/output handler when I tried to dump a table with cube columns, using a tool [*1] that uses binary data over libpq. $ pg2arrow -d postgres -t my_table ../utils/pgsql_client.c:351 SQL execution failed: ERROR: no binary out

Re: TRUNCATE on foreign table

2021-02-10 Thread Kohei KaiGai
2021年2月10日(水) 13:55 Ashutosh Bapat : > > On Tue, Feb 9, 2021 at 7:45 PM Kazutaka Onishi wrote: > > > > > IIUC, "truncatable" would be set to "false" for relations which do not > > > have physical storage e.g. views but will be true for regular tables. > > > > "truncatable" option is just for the f

Re: Is MaxHeapAttributeNumber a reasonable restriction for foreign-tables?

2021-02-04 Thread Kohei KaiGai
2021年2月4日(木) 23:45 Tom Lane : > > Amit Langote writes: > > On Thu, Feb 4, 2021 at 4:24 PM Kohei KaiGai wrote: > >> I think that MaxHeapAttributeNumber is a senseless restriction for foreign- > >> tables. How about your opinions? > > > My first re

Is MaxHeapAttributeNumber a reasonable restriction for foreign-tables?

2021-02-03 Thread Kohei KaiGai
Hello, I noticed that CheckAttributeNamesTypes() prevents to create a table that has more than MaxHeapAttributeNumber (1600) columns, for foreign-table also. IIUC, this magic number comes from length of the null-bitmap can be covered with t_hoff in HeapTupleHeaderData. For heap-tables, it seems to

Re: [BUG?] postgres_fdw incorrectly updates remote table if it has inherited children.

2020-03-02 Thread Kohei KaiGai
Fujita-san, > Unfortunately, I didn't have time to work on that (and won't in the > development cycle for PG13.) > > > Indeed, it is not an ideal query plan to execute for each updated rows... > > > > postgres=# explain select * from rtable_parent where tableoid = 126397 > > and ctid = '(0,11)'::t

Re: [BUG?] postgres_fdw incorrectly updates remote table if it has inherited children.

2020-02-29 Thread Kohei KaiGai
echanism would have the feature, it should not be a complicated problem. Best regards, 2020年3月1日(日) 12:39 Amit Langote : > > Hi, > > On Sun, Mar 1, 2020 at 12:00 PM Kohei KaiGai wrote: > > > > Hello, > > > > I noticed the following scenario under the devel

[BUG?] postgres_fdw incorrectly updates remote table if it has inherited children.

2020-02-29 Thread Kohei KaiGai
Hello, I noticed the following scenario under the development of truncate support on FDW. In case when 'ftable' maps a remote table that has inherited children,... postgres=# create table rtable_parent (id int, label text, x text); CREATE TABLE postgres=# create table rtable_child () inherits (r

Re: TRUNCATE on foreign tables

2020-02-29 Thread Kohei KaiGai
Hello, The attached is revised version. > > If callback is invoked with a foreign-relation that is specified by TRUNCATE > > command with ONLY, it seems to me reasonable that remote TRUNCATE > > command specifies the relation on behalf of the foreign table with ONLY. > > > > So, if ExecForeignTru

PL/Python - lifetime of variables?

2020-02-18 Thread Kohei KaiGai
Hello, I noticed that variables in PL/Python are not released at the end of procedure. Does it expected behavior? See this example below: https://github.com/heterodb/pg-strom/blob/master/test/input/arrow_python.source#L53 This PL/Python function maps a GPU buffer as cupy.ndarray object by cupy_s

Re: Is custom MemoryContext prohibited?

2020-02-10 Thread Kohei KaiGai
2020年2月10日(月) 13:53 Craig Ringer : > > On Thu, 6 Feb 2020 at 11:09, Andres Freund wrote: > > > I wasn't advocating for making plannodes.h etc frontend usable. I think > > that's a fairly different discussion than making enum NodeTag, > > pg_list.h, memutils.h available. I don't see them having ac

Re: Is custom MemoryContext prohibited?

2020-01-28 Thread Kohei KaiGai
2020年1月29日(水) 11:06 Thomas Munro : > > On Wed, Jan 29, 2020 at 2:49 PM Kohei KaiGai wrote: > > 2020年1月29日(水) 4:27 Thomas Munro : > > > On Tue, Jan 28, 2020 at 2:55 PM Kohei KaiGai wrote: > > > FWIW the code in https://commitfest.postgresql.org/26/2325/ ran into

Re: Is custom MemoryContext prohibited?

2020-01-28 Thread Kohei KaiGai
2020年1月29日(水) 4:27 Thomas Munro : > > On Tue, Jan 28, 2020 at 2:55 PM Kohei KaiGai wrote: > > I noticed MemoryContextIsValid() called by various kinds of memory context > > routines checks its node-tag as follows: > > > > #define MemoryContextIsValid(context

Re: Is custom MemoryContext prohibited?

2020-01-28 Thread Kohei KaiGai
2020年1月29日(水) 2:18 Robert Haas : > > On Tue, Jan 28, 2020 at 11:24 AM Tom Lane wrote: > > I strongly object to having the subtype field be just "char". > > I want it to be declared "MemoryContextType", so that gdb will > > still be telling me explicitly what struct type this really is. > > I reali

Re: Is custom MemoryContext prohibited?

2020-01-28 Thread Kohei KaiGai
2020年1月29日(水) 0:56 Tomas Vondra : > > On Tue, Jan 28, 2020 at 11:32:49PM +0900, Kohei KaiGai wrote: > >2020年1月28日(火) 23:09 Tomas Vondra : > >> > >> On Tue, Jan 28, 2020 at 10:55:11AM +0900, Kohei KaiGai wrote: > >> >Hello, > >> > > >&g

Re: Is custom MemoryContext prohibited?

2020-01-28 Thread Kohei KaiGai
2020年1月28日(火) 23:09 Tomas Vondra : > > On Tue, Jan 28, 2020 at 10:55:11AM +0900, Kohei KaiGai wrote: > >Hello, > > > >I noticed MemoryContextIsValid() called by various kinds of memory context > >routines checks its node-tag as follows: > > >

Is custom MemoryContext prohibited?

2020-01-27 Thread Kohei KaiGai
Hello, I noticed MemoryContextIsValid() called by various kinds of memory context routines checks its node-tag as follows: #define MemoryContextIsValid(context) \ ((context) != NULL && \ (IsA((context), AllocSetContext) || \ IsA((context), SlabContext) || \ IsA((context), Gen

Re: TRUNCATE on foreign tables

2020-01-27 Thread Kohei KaiGai
2020年1月21日(火) 15:38 Michael Paquier : > > On Mon, Jan 20, 2020 at 10:50:21PM +0900, Michael Paquier wrote: > > I have spent a good amount of time polishing 0001, tweaking the docs, > > comments, error messages and a bit its logic. I am getting > > comfortable with it, but it still needs an extra l

Re: TRUNCATE on foreign tables

2020-01-19 Thread Kohei KaiGai
2020年1月20日(月) 11:09 Kyotaro Horiguchi : > > Hello. > > At Fri, 17 Jan 2020 22:49:41 +0900, Kohei KaiGai wrote > in > > The v3 patch updated the points below: > > Did you attached it? > Sorry, it was a midnight job on Friday. Please check the attached patch. Best re

Re: TRUNCATE on foreign tables

2020-01-17 Thread Kohei KaiGai
hael Paquier : > > On Wed, Jan 15, 2020 at 11:33:07PM +0900, Kohei KaiGai wrote: > > 2020年1月15日(水) 17:11 Michael Paquier : > >> I have done a quick read through the patch. You have modified the > >> patch to pass down to the callback a list of relation OIDs to exec

Re: TRUNCATE on foreign tables

2020-01-15 Thread Kohei KaiGai
2020年1月15日(水) 17:11 Michael Paquier : > > On Tue, Jan 14, 2020 at 06:16:17PM +0900, Kohei KaiGai wrote: > > The "frels_list" is a list of foreign tables that are connected to a > > particular > > foreign server, thus, the server-id pulled out by foreign tables i

Re: TRUNCATE on foreign tables

2020-01-14 Thread Kohei KaiGai
s the manner of usual TRUNCATE command. Best regards, 2020年1月8日(水) 1:08 Kohei KaiGai : > > 2020年1月7日(火) 16:03 Michael Paquier : > > > > On Mon, Jan 06, 2020 at 04:32:39PM -0500, Stephen Frost wrote: > > > RESTRICT, yes. I don't know about ONLY being sensible as

Re: How to retain lesser paths at add_path()?

2020-01-13 Thread Kohei KaiGai
s, 2020年1月11日(土) 21:27 Tomas Vondra : > > On Sat, Jan 11, 2020 at 05:07:11PM +0900, Kohei KaiGai wrote: > >Hi, > > > >The proposition I posted at 10th-Oct proposed to have a separate list to > >retain > >lesser paths not to expand the path_list length, but h

Re: How to retain lesser paths at add_path()?

2020-01-11 Thread Kohei KaiGai
Hi, The proposition I posted at 10th-Oct proposed to have a separate list to retain lesser paths not to expand the path_list length, but here are no comments by others at that time. Indeed, the latest patch has not been updated yet. Please wait for a few days. I'll refresh the patch again. Thanks

Re: TRUNCATE on foreign tables

2020-01-07 Thread Kohei KaiGai
2020年1月7日(火) 16:03 Michael Paquier : > > On Mon, Jan 06, 2020 at 04:32:39PM -0500, Stephen Frost wrote: > > RESTRICT, yes. I don't know about ONLY being sensible as we don't > > really deal with inheritance and foreign tables very cleanly today, as I > > said up-thread, so I'm not sure if we reall

Re: TRUNCATE on foreign tables

2020-01-02 Thread Kohei KaiGai
2020年1月2日(木) 20:56 Alvaro Herrera : > > On 2020-Jan-02, Kohei KaiGai wrote: > > > 2020年1月2日(木) 12:16 Alvaro Herrera : > > > > > > I think this would need to preserve the notion of multi-table truncates. > > > Otherwise it won't be possible to tru

Re: TRUNCATE on foreign tables

2020-01-01 Thread Kohei KaiGai
2020年1月2日(木) 12:16 Alvaro Herrera : > > On 2020-Jan-01, Kohei KaiGai wrote: > > > Hello, > > > > The attached patch adds TRUNCATE support on foreign table. > > > > This patch adds an optional callback ExecForeignTruncate(Relation rel) > > to FdwRoutine.

Re: TRUNCATE on foreign tables

2019-12-31 Thread Kohei KaiGai
course, it is not clear to define the concept of "truncate" on some FDW drivers. In this case, TRUNCATE command prohibits to apply these foreign tables. 2019 is not finished at everywhere on the earth yet, so I believe it is Ok to add this patch to CF-2020:Jan. Best regards, 2020年1月1日(水

TRUNCATE on foreign tables

2019-12-31 Thread Kohei KaiGai
Hello, We right now don't support TRUNCATE on foreign tables. It may be a strange missing piece and restriction of operations. For example, if a partitioned table contains some foreign tables in its leaf, user cannot use TRUNCATE command to clean up the partitioned table. Probably, API design is

Re: Asymmetric partition-wise JOIN

2019-12-26 Thread Kohei KaiGai
ression test finished correctly. Best regards, 2019年12月1日(日) 12:24 Michael Paquier : > > On Sat, Aug 24, 2019 at 05:33:01PM +0900, Kohei KaiGai wrote: > > On the other hands, it eventually consumes almost equivalent amount > > of memory to load the inner relations, if no leafs

Re: How to retain lesser paths at add_path()?

2019-10-09 Thread Kohei KaiGai
2019年10月8日(火) 1:56 Tom Lane : > > Kohei KaiGai writes: > > 2019年10月7日(月) 23:44 Robert Haas : > >> But if we want to stick with the ad-hoc method, we could also just > >> have four possible return values: dominates, dominated-by, both, or > >> neither. >

Re: How to retain lesser paths at add_path()?

2019-10-07 Thread Kohei KaiGai
2019年10月7日(月) 23:44 Robert Haas : > > On Mon, Oct 7, 2019 at 9:56 AM Tom Lane wrote: > > We could imagine, maybe, that a hook for the purpose of allowing an > > additional dimension to be considered would be essentially a path > > comparison function, returning -1, +1, or 0 depending on whether >

Re: add a MAC check for TRUNCATE

2019-09-02 Thread Kohei KaiGai
Hello Yuli, 2019年7月25日(木) 3:52 Yuli Khodorkovskiy : > Since all DAC checks should have corresponding MAC, this patch adds a > hook to allow extensions to implement a MAC check on TRUNCATE. I have > also implemented this access check in the sepgsql extension. > > One important thing to note is that

Re: Asymmetric partition-wise JOIN

2019-08-24 Thread Kohei KaiGai
2019年8月24日(土) 7:02 Thomas Munro : > > On Fri, Aug 23, 2019 at 4:05 AM Kohei KaiGai wrote: > > We can consider the table join ptable X t1 above is equivalent to: > > (ptable_p0 + ptable_p1 + ptable_p2) X t1 > > = (ptable_p0 X t1) + (ptable_p1 X t1) + (ptable_p2 X t1) >

Re: Asymmetric partition-wise JOIN

2019-08-22 Thread Kohei KaiGai
for this feature? Best regards, 2019年8月12日(月) 15:03 Kohei KaiGai : > > Hello, > > PostgreSQL optimizer right now considers join pairs on only > non-partition - non-partition or > partition-leaf - partition-leaf relations. On the other hands, it is > harmless and makes sen

Asymmetric partition-wise JOIN

2019-08-11 Thread Kohei KaiGai
Hello, PostgreSQL optimizer right now considers join pairs on only non-partition - non-partition or partition-leaf - partition-leaf relations. On the other hands, it is harmless and makes sense to consider a join pair on non-partition - partition-leaf. See the example below. ptable is partitioned

Re: How to retain lesser paths at add_path()?

2019-08-11 Thread Kohei KaiGai
hether old_path that is dominated by new_path shall be removed, or not. If old_path survived, extension can re-use the path at the upper level to construct a special path. Best regards, 2019年8月1日(木) 21:14 Kohei KaiGai : > > 2019年8月1日(木) 19:24 Tomas Vondra : > > > > On Thu, A

Re: How to retain lesser paths at add_path()?

2019-08-01 Thread Kohei KaiGai
2019年8月1日(木) 19:24 Tomas Vondra : > > On Thu, Aug 01, 2019 at 06:28:08PM +0900, Kohei KaiGai wrote: > >2019年8月1日(木) 16:19 Richard Guo : > >> > >> On Thu, Aug 1, 2019 at 2:12 PM Kohei KaiGai wrote: > >>> > >>> 2019年8月1日(木) 1:41 Tom Lane : >

Re: How to retain lesser paths at add_path()?

2019-08-01 Thread Kohei KaiGai
2019年8月1日(木) 16:19 Richard Guo : > > On Thu, Aug 1, 2019 at 2:12 PM Kohei KaiGai wrote: >> >> 2019年8月1日(木) 1:41 Tom Lane : >> > >> > Robert Haas writes: >> > > Yeah, but I have to admit that this whole design makes me kinda >> > > uncom

Re: How to retain lesser paths at add_path()?

2019-07-31 Thread Kohei KaiGai
2019年8月1日(木) 1:41 Tom Lane : > > Robert Haas writes: > > Yeah, but I have to admit that this whole design makes me kinda > > uncomfortable. Every time somebody comes up with a new figure of > > merit, it increases not only the number of paths retained but also the > > cost of comparing two paths

How to retain lesser paths at add_path()?

2019-07-30 Thread Kohei KaiGai
Hello, When we add a new path using add_path(), it checks estimated cost and path-keys, then it also removes dominated paths, if any. Do we have a reasonable way to retain these "dominated" paths? Once it is considered lesser paths at a level, however, it may have a combined cheaper cost with uppe

Re: How to know referenced sub-fields of a composite type?

2019-05-30 Thread Kohei KaiGai
2019/05/30 16:33、Amit Langote のメール: >> On 2019/05/29 15:50, Kohei KaiGai wrote: >> 2019年5月29日(水) 13:26 Amit Langote : >>>> It means we can skip to load the sub-fields unreferenced, if >>>> query-planner can handle >>>> referenced and unreferenced s

Re: How to know referenced sub-fields of a composite type?

2019-05-28 Thread Kohei KaiGai
Hi Amit, 2019年5月29日(水) 13:26 Amit Langote : > > Kaigai-san, > > On 2019/05/29 12:13, Kohei KaiGai wrote: > > One interesting data type in Apache Arrow is "Struct" data type. It is > > equivalent to composite > > type in PostgreSQL. The "Struct"

How to know referenced sub-fields of a composite type?

2019-05-28 Thread Kohei KaiGai
Hello, A recent revision of PG-Strom has its columnar-storage using Apache Arrow format files on FDW infrastructure. Because of the columnar nature, it allows to load the values which are referenced by the query, thus, maximizes efficiency of the storage bandwidth. http://heterodb.github.io/pg-str

Re: add_partial_path() may remove dominated path but still in use

2019-02-05 Thread Kohei KaiGai
Hello, Let me remind the thread again. I'm waiting for the fix getting committed for a month... 2019年1月22日(火) 20:50 Kohei KaiGai : > > Let me remind the thread. > If no more comments, objections, or better ideas, please commit this fix. > > Thanks, > > 2019年1月17日(

[ANN] pg2arrow

2019-01-27 Thread Kohei KaiGai
Hello, I made a utility program to dump PostgreSQL database in Apache Arrow format. Apache Arrow is a kind of data format for columnar-based structured data; actively developed by Spark and comprehensive communities. It is suitable data representation for static and read-only but large number of

Re: add_partial_path() may remove dominated path but still in use

2019-01-22 Thread Kohei KaiGai
Let me remind the thread. If no more comments, objections, or better ideas, please commit this fix. Thanks, 2019年1月17日(木) 18:29 Kyotaro HORIGUCHI : > > Hello, sorry for the absence. > > At Fri, 11 Jan 2019 11:36:43 -0500, Robert Haas wrote > in > > On Thu, Jan 10, 2019 at

Re: add_partial_path() may remove dominated path but still in use

2019-01-10 Thread Kohei KaiGai
2019年1月11日(金) 5:52 Robert Haas : > > On Wed, Jan 9, 2019 at 12:44 AM Kohei KaiGai wrote: > > So, is it sufficient if set_rel_pathlist_hook is just relocated in > > front of the generate_gather_paths? > > If we have no use case for the second hook, here is little n

Re: add_partial_path() may remove dominated path but still in use

2019-01-08 Thread Kohei KaiGai
2019年1月9日(水) 13:18 Kyotaro HORIGUCHI : > > At Sun, 30 Dec 2018 12:31:22 +0900, Kohei KaiGai wrote > in > > 2018年12月30日(日) 4:12 Tom Lane : > > On the other hands, the later hook must be dedicated to add regular paths, > > and also provides a chance for extensions t

Re: add_partial_path() may remove dominated path but still in use

2019-01-03 Thread Kohei KaiGai
ion) AND (l_shipdate < '1995-01-01'::date) AND (l_shipdate >= '1994-01-01'::date)) (8 rows) Thanks, 2019年1月2日(水) 22:34 Kohei KaiGai : > > 2018年12月31日(月) 22:25 Amit Kapila : > > > > On Mon, Dec 31, 201

Re: add_partial_path() may remove dominated path but still in use

2019-01-02 Thread Kohei KaiGai
2018年12月31日(月) 22:25 Amit Kapila : > > On Mon, Dec 31, 2018 at 5:48 PM Kohei KaiGai wrote: > > > > 2018年12月31日(月) 13:10 Amit Kapila : > > > > > > On Sun, Dec 30, 2018 at 9:01 AM Kohei KaiGai wrote: > > > > 2018年12月30日(日) 4:12 Tom Lane : > > &

Re: add_partial_path() may remove dominated path but still in use

2018-12-31 Thread Kohei KaiGai
2018年12月31日(月) 13:10 Amit Kapila : > > On Sun, Dec 30, 2018 at 9:01 AM Kohei KaiGai wrote: > > 2018年12月30日(日) 4:12 Tom Lane : > > > > > > Kohei KaiGai writes: > > > > 2018年12月29日(土) 1:44 Tom Lane : > > > >> However, first I'd lik

Re: add_partial_path() may remove dominated path but still in use

2018-12-29 Thread Kohei KaiGai
2018年12月30日(日) 4:12 Tom Lane : > > Kohei KaiGai writes: > > 2018年12月29日(土) 1:44 Tom Lane : > >> However, first I'd like to know why this situation is arising in the first > >> place. To have the situation you're describing, we'd have to have >

Re: add_partial_path() may remove dominated path but still in use

2018-12-28 Thread Kohei KaiGai
2018年12月29日(土) 1:44 Tom Lane : > > Kohei KaiGai writes: > > I've investigated a crash report of PG-Strom for a few days, then I doubt > > add_partial_path() can unexpectedly release dominated old partial path > > but still referenced by other Gather node, and it lead

add_partial_path() may remove dominated path but still in use

2018-12-27 Thread Kohei KaiGai
Hello, I've investigated a crash report of PG-Strom for a few days, then I doubt add_partial_path() can unexpectedly release dominated old partial path but still referenced by other Gather node, and it leads unexpected system crash. Please check at the gpuscan.c:373 https://github.com/heterodb/pg

Re: lbound1 default in buildint2vector/buildoidvector

2018-11-15 Thread Kohei KaiGai
2018年11月15日(木) 12:41 Tom Lane : > > Kohei KaiGai writes: > > I noticed buildint2vector / buildoidvector assigns lbound1=0 as default > > value, but array type shall have lbound1=1 in the default. > > Is there some reasons for the difference? > > Backwards compatibili

lbound1 default in buildint2vector/buildoidvector

2018-11-14 Thread Kohei KaiGai
Hello, I noticed buildint2vector / buildoidvector assigns lbound1=0 as default value, but array type shall have lbound1=1 in the default. Is there some reasons for the difference? When I made a simple C-function that returns result of int2vector which carries attribute numbers of the argument. p

Re: [report] memory leaks in COPY FROM on partitioned table

2018-08-05 Thread Kohei KaiGai
2018-08-06 1:50 GMT+09:00 Alvaro Herrera : >> Now, it consumed about 60MB rss at the beginning of COPY FROM, and it >> grows up very slowly during the COPY FROM execution, then grew up to >> 250MB before completion. >> We may have another memory blocks which are not released during >> execution, ho

Re: [report] memory leaks in COPY FROM on partitioned table

2018-08-04 Thread Kohei KaiGai
2018-08-03 12:38 GMT+09:00 Alvaro Herrera : > On 2018-Aug-03, Kohei KaiGai wrote: > >> 2018-08-02 5:38 GMT+09:00 Alvaro Herrera : >> > On 2018-Aug-01, Alvaro Herrera wrote: >> > >> >> Right, makes sense. Pushed that way. >> > >> > Ka

Re: [report] memory leaks in COPY FROM on partitioned table

2018-08-02 Thread Kohei KaiGai
2018-08-02 5:38 GMT+09:00 Alvaro Herrera : > On 2018-Aug-01, Alvaro Herrera wrote: > >> Right, makes sense. Pushed that way. > > KaiGai, if you can please confirm that the pushed change fixes your test > case, I'd appreciate it. > Can you wait for a few days? I can drop the test dataset and reuse

Re: [report] memory leaks in COPY FROM on partitioned table

2018-07-23 Thread Kohei KaiGai
900, Kohei KaiGai wrote: >> In PG11beta2, my backend process gets terminated during COPY FROM of >> large text file (544GB) on partitioned table. >> The kernel log says OOM Killer send SIGKILL due to memory pressure. >> In fact, 63GB of system physical 64GB was consumed by the

[report] memory leaks in COPY FROM on partitioned table

2018-07-23 Thread Kohei KaiGai
Hello, In PG11beta2, my backend process gets terminated during COPY FROM of large text file (544GB) on partitioned table. The kernel log says OOM Killer send SIGKILL due to memory pressure. In fact, 63GB of system physical 64GB was consumed by the PostgreSQL backend just before the termination. O

[ANN] PG-Strom v2.0 is released

2018-04-17 Thread Kohei KaiGai
Hello, PG-Strom v2.0 is released. It is an extension module for PostgreSQL to accelerate reporting and analytics workloads towards large scale data set using GPU. Major enhancement in PG-Strom v2.0 includes: - Overall redesign of the internal infrastructure to manage GPU and stabilization - CPU+

Re: FP16 Support?

2018-01-08 Thread Kohei KaiGai
l or later) supports "half" precision data type by the hardware, so it might be valuable for somebody. Thanks, 2017-11-14 14:49 GMT+09:00 Kohei KaiGai : > 2017-11-14 10:33 GMT+09:00 Thomas Munro : >> On Tue, Nov 14, 2017 at 1:11 PM, Kohei KaiGai wrote: >>> Any opinions

Re: FP16 Support?

2017-11-13 Thread Kohei KaiGai
2017-11-14 10:33 GMT+09:00 Thomas Munro : > On Tue, Nov 14, 2017 at 1:11 PM, Kohei KaiGai wrote: >> Any opinions? > > The only reason I can think of for having it in core is that you might > want to use standard SQL notation FLOAT(10) to refer to it. Right now > our pa

Re: FP16 Support?

2017-11-13 Thread Kohei KaiGai
2017-11-14 10:32 GMT+09:00 Andres Freund : > On 2017-11-13 20:21:47 -0500, Tom Lane wrote: >> Kohei KaiGai writes: >> > How about your thought for support of half-precision floating point, >> > FP16 in short? >> >> This sounds like a whole lotta work for li

Re: FP16 Support?

2017-11-13 Thread Kohei KaiGai
2017-11-14 10:21 GMT+09:00 Tom Lane : > Kohei KaiGai writes: >> How about your thought for support of half-precision floating point, >> FP16 in short? > > This sounds like a whole lotta work for little if any gain. There's not > going to be any useful performan

FP16 Support?

2017-11-13 Thread Kohei KaiGai
Hello, How about your thought for support of half-precision floating point, FP16 in short? https://en.wikipedia.org/wiki/Half-precision_floating-point_format Probably, it does not make sense for most of our known workloads. Our supported hardware platform does not support FP16 operations, thus,