While looking at a recent complaint about bad planning, I was reminded that jsonb's @> and related operators use "contsel" as their selectivity estimator. This is really bad, because (a) contsel is only a stub, yielding a fixed default estimate, and (b) that default is 0.001, meaning we estimate these operators as five times more selective than equality, which is surely pretty silly.
There's a good model for improving this in ltree's ltreeparentsel(): for any "var OP constant" query, we can try applying the operator to all of the column's MCV and histogram values, taking the latter as being a random sample of the non-MCV values. That code is actually 100% generic except for the question of exactly what default selectivity ought to be plugged in when we don't have stats. Hence, the attached draft patch moves that logic into a generic function in selfuncs.c, and then invents "matchsel" and "matchjoinsel" generic estimators that have a default estimate of twice DEFAULT_EQ_SEL. (I'm not especially wedded to that number, but it seemed like a reasonable starting point.) There were a couple of other operators that seemed to be inappropriately using contsel, so I changed all of these to use matchsel: @>(tsquery,tsquery) | tsq_mcontains <@(tsquery,tsquery) | tsq_mcontained @@(text,text) | ts_match_tt @@(text,tsquery) | ts_match_tq -|-(anyrange,anyrange) | range_adjacent @>(jsonb,jsonb) | jsonb_contains ?(jsonb,text) | jsonb_exists ?|(jsonb,text[]) | jsonb_exists_any ?&(jsonb,text[]) | jsonb_exists_all <@(jsonb,jsonb) | jsonb_contained @?(jsonb,jsonpath) | jsonb_path_exists_opr @@(jsonb,jsonpath) | jsonb_path_match_opr Note: you might think that we should just shove this generic logic into contsel itself, and maybe areasel and patternsel while at it. However, that would be pretty useless for these functions' intended usage with the geometric operators, because we collect neither MCV nor histogram stats for the geometric data types, making the extra complexity worthless. Pending somebody putting some effort into estimation for the geometric data types, I think we should just get out of the business of having non-geometric types relying on these estimators. This patch is not complete, because I didn't look at changing the contrib modules, and grep says at least some of them are using contsel for non-geometric data types. But I thought I'd put it up for discussion at this stage. regards, tom lane
diff --git a/contrib/ltree/ltree_op.c b/contrib/ltree/ltree_op.c index 070868f..2791037 100644 --- a/contrib/ltree/ltree_op.c +++ b/contrib/ltree/ltree_op.c @@ -559,8 +559,6 @@ ltree2text(PG_FUNCTION_ARGS) } -#define DEFAULT_PARENT_SEL 0.001 - /* * ltreeparentsel - Selectivity of parent relationship for ltree data types. */ @@ -571,101 +569,12 @@ ltreeparentsel(PG_FUNCTION_ARGS) Oid operator = PG_GETARG_OID(1); List *args = (List *) PG_GETARG_POINTER(2); int varRelid = PG_GETARG_INT32(3); - VariableStatData vardata; - Node *other; - bool varonleft; double selec; - /* - * If expression is not variable <@ something or something <@ variable, - * then punt and return a default estimate. - */ - if (!get_restriction_variable(root, args, varRelid, - &vardata, &other, &varonleft)) - PG_RETURN_FLOAT8(DEFAULT_PARENT_SEL); - - /* - * If the something is a NULL constant, assume operator is strict and - * return zero, ie, operator will never return TRUE. - */ - if (IsA(other, Const) && - ((Const *) other)->constisnull) - { - ReleaseVariableStats(vardata); - PG_RETURN_FLOAT8(0.0); - } - - if (IsA(other, Const)) - { - /* Variable is being compared to a known non-null constant */ - Datum constval = ((Const *) other)->constvalue; - FmgrInfo contproc; - double mcvsum; - double mcvsel; - double nullfrac; - int hist_size; - - fmgr_info(get_opcode(operator), &contproc); - - /* - * Is the constant "<@" to any of the column's most common values? - */ - mcvsel = mcv_selectivity(&vardata, &contproc, constval, varonleft, - &mcvsum); - - /* - * If the histogram is large enough, see what fraction of it the - * constant is "<@" to, and assume that's representative of the - * non-MCV population. Otherwise use the default selectivity for the - * non-MCV population. - */ - selec = histogram_selectivity(&vardata, &contproc, - constval, varonleft, - 10, 1, &hist_size); - if (selec < 0) - { - /* Nope, fall back on default */ - selec = DEFAULT_PARENT_SEL; - } - else if (hist_size < 100) - { - /* - * For histogram sizes from 10 to 100, we combine the histogram - * and default selectivities, putting increasingly more trust in - * the histogram for larger sizes. - */ - double hist_weight = hist_size / 100.0; - - selec = selec * hist_weight + - DEFAULT_PARENT_SEL * (1.0 - hist_weight); - } - - /* In any case, don't believe extremely small or large estimates. */ - if (selec < 0.0001) - selec = 0.0001; - else if (selec > 0.9999) - selec = 0.9999; - - if (HeapTupleIsValid(vardata.statsTuple)) - nullfrac = ((Form_pg_statistic) GETSTRUCT(vardata.statsTuple))->stanullfrac; - else - nullfrac = 0.0; - - /* - * Now merge the results from the MCV and histogram calculations, - * realizing that the histogram covers only the non-null values that - * are not listed in MCV. - */ - selec *= 1.0 - nullfrac - mcvsum; - selec += mcvsel; - } - else - selec = DEFAULT_PARENT_SEL; - - ReleaseVariableStats(vardata); - - /* result should be in range, but make sure... */ - CLAMP_PROBABILITY(selec); + /* Use generic restriction selectivity logic, with default 0.001. */ + selec = generic_restriction_selectivity(root, operator, + args, varRelid, + 0.001); PG_RETURN_FLOAT8((float8) selec); } diff --git a/doc/src/sgml/xoper.sgml b/doc/src/sgml/xoper.sgml index 132056f..610545a 100644 --- a/doc/src/sgml/xoper.sgml +++ b/doc/src/sgml/xoper.sgml @@ -283,6 +283,18 @@ column OP constant </para> <para> + Another useful built-in selectivity estimation function + is <function>matchsel</function>, which will work for almost any + binary operator, if standard MCV and/or histogram statistics are + collected for the input data type(s). Its default estimate is set to + twice the default estimate used in <function>eqsel</function>, making + it most suitable for comparison operators that are somewhat less + strict than equality. (Or you could call the + underlying <function>generic_restriction_selectivity</function> + function, providing a different default estimate.) + </para> + + <para> There are additional selectivity estimation functions designed for geometric operators in <filename>src/backend/utils/adt/geo_selfuncs.c</filename>: <function>areasel</function>, <function>positionsel</function>, and <function>contsel</function>. At this writing these are just stubs, but you might want @@ -319,6 +331,7 @@ table1.column1 OP table2.column2 <member><function>scalarlejoinsel</function> for <literal><=</literal></member> <member><function>scalargtjoinsel</function> for <literal>></literal></member> <member><function>scalargejoinsel</function> for <literal>>=</literal></member> + <member><function>matchjoinsel</function> for generic matching operators</member> <member><function>areajoinsel</function> for 2D area-based comparisons</member> <member><function>positionjoinsel</function> for 2D position-based comparisons</member> <member><function>contjoinsel</function> for 2D containment-based comparisons</member> diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index 0be26fe..2e68338 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -829,6 +829,132 @@ histogram_selectivity(VariableStatData *vardata, FmgrInfo *opproc, } /* + * generic_restriction_selectivity - Selectivity for almost anything + * + * This function estimates selectivity for operators that we don't have any + * special knowledge about, but are on data types that we collect standard + * MCV and/or histogram statistics for. (Additional assumptions are that + * the operator is strict and immutable, or at least stable.) + * + * If we have "VAR OP CONST" or "CONST OP VAR", selectivity is estimated by + * applying the operator to each element of the column's MCV and/or histogram + * stats, and merging the results using the assumption that the histogram is + * a reasonable random sample of the column's non-MCV population. Note that + * if the operator's semantics are related to the histogram ordering, this + * might not be such a great assumption; other functions such as + * scalarineqsel() are probably a better match in such cases. + * + * Otherwise, fall back to the default selectivity provided by the caller. + */ +double +generic_restriction_selectivity(PlannerInfo *root, Oid operator, + List *args, int varRelid, + double default_selectivity) +{ + double selec; + VariableStatData vardata; + Node *other; + bool varonleft; + + /* + * If expression is not variable OP something or something OP variable, + * then punt and return the default estimate. + */ + if (!get_restriction_variable(root, args, varRelid, + &vardata, &other, &varonleft)) + return default_selectivity; + + /* + * If the something is a NULL constant, assume operator is strict and + * return zero, ie, operator will never return TRUE. + */ + if (IsA(other, Const) && + ((Const *) other)->constisnull) + { + ReleaseVariableStats(vardata); + return 0.0; + } + + if (IsA(other, Const)) + { + /* Variable is being compared to a known non-null constant */ + Datum constval = ((Const *) other)->constvalue; + FmgrInfo opproc; + double mcvsum; + double mcvsel; + double nullfrac; + int hist_size; + + fmgr_info(get_opcode(operator), &opproc); + + /* + * Calculate the selectivity for the column's most common values. + */ + mcvsel = mcv_selectivity(&vardata, &opproc, constval, varonleft, + &mcvsum); + + /* + * If the histogram is large enough, see what fraction of it matches + * the query, and assume that's representative of the non-MCV + * population. Otherwise use the default selectivity for the non-MCV + * population. + */ + selec = histogram_selectivity(&vardata, &opproc, + constval, varonleft, + 10, 1, &hist_size); + if (selec < 0) + { + /* Nope, fall back on default */ + selec = default_selectivity; + } + else if (hist_size < 100) + { + /* + * For histogram sizes from 10 to 100, we combine the histogram + * and default selectivities, putting increasingly more trust in + * the histogram for larger sizes. + */ + double hist_weight = hist_size / 100.0; + + selec = selec * hist_weight + + default_selectivity * (1.0 - hist_weight); + } + + /* In any case, don't believe extremely small or large estimates. */ + if (selec < 0.0001) + selec = 0.0001; + else if (selec > 0.9999) + selec = 0.9999; + + /* Don't forget to account for nulls. */ + if (HeapTupleIsValid(vardata.statsTuple)) + nullfrac = ((Form_pg_statistic) GETSTRUCT(vardata.statsTuple))->stanullfrac; + else + nullfrac = 0.0; + + /* + * Now merge the results from the MCV and histogram calculations, + * realizing that the histogram covers only the non-null values that + * are not listed in MCV. + */ + selec *= 1.0 - nullfrac - mcvsum; + selec += mcvsel; + } + else + { + /* Comparison value is not constant, so we can't do anything */ + selec = default_selectivity; + } + + ReleaseVariableStats(vardata); + + /* result should be in range, but make sure... */ + CLAMP_PROBABILITY(selec); + + return selec; +} + +/* * ineq_histogram_selectivity - Examine the histogram for scalarineqsel * * Determine the fraction of the variable's histogram population that @@ -2916,6 +3042,40 @@ fail: /* + * matchsel -- generic matching-operator selectivity support + * + * Use these for any operators that (a) are on data types for which we collect + * standard statistics, and (b) have behavior for which the default estimate + * of 2*DEFAULT_EQ_SEL is sane. Typically that is good for match-like + * operators. + */ + +Datum +matchsel(PG_FUNCTION_ARGS) +{ + PlannerInfo *root = (PlannerInfo *) PG_GETARG_POINTER(0); + Oid operator = PG_GETARG_OID(1); + List *args = (List *) PG_GETARG_POINTER(2); + int varRelid = PG_GETARG_INT32(3); + double selec; + + /* Use generic restriction selectivity logic. */ + selec = generic_restriction_selectivity(root, operator, + args, varRelid, + 2 * DEFAULT_EQ_SEL); + + PG_RETURN_FLOAT8((float8) selec); +} + +Datum +matchjoinsel(PG_FUNCTION_ARGS) +{ + /* Just punt, for the moment. */ + PG_RETURN_FLOAT8(2 * DEFAULT_EQ_SEL); +} + + +/* * Helper routine for estimate_num_groups: add an item to a list of * GroupVarInfos, but only if it's not known equal to any of the existing * entries. diff --git a/src/include/catalog/pg_operator.dat b/src/include/catalog/pg_operator.dat index 7c135da..6c9bbb3 100644 --- a/src/include/catalog/pg_operator.dat +++ b/src/include/catalog/pg_operator.dat @@ -3016,18 +3016,18 @@ { oid => '3693', descr => 'contains', oprname => '@>', oprleft => 'tsquery', oprright => 'tsquery', oprresult => 'bool', oprcom => '<@(tsquery,tsquery)', - oprcode => 'tsq_mcontains', oprrest => 'contsel', oprjoin => 'contjoinsel' }, + oprcode => 'tsq_mcontains', oprrest => 'matchsel', oprjoin => 'matchjoinsel' }, { oid => '3694', descr => 'is contained by', oprname => '<@', oprleft => 'tsquery', oprright => 'tsquery', oprresult => 'bool', oprcom => '@>(tsquery,tsquery)', - oprcode => 'tsq_mcontained', oprrest => 'contsel', oprjoin => 'contjoinsel' }, + oprcode => 'tsq_mcontained', oprrest => 'matchsel', oprjoin => 'matchjoinsel' }, { oid => '3762', descr => 'text search match', oprname => '@@', oprleft => 'text', oprright => 'text', oprresult => 'bool', - oprcode => 'ts_match_tt', oprrest => 'contsel', oprjoin => 'contjoinsel' }, + oprcode => 'ts_match_tt', oprrest => 'matchsel', oprjoin => 'matchjoinsel' }, { oid => '3763', descr => 'text search match', oprname => '@@', oprleft => 'text', oprright => 'tsquery', - oprresult => 'bool', oprcode => 'ts_match_tq', oprrest => 'contsel', - oprjoin => 'contjoinsel' }, + oprresult => 'bool', oprcode => 'ts_match_tq', oprrest => 'matchsel', + oprjoin => 'matchjoinsel' }, # generic record comparison operators { oid => '2988', oid_symbol => 'RECORD_EQ_OP', descr => 'equal', @@ -3178,7 +3178,7 @@ { oid => '3897', descr => 'is adjacent to', oprname => '-|-', oprleft => 'anyrange', oprright => 'anyrange', oprresult => 'bool', oprcom => '-|-(anyrange,anyrange)', - oprcode => 'range_adjacent', oprrest => 'contsel', oprjoin => 'contjoinsel' }, + oprcode => 'range_adjacent', oprrest => 'matchsel', oprjoin => 'matchjoinsel' }, { oid => '3898', descr => 'range union', oprname => '+', oprleft => 'anyrange', oprright => 'anyrange', oprresult => 'anyrange', oprcom => '+(anyrange,anyrange)', @@ -3258,22 +3258,22 @@ { oid => '3246', descr => 'contains', oprname => '@>', oprleft => 'jsonb', oprright => 'jsonb', oprresult => 'bool', oprcom => '<@(jsonb,jsonb)', oprcode => 'jsonb_contains', - oprrest => 'contsel', oprjoin => 'contjoinsel' }, + oprrest => 'matchsel', oprjoin => 'matchjoinsel' }, { oid => '3247', descr => 'key exists', oprname => '?', oprleft => 'jsonb', oprright => 'text', oprresult => 'bool', - oprcode => 'jsonb_exists', oprrest => 'contsel', oprjoin => 'contjoinsel' }, + oprcode => 'jsonb_exists', oprrest => 'matchsel', oprjoin => 'matchjoinsel' }, { oid => '3248', descr => 'any key exists', oprname => '?|', oprleft => 'jsonb', oprright => '_text', oprresult => 'bool', - oprcode => 'jsonb_exists_any', oprrest => 'contsel', - oprjoin => 'contjoinsel' }, + oprcode => 'jsonb_exists_any', oprrest => 'matchsel', + oprjoin => 'matchjoinsel' }, { oid => '3249', descr => 'all keys exist', oprname => '?&', oprleft => 'jsonb', oprright => '_text', oprresult => 'bool', - oprcode => 'jsonb_exists_all', oprrest => 'contsel', - oprjoin => 'contjoinsel' }, + oprcode => 'jsonb_exists_all', oprrest => 'matchsel', + oprjoin => 'matchjoinsel' }, { oid => '3250', descr => 'is contained by', oprname => '<@', oprleft => 'jsonb', oprright => 'jsonb', oprresult => 'bool', oprcom => '@>(jsonb,jsonb)', oprcode => 'jsonb_contained', - oprrest => 'contsel', oprjoin => 'contjoinsel' }, + oprrest => 'matchsel', oprjoin => 'matchjoinsel' }, { oid => '3284', descr => 'concatenate', oprname => '||', oprleft => 'jsonb', oprright => 'jsonb', oprresult => 'jsonb', oprcode => 'jsonb_concat' }, @@ -3292,10 +3292,10 @@ { oid => '4012', descr => 'jsonpath exists', oprname => '@?', oprleft => 'jsonb', oprright => 'jsonpath', oprresult => 'bool', oprcode => 'jsonb_path_exists_opr(jsonb,jsonpath)', - oprrest => 'contsel', oprjoin => 'contjoinsel' }, + oprrest => 'matchsel', oprjoin => 'matchjoinsel' }, { oid => '4013', descr => 'jsonpath match', oprname => '@@', oprleft => 'jsonb', oprright => 'jsonpath', oprresult => 'bool', oprcode => 'jsonb_path_match_opr(jsonb,jsonpath)', - oprrest => 'contsel', oprjoin => 'contjoinsel' }, + oprrest => 'matchsel', oprjoin => 'matchjoinsel' }, ] diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 07a86c7..088d106 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10574,6 +10574,15 @@ prosrc => 'shift_jis_2004_to_euc_jis_2004', probin => '$libdir/euc2004_sjis2004' }, +{ oid => '8387', + descr => 'restriction selectivity for generic matching operators', + proname => 'matchsel', provolatile => 's', prorettype => 'float8', + proargtypes => 'internal oid internal int4', prosrc => 'matchsel' }, +{ oid => '8388', descr => 'join selectivity for generic matching operators', + proname => 'matchjoinsel', provolatile => 's', prorettype => 'float8', + proargtypes => 'internal oid internal int2 internal', + prosrc => 'matchjoinsel' }, + # replication/origin.h { oid => '6003', descr => 'create a replication origin', proname => 'pg_replication_origin_create', provolatile => 'v', diff --git a/src/include/utils/selfuncs.h b/src/include/utils/selfuncs.h index 1c9570f..faa01cf 100644 --- a/src/include/utils/selfuncs.h +++ b/src/include/utils/selfuncs.h @@ -148,6 +148,9 @@ extern double histogram_selectivity(VariableStatData *vardata, FmgrInfo *opproc, Datum constval, bool varonleft, int min_hist_size, int n_skip, int *hist_size); +extern double generic_restriction_selectivity(PlannerInfo *root, Oid operator, + List *args, int varRelid, + double default_selectivity); extern double ineq_histogram_selectivity(PlannerInfo *root, VariableStatData *vardata, FmgrInfo *opproc, bool isgt, bool iseq,