MartijnVisser commented on code in PR #29864: URL: https://github.com/apache/flink/pull/29864#discussion_r4562933067
########## docs/data/functions/window/dense_rank.yml: ########## @@ -0,0 +1,65 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: DENSE_RANK +summary: Returns the rank of each row within its partition, without gaps after ties. + +syntax: + sql: DENSE_RANK() OVER ([PARTITION BY expression [, ...]] ORDER BY expression [ASC | DESC] [, ...]) + +returns: + description: One plus the number of distinct `ORDER BY` values that come before the current row in its partition. Rows with equal `ORDER BY` values, called peers, get the same rank, and the next rank follows without a gap, such as `1, 2, 2, 3`. + type: "`BIGINT NOT NULL`." + +notes: + - The window needs an `ORDER BY` and can't have a frame clause, such as `ROWS BETWEEN 1 PRECEDING AND CURRENT ROW`. + - Not supported in streaming mode. Use `ROW_NUMBER` instead. Review Comment: Same as for RANK, this runs as a streaming OVER aggregation. ########## docs/data/functions/window/lag.yml: ########## @@ -0,0 +1,105 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: LAG +summary: Returns a value from a row before the current row in its partition. + +syntax: + sql: LAG(expression [, offset [, default]]) OVER ([PARTITION BY expression [, ...]] [ORDER BY expression [ASC | DESC] [, ...]]) + java: | + lag($("expression")) + lag($("expression"), offset) + lag($("expression"), offset, defaultValue) + python: | + lag(col("expression")) + lag(col("expression"), offset) + lag(col("expression"), offset, default) + +arguments: + - name: expression + type: Any type + required: true + description: The value to read from the other row. + - name: offset + type: An integer + required: false + default: "`1`" + description: How many rows before the current row to look. `0` returns the value of the current row. A negative offset looks the other way, so `LAG(x, -1)` returns the same as `LEAD(x, 1)` in batch mode. + - name: default + type: The type of `expression` + required: false + default: NULL + description: The value to return if there is no row at that offset. It is evaluated on the current row. Give it the type of `expression`, such as `CAST(0 AS BIGINT)` for a `BIGINT` expression. + +returns: + description: The value of `expression` in the row that comes `offset` rows before the current row in its partition, by the `ORDER BY` of the window. + nulls: Returns NULL if that row has a NULL value, or if there is no such row and `default` is NULL or omitted. + type: The type of `expression`. NOT NULL if `default` is NOT NULL. Review Comment: This is the behavior of FLINK-40738: with a NOT NULL default, a NULL at the offset row isn't returned, I got `-1`. I think the intended type is nullable unless both are NOT NULL. ########## docs/data/functions/window/lead.yml: ########## @@ -0,0 +1,105 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: LEAD +summary: Returns a value from a row after the current row in its partition. + +syntax: + sql: LEAD(expression [, offset [, default]]) OVER ([PARTITION BY expression [, ...]] [ORDER BY expression [ASC | DESC] [, ...]]) + java: | + lead($("expression")) + lead($("expression"), offset) + lead($("expression"), offset, defaultValue) + python: | + lead(col("expression")) + lead(col("expression"), offset) + lead(col("expression"), offset, default) + +arguments: + - name: expression + type: Any type + required: true + description: The value to read from the other row. + - name: offset + type: An integer + required: false + default: "`1`" + description: How many rows after the current row to look. `0` returns the value of the current row. A negative offset looks the other way, so `LEAD(x, -1)` returns the same as `LAG(x, 1)` in batch mode. + - name: default + type: The type of `expression` + required: false + default: NULL + description: The value to return if there is no row at that offset. It is evaluated on the current row. Give it the type of `expression`, such as `CAST(0 AS BIGINT)` for a `BIGINT` expression. + +returns: + description: The value of `expression` in the row that comes `offset` rows after the current row in its partition, by the `ORDER BY` of the window. + nulls: Returns NULL if that row has a NULL value, or if there is no such row and `default` is NULL or omitted. + type: The type of `expression`. NOT NULL if `default` is NOT NULL. Review Comment: Same as for LAG, FLINK-40738. ########## docs/data/functions/window/lag.yml: ########## @@ -0,0 +1,105 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: LAG +summary: Returns a value from a row before the current row in its partition. + +syntax: + sql: LAG(expression [, offset [, default]]) OVER ([PARTITION BY expression [, ...]] [ORDER BY expression [ASC | DESC] [, ...]]) + java: | + lag($("expression")) + lag($("expression"), offset) + lag($("expression"), offset, defaultValue) + python: | + lag(col("expression")) + lag(col("expression"), offset) + lag(col("expression"), offset, default) + +arguments: + - name: expression + type: Any type + required: true + description: The value to read from the other row. + - name: offset + type: An integer + required: false + default: "`1`" + description: How many rows before the current row to look. `0` returns the value of the current row. A negative offset looks the other way, so `LAG(x, -1)` returns the same as `LEAD(x, 1)` in batch mode. + - name: default + type: The type of `expression` + required: false + default: NULL + description: The value to return if there is no row at that offset. It is evaluated on the current row. Give it the type of `expression`, such as `CAST(0 AS BIGINT)` for a `BIGINT` expression. + +returns: + description: The value of `expression` in the row that comes `offset` rows before the current row in its partition, by the `ORDER BY` of the window. + nulls: Returns NULL if that row has a NULL value, or if there is no such row and `default` is NULL or omitted. + type: The type of `expression`. NOT NULL if `default` is NOT NULL. + +notes: + - The window can't have a frame clause, such as `ROWS BETWEEN 1 PRECEDING AND CURRENT ROW`. + - Without an `ORDER BY`, the order of the rows is undefined. + - In streaming mode, the window must be ordered by a time attribute in ascending order. Review Comment: An ascending non-time column works too, as the OVER aggregation page says. `ORDER BY id` runs fine in streaming mode. ########## docs/data/functions/regular-expressions/regexp_instr.yml: ########## @@ -0,0 +1,92 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: REGEXP_INSTR +summary: Returns the position of the first match of a Java regular expression in a string. + +syntax: + sql: REGEXP_INSTR(str, regex) + java: $("str").regexpInstr(regex) + python: col("str").regexp_instr(regex) + +arguments: + - name: str + type: CHAR, VARCHAR, STRING + required: true + description: The string to search. + - name: regex + type: CHAR, VARCHAR, STRING + required: true + description: A Java regular expression. + +returns: + description: The position in `str` where the first match of `regex` starts, counting from 1. `0` if `regex` doesn't match. + nulls: Returns NULL if any argument is NULL or if `regex` doesn't compile. + type: "`INT`, nullable." + +notes: + - Positions count UTF-16 code units, so a character outside the Basic Multilingual Plane, such as an emoji, counts as 2. `POSITION` counts it as 1. Review Comment: This is the open FLINK-41319. Should the note link the ticket? ########## docs/data/functions/utility/typeof.yml: ########## @@ -0,0 +1,89 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: TYPEOF +summary: Returns the data type of an expression as a string. + +syntax: + sql: TYPEOF(input [, force_serializable]) + java: | + call("TYPEOF", $("input")) + call("TYPEOF", $("input"), true) + python: | + call("TYPEOF", col("input")) + call("TYPEOF", col("input"), True) + +arguments: + - name: input + type: Any type + required: true + description: The expression whose type to return. + - name: force_serializable + type: BOOLEAN literal + required: false + default: "`FALSE`" + description: "`TRUE` returns the full string that can be parsed back into the type, instead of the short one." + +returns: + description: The data type of `input`, including `NOT NULL` for a type that can't be NULL. + nulls: Returns NULL if `force_serializable` is `TRUE` and the type has no serializable string, such as `CHAR(0)`, the type of the literal `''`. + type: "`STRING`, nullable." + +notes: + - The type is computed when the statement is planned, so every row gets the same string. Review Comment: Not with a UNION ALL: `TYPEOF(x)` over `SELECT 1 UNION ALL SELECT CAST(NULL AS INT)` gives `INT NOT NULL` for one row and `INT` for the other. ########## docs/README.md: ########## @@ -283,3 +285,49 @@ Renders a link to a file in the Apache Flink Python Documentation. ``` Renders a link to the apache flink download page. + +## Built-in function pages + +Pages for built-in SQL functions live under `content/docs/sql/built-in-functions/<category>/`. They aren't written in Markdown. A content adapter generates each page, in English and Chinese, from one YAML file in `data/functions/`. + +### Adding a function + +1. Add `data/functions/<category>/<name>.yml`. `<category>` is the directory of the category under `content/docs/sql/built-in-functions/`, such as `regular-expressions`. `<name>` is the function name in lower case, such as `regexp_extract`, and the page is served at `<category>/regexp-extract/`. Use `data/functions/regular-expressions/regexp_extract.yml` as the reference for the fields. +2. Add a row for the function to its category in `data/sql_functions.yml` and `data/sql_functions_zh.yml`. Keep the rows in alphabetical order of the function name, the order of the pages in the sidebar. Give it `sql`, `table` and `page: regexp-extract`, without a `description`. The table shows the function's name, links it to the page and shows the page's summary. +3. Build the docs as described above and check the page in both languages. A broken `ref` fails the build, locally and in CI. + +The Chinese page renders the same YAML, so the English text is its placeholder until someone translates it. If you aren't a native Chinese speaker, don't translate: add no Chinese file and leave the translation to native speakers. A translation goes in `data/functions_zh/<category>/<name>.yml`, and the fields it sets replace the English ones. + +### What goes on a page + +Every page has the same sections, in this order. A section is left out when the YAML has no field for it. + +| Section | What it says | +|-------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Summary | One sentence on what the function returns. The category table shows it too. | +| See also | Related functions, listed under the summary. | +| Syntax | The SQL call, and the Java and Python Table API calls where they exist. There is no Scala tab, since the Scala API is deprecated. | +| Arguments | For each argument: the accepted types in plain words, whether it is required, its default and what it means. | +| Returns | What the function returns, then the cases that return NULL, then the return type and whether it is nullable. | +| Usage notes | What a user of the function needs and can't tell from the sections above, such as limits and determinism. | +| Examples | A list of the scenarios, then each scenario with a title, one sentence, the SQL and its result. A Table API-only function shows Java and Python code instead. | + +NULL handling has no section of its own. It goes into Returns, and the examples show it. + +### What to leave out + +A page tells a user how to use the function, not everything the code does. Leave out: + +- Behavior that all functions share, such as that an untyped `NULL` needs a cast, or that arguments of the wrong type fail at planning time. +- Internal behavior, such as how the planner rewrites a call or how its code is generated. +- Exact error messages. Say when a call fails instead, such as "Not supported in streaming mode." +- Types and inputs that few users would try. Name the accepted types in plain words, as other engines do, such as "Any comparable type". +- Common sense, such as that a call needs its parentheses. +- Bugs. A page describes the intended behavior, and a bug goes to Jira. Review Comment: Users on master still hit these. I think a note that links the open ticket helps more than leaving the bug out, for example FLINK-40738 for LAG. ########## docs/data/functions/window/rank.yml: ########## @@ -0,0 +1,80 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: RANK +summary: Returns the rank of each row within its partition, with gaps after ties. + +syntax: + sql: RANK() OVER ([PARTITION BY expression [, ...]] ORDER BY expression [ASC | DESC] [, ...]) + +returns: + description: One plus the number of rows that come before the current row in its partition, by the `ORDER BY` of the window. Rows with equal `ORDER BY` values, called peers, get the same rank, and the next rank skips the numbers they used, such as `1, 2, 2, 4`. + type: "`BIGINT NOT NULL`." + +notes: + - The window needs an `ORDER BY` and can't have a frame clause, such as `ROWS BETWEEN 1 PRECEDING AND CURRENT ROW`. + - Not supported in streaming mode. Use `ROW_NUMBER` instead. Review Comment: RANK does run in streaming mode as an OVER aggregation, `OverAggregateITCase#testRankOnOver` covers it. With ties the result is wrong though, that is FLINK-41099. Only Top-N shapes fail. ########## docs/data/functions/cast/cast.yml: ########## @@ -0,0 +1,109 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: CAST +summary: Converts a value to another data type, and fails the job if the value doesn't convert. + +syntax: + sql: CAST(value AS type) + java: | + $("value").cast(DataTypes.INT()) + $("value").cast(DataTypes.of("INT")) + python: col("value").cast(DataTypes.INT()) + +arguments: + - name: value + type: Any type + required: true + description: The value to convert. + - name: type + type: A data type + required: true + description: The target type, such as `INT` or `DECIMAL(10, 2)`. The [casting table]({{< ref "docs/sql/reference/data-types" >}}#casting) lists the types that each type can be cast to. + +returns: + description: "`value` converted to `type`." + nulls: Returns NULL if `value` is NULL. + type: "`type`. NOT NULL if `value` is NOT NULL, except for a cast from `VARIANT` to another type, which is always nullable." + +notes: + - A type pair that the casting table doesn't allow fails the statement at planning time, such as `BOOLEAN` to `DATE`. + - A value that doesn't convert, such as `'abc'` to `INT`, fails the job at runtime. To get NULL instead, use `TRY_CAST`. + - Casting to `CHAR(n)` cuts the result to `n` characters, whatever the source type, and pads a shorter one with spaces. `CAST(12345 AS CHAR(3))` returns `123`. Casting to `VARCHAR(n)` only cuts. `BINARY(n)` cuts and pads with zero bytes, and `VARBINARY(n)` only cuts. + - Casting bytes to a string fails the job if the bytes aren't valid UTF-8. To get NULL instead, use `TRY_CAST`, or repair the bytes first with `MAKE_VALID_UTF8`. Setting `table.exec.legacy-bytes-to-string-cast` to `true` replaces each invalid sequence with U+FFFD in every query of the session. + - Casts to an integer type don't check the range. An integer or a `DECIMAL` that doesn't fit wraps around, so `CAST(300 AS TINYINT)` returns `44`. A fraction is truncated toward zero. Review Comment: This only holds for numeric sources, `CAST('300' AS TINYINT)` fails with an overflow. ########## docs/data/functions/cast/cast.yml: ########## @@ -0,0 +1,109 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: CAST +summary: Converts a value to another data type, and fails the job if the value doesn't convert. + +syntax: + sql: CAST(value AS type) + java: | + $("value").cast(DataTypes.INT()) + $("value").cast(DataTypes.of("INT")) + python: col("value").cast(DataTypes.INT()) + +arguments: + - name: value + type: Any type + required: true + description: The value to convert. + - name: type + type: A data type + required: true + description: The target type, such as `INT` or `DECIMAL(10, 2)`. The [casting table]({{< ref "docs/sql/reference/data-types" >}}#casting) lists the types that each type can be cast to. + +returns: + description: "`value` converted to `type`." + nulls: Returns NULL if `value` is NULL. + type: "`type`. NOT NULL if `value` is NOT NULL, except for a cast from `VARIANT` to another type, which is always nullable." + +notes: + - A type pair that the casting table doesn't allow fails the statement at planning time, such as `BOOLEAN` to `DATE`. + - A value that doesn't convert, such as `'abc'` to `INT`, fails the job at runtime. To get NULL instead, use `TRY_CAST`. + - Casting to `CHAR(n)` cuts the result to `n` characters, whatever the source type, and pads a shorter one with spaces. `CAST(12345 AS CHAR(3))` returns `123`. Casting to `VARCHAR(n)` only cuts. `BINARY(n)` cuts and pads with zero bytes, and `VARBINARY(n)` only cuts. + - Casting bytes to a string fails the job if the bytes aren't valid UTF-8. To get NULL instead, use `TRY_CAST`, or repair the bytes first with `MAKE_VALID_UTF8`. Setting `table.exec.legacy-bytes-to-string-cast` to `true` replaces each invalid sequence with U+FFFD in every query of the session. + - Casts to an integer type don't check the range. An integer or a `DECIMAL` that doesn't fit wraps around, so `CAST(300 AS TINYINT)` returns `44`. A fraction is truncated toward zero. + - For a `DECIMAL` target, extra digits after the point are rounded half away from zero, so `CAST(-1.25 AS DECIMAL(2, 1))` returns `-1.3`. A number that still doesn't fit the precision returns NULL. Review Comment: With a NOT NULL input the job fails instead: `CAST(12345 AS DECIMAL(3, 1))` is typed NOT NULL, so the enforcer rejects the NULL. The nulls line also misses this case. ########## docs/data/functions/cast/try_cast.yml: ########## @@ -0,0 +1,110 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: TRY_CAST +summary: Converts a value to another data type, and returns NULL if the value doesn't convert. + +syntax: + sql: TRY_CAST(value AS type) + java: | + $("value").tryCast(DataTypes.INT()) + $("value").tryCast(DataTypes.of("INT")) + python: col("value").try_cast(DataTypes.INT()) + +arguments: + - name: value + type: Any type + required: true + description: The value to convert. + - name: type + type: A data type + required: true + description: The target type, such as `INT` or `DECIMAL(10, 2)`. The [casting table]({{< ref "docs/sql/reference/data-types" >}}#casting) lists the types that each type can be cast to. + +returns: + description: "`value` converted to `type`, like `CAST`." + nulls: Returns NULL if `value` is NULL, if `CAST` returns NULL for it, or if it doesn't convert, where `CAST` fails the job. + type: "`type`, always nullable." + +notes: + - A type pair that the casting table doesn't allow still fails the statement at planning time, as with `CAST`. + - "`TRY_CAST` catches only the errors of its own conversion. An error in `value`, such as a nested `CAST` that fails, still fails the job." + - If one element of an array, a map or a row doesn't convert, the whole value is NULL. + - Casts between numeric types don't fail, so `TRY_CAST` doesn't return NULL for them. `TRY_CAST(300 AS TINYINT)` wraps around to `44`, while `TRY_CAST('300' AS TINYINT)` returns NULL. Review Comment: `TRY_CAST(300 AS DECIMAL(2, 0))` returns NULL, so this doesn't hold for a `DECIMAL` target. ########## docs/data/functions/regular-expressions/regexp.yml: ########## @@ -0,0 +1,95 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: REGEXP +summary: Returns whether a string contains a match for a Java regular expression. + +syntax: + sql: REGEXP(str, regex) + java: $("str").regexp(regex) + python: col("str").regexp(regex) + +arguments: + - name: str + type: CHAR, VARCHAR, STRING + required: true + description: The string to search. + - name: regex + type: CHAR, VARCHAR, STRING + required: true + description: A Java regular expression. + +returns: + description: "`TRUE` if `regex` matches somewhere in `str`, `FALSE` if it doesn't." Review Comment: An empty `regex` returns FALSE, although it matches everywhere. The old row said so and `RegexpFunctionsITCase` asserts it, so I think the page needs it too. ########## docs/data/functions/utility/descriptor.yml: ########## @@ -0,0 +1,58 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License + +name: DESCRIPTOR +summary: Passes a list of column names to a table function. + +syntax: + sql: DESCRIPTOR(column1 [, column2]*) + java: descriptor("column1", "column2") Review Comment: The Python call is missing, `descriptor(*column_names)` exists in `pyflink/table/expressions.py`. ########## docs/README.md: ########## @@ -283,3 +285,49 @@ Renders a link to a file in the Apache Flink Python Documentation. ``` Renders a link to the apache flink download page. + +## Built-in function pages + +Pages for built-in SQL functions live under `content/docs/sql/built-in-functions/<category>/`. They aren't written in Markdown. A content adapter generates each page, in English and Chinese, from one YAML file in `data/functions/`. + +### Adding a function + +1. Add `data/functions/<category>/<name>.yml`. `<category>` is the directory of the category under `content/docs/sql/built-in-functions/`, such as `regular-expressions`. `<name>` is the function name in lower case, such as `regexp_extract`, and the page is served at `<category>/regexp-extract/`. Use `data/functions/regular-expressions/regexp_extract.yml` as the reference for the fields. +2. Add a row for the function to its category in `data/sql_functions.yml` and `data/sql_functions_zh.yml`. Keep the rows in alphabetical order of the function name, the order of the pages in the sidebar. Give it `sql`, `table` and `page: regexp-extract`, without a `description`. The table shows the function's name, links it to the page and shows the page's summary. +3. Build the docs as described above and check the page in both languages. A broken `ref` fails the build, locally and in CI. + +The Chinese page renders the same YAML, so the English text is its placeholder until someone translates it. If you aren't a native Chinese speaker, don't translate: add no Chinese file and leave the translation to native speakers. A translation goes in `data/functions_zh/<category>/<name>.yml`, and the fields it sets replace the English ones. Review Comment: This drops about 1,000 characters of existing Chinese text from the converted rows. Could you open a chinese-translation ticket and link it on FLINK-41433? ########## docs/layouts/partials/built-in-functions/pages.html: ########## @@ -0,0 +1,47 @@ +{{/* +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +*/}} +{{/* + Adds a page for every function in data/functions/<category>/<name>.yml. The content adapter + _content.gotmpl of each language calls this partial. For the zh pages, a file with the same + path under data/functions_zh/ replaces the fields it sets, so existing Chinese text is kept + and the other sections fall back to English. +*/}} +{{ $adapter := . }} +{{ $overrides := dict }} +{{ if eq site.Language.Lang "zh" }} + {{ $overrides = index site.Data "functions_zh" | default dict }} +{{ end }} +{{ range $category, $functions := site.Data.functions }} + {{ range $key, $function := $functions }} + {{ with index $overrides $category }} + {{ with index . $key }} + {{ $function = merge $function . }} + {{ end }} + {{ end }} + {{ $slug := replace $key "_" "-" }} + {{ $markdown := partial "built-in-functions/markdown.html" (dict "function" $function "slug" $slug) }} + {{ $adapter.AddPage (dict + "kind" "page" + "path" (printf "%s/%s" $category $slug) + "title" $function.name + "params" (dict "builtinfunction" true "summary" $function.summary) Review Comment: Edit This Page on a function page points to `_content.gotmpl`, since that is its `.File.Path`. Passing the YAML path as a param and using it in `footer.html` would fix that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
