[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[ https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660300#comment-16660300 ] Timo Walther commented on FLINK-9737: - Fixed in 1.7.0: 6c9193ff288919a7ac8dfdb5773b92d84f843a4c, d54fde128acc0b6c53e4792fd0d6eeaa98549068, 13fd5d5c380a1a37127bd22171e75d3947cc90e3 > Allow to define Table Version Functions from SQL environment file > - > > Key: FLINK-9737 > URL: https://issues.apache.org/jira/browse/FLINK-9737 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Affects Versions: 1.5.0 >Reporter: Piotr Nowojski >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > > It should be possible to define TVF from SQL environment file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[ https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660293#comment-16660293 ] ASF GitHub Bot commented on FLINK-9737: --- twalthr commented on issue #6897: [FLINK-9737] [FLINK-8880] [sql-client] Support defining temporal tables in environment files URL: https://github.com/apache/flink/pull/6897#issuecomment-432158343 Thanks for the feedback @dawidwys and @pnowojski. I addressed your comments and will merge this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Allow to define Table Version Functions from SQL environment file > - > > Key: FLINK-9737 > URL: https://issues.apache.org/jira/browse/FLINK-9737 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Affects Versions: 1.5.0 >Reporter: Piotr Nowojski >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > > It should be possible to define TVF from SQL environment file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660232#comment-16660232
]
ASF GitHub Bot commented on FLINK-9737:
---
twalthr commented on a change in pull request #6897: [FLINK-9737] [FLINK-8880]
[sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r227270152
##
File path: docs/dev/table/sqlClient.md
##
@@ -503,7 +504,49 @@ Views created within a CLI session can also be removed
again using the `DROP VIE
DROP VIEW MyNewView;
{% endhighlight %}
-Attention The definition of views is
limited to the mentioned syntax above. Defining a schema for views or escape
whitespaces in table names will be supported in future versions.
+Attention The definition of views in
the CLI is limited to the mentioned syntax above. Defining a schema for views
or escaping whitespaces in table names will be supported in future versions.
+
+{% top %}
+
+Temporal Tables
+---
+
+A [temporal table](./streaming/temporal_tables.html) allows for a
(parameterized) view on a changing history table that returns the content of a
table at a specific point in time. This is especially useful for joining a
table with the content of another table at a particular timestamp. More
information can be found in the [temporal table
joins](./streaming/joins.html#join-with-a-temporal-table) page.
+
+The following example shows how to define a temporal table
`SourceTemporalTable`:
+
+{% highlight yaml %}
+tables:
+
+ # Define the table source (or view) that contains updates to a temporal table
+ - name: HistorySource
+type: source-table
+update-mode: append
+connector: # ...
+format: # ...
+schema:
+ - name: integerField
+type: INT
+ - name: stringField
+type: VARCHAR
+ - name: rowtimeField
+type: TIMESTAMP
+rowtime:
+ timestamps:
+type: from-field
+from: rowtimeField
+ watermarks:
+type: from-source
+
+ # Define a temporal table over the changing history table with time
attribute and primary key
+ - name: SourceTemporalTable
+type: temporal-table
+changing-table: HistorySource
Review comment:
In the documentation we use "changing history table". I'm fine with
`history-table`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It should be possible to define TVF from SQL environment file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660228#comment-16660228
]
ASF GitHub Bot commented on FLINK-9737:
---
pnowojski commented on a change in pull request #6897: [FLINK-9737]
[FLINK-8880] [sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r227267053
##
File path: docs/dev/table/sqlClient.md
##
@@ -503,7 +504,49 @@ Views created within a CLI session can also be removed
again using the `DROP VIE
DROP VIEW MyNewView;
{% endhighlight %}
-Attention The definition of views is
limited to the mentioned syntax above. Defining a schema for views or escape
whitespaces in table names will be supported in future versions.
+Attention The definition of views in
the CLI is limited to the mentioned syntax above. Defining a schema for views
or escaping whitespaces in table names will be supported in future versions.
+
+{% top %}
+
+Temporal Tables
+---
+
+A [temporal table](./streaming/temporal_tables.html) allows for a
(parameterized) view on a changing history table that returns the content of a
table at a specific point in time. This is especially useful for joining a
table with the content of another table at a particular timestamp. More
information can be found in the [temporal table
joins](./streaming/joins.html#join-with-a-temporal-table) page.
+
+The following example shows how to define a temporal table
`SourceTemporalTable`:
+
+{% highlight yaml %}
+tables:
+
+ # Define the table source (or view) that contains updates to a temporal table
+ - name: HistorySource
+type: source-table
+update-mode: append
+connector: # ...
+format: # ...
+schema:
+ - name: integerField
+type: INT
+ - name: stringField
+type: VARCHAR
+ - name: rowtimeField
+type: TIMESTAMP
+rowtime:
+ timestamps:
+type: from-field
+from: rowtimeField
+ watermarks:
+type: from-source
+
+ # Define a temporal table over the changing history table with time
attribute and primary key
+ - name: SourceTemporalTable
+type: temporal-table
+changing-table: HistorySource
Review comment:
`changing-table` is not very good name and it's inconsistent with already
used nomenclature, that even you are using here :) (`History***`). Maybe change
it to `underlying-table`, `underlying-history-table`, `history-table` or
something like that? If you think that there is some better name, please also
make the effort to rename existing `underlyingHistoryTable` occurrences in the
source code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It should be possible to define TVF from SQL environment file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660227#comment-16660227
]
ASF GitHub Bot commented on FLINK-9737:
---
pnowojski commented on a change in pull request #6897: [FLINK-9737]
[FLINK-8880] [sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r227268384
##
File path: docs/dev/table/sqlClient.md
##
@@ -503,7 +504,49 @@ Views created within a CLI session can also be removed
again using the `DROP VIE
DROP VIEW MyNewView;
{% endhighlight %}
-Attention The definition of views is
limited to the mentioned syntax above. Defining a schema for views or escape
whitespaces in table names will be supported in future versions.
+Attention The definition of views in
the CLI is limited to the mentioned syntax above. Defining a schema for views
or escaping whitespaces in table names will be supported in future versions.
+
+{% top %}
+
+Temporal Tables
+---
+
+A [temporal table](./streaming/temporal_tables.html) allows for a
(parameterized) view on a changing history table that returns the content of a
table at a specific point in time. This is especially useful for joining a
table with the content of another table at a particular timestamp. More
information can be found in the [temporal table
joins](./streaming/joins.html#join-with-a-temporal-table) page.
+
+The following example shows how to define a temporal table
`SourceTemporalTable`:
+
+{% highlight yaml %}
+tables:
+
+ # Define the table source (or view) that contains updates to a temporal table
+ - name: HistorySource
+type: source-table
+update-mode: append
+connector: # ...
+format: # ...
+schema:
+ - name: integerField
+type: INT
+ - name: stringField
+type: VARCHAR
+ - name: rowtimeField
+type: TIMESTAMP
+rowtime:
+ timestamps:
+type: from-field
+from: rowtimeField
+ watermarks:
+type: from-source
+
+ # Define a temporal table over the changing history table with time
attribute and primary key
+ - name: SourceTemporalTable
+type: temporal-table
+changing-table: HistorySource
+primary-key: integerField
+time-attribute: rowtimeField
Review comment:
change to:
```
time-attribute: rowtimeField # could also be a proctime field
```
?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It should be possible to define TVF from SQL environment file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660226#comment-16660226
]
ASF GitHub Bot commented on FLINK-9737:
---
pnowojski commented on a change in pull request #6897: [FLINK-9737]
[FLINK-8880] [sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r227267053
##
File path: docs/dev/table/sqlClient.md
##
@@ -503,7 +504,49 @@ Views created within a CLI session can also be removed
again using the `DROP VIE
DROP VIEW MyNewView;
{% endhighlight %}
-Attention The definition of views is
limited to the mentioned syntax above. Defining a schema for views or escape
whitespaces in table names will be supported in future versions.
+Attention The definition of views in
the CLI is limited to the mentioned syntax above. Defining a schema for views
or escaping whitespaces in table names will be supported in future versions.
+
+{% top %}
+
+Temporal Tables
+---
+
+A [temporal table](./streaming/temporal_tables.html) allows for a
(parameterized) view on a changing history table that returns the content of a
table at a specific point in time. This is especially useful for joining a
table with the content of another table at a particular timestamp. More
information can be found in the [temporal table
joins](./streaming/joins.html#join-with-a-temporal-table) page.
+
+The following example shows how to define a temporal table
`SourceTemporalTable`:
+
+{% highlight yaml %}
+tables:
+
+ # Define the table source (or view) that contains updates to a temporal table
+ - name: HistorySource
+type: source-table
+update-mode: append
+connector: # ...
+format: # ...
+schema:
+ - name: integerField
+type: INT
+ - name: stringField
+type: VARCHAR
+ - name: rowtimeField
+type: TIMESTAMP
+rowtime:
+ timestamps:
+type: from-field
+from: rowtimeField
+ watermarks:
+type: from-source
+
+ # Define a temporal table over the changing history table with time
attribute and primary key
+ - name: SourceTemporalTable
+type: temporal-table
+changing-table: HistorySource
Review comment:
`changing-table` is not very good name and it's inconsistent with already
used nomenclature, that even you are using here. Please change it to
`underlying-table`, `underlying-history-table`, `history-table` or something
like that. If you think that there is some better name, please also make the
effort to rename existing `underlyingHistoryTable` occurrences in the source
code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It should be possible to define TVF from SQL environment file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660201#comment-16660201
]
ASF GitHub Bot commented on FLINK-9737:
---
twalthr commented on a change in pull request #6897: [FLINK-9737] [FLINK-8880]
[sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r227255346
##
File path:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/DescriptorProperties.scala
##
@@ -185,6 +196,18 @@ class DescriptorProperties(normalizeKeys: Boolean = true)
{
putIndexedVariableProperties(key,
propertySets.asScala.map(_.asScala.toMap))
}
+ /**
+* Adds an array of properties under a common key.
+*
+* The key consumer receives the array key and value for inserting the
array element.
+*/
+ def putArray[E](key: String, values: JList[E], keyConsumer:
BiConsumer[String, E]): Unit = {
Review comment:
Yes, you are right. This method was also one of the methods where I thought
it would make sense to make the `put/get/validate` methods more modular. I'm
fine with removing these changes for now and open a follow-up issue. The
temporal tables don't support composite keys so far anyway.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It should be possible to define TVF from SQL environment file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660200#comment-16660200
]
ASF GitHub Bot commented on FLINK-9737:
---
twalthr commented on a change in pull request #6897: [FLINK-9737] [FLINK-8880]
[sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r227254032
##
File path: docs/dev/table/sqlClient.md
##
@@ -503,7 +504,49 @@ Views created within a CLI session can also be removed
again using the `DROP VIE
DROP VIEW MyNewView;
{% endhighlight %}
-Attention The definition of views is
limited to the mentioned syntax above. Defining a schema for views or escape
whitespaces in table names will be supported in future versions.
+Attention The definition of views in
the CLI is limited to the mentioned syntax above. Defining a schema for views
or escaping whitespaces in table names will be supported in future versions.
+
+{% top %}
+
+Temporal Tables
+---
+
+A [temporal table](./streaming/temporal_tables.html) allows for a
(parameterized) view on a changing history table that returns the content of a
table at a specific point in time. This is especially useful for joining a
table with the content of another table at a particular timestamp. More
information can be found in the [temporal table
joins](./streaming/joins.html#join-with-a-temporal-table) page.
+
+The following example shows how to define a temporal table
`SourceTemporalTable`:
+
+{% highlight yaml %}
+tables:
+
+ # Define the table source (or view) that contains updates to a temporal table
+ - name: HistorySource
+type: source-table
+update-mode: append
+connector: # ...
+format: # ...
+schema:
+ - name: integerField
+type: INT
+ - name: stringField
+type: VARCHAR
+ - name: rowtimeField
+type: TIMESTAMP
+rowtime:
+ timestamps:
+type: from-field
+from: rowtimeField
+ watermarks:
+type: from-source
+
+ # Define a temporal table over the changing history table with time
attribute and primary key
+ - name: SourceTemporalTable
+type: temporal-table
+changing-table: HistorySource
+primary-key: integerField
+time-attribute: rowtimeField
+{% endhighlight %}
+
+Similar to views, temporal tables are registered in the order in which they
are defined in the environment file.
Review comment:
I added the following lines:
"As shown in the example, definitions of table sources, views, and temporal
tables can be mixed with each other. They are registered in the order in which
they are defined in the environment file. For example, a temporal table can
reference a view which can depend on another view or table source."
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It should be possible to define TVF from SQL environment file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659497#comment-16659497
]
ASF GitHub Bot commented on FLINK-9737:
---
dawidwys commented on a change in pull request #6897: [FLINK-9737] [FLINK-8880]
[sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r227000607
##
File path:
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/config/Environment.java
##
@@ -42,99 +45,72 @@
*/
public class Environment {
- private Map tables;
+ public static final String EXECUTION = "execution";
Review comment:
nit: `EXECUTION` -> `EXECUTION_(ENTRY)_KEY`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It should be possible to define TVF from SQL environment file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659499#comment-16659499
]
ASF GitHub Bot commented on FLINK-9737:
---
dawidwys commented on a change in pull request #6897: [FLINK-9737] [FLINK-8880]
[sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r226986000
##
File path:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/descriptors/DescriptorProperties.scala
##
@@ -185,6 +196,18 @@ class DescriptorProperties(normalizeKeys: Boolean = true)
{
putIndexedVariableProperties(key,
propertySets.asScala.map(_.asScala.toMap))
}
+ /**
+* Adds an array of properties under a common key.
+*
+* The key consumer receives the array key and value for inserting the
array element.
+*/
+ def putArray[E](key: String, values: JList[E], keyConsumer:
BiConsumer[String, E]): Unit = {
Review comment:
Do we need this method? This method looks a bit strange as it does not use
any internal state of the `DescriptorProperties`. It may event not put anything
to the descriptor as the whole insertion logic is covered by the `keyConsumer`.
As it is used only in tests, could we move it there as a utility method?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It should be possible to define TVF from SQL environment file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659498#comment-16659498
]
ASF GitHub Bot commented on FLINK-9737:
---
dawidwys commented on a change in pull request #6897: [FLINK-9737] [FLINK-8880]
[sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r227005801
##
File path:
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/config/entries/TableEntry.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.client.config.entries;
+
+import org.apache.flink.table.client.SqlClientException;
+import org.apache.flink.table.client.config.ConfigUtil;
+import org.apache.flink.table.descriptors.DescriptorProperties;
+
+import java.util.Arrays;
+import java.util.Map;
+
+/**
+ * Describes a table-like configuration entry.
+ */
+public abstract class TableEntry extends ConfigEntry {
+
+ public static final String TABLES_NAME = "name";
+
+ private static final String TABLES_TYPE = "type";
+
+ @Deprecated
+ private static final String TABLES_TYPE_VALUE_SOURCE = "source";
+
+ @Deprecated
+ private static final String TABLES_TYPE_VALUE_SINK = "sink";
+
+ @Deprecated
+ private static final String TABLES_TYPE_VALUE_BOTH = "both";
+
+ private static final String TABLES_TYPE_VALUE_SOURCE_TABLE =
"source-table";
+
+ private static final String TABLES_TYPE_VALUE_SINK_TABLE = "sink-table";
+
+ private static final String TABLES_TYPE_VALUE_SOURCE_SINK_TABLE =
"source-sink-table";
+
+ private static final String TABLES_TYPE_VALUE_VIEW = "view";
+
+ private static final String TABLES_TYPE_VALUE_TEMPORAL_TABLE =
"temporal-table";
+
+ private final String name;
+
+ protected TableEntry(String name, DescriptorProperties properties) {
+ super(properties);
+ this.name = name;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public static TableEntry create(Map config) {
+ return create(ConfigUtil.normalizeYaml(config));
+ }
+
+ private static TableEntry create(DescriptorProperties properties) {
+ properties.validateString(TABLES_NAME, false, 1);
+ properties.validateEnumValues(
+ TABLES_TYPE,
+ false,
+ Arrays.asList(
+ TABLES_TYPE_VALUE_SOURCE,
+ TABLES_TYPE_VALUE_SOURCE_TABLE,
+ TABLES_TYPE_VALUE_SINK,
+ TABLES_TYPE_VALUE_SINK_TABLE,
+ TABLES_TYPE_VALUE_BOTH,
+ TABLES_TYPE_VALUE_SOURCE_SINK_TABLE,
+ TABLES_TYPE_VALUE_VIEW,
+ TABLES_TYPE_VALUE_TEMPORAL_TABLE));
+
+ final String name = properties.getString(TABLES_NAME);
+
+ final DescriptorProperties cleanedProperties =
+ properties.removeKeys(Arrays.asList(TABLES_NAME,
TABLES_TYPE));
Review comment:
How about rename the `removeKeys` method to e.g. `withoutKeys`? This is one
of few methods (the only one?) that does not mutate the internal state. I
think this should be somehow more obvious from the name itself.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components:
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659501#comment-16659501
]
ASF GitHub Bot commented on FLINK-9737:
---
dawidwys commented on a change in pull request #6897: [FLINK-9737] [FLINK-8880]
[sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r226996878
##
File path: docs/dev/table/sqlClient.md
##
@@ -503,7 +504,49 @@ Views created within a CLI session can also be removed
again using the `DROP VIE
DROP VIEW MyNewView;
{% endhighlight %}
-Attention The definition of views is
limited to the mentioned syntax above. Defining a schema for views or escape
whitespaces in table names will be supported in future versions.
+Attention The definition of views in
the CLI is limited to the mentioned syntax above. Defining a schema for views
or escaping whitespaces in table names will be supported in future versions.
+
+{% top %}
+
+Temporal Tables
+---
+
+A [temporal table](./streaming/temporal_tables.html) allows for a
(parameterized) view on a changing history table that returns the content of a
table at a specific point in time. This is especially useful for joining a
table with the content of another table at a particular timestamp. More
information can be found in the [temporal table
joins](./streaming/joins.html#join-with-a-temporal-table) page.
+
+The following example shows how to define a temporal table
`SourceTemporalTable`:
+
+{% highlight yaml %}
+tables:
+
+ # Define the table source (or view) that contains updates to a temporal table
+ - name: HistorySource
+type: source-table
+update-mode: append
+connector: # ...
+format: # ...
+schema:
+ - name: integerField
+type: INT
+ - name: stringField
+type: VARCHAR
+ - name: rowtimeField
+type: TIMESTAMP
+rowtime:
+ timestamps:
+type: from-field
+from: rowtimeField
+ watermarks:
+type: from-source
+
+ # Define a temporal table over the changing history table with time
attribute and primary key
+ - name: SourceTemporalTable
+type: temporal-table
+changing-table: HistorySource
+primary-key: integerField
+time-attribute: rowtimeField
+{% endhighlight %}
+
+Similar to views, temporal tables are registered in the order in which they
are defined in the environment file.
Review comment:
Should we somewhere mention that we can mix views/tables/temporal tables,
and the order is important there? So that it is clear we can have e.g. a chain
`view A -> temporal table B depends on A -> view C depends on B`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It should be possible to define TVF from SQL environment file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[
https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659500#comment-16659500
]
ASF GitHub Bot commented on FLINK-9737:
---
dawidwys commented on a change in pull request #6897: [FLINK-9737] [FLINK-8880]
[sql-client] Support defining temporal tables in environment files
URL: https://github.com/apache/flink/pull/6897#discussion_r227000688
##
File path:
flink-libraries/flink-sql-client/src/main/java/org/apache/flink/table/client/config/Environment.java
##
@@ -42,99 +45,72 @@
*/
public class Environment {
- private Map tables;
+ public static final String EXECUTION = "execution";
- private Map views;
+ public static final String DEPLOYMENT = "deployment";
Review comment:
nit: `DEPLOYMENT` -> `DEPLOYMENT_(ENTRY)_KEY`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow to define Table Version Functions from SQL environment file
> -
>
> Key: FLINK-9737
> URL: https://issues.apache.org/jira/browse/FLINK-9737
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.5.0
>Reporter: Piotr Nowojski
>Assignee: Timo Walther
>Priority: Major
> Labels: pull-request-available
>
> It should be possible to define TVF from SQL environment file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[ https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659352#comment-16659352 ] ASF GitHub Bot commented on FLINK-9737: --- twalthr opened a new pull request #6897: [FLINK-9737] [FLINK-8880] [sql-client] Support defining temporal tables in environment files URL: https://github.com/apache/flink/pull/6897 ## What is the purpose of the change This PR introduces the definition of temporal tables also to the SQL Client. Similar to table views, temporal table definitions need to reference a different table source or table view. Thus, the definition order is important which is why this code refactors the parsing of environment files. The recently introduced top-level section `views` has been integrated into `tables`; temporal tables are defined there as well. As a side effect of the refactoring, properties are also validated for data type and value bounds now. This PR depends on #6893. ## Brief change log - Additional auxiliary methods for property handling - Refactoring for similar treatment of table sources, sinks, source-sinks, views, and temporal tables - Introduction of a basic validation for all configuration entries ## Verifying this change - `org.apache.flink.table.client.gateway.local.ExecutionContextTest` has been extended - Existing tests have been slightly refactored ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Allow to define Table Version Functions from SQL environment file > - > > Key: FLINK-9737 > URL: https://issues.apache.org/jira/browse/FLINK-9737 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Affects Versions: 1.5.0 >Reporter: Piotr Nowojski >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > > It should be possible to define TVF from SQL environment file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9737) Allow to define Table Version Functions from SQL environment file
[ https://issues.apache.org/jira/browse/FLINK-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532437#comment-16532437 ] Fabian Hueske commented on FLINK-9737: -- I think the syntax should be coordinated with FLINK-8863. > Allow to define Table Version Functions from SQL environment file > - > > Key: FLINK-9737 > URL: https://issues.apache.org/jira/browse/FLINK-9737 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Affects Versions: 1.5.0 >Reporter: Piotr Nowojski >Priority: Major > > It should be possible to define TVF from SQL environment file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
