gaoyunhaii commented on a change in pull request #14229:
URL: https://github.com/apache/flink/pull/14229#discussion_r533120403
##########
File path: docs/dev/table/connectors/formats/avro-confluent.md
##########
@@ -45,6 +45,8 @@ Dependencies
connector=connector
%}
+**NOTE:** If you need make an uber-jar from the project which used multiple
table formats, please see [shade table formats]({% link
dev/table/connectors/formats/index.md %}#shade-table-formats) for more
information.
Review comment:
We might change `make` -> `to build`, and `used` -> `uses`
##########
File path: docs/dev/table/connectors/formats/canal.md
##########
@@ -51,7 +51,9 @@ Dependencies
connector=connector
%}
-*Note: please refer to [Canal
documentation](https://github.com/alibaba/canal/wiki) about how to deploy Canal
to synchronize changelog to message queues.*
+
+**NOTE:** Please refer to [Canal
documentation](https://github.com/alibaba/canal/wiki) about how to deploy Canal
to synchronize changelog to message queues.
+**NOTE:** If you need make an uber-jar from the project which used multiple
table formats, please see [shade table formats]({% link
dev/table/connectors/formats/index.md %}#shade-table-formats) for more
information.
Review comment:
There should be one empty line between the two notes, otherwise the two
notes will be viewed as the same paragraph and will not have line break.
##########
File path: docs/dev/table/connectors/formats/index.md
##########
@@ -90,3 +90,61 @@ Flink supports the following formats:
</tr>
</tbody>
</table>
+
+
+Shade Table Formats
+-------------------
+ Flink uses Java's [Service Provider Interfaces
(SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) to load
the factory of table format by table format identifier, the SPI configuration
file `org.apache.flink.table.factories.Factory` for every table format is under
the same directory `META-INF/services`, these META-INF/services files will
overlap each other when building a jar-with-dependency of the project if your
project has used more than one table format, this will lead Flink can not load
these table format factories correctly.
+ In this situation, a recommended way is shading these table formats and using
maven-shade-plugin
[ServicesResourceTransformer](https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html)
to merge these META-INF/services files properly.
Review comment:
- `shading these table formats `: I think we should not need to shade
these table formats, to me shading should refers to the action like change the
package of `io.netty` to `org.apache.flink.shaded.io.netty`. We only use the
transformer to merge the configuration files.
- `these META-INF/services files`: we might use `the configuration files
under META-INF/services` to be more precise ?
##########
File path: docs/dev/table/connectors/formats/debezium.md
##########
@@ -59,7 +59,9 @@ Dependencies
</div>
</div>
-*Note: please refer to [Debezium
documentation](https://debezium.io/documentation/reference/1.3/index.html)
about how to setup a Debezium Kafka Connect to synchronize changelog to Kafka
topics.*
+**NOTE:** please refer to [Debezium
documentation](https://debezium.io/documentation/reference/1.3/index.html)
about how to setup a Debezium Kafka Connect to synchronize changelog to Kafka
topics.
+**NOTE:** If you need make an uber-jar from the project which used multiple
table formats, please see [shade table formats]({% link
dev/table/connectors/formats/index.md %}#shade-table-formats) for more
information.
Review comment:
Similarly, there should be one empty line between the two notes.
##########
File path: docs/dev/table/connectors/formats/debezium.zh.md
##########
@@ -59,7 +59,8 @@ Flink 还支持将 Flink SQL 中的 INSERT / UPDATE / DELETE 消息编码为 Deb
</div>
</div>
-*注意: 请参考 [Debezium
文档](https://debezium.io/documentation/reference/1.3/index.html)，了解如何设置 Debezium
Kafka Connect 用来将变更日志同步到 Kafka 主题。*
+**注意:** 请参考 [Debezium
文档](https://debezium.io/documentation/reference/1.3/index.html)，了解如何设置 Debezium
Kafka Connect 用来将变更日志同步到 Kafka 主题。
+**注意:** 如果你需要从一个包含多个表格式的工程创建 uber-jar, 请查看 [shade 表格式]({% link
dev/table/connectors/formats/index.zh.md %}#shade-表格式) 页面获取更多信息。
Review comment:
Similarly, there should be one empty line between the two notes.
##########
File path: docs/dev/table/connectors/formats/index.md
##########
@@ -90,3 +90,61 @@ Flink supports the following formats:
</tr>
</tbody>
</table>
+
+
+Shade Table Formats
+-------------------
+ Flink uses Java's [Service Provider Interfaces
(SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) to load
the factory of table format by table format identifier, the SPI configuration
file `org.apache.flink.table.factories.Factory` for every table format is under
the same directory `META-INF/services`, these META-INF/services files will
overlap each other when building a jar-with-dependency of the project if your
project has used more than one table format, this will lead Flink can not load
these table format factories correctly.
+ In this situation, a recommended way is shading these table formats and using
maven-shade-plugin
[ServicesResourceTransformer](https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html)
to merge these META-INF/services files properly.
+
+Given the pom.xml file content of example that contains `flink-avro` and
`flink-parquet` table formats in a project.
+
+{% highlight xml %}
+
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>org.example</groupId>
+ <artifactId>myProject</artifactId>
+ <version>1.0-SNAPSHOT</version>
+
+ <dependencies>
+ <!-- other project dependencies ...-->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-orc_2.11</artifactId>
+ <version>${flink-version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-parquet_2.11</artifactId>
+ <version>${flink-version}</version>
+ </dependency>
+
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>shade</id>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <transformers combine.children="append">
+ <!-- The service transformer is needed to
merge META-INF/services files -->
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
Review comment:
We might add one line of `<!-- ... -->` to show that we might add other
transformers
##########
File path: docs/dev/table/connectors/formats/index.md
##########
@@ -90,3 +90,61 @@ Flink supports the following formats:
</tr>
</tbody>
</table>
+
+
+Shade Table Formats
+-------------------
+ Flink uses Java's [Service Provider Interfaces
(SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) to load
the factory of table format by table format identifier, the SPI configuration
file `org.apache.flink.table.factories.Factory` for every table format is under
the same directory `META-INF/services`, these META-INF/services files will
overlap each other when building a jar-with-dependency of the project if your
project has used more than one table format, this will lead Flink can not load
these table format factories correctly.
+ In this situation, a recommended way is shading these table formats and using
maven-shade-plugin
[ServicesResourceTransformer](https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html)
to merge these META-INF/services files properly.
Review comment:
If we want to make this line in a separate paragraph there should be one
empty line between the two lines.
##########
File path: docs/dev/table/connectors/formats/canal.zh.md
##########
@@ -51,7 +51,8 @@ Flink 还支持将 Flink SQL 中的 INSERT / UPDATE / DELETE 消息编码为 Can
connector=connector
%}
-*注意：有关如何部署 Canal 以将变更日志同步到消息队列，请参阅 [Canal
文档](https://github.com/alibaba/canal/wiki)。*
+**注意:** 有关如何部署 Canal 以将变更日志同步到消息队列，请参阅 [Canal
文档](https://github.com/alibaba/canal/wiki)。
+**注意:** 如果你需要从一个包含多个表格式的工程创建 uber-jar, 请查看 [shade 表格式]({% link
dev/table/connectors/formats/index.zh.md %}#shade-表格式) 页面获取更多信息。
Review comment:
Similarly, there should be one empty line between the two notes.
##########
File path: docs/dev/table/connectors/formats/index.md
##########
@@ -90,3 +90,61 @@ Flink supports the following formats:
</tr>
</tbody>
</table>
+
+
+Shade Table Formats
+-------------------
+ Flink uses Java's [Service Provider Interfaces
(SPI)](https://docs.oracle.com/javase/tutorial/sound/SPI-intro.html) to load
the factory of table format by table format identifier, the SPI configuration
file `org.apache.flink.table.factories.Factory` for every table format is under
the same directory `META-INF/services`, these META-INF/services files will
overlap each other when building a jar-with-dependency of the project if your
project has used more than one table format, this will lead Flink can not load
these table format factories correctly.
Review comment:
How about we change
- `load the factory of table factory by table format identifier,` -> `load
the table format factories by their identifiers.`
- `the SPI configuration file ...` -> `Since the SPI configuration file
named...`
- `these META-INF/services files ` -> `these configuration files` ? I think
the files are the configuration files that get overrided
- `overlap` -> `override` ?
- `a jar-with-dependency` -> `the uber-jar` ? Since we use uber-jar in other
places and might better to unify the names.
- `this will... ` -> `which will... `
- `lead Flink can not` -> `cause Flink to fail to`, it seems lead must be
used as `lead to + nonu`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
