yunqingmoswu commented on code in PR #417: URL: https://github.com/apache/incubator-inlong-website/pull/417#discussion_r898642591
########## docs/modules/sort/datastream_example.md: ########## @@ -6,1716 +6,149 @@ sidebar_position: 3 # Examples To make it easier for you to create InLong-Sort jobs, here we list some data stream configuration examples. +The following will introduce SQL, Dashboard, Manager Client Tools methods to use Inlong Sort. -## MySQL to Kafka +# Precondition Review Comment: General description of the precondition required. ########## docs/modules/sort/datastream_example.md: ########## @@ -6,1716 +6,149 @@ sidebar_position: 3 # Examples To make it easier for you to create InLong-Sort jobs, here we list some data stream configuration examples. +The following will introduce SQL, Dashboard, Manager Client Tools methods to use Inlong Sort. -## MySQL to Kafka +# Precondition -- Single table or sub-database sub-table sync example: +Step1. You can build a Flink Standalone single cluster for use. [Flink Standalone Mode](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/) -```json -{ - "groupId":"1", - "streams":[ - { - "streamId":"1", - "nodes":[ - { - "type":"mysqlExtract", - "id":"1", - "name":"mysql_input", - "fields":[ - { - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - { - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - { - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - ], - "properties":{ - "append-mode":"true" - }, - "primaryKey":"id", - "tableNames":[ - "YOUR_TABLE" - ], - "hostname":"YOUR_MYSQL_HOST", - "username":"YOUR_USERNAME", - "password":"YOUR_PASSWORD", - "database":"YOUR_DATABASE", - "port":3306, - "incrementalSnapshotEnabled":true - }, - { - "type":"kafkaLoad", - "id":"2", - "name":"kafka_output", - "fields":[ - { - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - { - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - { - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - ], - "fieldRelationShips":[ - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - "outputField":{ - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - } - }, - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - "outputField":{ - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - } - }, - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - }, - "outputField":{ - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - } - ], - "topic":"YOUR_TOPIC", - "bootstrapServers":"YOUR_KAFKA_BOOTSTRAP_SERVERS", - "format":{ - "type":"jsonFormat", - "failOnMissingField":false, - "ignoreParseErrors":true, - "timestampFormatStandard":"SQL", - "mapNullKeyMode":"DROP", - "mapNullKeyLiteral":"null", - "encodeDecimalAsPlainNumber":true - }, - "primaryKey":"id" - } - ], - "relations":[ - { - "type":"baseRelation", - "inputs":[ - "1" - ], - "outputs":[ - "2" - ] - } - ] - } - ] -} -``` - -- Whole-database migration example： +Step2. Install MySQL and turn on binlog. [MySQL Installation Guide](https://dev.mysql.com/doc/mysql-installation-excerpt/5.7/en/) and [MySQL binlog](https://dev.mysql.com/doc/refman/5.7/en/replication-howto-masterbaseconfig.html) Review Comment: It is recommended to change to "Prepare a MySQL environment and enable binlog for it. Please refer to [MySQL Installation Guide](https://dev.mysql.com/doc/mysql-installation-excerpt/5.7/en/) if there is no MySQL environment. Please refer to [MySQL Extract Load docs](xxx) about enable binlog for MySQL if the binlog is not enabled, of course it is also gotten directly refer to [MySQL binlog](https://dev.mysql.com/doc/refman/5.7/en/replication-howto-masterbaseconfig.html)" ########## docs/modules/sort/datastream_example.md: ########## @@ -6,1716 +6,149 @@ sidebar_position: 3 # Examples To make it easier for you to create InLong-Sort jobs, here we list some data stream configuration examples. +The following will introduce SQL, Dashboard, Manager Client Tools methods to use Inlong Sort. -## MySQL to Kafka +# Precondition -- Single table or sub-database sub-table sync example: +Step1. You can build a Flink Standalone single cluster for use. [Flink Standalone Mode](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/) -```json -{ - "groupId":"1", - "streams":[ - { - "streamId":"1", - "nodes":[ - { - "type":"mysqlExtract", - "id":"1", - "name":"mysql_input", - "fields":[ - { - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - { - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - { - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - ], - "properties":{ - "append-mode":"true" - }, - "primaryKey":"id", - "tableNames":[ - "YOUR_TABLE" - ], - "hostname":"YOUR_MYSQL_HOST", - "username":"YOUR_USERNAME", - "password":"YOUR_PASSWORD", - "database":"YOUR_DATABASE", - "port":3306, - "incrementalSnapshotEnabled":true - }, - { - "type":"kafkaLoad", - "id":"2", - "name":"kafka_output", - "fields":[ - { - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - { - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - { - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - ], - "fieldRelationShips":[ - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - "outputField":{ - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - } - }, - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - "outputField":{ - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - } - }, - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - }, - "outputField":{ - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - } - ], - "topic":"YOUR_TOPIC", - "bootstrapServers":"YOUR_KAFKA_BOOTSTRAP_SERVERS", - "format":{ - "type":"jsonFormat", - "failOnMissingField":false, - "ignoreParseErrors":true, - "timestampFormatStandard":"SQL", - "mapNullKeyMode":"DROP", - "mapNullKeyLiteral":"null", - "encodeDecimalAsPlainNumber":true - }, - "primaryKey":"id" - } - ], - "relations":[ - { - "type":"baseRelation", - "inputs":[ - "1" - ], - "outputs":[ - "2" - ] - } - ] - } - ] -} -``` - -- Whole-database migration example： +Step2. Install MySQL and turn on binlog. [MySQL Installation Guide](https://dev.mysql.com/doc/mysql-installation-excerpt/5.7/en/) and [MySQL binlog](https://dev.mysql.com/doc/refman/5.7/en/replication-howto-masterbaseconfig.html) +(You can also directly refer to Inlong MySQL Extract Node doc about MySQL turn on binlog information.) -```json -{ - "groupId":"1", - "streams":[ - { - "streamId":"1", - "nodes":[ - { - "type":"mysqlExtract", - "id":"1", - "name":"mysql_input", - "fields":[ - { - "type":"builtin", - "name":"data", - "formatInfo":{ - "type":"string" - }, - "builtinField":"MYSQL_METADATA_DATA" - } - ], - "properties":{ - "append-mode":"true", - "migrate-all":"true" - }, - "tableNames":[ - "[\\s\\S]*.*" - ], - "hostname":"YOUR_MYSQL_HOST", - "username":"YOUR_USERNAME", - "password":"YOUR_PASSWORD", - "database":"[\\s\\S]*.*", - "port":3306, - "incrementalSnapshotEnabled":false - }, - { - "type":"kafkaLoad", - "id":"2", - "name":"kafka_output", - "fields":[ - { - "type":"base", - "name":"data", - "formatInfo":{ - "type":"string" - } - } - ], - "fieldRelationShips":[ - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"data", - "formatInfo":{ - "type":"string" - } - }, - "outputField":{ - "type":"base", - "name":"data", - "formatInfo":{ - "type":"string" - } - } - } - ], - "topic":"YOUR_TOPIC", - "bootstrapServers":"YOUR_KAFKA_BOOTSTRAP_SERVERS", - "format":{ - "type":"csvFormat", - "fieldDelimiter":",", - "disableQuoteCharacter":true, - "quoteCharacter":null, - "allowComments":false, - "ignoreParseErrors":true, - "arrayElementDelimiter":";", - "escapeCharacter":null, - "nullLiteral":null - } - } - ], - "relations":[ - { - "type":"baseRelation", - "inputs":[ - "1" - ], - "outputs":[ - "2" - ] - } - ] - } - ] -} -``` +Step3. Install Kafka and Hadoop. [Kafka Installation Guide](https://kafka.apache.org/quickstart) and [Hadoop Installation Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html) -## Kafka to Hive +Step4. Install Hive and turn on metastore service. [Hive Installation Guide](https://cwiki.apache.org/confluence/display/Hive//GettingStarted) Review Comment: It is recommended to change to "Prepare a Hive environment and enable metastore service. Please refer to [Hive Installation Guide](https://cwiki.apache.org/confluence/display/Hive//GettingStarted) if there is no Hive environment." ########## docs/modules/sort/datastream_example.md: ########## @@ -6,1716 +6,149 @@ sidebar_position: 3 # Examples To make it easier for you to create InLong-Sort jobs, here we list some data stream configuration examples. +The following will introduce SQL, Dashboard, Manager Client Tools methods to use Inlong Sort. -## MySQL to Kafka Review Comment: It is recommended to add the general description of MySQL -> Kafka and Kafka -> Hive. ########## docs/modules/sort/datastream_example.md: ########## @@ -6,1716 +6,149 @@ sidebar_position: 3 # Examples To make it easier for you to create InLong-Sort jobs, here we list some data stream configuration examples. +The following will introduce SQL, Dashboard, Manager Client Tools methods to use Inlong Sort. -## MySQL to Kafka +# Precondition -- Single table or sub-database sub-table sync example: +Step1. You can build a Flink Standalone single cluster for use. [Flink Standalone Mode](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/) -```json -{ - "groupId":"1", - "streams":[ - { - "streamId":"1", - "nodes":[ - { - "type":"mysqlExtract", - "id":"1", - "name":"mysql_input", - "fields":[ - { - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - { - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - { - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - ], - "properties":{ - "append-mode":"true" - }, - "primaryKey":"id", - "tableNames":[ - "YOUR_TABLE" - ], - "hostname":"YOUR_MYSQL_HOST", - "username":"YOUR_USERNAME", - "password":"YOUR_PASSWORD", - "database":"YOUR_DATABASE", - "port":3306, - "incrementalSnapshotEnabled":true - }, - { - "type":"kafkaLoad", - "id":"2", - "name":"kafka_output", - "fields":[ - { - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - { - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - { - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - ], - "fieldRelationShips":[ - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - "outputField":{ - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - } - }, - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - "outputField":{ - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - } - }, - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - }, - "outputField":{ - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - } - ], - "topic":"YOUR_TOPIC", - "bootstrapServers":"YOUR_KAFKA_BOOTSTRAP_SERVERS", - "format":{ - "type":"jsonFormat", - "failOnMissingField":false, - "ignoreParseErrors":true, - "timestampFormatStandard":"SQL", - "mapNullKeyMode":"DROP", - "mapNullKeyLiteral":"null", - "encodeDecimalAsPlainNumber":true - }, - "primaryKey":"id" - } - ], - "relations":[ - { - "type":"baseRelation", - "inputs":[ - "1" - ], - "outputs":[ - "2" - ] - } - ] - } - ] -} -``` - -- Whole-database migration example： +Step2. Install MySQL and turn on binlog. [MySQL Installation Guide](https://dev.mysql.com/doc/mysql-installation-excerpt/5.7/en/) and [MySQL binlog](https://dev.mysql.com/doc/refman/5.7/en/replication-howto-masterbaseconfig.html) +(You can also directly refer to Inlong MySQL Extract Node doc about MySQL turn on binlog information.) -```json -{ - "groupId":"1", - "streams":[ - { - "streamId":"1", - "nodes":[ - { - "type":"mysqlExtract", - "id":"1", - "name":"mysql_input", - "fields":[ - { - "type":"builtin", - "name":"data", - "formatInfo":{ - "type":"string" - }, - "builtinField":"MYSQL_METADATA_DATA" - } - ], - "properties":{ - "append-mode":"true", - "migrate-all":"true" - }, - "tableNames":[ - "[\\s\\S]*.*" - ], - "hostname":"YOUR_MYSQL_HOST", - "username":"YOUR_USERNAME", - "password":"YOUR_PASSWORD", - "database":"[\\s\\S]*.*", - "port":3306, - "incrementalSnapshotEnabled":false - }, - { - "type":"kafkaLoad", - "id":"2", - "name":"kafka_output", - "fields":[ - { - "type":"base", - "name":"data", - "formatInfo":{ - "type":"string" - } - } - ], - "fieldRelationShips":[ - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"data", - "formatInfo":{ - "type":"string" - } - }, - "outputField":{ - "type":"base", - "name":"data", - "formatInfo":{ - "type":"string" - } - } - } - ], - "topic":"YOUR_TOPIC", - "bootstrapServers":"YOUR_KAFKA_BOOTSTRAP_SERVERS", - "format":{ - "type":"csvFormat", - "fieldDelimiter":",", - "disableQuoteCharacter":true, - "quoteCharacter":null, - "allowComments":false, - "ignoreParseErrors":true, - "arrayElementDelimiter":";", - "escapeCharacter":null, - "nullLiteral":null - } - } - ], - "relations":[ - { - "type":"baseRelation", - "inputs":[ - "1" - ], - "outputs":[ - "2" - ] - } - ] - } - ] -} -``` +Step3. Install Kafka and Hadoop. [Kafka Installation Guide](https://kafka.apache.org/quickstart) and [Hadoop Installation Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html) Review Comment: It is recommended to change to "Prepare a Kafka and Hadoop environment. Please refer to[Kafka Installation Guide](https://kafka.apache.org/quickstart) or [Hadoop Installation Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html) if there is no Kafka environment or Hadoop environment." ########## docs/modules/sort/datastream_example.md: ########## @@ -6,1716 +6,149 @@ sidebar_position: 3 # Examples To make it easier for you to create InLong-Sort jobs, here we list some data stream configuration examples. +The following will introduce SQL, Dashboard, Manager Client Tools methods to use Inlong Sort. -## MySQL to Kafka +# Precondition -- Single table or sub-database sub-table sync example: +Step1. You can build a Flink Standalone single cluster for use. [Flink Standalone Mode](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/) -```json -{ - "groupId":"1", - "streams":[ - { - "streamId":"1", - "nodes":[ - { - "type":"mysqlExtract", - "id":"1", - "name":"mysql_input", - "fields":[ - { - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - { - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - { - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - ], - "properties":{ - "append-mode":"true" - }, - "primaryKey":"id", - "tableNames":[ - "YOUR_TABLE" - ], - "hostname":"YOUR_MYSQL_HOST", - "username":"YOUR_USERNAME", - "password":"YOUR_PASSWORD", - "database":"YOUR_DATABASE", - "port":3306, - "incrementalSnapshotEnabled":true - }, - { - "type":"kafkaLoad", - "id":"2", - "name":"kafka_output", - "fields":[ - { - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - { - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - { - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - ], - "fieldRelationShips":[ - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - "outputField":{ - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - } - }, - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - "outputField":{ - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - } - }, - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - }, - "outputField":{ - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - } - ], - "topic":"YOUR_TOPIC", - "bootstrapServers":"YOUR_KAFKA_BOOTSTRAP_SERVERS", - "format":{ - "type":"jsonFormat", - "failOnMissingField":false, - "ignoreParseErrors":true, - "timestampFormatStandard":"SQL", - "mapNullKeyMode":"DROP", - "mapNullKeyLiteral":"null", - "encodeDecimalAsPlainNumber":true - }, - "primaryKey":"id" - } - ], - "relations":[ - { - "type":"baseRelation", - "inputs":[ - "1" - ], - "outputs":[ - "2" - ] - } - ] - } - ] -} -``` - -- Whole-database migration example： +Step2. Install MySQL and turn on binlog. [MySQL Installation Guide](https://dev.mysql.com/doc/mysql-installation-excerpt/5.7/en/) and [MySQL binlog](https://dev.mysql.com/doc/refman/5.7/en/replication-howto-masterbaseconfig.html) +(You can also directly refer to Inlong MySQL Extract Node doc about MySQL turn on binlog information.) -```json -{ - "groupId":"1", - "streams":[ - { - "streamId":"1", - "nodes":[ - { - "type":"mysqlExtract", - "id":"1", - "name":"mysql_input", - "fields":[ - { - "type":"builtin", - "name":"data", - "formatInfo":{ - "type":"string" - }, - "builtinField":"MYSQL_METADATA_DATA" - } - ], - "properties":{ - "append-mode":"true", - "migrate-all":"true" - }, - "tableNames":[ - "[\\s\\S]*.*" - ], - "hostname":"YOUR_MYSQL_HOST", - "username":"YOUR_USERNAME", - "password":"YOUR_PASSWORD", - "database":"[\\s\\S]*.*", - "port":3306, - "incrementalSnapshotEnabled":false - }, - { - "type":"kafkaLoad", - "id":"2", - "name":"kafka_output", - "fields":[ - { - "type":"base", - "name":"data", - "formatInfo":{ - "type":"string" - } - } - ], - "fieldRelationShips":[ - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"data", - "formatInfo":{ - "type":"string" - } - }, - "outputField":{ - "type":"base", - "name":"data", - "formatInfo":{ - "type":"string" - } - } - } - ], - "topic":"YOUR_TOPIC", - "bootstrapServers":"YOUR_KAFKA_BOOTSTRAP_SERVERS", - "format":{ - "type":"csvFormat", - "fieldDelimiter":",", - "disableQuoteCharacter":true, - "quoteCharacter":null, - "allowComments":false, - "ignoreParseErrors":true, - "arrayElementDelimiter":";", - "escapeCharacter":null, - "nullLiteral":null - } - } - ], - "relations":[ - { - "type":"baseRelation", - "inputs":[ - "1" - ], - "outputs":[ - "2" - ] - } - ] - } - ] -} -``` +Step3. Install Kafka and Hadoop. [Kafka Installation Guide](https://kafka.apache.org/quickstart) and [Hadoop Installation Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html) -## Kafka to Hive +Step4. Install Hive and turn on metastore service. [Hive Installation Guide](https://cwiki.apache.org/confluence/display/Hive//GettingStarted) -```json -{ - "groupId":"1", - "streams":[ - { - "streamId":"1", - "nodes":[ - { - "type":"kafkaExtract", - "id":"1", - "name":"kafka_input", - "fields":[ - { - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - { - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - { - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - ], - "topic":"YOUR_TOPIC", - "bootstrapServers":"YOUR_KAFKA_BOOTSTRAP_SERVERS", - "format":{ - "type":"jsonFormat", - "failOnMissingField":false, - "ignoreParseErrors":true, - "timestampFormatStandard":"SQL", - "mapNullKeyMode":"DROP", - "mapNullKeyLiteral":"null", - "encodeDecimalAsPlainNumber":true - }, - "scanStartupMode":"EARLIEST_OFFSET" - }, - { - "type":"hiveLoad", - "id":"2", - "name":"hive_output", - "fields":[ - { - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - { - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - { - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - ], - "fieldRelationShips":[ - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - }, - "outputField":{ - "type":"base", - "name":"id", - "formatInfo":{ - "type":"long" - } - } - }, - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - }, - "outputField":{ - "type":"base", - "name":"name", - "formatInfo":{ - "type":"string" - } - } - }, - { - "type":"fieldRelationShip", - "inputField":{ - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - }, - "outputField":{ - "type":"base", - "name":"age", - "formatInfo":{ - "type":"int" - } - } - } - ], - "sinkParallelism":1, - "catalogName":"hivecatlog", - "database":"YOUR_DATABASE", - "tableName":"YOUR_TABLE_NAME", - "hiveConfDir":"YOUR_HIVE_CONF_DIR", - "hiveVersion":"3.1.2", - "hadoopConfDir":"YOUR_HADOOP_CONF_DIR" - } - ], - "relations":[ - { - "type":"baseRelation", - "inputs":[ - "1" - ], - "outputs":[ - "2" - ] - } - ] - } - ] -} -``` +Step5. Download Inlong installation package [inlong-distribution-(version)-incubating-bin.tar.gz](https://inlong.apache.org/download/main) +and choose MySQL,Kafka,Hive connector dependencies in the [inlong-distribution-(version)-incubating-sort-connectors.tar.gz](https://inlong.apache.org/download/main) -## Transform examples +Step6. Put sort-dist-[version].jar and MySQL,Kafka,Hive connector jars into FLINK_HOME/lib . -Currently only supports string split, string regex replace, string regex replace first matched value, data distinct,data filter, regular join, and etc. -The following takes kafka to kafka as an example to illustrate the usage of each transform. +Note: sort-dist-[version].jar in the inlong-sort package. [inlong-distribution-(version)-incubating-bin.tar.gz](https://inlong.apache.org/download/main) -- String Split: +# Usage for SQL Review Comment: Usage for SQL -> Usage for SQL API ########## docs/modules/sort/datastream_example.md: ########## @@ -6,1716 +6,149 @@ sidebar_position: 3 # Examples To make it easier for you to create InLong-Sort jobs, here we list some data stream configuration examples. +The following will introduce SQL, Dashboard, Manager Client Tools methods to use Inlong Sort. -## MySQL to Kafka +# Precondition -- Single table or sub-database sub-table sync example: +Step1. You can build a Flink Standalone single cluster for use. [Flink Standalone Mode](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/) Review Comment: It is recommended to change to "Prepare a Flink environment. Please refer to [Flink Standalone Mode](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/), if there is no Flink environment." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
