Re: [PR] [HUDI-6330][DOCS] Update user doc to show how to use consistent bucket index for Flink engine [hudi]
xushiyan merged PR #10977: URL: https://github.com/apache/hudi/pull/10977 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6330][DOCS] Update user doc to show how to use consistent bucket index for Flink engine [hudi]
beyond1920 commented on code in PR #10977:
URL: https://github.com/apache/hudi/pull/10977#discussion_r1555701329
##
website/docs/sql_dml.md:
##
@@ -390,3 +390,70 @@ and `clean.async.enabled` options are used to disable the
compaction and cleanin
This is done to ensure that the compaction and cleaning services are not
executed twice for the same table.
+### Consistent hashing index (Experimental)
+
+We have introduced the Consistent Hashing Index since [0.13.0
release](/releases/release-0.13.0#consistent-hashing-index). In comparison to
the static hashing index ([Bucket
Index](/releases/release-0.11.0#bucket-index)), the consistent hashing index
offers dynamic scalability of data buckets for the writer.
+You can find the
[RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md) for the
design of this feature.
+In the 0.13.X release, the Consistent Hashing Index is supported only for
Spark engine. And since [release
0.14.0](/releases/release-0.14.0#consistent-hashing-index-support), the index
is supported for Flink engine.
+
+In the below example, we have a streaming ingestion pipeline that written to
the table with consistent bucket index.
+To utilize this feature, configure the option `index.type` as `BUCKET` and set
`hoodie.index.bucket.engine` to `CONSISTENT_HASHING`.
+When enabling the consistent hashing index, it's important to enable
clustering scheduling within the writer. During this process, the writer will
perform dual writes for both the old and new data buckets while the clustering
is pending. Although the dual write does not impact correctness, it is strongly
recommended to execute clustering as quickly as possible.
+
+```sql
+-- set the interval as 30 seconds
+execution.checkpointing.interval: 3
+state.backend: rocksdb
+
+-- This is a datagen source that can generates records continuously
+CREATE TABLE sourceT (
+uuid varchar(20),
+name varchar(10),
+age int,
+ts timestamp(3),
+`partition` as 'par1'
+) WITH (
+'connector' = 'datagen',
+'rows-per-second' = '200'
+);
+
+-- Create the hudi table with consistent bucket index
+CREATE TABLE t1(
+uuid VARCHAR(20) PRIMARY KEY NOT ENFORCED,
+name VARCHAR(10),
+age INT,
+ts TIMESTAMP(3),
+`partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+'connector'='hudi',
+'path' = '${work_path}/hudi-demo/hudiT',
+'table.type' = 'MERGE_ON_READ',
+'index.type' = 'BUCKET',
+'clustering.schedule.enabled'='true',
+'hoodie.index.bucket.engine'='CONSISTENT_HASHING',
+
'hoodie.clustering.plan.strategy.class'='org.apache.hudi.client.clustering.plan.strategy.FlinkConsistentBucketClusteringPlanStrategy',
+
'hoodie.clustering.execution.strategy.class'='org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy',
+'hoodie.bucket.index.num.buckets'='8',
+'hoodie.bucket.index.max.num.buckets'='128',
+'hoodie.bucket.index.min.num.buckets'='8',
+'hoodie.bucket.index.split.threshold'='1.5',
+'write.tasks'='2'
+);
+
+-- submit the pipelines
+insert into t1 select * from sourceT;
+
+select * from t1 limit 20;
+```
+
+:::caution
+Consistent Hashing Index is supported for Flink engine since [release
0.14.0](/releases/release-0.14.0#consistent-hashing-index-support) and
currently there are some limitations to use it as of 0.14.0:
+
+- This index is supported only for MOR table. This limitation also exists even
if using Spark engine.
+- It does not work with metadata table enabled. This limitation also exists
even if using Spark engine.
+- Consistent hashing index does not work with bulk-insert using Flink engine
yet, please use simple bucket index or Spark engine for bulk-insert pipelines.
+- The resize plan which generated by Flink engine does not support merge small
file groups yet, but only support split big file group.
+- The resize plan should be executed through an offline Spark job.
Review Comment:
Flink engine does not support execute resize plan yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-6330][DOCS] Update user doc to show how to use consistent bucket index for Flink engine [hudi]
beyond1920 commented on code in PR #10977: URL: https://github.com/apache/hudi/pull/10977#discussion_r1555696442 ## website/docs/sql_dml.md: ## @@ -390,3 +390,70 @@ and `clean.async.enabled` options are used to disable the compaction and cleanin This is done to ensure that the compaction and cleaning services are not executed twice for the same table. +### Consistent hashing index (Experimental) + +We have introduced the Consistent Hashing Index since [0.13.0 release](/releases/release-0.13.0#consistent-hashing-index). In comparison to the static hashing index ([Bucket Index](/releases/release-0.11.0#bucket-index)), the consistent hashing index offers dynamic scalability of data buckets for the writer. +You can find the [RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md) for the design of this feature. +In the 0.13.X release, the Consistent Hashing Index is supported only for Spark engine. And since [release 0.14.0](/releases/release-0.14.0#consistent-hashing-index-support), the index is supported for Flink engine. + +In the below example, we have a streaming ingestion pipeline that written to the table with consistent bucket index. +To utilize this feature, configure the option `index.type` as `BUCKET` and set `hoodie.index.bucket.engine` to `CONSISTENT_HASHING`. +When enabling the consistent hashing index, it's important to enable clustering scheduling within the writer. During this process, the writer will perform dual writes for both the old and new data buckets while the clustering is pending. Although the dual write does not impact correctness, it is strongly recommended to execute clustering as quickly as possible. + +```sql +-- set the interval as 30 seconds +execution.checkpointing.interval: 3 +state.backend: rocksdb + +-- This is a datagen source that can generates records continuously Review Comment: I prefer to add the source table schema here in order to keep the demo complete. Then users could conveniently copy the complete demo and run it in SqlClient. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6330][DOCS] Update user doc to show how to use consistent bucket index for Flink engine [hudi]
beyond1920 commented on code in PR #10977: URL: https://github.com/apache/hudi/pull/10977#discussion_r1555693313 ## website/docs/sql_dml.md: ## @@ -390,3 +390,70 @@ and `clean.async.enabled` options are used to disable the compaction and cleanin This is done to ensure that the compaction and cleaning services are not executed twice for the same table. +### Consistent hashing index (Experimental) + +We have introduced the Consistent Hashing Index since [0.13.0 release](/releases/release-0.13.0#consistent-hashing-index). In comparison to the static hashing index ([Bucket Index](/releases/release-0.11.0#bucket-index)), the consistent hashing index offers dynamic scalability of data buckets for the writer. +You can find the [RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md) for the design of this feature. +In the 0.13.X release, the Consistent Hashing Index is supported only for Spark engine. And since [release 0.14.0](/releases/release-0.14.0#consistent-hashing-index-support), the index is supported for Flink engine. + +In the below example, we have a streaming ingestion pipeline that written to the table with consistent bucket index. +To utilize this feature, configure the option `index.type` as `BUCKET` and set `hoodie.index.bucket.engine` to `CONSISTENT_HASHING`. +When enabling the consistent hashing index, it's important to enable clustering scheduling within the writer. During this process, the writer will perform dual writes for both the old and new data buckets while the clustering is pending. Although the dual write does not impact correctness, it is strongly recommended to execute clustering as quickly as possible. + +```sql +-- set the interval as 30 seconds +execution.checkpointing.interval: 3 +state.backend: rocksdb Review Comment: No. In order to commit the dataset, the checkpoint needs to be enabled, here is an example configuration for a `flink-conf.yaml`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6330][DOCS] Update user doc to show how to use consistent bucket index for Flink engine [hudi]
danny0405 commented on code in PR #10977:
URL: https://github.com/apache/hudi/pull/10977#discussion_r1555483875
##
website/docs/sql_dml.md:
##
@@ -390,3 +390,70 @@ and `clean.async.enabled` options are used to disable the
compaction and cleanin
This is done to ensure that the compaction and cleaning services are not
executed twice for the same table.
+### Consistent hashing index (Experimental)
+
+We have introduced the Consistent Hashing Index since [0.13.0
release](/releases/release-0.13.0#consistent-hashing-index). In comparison to
the static hashing index ([Bucket
Index](/releases/release-0.11.0#bucket-index)), the consistent hashing index
offers dynamic scalability of data buckets for the writer.
+You can find the
[RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md) for the
design of this feature.
+In the 0.13.X release, the Consistent Hashing Index is supported only for
Spark engine. And since [release
0.14.0](/releases/release-0.14.0#consistent-hashing-index-support), the index
is supported for Flink engine.
+
+In the below example, we have a streaming ingestion pipeline that written to
the table with consistent bucket index.
+To utilize this feature, configure the option `index.type` as `BUCKET` and set
`hoodie.index.bucket.engine` to `CONSISTENT_HASHING`.
+When enabling the consistent hashing index, it's important to enable
clustering scheduling within the writer. During this process, the writer will
perform dual writes for both the old and new data buckets while the clustering
is pending. Although the dual write does not impact correctness, it is strongly
recommended to execute clustering as quickly as possible.
+
+```sql
+-- set the interval as 30 seconds
+execution.checkpointing.interval: 3
+state.backend: rocksdb
+
+-- This is a datagen source that can generates records continuously
+CREATE TABLE sourceT (
+uuid varchar(20),
+name varchar(10),
+age int,
+ts timestamp(3),
+`partition` as 'par1'
+) WITH (
+'connector' = 'datagen',
+'rows-per-second' = '200'
+);
+
+-- Create the hudi table with consistent bucket index
+CREATE TABLE t1(
+uuid VARCHAR(20) PRIMARY KEY NOT ENFORCED,
+name VARCHAR(10),
+age INT,
+ts TIMESTAMP(3),
+`partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+'connector'='hudi',
+'path' = '${work_path}/hudi-demo/hudiT',
+'table.type' = 'MERGE_ON_READ',
+'index.type' = 'BUCKET',
+'clustering.schedule.enabled'='true',
+'hoodie.index.bucket.engine'='CONSISTENT_HASHING',
+
'hoodie.clustering.plan.strategy.class'='org.apache.hudi.client.clustering.plan.strategy.FlinkConsistentBucketClusteringPlanStrategy',
+
'hoodie.clustering.execution.strategy.class'='org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy',
+'hoodie.bucket.index.num.buckets'='8',
+'hoodie.bucket.index.max.num.buckets'='128',
+'hoodie.bucket.index.min.num.buckets'='8',
+'hoodie.bucket.index.split.threshold'='1.5',
+'write.tasks'='2'
+);
+
+-- submit the pipelines
+insert into t1 select * from sourceT;
+
+select * from t1 limit 20;
+```
+
+:::caution
+Consistent Hashing Index is supported for Flink engine since [release
0.14.0](/releases/release-0.14.0#consistent-hashing-index-support) and
currently there are some limitations to use it as of 0.14.0:
+
+- This index is supported only for MOR table. This limitation also exists even
if using Spark engine.
+- It does not work with metadata table enabled. This limitation also exists
even if using Spark engine.
+- Consistent hashing index does not work with bulk-insert using Flink engine
yet, please use simple bucket index or Spark engine for bulk-insert pipelines.
+- The resize plan which generated by Flink engine does not support merge small
file groups yet, but only support split big file group.
+- The resize plan should be executed through an offline Spark job.
Review Comment:
Spark or Flink job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-6330][DOCS] Update user doc to show how to use consistent bucket index for Flink engine [hudi]
danny0405 commented on code in PR #10977:
URL: https://github.com/apache/hudi/pull/10977#discussion_r1555483571
##
website/docs/sql_dml.md:
##
@@ -390,3 +390,70 @@ and `clean.async.enabled` options are used to disable the
compaction and cleanin
This is done to ensure that the compaction and cleaning services are not
executed twice for the same table.
+### Consistent hashing index (Experimental)
+
+We have introduced the Consistent Hashing Index since [0.13.0
release](/releases/release-0.13.0#consistent-hashing-index). In comparison to
the static hashing index ([Bucket
Index](/releases/release-0.11.0#bucket-index)), the consistent hashing index
offers dynamic scalability of data buckets for the writer.
+You can find the
[RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md) for the
design of this feature.
+In the 0.13.X release, the Consistent Hashing Index is supported only for
Spark engine. And since [release
0.14.0](/releases/release-0.14.0#consistent-hashing-index-support), the index
is supported for Flink engine.
+
+In the below example, we have a streaming ingestion pipeline that written to
the table with consistent bucket index.
+To utilize this feature, configure the option `index.type` as `BUCKET` and set
`hoodie.index.bucket.engine` to `CONSISTENT_HASHING`.
+When enabling the consistent hashing index, it's important to enable
clustering scheduling within the writer. During this process, the writer will
perform dual writes for both the old and new data buckets while the clustering
is pending. Although the dual write does not impact correctness, it is strongly
recommended to execute clustering as quickly as possible.
+
+```sql
+-- set the interval as 30 seconds
+execution.checkpointing.interval: 3
+state.backend: rocksdb
+
+-- This is a datagen source that can generates records continuously
+CREATE TABLE sourceT (
+uuid varchar(20),
+name varchar(10),
+age int,
+ts timestamp(3),
+`partition` as 'par1'
+) WITH (
+'connector' = 'datagen',
+'rows-per-second' = '200'
+);
+
+-- Create the hudi table with consistent bucket index
+CREATE TABLE t1(
+uuid VARCHAR(20) PRIMARY KEY NOT ENFORCED,
+name VARCHAR(10),
+age INT,
+ts TIMESTAMP(3),
+`partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+'connector'='hudi',
+'path' = '${work_path}/hudi-demo/hudiT',
+'table.type' = 'MERGE_ON_READ',
+'index.type' = 'BUCKET',
+'clustering.schedule.enabled'='true',
+'hoodie.index.bucket.engine'='CONSISTENT_HASHING',
+
'hoodie.clustering.plan.strategy.class'='org.apache.hudi.client.clustering.plan.strategy.FlinkConsistentBucketClusteringPlanStrategy',
+
'hoodie.clustering.execution.strategy.class'='org.apache.hudi.client.clustering.run.strategy.SparkConsistentBucketClusteringExecutionStrategy',
+'hoodie.bucket.index.num.buckets'='8',
+'hoodie.bucket.index.max.num.buckets'='128',
+'hoodie.bucket.index.min.num.buckets'='8',
+'hoodie.bucket.index.split.threshold'='1.5',
+'write.tasks'='2'
+);
+
+-- submit the pipelines
+insert into t1 select * from sourceT;
+
+select * from t1 limit 20;
+```
+
+:::caution
+Consistent Hashing Index is supported for Flink engine since [release
0.14.0](/releases/release-0.14.0#consistent-hashing-index-support) and
currently there are some limitations to use it as of 0.14.0:
+
+- This index is supported only for MOR table. This limitation also exists even
if using Spark engine.
+- It does not work with metadata table enabled. This limitation also exists
even if using Spark engine.
+- Consistent hashing index does not work with bulk-insert using Flink engine
yet, please use simple bucket index or Spark engine for bulk-insert pipelines.
+- The resize plan which generated by Flink engine does not support merge small
file groups yet, but only support split big file group.
Review Comment:
The resize plan which generated by Flink engine only supports merging small
file groups, the file splitting is not supported yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-6330][DOCS] Update user doc to show how to use consistent bucket index for Flink engine [hudi]
danny0405 commented on code in PR #10977: URL: https://github.com/apache/hudi/pull/10977#discussion_r1555476887 ## website/docs/sql_dml.md: ## @@ -390,3 +390,70 @@ and `clean.async.enabled` options are used to disable the compaction and cleanin This is done to ensure that the compaction and cleaning services are not executed twice for the same table. +### Consistent hashing index (Experimental) + +We have introduced the Consistent Hashing Index since [0.13.0 release](/releases/release-0.13.0#consistent-hashing-index). In comparison to the static hashing index ([Bucket Index](/releases/release-0.11.0#bucket-index)), the consistent hashing index offers dynamic scalability of data buckets for the writer. +You can find the [RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md) for the design of this feature. +In the 0.13.X release, the Consistent Hashing Index is supported only for Spark engine. And since [release 0.14.0](/releases/release-0.14.0#consistent-hashing-index-support), the index is supported for Flink engine. + +In the below example, we have a streaming ingestion pipeline that written to the table with consistent bucket index. +To utilize this feature, configure the option `index.type` as `BUCKET` and set `hoodie.index.bucket.engine` to `CONSISTENT_HASHING`. +When enabling the consistent hashing index, it's important to enable clustering scheduling within the writer. During this process, the writer will perform dual writes for both the old and new data buckets while the clustering is pending. Although the dual write does not impact correctness, it is strongly recommended to execute clustering as quickly as possible. + +```sql +-- set the interval as 30 seconds +execution.checkpointing.interval: 3 +state.backend: rocksdb Review Comment: Are these two lines standard SQL? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6330][DOCS] Update user doc to show how to use consistent bucket index for Flink engine [hudi]
danny0405 commented on code in PR #10977: URL: https://github.com/apache/hudi/pull/10977#discussion_r1555478222 ## website/docs/sql_dml.md: ## @@ -390,3 +390,70 @@ and `clean.async.enabled` options are used to disable the compaction and cleanin This is done to ensure that the compaction and cleaning services are not executed twice for the same table. +### Consistent hashing index (Experimental) + +We have introduced the Consistent Hashing Index since [0.13.0 release](/releases/release-0.13.0#consistent-hashing-index). In comparison to the static hashing index ([Bucket Index](/releases/release-0.11.0#bucket-index)), the consistent hashing index offers dynamic scalability of data buckets for the writer. +You can find the [RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md) for the design of this feature. +In the 0.13.X release, the Consistent Hashing Index is supported only for Spark engine. And since [release 0.14.0](/releases/release-0.14.0#consistent-hashing-index-support), the index is supported for Flink engine. + +In the below example, we have a streaming ingestion pipeline that written to the table with consistent bucket index. +To utilize this feature, configure the option `index.type` as `BUCKET` and set `hoodie.index.bucket.engine` to `CONSISTENT_HASHING`. +When enabling the consistent hashing index, it's important to enable clustering scheduling within the writer. During this process, the writer will perform dual writes for both the old and new data buckets while the clustering is pending. Although the dual write does not impact correctness, it is strongly recommended to execute clustering as quickly as possible. + +```sql +-- set the interval as 30 seconds +execution.checkpointing.interval: 3 +state.backend: rocksdb + +-- This is a datagen source that can generates records continuously Review Comment: can generate, BTW do we need the source table schema here, just the sink DDL definition might be okay. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6330][DOCS] Update user doc to show how to use consistent bucket index for Flink engine [hudi]
danny0405 commented on code in PR #10977: URL: https://github.com/apache/hudi/pull/10977#discussion_r1555475088 ## website/docs/sql_dml.md: ## @@ -390,3 +390,70 @@ and `clean.async.enabled` options are used to disable the compaction and cleanin This is done to ensure that the compaction and cleaning services are not executed twice for the same table. +### Consistent hashing index (Experimental) + +We have introduced the Consistent Hashing Index since [0.13.0 release](/releases/release-0.13.0#consistent-hashing-index). In comparison to the static hashing index ([Bucket Index](/releases/release-0.11.0#bucket-index)), the consistent hashing index offers dynamic scalability of data buckets for the writer. +You can find the [RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md) for the design of this feature. +In the 0.13.X release, the Consistent Hashing Index is supported only for Spark engine. And since [release 0.14.0](/releases/release-0.14.0#consistent-hashing-index-support), the index is supported for Flink engine. + +In the below example, we have a streaming ingestion pipeline that written to the table with consistent bucket index. Review Comment: `that written` -> `that writes` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-6330][DOCS] Update user doc to show how to use consistent bucket index for Flink engine [hudi]
beyond1920 opened a new pull request, #10977: URL: https://github.com/apache/hudi/pull/10977 ### Change Logs Update user doc to show how to use consistent bucket index for Flink engine ### Impact None ### Risk level (write none, low medium or high below) None ### Documentation Update None ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
