[GitHub] [hudi] hudi-bot commented on pull request #4584: [HUDI-3198] optimize create table based on an existing hudi table
hudi-bot commented on pull request #4584: URL: https://github.com/apache/hudi/pull/4584#issuecomment-1012892764 ## CI report: * 712fb15b14fd5d2a224c0ebe7c8f2d71b933c5f5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5188) * 44395d6c04d1265218c5d96ed9bf9ca091570477 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5232) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4584: [HUDI-3198] optimize create table based on an existing hudi table
hudi-bot removed a comment on pull request #4584: URL: https://github.com/apache/hudi/pull/4584#issuecomment-1012886153 ## CI report: * 712fb15b14fd5d2a224c0ebe7c8f2d71b933c5f5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5188) * 44395d6c04d1265218c5d96ed9bf9ca091570477 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4584: [HUDI-3198] optimize create table based on an existing hudi table
hudi-bot commented on pull request #4584: URL: https://github.com/apache/hudi/pull/4584#issuecomment-1012886153 ## CI report: * 712fb15b14fd5d2a224c0ebe7c8f2d71b933c5f5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5188) * 44395d6c04d1265218c5d96ed9bf9ca091570477 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] Carl-Zhou-CN commented on issue #4597: [SUPPORT] - Hudi Upserts Not working
Carl-Zhou-CN commented on issue #4597: URL: https://github.com/apache/hudi/issues/4597#issuecomment-1012886395 @harishraju-govindaraju hi,Because what you changed happened to be the partition field，the default index of hudi can only guarantee upsert under the partition. If you need global upsert, you need to configure hoodie.index.type:GLOBAL_BLOOM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4587: [HUDI-3236] use fields'comments persisted in catalog to fill in schema
hudi-bot removed a comment on pull request #4587: URL: https://github.com/apache/hudi/pull/4587#issuecomment-1012879648 ## CI report: * 790f50ec670c9247908c3c0fcdcaac34d0e02a21 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5202) * dec3b88d3cd1ea9475ce35b8789a721d940ae3f2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4584: [HUDI-3198] optimize create table based on an existing hudi table
hudi-bot removed a comment on pull request #4584: URL: https://github.com/apache/hudi/pull/4584#issuecomment-1011959705 ## CI report: * 712fb15b14fd5d2a224c0ebe7c8f2d71b933c5f5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5188) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4587: [HUDI-3236] use fields'comments persisted in catalog to fill in schema
hudi-bot commented on pull request #4587: URL: https://github.com/apache/hudi/pull/4587#issuecomment-1012886240 ## CI report: * 790f50ec670c9247908c3c0fcdcaac34d0e02a21 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5202) * dec3b88d3cd1ea9475ce35b8789a721d940ae3f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5231) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4587: [HUDI-3236] use fields'comments persisted in catalog to fill in schema
hudi-bot commented on pull request #4587: URL: https://github.com/apache/hudi/pull/4587#issuecomment-1012879648 ## CI report: * 790f50ec670c9247908c3c0fcdcaac34d0e02a21 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5202) * dec3b88d3cd1ea9475ce35b8789a721d940ae3f2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4587: [HUDI-3236] use fields'comments persisted in catalog to fill in schema
hudi-bot removed a comment on pull request #4587: URL: https://github.com/apache/hudi/pull/4587#issuecomment-1012282006 ## CI report: * 790f50ec670c9247908c3c0fcdcaac34d0e02a21 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5202) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4083: [HUDI-2837] Add support for using database name in incremental query
hudi-bot commented on pull request #4083: URL: https://github.com/apache/hudi/pull/4083#issuecomment-1012866728 ## CI report: * 00221c82e8b1693280fd72625eafcd503d54323c UNKNOWN * 46053bb143d1fd1274ac466197cc9361708e738b UNKNOWN * 2722bfcfd29a95f27338c1c8b026185472eefba0 UNKNOWN * f6101d298a6c7b26297da1d16913de9eee7a19cb UNKNOWN * dd027843f8367b044f8273940c244267330f5327 UNKNOWN * 8b32ba0ed32b5f7d7098752810e3b8e8a6bad6b6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5143) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5147) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5149) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5167) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5181) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5180) * b892126838427c1d2b018819cd03d18c4ce51de0 UNKNOWN * 6359d0f9c6e218f2d77f44560cc3a1bead7b1ac6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4083: [HUDI-2837] Add support for using database name in incremental query
hudi-bot removed a comment on pull request #4083: URL: https://github.com/apache/hudi/pull/4083#issuecomment-1012844381 ## CI report: * 00221c82e8b1693280fd72625eafcd503d54323c UNKNOWN * 46053bb143d1fd1274ac466197cc9361708e738b UNKNOWN * 2722bfcfd29a95f27338c1c8b026185472eefba0 UNKNOWN * f6101d298a6c7b26297da1d16913de9eee7a19cb UNKNOWN * dd027843f8367b044f8273940c244267330f5327 UNKNOWN * 8b32ba0ed32b5f7d7098752810e3b8e8a6bad6b6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5143) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5147) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5149) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5167) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5181) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5180) * b892126838427c1d2b018819cd03d18c4ce51de0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] rmahindra123 commented on a change in pull request #4594: [MINOR] - Publishing Debezium Blog
rmahindra123 commented on a change in pull request #4594: URL: https://github.com/apache/hudi/pull/4594#discussion_r784622438 ## File path: website/blog/2022-01-14-change-data-capture-with-debezium-and-apache-hudi.md ## @@ -0,0 +1,186 @@ +--- +title: "Change Data Capture with Debezium and Apache Hudi" +excerpt: "A review of new Debezium source connector for Apache Hudi" +author: Rajesh Mahindra +category: blog +--- + +As of Hudi v0.10.0, we are excited to announce the availability of [Debezium](https://debezium.io/) sources for [Deltastreamer](https://hudi.apache.org/docs/hoodie_deltastreamer) that provide the ingestion of change capture data (CDC) from Postgres and Mysql databases to your data lake. For more details, please refer to the original [RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-39/rfc-39.md). + + + +## Background + + +When you want to perform analytics on data from transactional databases like Postgres or Mysql you typically need to bring this data into an OLAP system such as a data warehouse or a data lake through a process called [Change Data Capture](https://debezium.io/documentation/faq/#what_is_change_data_capture) (CDC). Debezium is a popular tool that makes CDC easy. It provides a way to capture row-level changes in your databases by [reading changelogs](https://debezium.io/blog/2018/07/19/advantages-of-log-based-change-data-capture/). By doing so, Debezium avoids increased CPU load on your database and ensures you capture all changes including deletes. + +Now that [Apache Hudi](https://hudi.apache.org/docs/overview/) offers a Debezium source connector, CDC ingestion into a data lake is easier than ever with some [unique differentiated capabilities](https://hudi.apache.org/docs/use_cases). Hudi enables efficient update, merge, and delete transactions on a data lake. Hudi uniquely provides [Merge-On-Read](https://hudi.apache.org/docs/table_types#merge-on-read-table) writers which unlock [significantly lower latency](https://aws.amazon.com/blogs/big-data/how-amazon-transportation-service-enabled-near-real-time-event-analytics-at-petabyte-scale-using-aws-glue-with-apache-hudi/) ingestion than typical data lake writers with Spark or Flink. Last but not least, Apache Hudi offers [incremental queries](https://hudi.apache.org/docs/querying_data#spark-incr-query) so after capturing changes from your database, you can incrementally process these changes downstream throughout all of your subsequent ETL pipelines. + +## Design Overview + + +The architecture for an end-to-end CDC ingestion flow with Apache Hudi is shown above. The first component is the Debezium deployment, which consists of a Kafka cluster, schema registry (Confluent or Apicurio), and the Debezium connector. The Debezium connector continuously polls the changelogs from the database and writes an AVRO message with the changes for each database row to a dedicated Kafka topic per table. + +The second component is [Hudi Deltastreamer](https://hudi.apache.org/docs/hoodie_deltastreamer) that reads and processes the incoming Debezium records from Kafka for each table and writes (updates) the corresponding rows in a Hudi table on your cloud storage. + +To ingest the data from the database table into a Hudi table in near real-time, we implement two classes that can be plugged into the Deltastreamer. Firstly, we implemented a [Debezium source](https://github.com/apache/hudi/blob/83f8ed2ae3ba7fb20813cbb8768deae6244b020c/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/debezium/DebeziumSource.java). With Deltastreamer running in continuous mode, the source continuously reads and processes the Debezium change records in Avro format from the Kafka topic for a given table, and writes the updated record to the destination Hudi table. In addition to the columns from the database table, we also ingest some meta fields that are added by Debezium in the target Hudi table. The meta fields help us correctly merge updates and delete records. The records are read using the latest schema from the [Schema Registry](https://hudi.apache.org/docs/hoodie_deltastreamer#schema-providers). + +Secondly, we implement a custom [Debezium Payload](https://github.com/apache/hudi/blob/83f8ed2ae3ba7fb20813cbb8768deae6244b020c/hudi-common/src/main/java/org/apache/hudi/common/model/debezium/AbstractDebeziumAvroPayload.java) that essentially governs how Hudi records are merged when the same row is updated or deleted. When a new Hudi record is received for an existing row, the payload picks the latest record using the higher value of the appropriate column (FILEID and POS fields in MySql and LSN fields in Postgres). In the case that the latter event is a delete record, the payload implementation ensures that the record is hard deleted from the storage. Delete records are identified using the op field, which has a value of **d** for deletes. + +## Apache Hudi Configurations + +
[GitHub] [hudi] hudi-bot commented on pull request #4586: [HUDI-1558] Struct Stream Source Support Spark3
hudi-bot commented on pull request #4586: URL: https://github.com/apache/hudi/pull/4586#issuecomment-1012846304 ## CI report: * 0d4cc106c5ae7d077a55a2dfab3ea13b6c9e2054 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5226) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4586: [HUDI-1558] Struct Stream Source Support Spark3
hudi-bot removed a comment on pull request #4586: URL: https://github.com/apache/hudi/pull/4586#issuecomment-1012808196 ## CI report: * 2c2c2a5b8dadaaefce68b9d0730d6e18e8bcc0f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5212) * 0d4cc106c5ae7d077a55a2dfab3ea13b6c9e2054 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5226) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4535: [HUDI-3161] Add Call Produce Command for spark sql
hudi-bot removed a comment on pull request #4535: URL: https://github.com/apache/hudi/pull/4535#issuecomment-1012844657 ## CI report: * 49b18f6d40a8b859927dcc9d606d40fd4162f0b1 UNKNOWN * a39a6cda867038f96d379ff17b7e1216fa2326fb UNKNOWN * f56b53b80f3cfc8949eb2f4d14ee2a8a762252da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5045) * 360d4337ef225abe15188253f401217c9c944818 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5229) * 02f77dc177757252c7b238ae2643ababdec03748 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4535: [HUDI-3161] Add Call Produce Command for spark sql
hudi-bot commented on pull request #4535: URL: https://github.com/apache/hudi/pull/4535#issuecomment-1012846223 ## CI report: * 49b18f6d40a8b859927dcc9d606d40fd4162f0b1 UNKNOWN * a39a6cda867038f96d379ff17b7e1216fa2326fb UNKNOWN * 360d4337ef225abe15188253f401217c9c944818 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5229) * 02f77dc177757252c7b238ae2643ababdec03748 UNKNOWN * c8a3a975bf947b29a1f9c56eeb1ef0afbfa25d8c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4535: [WIP][HUDI-3161] Add Call Produce Command for spark sql
hudi-bot removed a comment on pull request #4535: URL: https://github.com/apache/hudi/pull/4535#issuecomment-1012834240 ## CI report: * 49b18f6d40a8b859927dcc9d606d40fd4162f0b1 UNKNOWN * a39a6cda867038f96d379ff17b7e1216fa2326fb UNKNOWN * f56b53b80f3cfc8949eb2f4d14ee2a8a762252da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5045) * 360d4337ef225abe15188253f401217c9c944818 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5229) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4180: [HUDI-2903] get table schema from the last commit with data written
hudi-bot removed a comment on pull request #4180: URL: https://github.com/apache/hudi/pull/4180#issuecomment-1012842641 ## CI report: * 5ad7241fbe98875c71a3aa4d394cd95f266ae5d9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5141) * 60055127509923dc406fe2c5fa7fdf037d75a37d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4180: [HUDI-2903] get table schema from the last commit with data written
hudi-bot commented on pull request #4180: URL: https://github.com/apache/hudi/pull/4180#issuecomment-1012844419 ## CI report: * 5ad7241fbe98875c71a3aa4d394cd95f266ae5d9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5141) * 60055127509923dc406fe2c5fa7fdf037d75a37d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5230) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4535: [WIP][HUDI-3161] Add Call Produce Command for spark sql
hudi-bot commented on pull request #4535: URL: https://github.com/apache/hudi/pull/4535#issuecomment-1012844657 ## CI report: * 49b18f6d40a8b859927dcc9d606d40fd4162f0b1 UNKNOWN * a39a6cda867038f96d379ff17b7e1216fa2326fb UNKNOWN * f56b53b80f3cfc8949eb2f4d14ee2a8a762252da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5045) * 360d4337ef225abe15188253f401217c9c944818 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5229) * 02f77dc177757252c7b238ae2643ababdec03748 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4083: [HUDI-2837] Add support for using database name in incremental query
hudi-bot commented on pull request #4083: URL: https://github.com/apache/hudi/pull/4083#issuecomment-1012844381 ## CI report: * 00221c82e8b1693280fd72625eafcd503d54323c UNKNOWN * 46053bb143d1fd1274ac466197cc9361708e738b UNKNOWN * 2722bfcfd29a95f27338c1c8b026185472eefba0 UNKNOWN * f6101d298a6c7b26297da1d16913de9eee7a19cb UNKNOWN * dd027843f8367b044f8273940c244267330f5327 UNKNOWN * 8b32ba0ed32b5f7d7098752810e3b8e8a6bad6b6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5143) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5147) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5149) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5167) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5181) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5180) * b892126838427c1d2b018819cd03d18c4ce51de0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4083: [HUDI-2837] Add support for using database name in incremental query
hudi-bot removed a comment on pull request #4083: URL: https://github.com/apache/hudi/pull/4083#issuecomment-1011865457 ## CI report: * 00221c82e8b1693280fd72625eafcd503d54323c UNKNOWN * 46053bb143d1fd1274ac466197cc9361708e738b UNKNOWN * 2722bfcfd29a95f27338c1c8b026185472eefba0 UNKNOWN * f6101d298a6c7b26297da1d16913de9eee7a19cb UNKNOWN * dd027843f8367b044f8273940c244267330f5327 UNKNOWN * 8b32ba0ed32b5f7d7098752810e3b8e8a6bad6b6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5143) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5147) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5149) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5167) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5181) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5180) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4180: [HUDI-2903] get table schema from the last commit with data written
hudi-bot commented on pull request #4180: URL: https://github.com/apache/hudi/pull/4180#issuecomment-1012842641 ## CI report: * 5ad7241fbe98875c71a3aa4d394cd95f266ae5d9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5141) * 60055127509923dc406fe2c5fa7fdf037d75a37d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4180: [HUDI-2903] get table schema from the last commit with data written
hudi-bot removed a comment on pull request #4180: URL: https://github.com/apache/hudi/pull/4180#issuecomment-1010849850 ## CI report: * 5ad7241fbe98875c71a3aa4d394cd95f266ae5d9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5141) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4535: [WIP][HUDI-3161] Add Call Produce Command for spark sql
hudi-bot removed a comment on pull request #4535: URL: https://github.com/apache/hudi/pull/4535#issuecomment-1012815228 ## CI report: * 49b18f6d40a8b859927dcc9d606d40fd4162f0b1 UNKNOWN * a39a6cda867038f96d379ff17b7e1216fa2326fb UNKNOWN * f56b53b80f3cfc8949eb2f4d14ee2a8a762252da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5045) * 360d4337ef225abe15188253f401217c9c944818 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4535: [WIP][HUDI-3161] Add Call Produce Command for spark sql
hudi-bot commented on pull request #4535: URL: https://github.com/apache/hudi/pull/4535#issuecomment-1012834240 ## CI report: * 49b18f6d40a8b859927dcc9d606d40fd4162f0b1 UNKNOWN * a39a6cda867038f96d379ff17b7e1216fa2326fb UNKNOWN * f56b53b80f3cfc8949eb2f4d14ee2a8a762252da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5045) * 360d4337ef225abe15188253f401217c9c944818 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5229) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4589: [MINOR] Fix the check condition in the `readFromVector` method to alway true
hudi-bot commented on pull request #4589: URL: https://github.com/apache/hudi/pull/4589#issuecomment-1012832907 ## CI report: * 1141cc55e5e354d5671f9d7e894b2fd85a8aa1de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5197) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5211) * 9694d86642166e74f603cf2a7a5cc38ae2204f20 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5228) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiaotianzhang01 commented on a change in pull request #4589: [MINOR] Fix the check condition in the `readFromVector` method to alway true
xiaotianzhang01 commented on a change in pull request #4589:
URL: https://github.com/apache/hudi/pull/4589#discussion_r784582353
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/AvroOrcUtils.java
##
@@ -521,9 +522,6 @@ public static Object readFromVector(TypeDescription type,
ColumnVector colVector
} else {
return ((BytesColumnVector) colVector).toString(vectorPos);
}
- case DATE:
-// convert to daysSinceEpoch for LogicalType.Date
Review comment:
Sorry, I think it's still a single PR focus feature fix, so I'll cancel
changes here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4589: [MINOR] Fix the check condition in the `readFromVector` method to alway true
hudi-bot removed a comment on pull request #4589: URL: https://github.com/apache/hudi/pull/4589#issuecomment-1012831570 ## CI report: * 1141cc55e5e354d5671f9d7e894b2fd85a8aa1de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5197) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5211) * 9694d86642166e74f603cf2a7a5cc38ae2204f20 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiaotianzhang01 commented on a change in pull request #4589: [MINOR] Fix the check condition in the `readFromVector` method to alway true
xiaotianzhang01 commented on a change in pull request #4589:
URL: https://github.com/apache/hudi/pull/4589#discussion_r784582594
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/AvroOrcUtils.java
##
@@ -521,9 +522,6 @@ public static Object readFromVector(TypeDescription type,
ColumnVector colVector
} else {
return ((BytesColumnVector) colVector).toString(vectorPos);
}
- case DATE:
-// convert to daysSinceEpoch for LogicalType.Date
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] xiaotianzhang01 commented on a change in pull request #4589: [MINOR] Fix the check condition in the `readFromVector` method to alway true
xiaotianzhang01 commented on a change in pull request #4589:
URL: https://github.com/apache/hudi/pull/4589#discussion_r784582353
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/AvroOrcUtils.java
##
@@ -521,9 +522,6 @@ public static Object readFromVector(TypeDescription type,
ColumnVector colVector
} else {
return ((BytesColumnVector) colVector).toString(vectorPos);
}
- case DATE:
-// convert to daysSinceEpoch for LogicalType.Date
Review comment:
Sorry, I think it's still a single PR focus feature fix, so I'll
Cancel changes here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] gnailJC opened a new issue #4599: [SUPPORT] support savepoint on MergeOnRead table
gnailJC opened a new issue #4599: URL: https://github.com/apache/hudi/issues/4599 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-subscr...@hudi.apache.org. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** I get a exception "Savepointing is not supported or MergeOnRead table types" when creating a savepoint on a MOR table on hudi-cli. Is it possible to support create savepoints on MOR table? **To Reproduce** Steps to reproduce the behavior: 1. connect --path oss://bucket1/test_mor 2. savepoint create --commit 20220101*** --sparkMaster spark://** 3. 4. **Expected behavior** A clear and concise description of what you expected to happen. **Environment Description** * Hudi version : 0.10.0 * Spark version : 3.0.3 * Hive version : * Hadoop version : 2.9.2 * Storage (HDFS/S3/GCS..) : oss * Running on Docker? (yes/no) : no **Additional context** Add any other context about the problem here. **Stacktrace** ```Add the stacktrace of the error.``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4589: [MINOR] Fix the check condition in the `readFromVector` method to alway true
hudi-bot removed a comment on pull request #4589: URL: https://github.com/apache/hudi/pull/4589#issuecomment-1012713610 ## CI report: * 1141cc55e5e354d5671f9d7e894b2fd85a8aa1de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5197) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5211) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4589: [MINOR] Fix the check condition in the `readFromVector` method to alway true
hudi-bot commented on pull request #4589: URL: https://github.com/apache/hudi/pull/4589#issuecomment-1012831570 ## CI report: * 1141cc55e5e354d5671f9d7e894b2fd85a8aa1de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5197) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5211) * 9694d86642166e74f603cf2a7a5cc38ae2204f20 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot removed a comment on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012796670 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 2a3b74e239bf96351eece45e928eac8e4f96d4ff Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5222) * 9fb948535b79336115a8e0fddfa955f7bfbff5f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5225) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot commented on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012822806 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 9fb948535b79336115a8e0fddfa955f7bfbff5f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5225) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated (5ce45c4 -> 7d163ee)
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 5ce45c4 [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation (#4514) add 7d163ee [MINOR] Fix local flaky test in TestFSUtils (#4596) No new revisions were added by this update. Summary of changes: .../java/org/apache/hudi/common/fs/TestFSUtils.java | 17 +++-- 1 file changed, 7 insertions(+), 10 deletions(-)
[GitHub] [hudi] xushiyan merged pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
xushiyan merged pull request #4596: URL: https://github.com/apache/hudi/pull/4596 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4535: [WIP][HUDI-3161] Add Call Produce Command for spark sql
hudi-bot removed a comment on pull request #4535: URL: https://github.com/apache/hudi/pull/4535#issuecomment-1008582494 ## CI report: * 49b18f6d40a8b859927dcc9d606d40fd4162f0b1 UNKNOWN * a39a6cda867038f96d379ff17b7e1216fa2326fb UNKNOWN * f56b53b80f3cfc8949eb2f4d14ee2a8a762252da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5045) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4535: [WIP][HUDI-3161] Add Call Produce Command for spark sql
hudi-bot commented on pull request #4535: URL: https://github.com/apache/hudi/pull/4535#issuecomment-1012815228 ## CI report: * 49b18f6d40a8b859927dcc9d606d40fd4162f0b1 UNKNOWN * a39a6cda867038f96d379ff17b7e1216fa2326fb UNKNOWN * f56b53b80f3cfc8949eb2f4d14ee2a8a762252da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5045) * 360d4337ef225abe15188253f401217c9c944818 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4586: [HUDI-1558] Struct Stream Source Support Spark3
hudi-bot commented on pull request #4586: URL: https://github.com/apache/hudi/pull/4586#issuecomment-1012808196 ## CI report: * 2c2c2a5b8dadaaefce68b9d0730d6e18e8bcc0f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5212) * 0d4cc106c5ae7d077a55a2dfab3ea13b6c9e2054 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5226) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4586: [HUDI-1558] Struct Stream Source Support Spark3
hudi-bot removed a comment on pull request #4586: URL: https://github.com/apache/hudi/pull/4586#issuecomment-1012807071 ## CI report: * 2c2c2a5b8dadaaefce68b9d0730d6e18e8bcc0f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5212) * 0d4cc106c5ae7d077a55a2dfab3ea13b6c9e2054 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4586: [HUDI-1558] Struct Stream Source Support Spark3
hudi-bot removed a comment on pull request #4586: URL: https://github.com/apache/hudi/pull/4586#issuecomment-1012727423 ## CI report: * 2c2c2a5b8dadaaefce68b9d0730d6e18e8bcc0f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5212) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4586: [HUDI-1558] Struct Stream Source Support Spark3
hudi-bot commented on pull request #4586: URL: https://github.com/apache/hudi/pull/4586#issuecomment-1012807071 ## CI report: * 2c2c2a5b8dadaaefce68b9d0730d6e18e8bcc0f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5212) * 0d4cc106c5ae7d077a55a2dfab3ea13b6c9e2054 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4589: [MINOR] Fix the check condition in the `readFromVector` method to alway true
danny0405 commented on a change in pull request #4589:
URL: https://github.com/apache/hudi/pull/4589#discussion_r784527723
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/AvroOrcUtils.java
##
@@ -521,9 +522,6 @@ public static Object readFromVector(TypeDescription type,
ColumnVector colVector
} else {
return ((BytesColumnVector) colVector).toString(vectorPos);
}
- case DATE:
-// convert to daysSinceEpoch for LogicalType.Date
Review comment:
Not sure what this patch fixed for ? Do you want to fix the NPE here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot removed a comment on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012777350 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5220) * f2baeebee296d7f79c325a4d4ca63b50d189733d UNKNOWN * 626cecc34f53634052875ee006cd969b6f0e927b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5223) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot commented on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012798940 ## CI report: * f2baeebee296d7f79c325a4d4ca63b50d189733d UNKNOWN * 626cecc34f53634052875ee006cd969b6f0e927b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5223) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot removed a comment on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012786245 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 2a3b74e239bf96351eece45e928eac8e4f96d4ff Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5222) * 9fb948535b79336115a8e0fddfa955f7bfbff5f2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot commented on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012796670 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 2a3b74e239bf96351eece45e928eac8e4f96d4ff Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5222) * 9fb948535b79336115a8e0fddfa955f7bfbff5f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5225) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] dongkelun commented on a change in pull request #4515: [HUDI-3158] Reduce warn logs in Spark SQL INSERT OVERWRITE
dongkelun commented on a change in pull request #4515:
URL: https://github.com/apache/hudi/pull/4515#discussion_r784521348
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java
##
@@ -156,6 +157,18 @@ public void createNewInstant(HoodieInstant instant) {
createFileInMetaPath(instant.getFileName(), Option.empty(), false);
}
+ public void createRequestedReplaceCommit(String instantTime, String
actionType) {
+try {
+ HoodieInstant instant = new HoodieInstant(State.REQUESTED, actionType,
instantTime);
+ LOG.info("Creating a new instant " + instant);
Review comment:
It's not necessary, just to be consistent with the previous method
`createNewInstant`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[hudi] branch master updated (195dac9 -> 5ce45c4)
This is an automated email from the ASF dual-hosted git repository. leesf pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 195dac9 [MINOR] Disable flaky tests to unlock CI (#4592) add 5ce45c4 [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation (#4514) No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/hudi/SparkAdapter.scala | 4 +- hudi-spark-datasource/README.md| 38 +++ hudi-spark-datasource/hudi-spark-common/pom.xml| 42 +++ .../apache/hudi/HoodieDatasetBulkInsertHelper.java | 0 .../java/org/apache/hudi/SparkRowWriteHelper.java | 0 .../SparkStreamingAsyncClusteringService.java | 0 .../async/SparkStreamingAsyncCompactService.java | 0 .../org/apache/hudi/sql/IExpressionEvaluator.java | 0 .../main/java/org/apache/hudi/sql/InsertMode.java | 0 ...org.apache.spark.sql.sources.DataSourceRegister | 0 .../main/scala/org/apache/hudi/DefaultSource.scala | 2 +- .../scala/org/apache/hudi/HoodieBootstrapRDD.scala | 0 .../org/apache/hudi/HoodieBootstrapRelation.scala | 0 .../org/apache/hudi/HoodieEmptyRelation.scala | 0 .../scala/org/apache/hudi/HoodieFileIndex.scala| 4 +- .../org/apache/hudi/HoodieMergeOnReadRDD.scala | 46 ++-- .../org/apache/hudi/HoodieSparkSqlWriter.scala | 10 +- .../org/apache/hudi/HoodieStreamingSink.scala | 0 .../scala/org/apache/hudi/HoodieWriterUtils.scala | 0 .../org/apache/hudi/IncrementalRelation.scala | 0 .../hudi/MergeOnReadIncrementalRelation.scala | 0 .../apache/hudi/MergeOnReadSnapshotRelation.scala | 4 +- .../src/main/scala/org/apache/hudi/package.scala | 0 .../spark/sql/avro/HoodieAvroDeserializer.scala| 0 .../spark/sql/avro/HoodieAvroSerializer.scala | 0 .../sql/catalyst/catalog/HoodieCatalogTable.scala | 26 +- .../spark/sql/catalyst/trees/HoodieLeafLike.scala | 0 .../apache/spark/sql/hive/HiveClientUtils.scala| 0 .../apache/spark/sql/hudi/DataSkippingUtils.scala | 0 .../apache/spark/sql/hudi/HoodieOptionConfig.scala | 2 +- .../spark/sql/hudi/HoodieSqlCommonUtils.scala} | 64 ++--- .../org/apache/spark/sql/hudi/SerDeUtils.scala | 0 .../AlterHoodieTableAddColumnsCommand.scala| 2 - .../AlterHoodieTableChangeColumnCommand.scala | 2 - .../AlterHoodieTableDropPartitionCommand.scala | 5 +- .../command/AlterHoodieTableRenameCommand.scala| 0 .../hudi/command/CreateHoodieTableCommand.scala| 10 +- .../hudi/command/HoodieLeafRunnableCommand.scala | 0 .../command/ShowHoodieTablePartitionsCommand.scala | 2 +- .../spark/sql/hudi/command/SqlKeyGenerator.scala | 2 +- .../hudi/command/TruncateHoodieTableCommand.scala | 0 .../hudi/command/ValidateDuplicateKeyPayload.scala | 45 .../hudi/command/payload/ExpressionCodeGen.scala | 0 .../hudi/command/payload/ExpressionPayload.scala | 0 .../sql/hudi/command/payload/SqlTypedRecord.scala | 0 .../sql/hudi/streaming/HoodieSourceOffset.scala| 0 .../sql/hudi/streaming/HoodieStreamSource.scala| 0 hudi-spark-datasource/hudi-spark/pom.xml | 12 + .../sql/hudi/HoodieSparkSessionExtension.scala | 2 +- .../org/apache/spark/sql/hudi/HoodieSqlUtils.scala | 289 + .../spark/sql/hudi/analysis/HoodieAnalysis.scala | 28 +- .../hudi/command/CompactionHoodiePathCommand.scala | 11 +- .../command/CompactionHoodieTableCommand.scala | 5 +- .../command/CompactionShowHoodieTableCommand.scala | 5 +- .../command/CreateHoodieTableAsSelectCommand.scala | 5 +- .../hudi/command/DeleteHoodieTableCommand.scala| 7 +- .../sql/hudi/command/DropHoodieTableCommand.scala | 2 +- .../command/InsertIntoHoodieTableCommand.scala | 33 +-- .../hudi/command/MergeIntoHoodieTableCommand.scala | 8 +- .../hudi/command/UpdateHoodieTableCommand.scala| 10 +- .../apache/hudi/functional/TestMORDataSource.scala | 2 - .../org/apache/spark/sql/hudi/TestAlterTable.scala | 2 +- hudi-spark-datasource/hudi-spark2-common/pom.xml | 19 ++ ...org.apache.spark.sql.sources.DataSourceRegister | 2 +- .../Spark2DefaultSource.scala} | 16 +- .../apache/spark/sql/adapter/Spark2Adapter.scala | 8 +- hudi-spark-datasource/hudi-spark3-common/pom.xml | 247 ++ .../apache/hudi/spark3/internal/DefaultSource.java | 0 .../HoodieBulkInsertDataInternalWriter.java| 0 .../HoodieBulkInsertDataInternalWriterFactory.java | 0 .../HoodieDataSourceInternalBatchWrite.java| 0 .../HoodieDataSourceInternalBatchWriteBuilder.java | 0 .../internal/HoodieDataSourceInternalTable.java| 0 .../spark3/internal/HoodieWriterCommitMessage.java | 0 .../apache/hudi/spark3/internal/ReflectUtil.java | 0 .../scala/org/apache/hudi/Spark3RowSerDe.scala | 0 .../apache/spark/
[GitHub] [hudi] leesf merged pull request #4514: [HUDI-3172] Refactor hudi existing modules to make more code reuse in V2 Implementation
leesf merged pull request #4514: URL: https://github.com/apache/hudi/pull/4514 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot removed a comment on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012785174 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 3cb5ff6636414fa2ca81c9740e073f694a696e5e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5218) * 2a3b74e239bf96351eece45e928eac8e4f96d4ff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5222) * 9fb948535b79336115a8e0fddfa955f7bfbff5f2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot commented on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012786245 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 2a3b74e239bf96351eece45e928eac8e4f96d4ff Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5222) * 9fb948535b79336115a8e0fddfa955f7bfbff5f2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot removed a comment on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012774985 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 3cb5ff6636414fa2ca81c9740e073f694a696e5e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5218) * 2a3b74e239bf96351eece45e928eac8e4f96d4ff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5222) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot commented on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012785174 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 3cb5ff6636414fa2ca81c9740e073f694a696e5e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5218) * 2a3b74e239bf96351eece45e928eac8e4f96d4ff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5222) * 9fb948535b79336115a8e0fddfa955f7bfbff5f2 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4531: [HUDI-3191][Stacked on 4520] Rebasing Hive's FileInputFormat onto `AbstractHoodieTableFileIndex`
hudi-bot commented on pull request #4531: URL: https://github.com/apache/hudi/pull/4531#issuecomment-1012781174 ## CI report: * e13878078c60d807592b5c3f53d1ec1368a9d78d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5219) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4531: [HUDI-3191][Stacked on 4520] Rebasing Hive's FileInputFormat onto `AbstractHoodieTableFileIndex`
hudi-bot removed a comment on pull request #4531: URL: https://github.com/apache/hudi/pull/4531#issuecomment-1012727355 ## CI report: * 38d1af9040aa65a7b1205087022b727db17ab049 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5216) * e13878078c60d807592b5c3f53d1ec1368a9d78d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5219) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HUDI-3066) Very slow file listing after enabling metadata for existing tables in 0.10.0 release
[
https://issues.apache.org/jira/browse/HUDI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17475935#comment-17475935
]
Harsha Teja Kanna edited comment on HUDI-3066 at 1/14/22, 5:04 AM:
---
With only base path in the load. File listing time(few seconds) is negligible
compared to query runtime. Thanks
Though I have to add a new column 'date' to every table to make the above
columns type clash resolved.
was (Author: h7kanna):
With only base path in the load. File listing time(few seconds) is negligible
compared to query runtime. Thanks
Though I have to add a new column 'date' to every table to make the above
columns type clash.
> Very slow file listing after enabling metadata for existing tables in 0.10.0
> release
>
>
> Key: HUDI-3066
> URL: https://issues.apache.org/jira/browse/HUDI-3066
> Project: Apache Hudi
> Issue Type: Bug
>Affects Versions: 0.10.0
> Environment: EMR 6.4.0
> Hudi version : 0.10.0
>Reporter: Harsha Teja Kanna
>Assignee: sivabalan narayanan
>Priority: Critical
> Labels: performance, pull-request-available
> Fix For: 0.11.0
>
> Attachments: Screen Shot 2021-12-18 at 6.16.29 PM.png, Screen Shot
> 2021-12-20 at 10.05.50 PM.png, Screen Shot 2021-12-20 at 10.17.44 PM.png,
> Screen Shot 2021-12-21 at 10.22.54 PM.png, Screen Shot 2021-12-21 at 10.24.12
> PM.png, metadata_files.txt, metadata_files_compacted.txt,
> metadata_timeline.txt, metadata_timeline_archived.txt,
> metadata_timeline_compacted.txt, stderr_part1.txt, stderr_part2.txt,
> timeline.txt, writer_log.txt
>
>
> After 'metadata table' is enabled, File listing takes long time.
> If metadata is enabled on Reader side(as shown below), it is taking even more
> time per file listing task
> {code:java}
> import org.apache.hudi.DataSourceReadOptions
> import org.apache.hudi.common.config.HoodieMetadataConfig
> val hadoopConf = spark.conf
> hadoopConf.set(HoodieMetadataConfig.ENABLE.key(), "true")
> val basePath = "s3a://datalake-hudi"
> val sessions = spark
> .read
> .format("org.apache.hudi")
> .option(DataSourceReadOptions.QUERY_TYPE.key(),
> DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL)
> .option(DataSourceReadOptions.READ_PATHS.key(),
> s"${basePath}/sessions_by_entrydate/entrydate=2021/*/*/*")
> .load()
> sessions.createOrReplaceTempView("sessions") {code}
> Existing tables (COW) have inline clustering on and have many replace commits.
> Logs seem to suggest the delay is in view.AbstractTableFileSystemView
> resetFileGroupsReplaced function or metadata.HoodieBackedTableMetadata
> Also many log messages in AbstractHoodieLogRecordReader
>
> 2021-12-18 23:17:54,056 INFO view.AbstractTableFileSystemView: Took 4118 ms
> to read 136 instants, 9731 replaced file groups
> 2021-12-18 23:37:46,086 INFO log.AbstractHoodieLogRecordReader: Number of
> remaining logblocks to merge 1
> 2021-12-18 23:37:46,090 INFO log.AbstractHoodieLogRecordReader: Reading a
> data block from file
> s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.76_0-20-515
> at instant 20211217035105329
> 2021-12-18 23:37:46,090 INFO log.AbstractHoodieLogRecordReader: Number of
> remaining logblocks to merge 1
> 2021-12-18 23:37:46,094 INFO log.HoodieLogFormatReader: Moving to the next
> reader for logfile
> HoodieLogFile\{pathStr='s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.121_0-57-663',
> fileLen=0}
> 2021-12-18 23:37:46,095 INFO log.AbstractHoodieLogRecordReader: Scanning log
> file
> HoodieLogFile\{pathStr='s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.20_0-35-613',
> fileLen=0}
> 2021-12-18 23:37:46,095 INFO s3a.S3AInputStream: Switching to Random IO seek
> policy
> 2021-12-18 23:37:46,096 INFO log.AbstractHoodieLogRecordReader: Reading a
> data block from file
> s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.62_0-34-377
> at instant 20211217022049877
> 2021-12-18 23:37:46,096 INFO log.AbstractHoodieLogRecordReader: Number of
> remaining logblocks to merge 1
> 2021-12-18 23:37:46,105 INFO log.HoodieLogFormatReader: Moving to the next
> reader for logfile
> HoodieLogFile\{pathStr='s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.86_0-20-362',
> fileLen=0}
> 2021-12-18 23:37:46,109 INFO log.AbstractHoodieLogRecordReader: Scanning log
> file
> HoodieLogFile\{pathStr='s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.121_0-57-663',
> fileLen=0}
> 2021-12-18 23:37:46,109 INFO s3a.S3AInputStream: Switching to Random IO seek
> policy
> 2021-12-18 23:37
[jira] [Comment Edited] (HUDI-3244) UnsupportedOperationException when bulk insert to hudi
[
https://issues.apache.org/jira/browse/HUDI-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17475893#comment-17475893
]
cdmikechen edited comment on HUDI-3244 at 1/14/22, 5:03 AM:
Need to merge pr https://github.com/apache/hudi/pull/4498
was (Author: chenxiang):
It seems like I imported *hudi-spark3-bundle_2.12* (2021.12.29) which I built
before, but its name is *hudi-spark3.1.2-bundle_2.12* now.
> UnsupportedOperationException when bulk insert to hudi
> --
>
> Key: HUDI-3244
> URL: https://issues.apache.org/jira/browse/HUDI-3244
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Reporter: cdmikechen
>Priority: Major
> Fix For: 0.10.1
>
>
> When I bulk insert to hudi, I catch this error.
> {code}
> java.lang.UnsupportedOperationException
> at java.base/java.util.Collections$UnmodifiableMap.put(Unknown Source)
> at
> org.apache.hudi.DataSourceUtils.mayBeOverwriteParquetWriteLegacyFormatProp(DataSourceUtils.java:321)
> at
> org.apache.hudi.spark3.internal.DefaultSource.getTable(DefaultSource.java:59)
> at
> org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:83)
> at
> org.apache.spark.sql.DataFrameWriter.getTable$1(DataFrameWriter.scala:322)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:338)
> at
> org.apache.hudi.HoodieSparkSqlWriter$.bulkInsertAsRow(HoodieSparkSqlWriter.scala:477)
> at
> org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:158)
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:164)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:127)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:126)
> at
> org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:962)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> at
> org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:962)
> at
> org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:414)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:398)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:287)
> at com.syzh.data.LoadRdbmsToHudi.writeHoodie(LoadRdbmsToHudi.scala:326)
> at com.syzh.data.LoadRdbmsToHudi.loadJdbc(LoadRdbmsToHudi.scala:91)
> at
> com.syzh.batch.spark.service.EtlService.ingestionTotalData(EtlService.java:114)
> at
> com.syzh.batch.spark.listener.EtlBatchListener.doTask(EtlBatchListener.java:154)
> at
> com.syzh.batch.spark.listener.EtlBatchListener.action(EtlBatchListener.java:110)
> at
> org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:685)
> at
> org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:679)
> at
> org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
> at
> org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:84)
> at
> org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCa
[jira] [Commented] (HUDI-3066) Very slow file listing after enabling metadata for existing tables in 0.10.0 release
[
https://issues.apache.org/jira/browse/HUDI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17475935#comment-17475935
]
Harsha Teja Kanna commented on HUDI-3066:
-
With only base path in the load. File listing time(few seconds) is negligible
compared to query runtime. Thanks
Though I have to add a new column 'date' to every table to make the above
columns type clash.
> Very slow file listing after enabling metadata for existing tables in 0.10.0
> release
>
>
> Key: HUDI-3066
> URL: https://issues.apache.org/jira/browse/HUDI-3066
> Project: Apache Hudi
> Issue Type: Bug
>Affects Versions: 0.10.0
> Environment: EMR 6.4.0
> Hudi version : 0.10.0
>Reporter: Harsha Teja Kanna
>Assignee: sivabalan narayanan
>Priority: Critical
> Labels: performance, pull-request-available
> Fix For: 0.11.0
>
> Attachments: Screen Shot 2021-12-18 at 6.16.29 PM.png, Screen Shot
> 2021-12-20 at 10.05.50 PM.png, Screen Shot 2021-12-20 at 10.17.44 PM.png,
> Screen Shot 2021-12-21 at 10.22.54 PM.png, Screen Shot 2021-12-21 at 10.24.12
> PM.png, metadata_files.txt, metadata_files_compacted.txt,
> metadata_timeline.txt, metadata_timeline_archived.txt,
> metadata_timeline_compacted.txt, stderr_part1.txt, stderr_part2.txt,
> timeline.txt, writer_log.txt
>
>
> After 'metadata table' is enabled, File listing takes long time.
> If metadata is enabled on Reader side(as shown below), it is taking even more
> time per file listing task
> {code:java}
> import org.apache.hudi.DataSourceReadOptions
> import org.apache.hudi.common.config.HoodieMetadataConfig
> val hadoopConf = spark.conf
> hadoopConf.set(HoodieMetadataConfig.ENABLE.key(), "true")
> val basePath = "s3a://datalake-hudi"
> val sessions = spark
> .read
> .format("org.apache.hudi")
> .option(DataSourceReadOptions.QUERY_TYPE.key(),
> DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL)
> .option(DataSourceReadOptions.READ_PATHS.key(),
> s"${basePath}/sessions_by_entrydate/entrydate=2021/*/*/*")
> .load()
> sessions.createOrReplaceTempView("sessions") {code}
> Existing tables (COW) have inline clustering on and have many replace commits.
> Logs seem to suggest the delay is in view.AbstractTableFileSystemView
> resetFileGroupsReplaced function or metadata.HoodieBackedTableMetadata
> Also many log messages in AbstractHoodieLogRecordReader
>
> 2021-12-18 23:17:54,056 INFO view.AbstractTableFileSystemView: Took 4118 ms
> to read 136 instants, 9731 replaced file groups
> 2021-12-18 23:37:46,086 INFO log.AbstractHoodieLogRecordReader: Number of
> remaining logblocks to merge 1
> 2021-12-18 23:37:46,090 INFO log.AbstractHoodieLogRecordReader: Reading a
> data block from file
> s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.76_0-20-515
> at instant 20211217035105329
> 2021-12-18 23:37:46,090 INFO log.AbstractHoodieLogRecordReader: Number of
> remaining logblocks to merge 1
> 2021-12-18 23:37:46,094 INFO log.HoodieLogFormatReader: Moving to the next
> reader for logfile
> HoodieLogFile\{pathStr='s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.121_0-57-663',
> fileLen=0}
> 2021-12-18 23:37:46,095 INFO log.AbstractHoodieLogRecordReader: Scanning log
> file
> HoodieLogFile\{pathStr='s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.20_0-35-613',
> fileLen=0}
> 2021-12-18 23:37:46,095 INFO s3a.S3AInputStream: Switching to Random IO seek
> policy
> 2021-12-18 23:37:46,096 INFO log.AbstractHoodieLogRecordReader: Reading a
> data block from file
> s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.62_0-34-377
> at instant 20211217022049877
> 2021-12-18 23:37:46,096 INFO log.AbstractHoodieLogRecordReader: Number of
> remaining logblocks to merge 1
> 2021-12-18 23:37:46,105 INFO log.HoodieLogFormatReader: Moving to the next
> reader for logfile
> HoodieLogFile\{pathStr='s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.86_0-20-362',
> fileLen=0}
> 2021-12-18 23:37:46,109 INFO log.AbstractHoodieLogRecordReader: Scanning log
> file
> HoodieLogFile\{pathStr='s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.121_0-57-663',
> fileLen=0}
> 2021-12-18 23:37:46,109 INFO s3a.S3AInputStream: Switching to Random IO seek
> policy
> 2021-12-18 23:37:46,110 INFO log.HoodieLogFormatReader: Moving to the next
> reader for logfile
> HoodieLogFile\{pathStr='s3a://datalake-hudi/sessions/.hoodie/metadata/files/.files-_20211216144130775001.log.77_0-35-590',
> fileLen=0}
> 2021-12-18 23:37:46,112 INFO log.AbstractHoodieLogRecordReader: Rea
[GitHub] [hudi] hudi-bot commented on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot commented on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012777350 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5220) * f2baeebee296d7f79c325a4d4ca63b50d189733d UNKNOWN * 626cecc34f53634052875ee006cd969b6f0e927b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5223) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot removed a comment on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012745095 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5220) * f2baeebee296d7f79c325a4d4ca63b50d189733d UNKNOWN * 626cecc34f53634052875ee006cd969b6f0e927b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4515: [HUDI-3158] Reduce warn logs in Spark SQL INSERT OVERWRITE
codope commented on a change in pull request #4515:
URL: https://github.com/apache/hudi/pull/4515#discussion_r784507846
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java
##
@@ -156,6 +157,18 @@ public void createNewInstant(HoodieInstant instant) {
createFileInMetaPath(instant.getFileName(), Option.empty(), false);
}
+ public void createRequestedReplaceCommit(String instantTime, String
actionType) {
+try {
+ HoodieInstant instant = new HoodieInstant(State.REQUESTED, actionType,
instantTime);
+ LOG.info("Creating a new instant " + instant);
Review comment:
Is this info log necessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Reopened] (HUDI-3244) UnsupportedOperationException when bulk insert to hudi
[
https://issues.apache.org/jira/browse/HUDI-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
cdmikechen reopened HUDI-3244:
--
> UnsupportedOperationException when bulk insert to hudi
> --
>
> Key: HUDI-3244
> URL: https://issues.apache.org/jira/browse/HUDI-3244
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Reporter: cdmikechen
>Priority: Major
> Fix For: 0.10.1
>
>
> When I bulk insert to hudi, I catch this error.
> {code}
> java.lang.UnsupportedOperationException
> at java.base/java.util.Collections$UnmodifiableMap.put(Unknown Source)
> at
> org.apache.hudi.DataSourceUtils.mayBeOverwriteParquetWriteLegacyFormatProp(DataSourceUtils.java:321)
> at
> org.apache.hudi.spark3.internal.DefaultSource.getTable(DefaultSource.java:59)
> at
> org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:83)
> at
> org.apache.spark.sql.DataFrameWriter.getTable$1(DataFrameWriter.scala:322)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:338)
> at
> org.apache.hudi.HoodieSparkSqlWriter$.bulkInsertAsRow(HoodieSparkSqlWriter.scala:477)
> at
> org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:158)
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:164)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:127)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:126)
> at
> org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:962)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> at
> org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:962)
> at
> org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:414)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:398)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:287)
> at com.syzh.data.LoadRdbmsToHudi.writeHoodie(LoadRdbmsToHudi.scala:326)
> at com.syzh.data.LoadRdbmsToHudi.loadJdbc(LoadRdbmsToHudi.scala:91)
> at
> com.syzh.batch.spark.service.EtlService.ingestionTotalData(EtlService.java:114)
> at
> com.syzh.batch.spark.listener.EtlBatchListener.doTask(EtlBatchListener.java:154)
> at
> com.syzh.batch.spark.listener.EtlBatchListener.action(EtlBatchListener.java:110)
> at
> org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:685)
> at
> org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:679)
> at
> org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
> at
> org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:84)
> at
> org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCache.java:678)
> at
> org.apache.curator.framework.recipes.cache.TreeCache.access$1400(TreeCache.java:69)
> at
> org.apache.curator.framework.recipes.cache.TreeCache$4.run(TreeCache.java:790)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.base/java.util.conc
[GitHub] [hudi] hudi-bot commented on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot commented on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012774985 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 3cb5ff6636414fa2ca81c9740e073f694a696e5e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5218) * 2a3b74e239bf96351eece45e928eac8e4f96d4ff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5222) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot removed a comment on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012728429 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 3cb5ff6636414fa2ca81c9740e073f694a696e5e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5218) * 2a3b74e239bf96351eece45e928eac8e4f96d4ff UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-3236) ALTER TABLE COMMENT old comment gets reverted
[
https://issues.apache.org/jira/browse/HUDI-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-3236:
-
Status: In Progress (was: Open)
> ALTER TABLE COMMENT old comment gets reverted
> -
>
> Key: HUDI-3236
> URL: https://issues.apache.org/jira/browse/HUDI-3236
> Project: Apache Hudi
> Issue Type: Bug
>Affects Versions: 0.10.1
>Reporter: Raymond Xu
>Assignee: Yann Byron
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.11.0
>
>
> {code:sql}
> create table if not exists cow_nonpt_nonpcf_tbl (
> id int,
> name string,
> price double
> ) using hudi
> options (
> type = 'cow',
> primaryKey = 'id'
> );
> insert into cow_nonpt_nonpcf_tbl select 1, 'a1', 20;
> ALTER TABLE cow_nonpt_nonpcf_tbl alter column id comment "primary id";
> DESC cow_nonpt_nonpcf_tbl;
> -- this works fine so far
> ALTER TABLE cow_nonpt_nonpcf_tbl alter column name comment "name column";
> DESC cow_nonpt_nonpcf_tbl;
> -- this saves the comment for name column
> -- but comment for id column was reverted back to NULL
> {code}
> reported while testing on 0.10.1-rc1 (spark 3.0.3, 3.1.2)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [hudi] hudi-bot commented on pull request #4590: Fix flakiness in TestFSUtils
hudi-bot commented on pull request #4590: URL: https://github.com/apache/hudi/pull/4590#issuecomment-1012749384 ## CI report: * 635641ee58bb67296840f49f20eef92f0b2fa642 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5221) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4590: Fix flakiness in TestFSUtils
hudi-bot removed a comment on pull request #4590: URL: https://github.com/apache/hudi/pull/4590#issuecomment-1012731166 ## CI report: * Unknown: [CANCELED](TBD) * 635641ee58bb67296840f49f20eef92f0b2fa642 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5221) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot removed a comment on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012734043 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5220) * f2baeebee296d7f79c325a4d4ca63b50d189733d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot commented on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012745095 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5220) * f2baeebee296d7f79c325a4d4ca63b50d189733d UNKNOWN * 626cecc34f53634052875ee006cd969b6f0e927b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4595: [MINOR] fix delete unused parameter in `TablePathUtils`
hudi-bot removed a comment on pull request #4595: URL: https://github.com/apache/hudi/pull/4595#issuecomment-1012713648 ## CI report: * 677d38039d4a10c39ccb7369d1dd52d4e6e6b435 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5215) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4595: [MINOR] fix delete unused parameter in `TablePathUtils`
hudi-bot commented on pull request #4595: URL: https://github.com/apache/hudi/pull/4595#issuecomment-1012743537 ## CI report: * 677d38039d4a10c39ccb7369d1dd52d4e6e6b435 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5215) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan closed pull request #4598: [MINOR] Flaky TestFSUtils
xushiyan closed pull request #4598: URL: https://github.com/apache/hudi/pull/4598 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4556: [HUDI-3191][Stacked on 4531] Removing duplicating file-listing process w/in Hive's MOR `FIleInputFormat`s
hudi-bot removed a comment on pull request #4556: URL: https://github.com/apache/hudi/pull/4556#issuecomment-1012696524 ## CI report: * 77d11131baabd1c4e3cc2050337daca4df5f6427 UNKNOWN * 3d9c2ae28da858d1e8476052c99391015effb7db UNKNOWN * 31b0669d7b638bd65a17b22a2ceb772f2627512c UNKNOWN * 096e26af39a477ffccbee0edd764493e12e15e71 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5124) * 97dd5ec7c9246f6d9f467b9076b790dd6017c5b4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5213) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4556: [HUDI-3191][Stacked on 4531] Removing duplicating file-listing process w/in Hive's MOR `FIleInputFormat`s
hudi-bot commented on pull request #4556: URL: https://github.com/apache/hudi/pull/4556#issuecomment-1012735569 ## CI report: * 77d11131baabd1c4e3cc2050337daca4df5f6427 UNKNOWN * 3d9c2ae28da858d1e8476052c99391015effb7db UNKNOWN * 31b0669d7b638bd65a17b22a2ceb772f2627512c UNKNOWN * 97dd5ec7c9246f6d9f467b9076b790dd6017c5b4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5213) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot removed a comment on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012732552 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5220) * f2baeebee296d7f79c325a4d4ca63b50d189733d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot commented on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012734043 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5220) * f2baeebee296d7f79c325a4d4ca63b50d189733d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot commented on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012732552 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5220) * f2baeebee296d7f79c325a4d4ca63b50d189733d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot removed a comment on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012728599 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5220) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4590: Fix flakiness in TestFSUtils
hudi-bot removed a comment on pull request #4590: URL: https://github.com/apache/hudi/pull/4590#issuecomment-1012729793 ## CI report: * Unknown: [CANCELED](TBD) * 635641ee58bb67296840f49f20eef92f0b2fa642 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4590: Fix flakiness in TestFSUtils
hudi-bot commented on pull request #4590: URL: https://github.com/apache/hudi/pull/4590#issuecomment-1012731166 ## CI report: * Unknown: [CANCELED](TBD) * 635641ee58bb67296840f49f20eef92f0b2fa642 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5221) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1296) Implement Spark DataSource using range metadata for file/partition pruning
[ https://issues.apache.org/jira/browse/HUDI-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17475930#comment-17475930 ] shibei commented on HUDI-1296: -- [~manojg] [~alexey.kudinkin] [~vinoth] Is there anyone in charge of this issue? If not, can I implement it based on [https://github.com/apache/hudi/pull/4352] ? > Implement Spark DataSource using range metadata for file/partition pruning > -- > > Key: HUDI-1296 > URL: https://issues.apache.org/jira/browse/HUDI-1296 > Project: Apache Hudi > Issue Type: Task > Components: Spark Integration >Affects Versions: 0.9.0 >Reporter: Vinoth Chandar >Assignee: Alexey Kudinkin >Priority: Blocker > Fix For: 0.11.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot removed a comment on pull request #4590: Fix flakiness in TestFSUtils
hudi-bot removed a comment on pull request #4590: URL: https://github.com/apache/hudi/pull/4590#issuecomment-1012680190 ## CI report: * d376f391d5c727a1a2e863ef2dbe3dce9c073c53 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5208) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4590: Fix flakiness in TestFSUtils
hudi-bot commented on pull request #4590: URL: https://github.com/apache/hudi/pull/4590#issuecomment-1012729793 ## CI report: * Unknown: [CANCELED](TBD) * 635641ee58bb67296840f49f20eef92f0b2fa642 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1631) Sort data when creating a new version of file
[ https://issues.apache.org/jira/browse/HUDI-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-1631: Status: In Progress (was: Open) > Sort data when creating a new version of file > - > > Key: HUDI-1631 > URL: https://issues.apache.org/jira/browse/HUDI-1631 > Project: Apache Hudi > Issue Type: Task >Reporter: satish >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > Add option to sort data by specific file when creating new version of file. > Anytime we open a file and write data (whehther to apply updates/add new > records), we want to sort data in the file by specified column(s). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1630) Add partitioner strategy for improving data locality
[ https://issues.apache.org/jira/browse/HUDI-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-1630: Story Points: 4 > Add partitioner strategy for improving data locality > > > Key: HUDI-1630 > URL: https://issues.apache.org/jira/browse/HUDI-1630 > Project: Apache Hudi > Issue Type: Task >Reporter: satish >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > We can use index/metadata information and co-locate records with same value > for specified column(s) in same fileId -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1629) Change partitioner abstraction to implement multiple strategies
[ https://issues.apache.org/jira/browse/HUDI-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-1629: Story Points: 4 > Change partitioner abstraction to implement multiple strategies > --- > > Key: HUDI-1629 > URL: https://issues.apache.org/jira/browse/HUDI-1629 > Project: Apache Hudi > Issue Type: Task >Reporter: satish >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > Existing UpsertPartitioner only considers file sizing to assign > inserts/updates. We also want to consider data locality and other factors. So > change partitioner abstraction to make it easy to implement and plug in other > strategies. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] nsivabalan commented on pull request #4590: Fix flakiness in TestFSUtils
nsivabalan commented on pull request #4590: URL: https://github.com/apache/hudi/pull/4590#issuecomment-1012728863 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1631) Sort data when creating a new version of file
[ https://issues.apache.org/jira/browse/HUDI-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-1631: Story Points: 3 > Sort data when creating a new version of file > - > > Key: HUDI-1631 > URL: https://issues.apache.org/jira/browse/HUDI-1631 > Project: Apache Hudi > Issue Type: Task >Reporter: satish >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > Add option to sort data by specific file when creating new version of file. > Anytime we open a file and write data (whehther to apply updates/add new > records), we want to sort data in the file by specified column(s). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot removed a comment on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot removed a comment on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012727465 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4596: [MINOR] Fix local flaky test in TestFSUtils
hudi-bot commented on pull request #4596: URL: https://github.com/apache/hudi/pull/4596#issuecomment-1012728599 ## CI report: * 4607243c0b3e59f5c110f40b00df1bfc96af7fbc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5220) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot commented on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012728429 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * 3cb5ff6636414fa2ca81c9740e073f694a696e5e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5218) * 2a3b74e239bf96351eece45e928eac8e4f96d4ff UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1631) Sort data when creating a new version of file
[ https://issues.apache.org/jira/browse/HUDI-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-1631: Sprint: Hudi-Sprint-Jan-10 > Sort data when creating a new version of file > - > > Key: HUDI-1631 > URL: https://issues.apache.org/jira/browse/HUDI-1631 > Project: Apache Hudi > Issue Type: Task >Reporter: satish >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > Add option to sort data by specific file when creating new version of file. > Anytime we open a file and write data (whehther to apply updates/add new > records), we want to sort data in the file by specified column(s). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] hudi-bot removed a comment on pull request #4333: [HUDI-431] Adding support for Parquet in MOR `LogBlock`s
hudi-bot removed a comment on pull request #4333: URL: https://github.com/apache/hudi/pull/4333#issuecomment-1012727208 ## CI report: * 286aa8b95627eaaa01114567797186263a830774 UNKNOWN * e722499ee75403ab62f646fdabca1a2c59570164 UNKNOWN * de0d4385394dc5d820964cefc872f099cee7a02b UNKNOWN * 93f3baa443153657ebe212f1c1b453776dc4cc82 UNKNOWN * d7f63eb9541962516366659ec0c2f86157397c4d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5214) * 3cb5ff6636414fa2ca81c9740e073f694a696e5e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5218) * 2a3b74e239bf96351eece45e928eac8e4f96d4ff UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1630) Add partitioner strategy for improving data locality
[ https://issues.apache.org/jira/browse/HUDI-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-1630: Sprint: Hudi-Sprint-Jan-10 > Add partitioner strategy for improving data locality > > > Key: HUDI-1630 > URL: https://issues.apache.org/jira/browse/HUDI-1630 > Project: Apache Hudi > Issue Type: Task >Reporter: satish >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > We can use index/metadata information and co-locate records with same value > for specified column(s) in same fileId -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1629) Change partitioner abstraction to implement multiple strategies
[ https://issues.apache.org/jira/browse/HUDI-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-1629: Sprint: Hudi-Sprint-Jan-10 > Change partitioner abstraction to implement multiple strategies > --- > > Key: HUDI-1629 > URL: https://issues.apache.org/jira/browse/HUDI-1629 > Project: Apache Hudi > Issue Type: Task >Reporter: satish >Assignee: Ethan Guo >Priority: Blocker > Fix For: 0.11.0 > > > Existing UpsertPartitioner only considers file sizing to assign > inserts/updates. We also want to consider data locality and other factors. So > change partitioner abstraction to make it easy to implement and plug in other > strategies. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [hudi] harishraju-govindaraju commented on issue #4597: [SUPPORT] - Hudi Upserts Not working
harishraju-govindaraju commented on issue #4597: URL: https://github.com/apache/hudi/issues/4597#issuecomment-1012727798 Spark version used is 2.4.3, Hudi - hudi-spark-bundle_2.11-0.10.0.jar,s3://bucket001/jars/spark-avro_2.11-2.4.4.jar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4586: [HUDI-1558] Struct Stream Source Support Spark3
hudi-bot commented on pull request #4586: URL: https://github.com/apache/hudi/pull/4586#issuecomment-1012727423 ## CI report: * 2c2c2a5b8dadaaefce68b9d0730d6e18e8bcc0f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5212) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
