[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support
pengzhiwei2018 commented on a change in pull request #2645:
URL: https://github.com/apache/hudi/pull/2645#discussion_r619757590
##
File path: pom.xml
##
@@ -112,6 +112,7 @@
3.0.0
3
+hudi-spark2
Review comment:
As I have describe above, this change should be include in this PR.
##
File path:
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCreateTable.scala
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import scala.collection.JavaConverters._
+import org.apache.hudi.common.model.HoodieRecord
+import org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+import org.apache.spark.sql.types.{DoubleType, IntegerType, LongType,
StringType, StructField}
+
+class TestCreateTable extends TestHoodieSqlBase {
+
+ test("Test Create Managed Hoodie Table") {
+val tableName = generateTableName
+// Create a managed table
+spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ | ) using hudi
+ | options (
+ | primaryKey = 'id',
+ | versionColumn = 'ts'
Review comment:
> can we stick close to the parameter names used in the datasource
write? We need not use the entire property name, but can we use same terms like
precombineColumn=ts instead?
Agree with this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2870: [HUDI-1837] Add optional instant range to log record scanner for log block filtering
codecov-commenter edited a comment on pull request #2870: URL: https://github.com/apache/hudi/pull/2870#issuecomment-825644626 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2870?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2870](https://codecov.io/gh/apache/hudi/pull/2870?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7af9300) into [master](https://codecov.io/gh/apache/hudi/commit/a5789c40673b36d40adab696706acc1446a286f8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a5789c4) will **decrease** coverage by `43.63%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2870?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2870 +/- ## - Coverage 53.00% 9.36% -43.64% + Complexity 3743 48 -3695 Files 488 54 -434 Lines 234351996-21439 Branches 2500 236 -2264 - Hits 12422 187-12235 + Misses 99131796 -8117 + Partials 1100 13 -1087 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.36% <ø> (-60.33%)` | `48.00 <ø> (-325.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2870?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2870/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2870/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2870/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2870/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2870/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <
[jira] [Updated] (HUDI-1837) Add optional instant range to log record scanner for log block filtering
[ https://issues.apache.org/jira/browse/HUDI-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-1837: - Summary: Add optional instant range to log record scanner for log block filtering (was: Add option for max log file size) > Add optional instant range to log record scanner for log block filtering > - > > Key: HUDI-1837 > URL: https://issues.apache.org/jira/browse/HUDI-1837 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support
pengzhiwei2018 commented on a change in pull request #2645:
URL: https://github.com/apache/hudi/pull/2645#discussion_r619743008
##
File path:
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/UuidKeyGenerator.java
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+
+/**
+ * A KeyGenerator which use the uuid as the record key.
+ */
+public class UuidKeyGenerator extends BuiltinKeyGenerator {
Review comment:
Yes, it make sense to me! I prefer to make this in a separate PR because
it need some work to pass the commit time. I have file a JIRA at
[1840](https://issues.apache.org/jira/browse/HUDI-1840)
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##
@@ -59,6 +60,8 @@
public static final String HOODIE_TABLE_VERSION_PROP_NAME =
"hoodie.table.version";
public static final String HOODIE_TABLE_PRECOMBINE_FIELD =
"hoodie.table.precombine.field";
public static final String HOODIE_TABLE_PARTITION_COLUMNS =
"hoodie.table.partition.columns";
+ public static final String HOODIE_TABLE_ROWKEY_FIELDS =
"hoodie.table.rowkey.fields";
+ public static final String HOODIE_TABLE_SCHEMA = "hoodie.table.schema";
Review comment:
make sense to me!
##
File path:
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCreateTable.scala
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import scala.collection.JavaConverters._
+import org.apache.hudi.common.model.HoodieRecord
+import org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+import org.apache.spark.sql.types.{DoubleType, IntegerType, LongType,
StringType, StructField}
+
+class TestCreateTable extends TestHoodieSqlBase {
+
+ test("Test Create Managed Hoodie Table") {
+val tableName = generateTableName
+// Create a managed table
+spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ | ) using hudi
+ | options (
+ | primaryKey = 'id',
+ | versionColumn = 'ts'
Review comment:
yes
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java
##
@@ -192,7 +193,7 @@ protected void initializeIncomingRecordsMap() {
long memoryForMerge =
IOUtils.getMaxMemoryPerPartitionMerge(taskContextSupplier, config.getProps());
LOG.info("MaxMemoryPerPartitionMerge => " + memoryForMerge);
this.keyToNewRecords = new ExternalSpillableMap<>(memoryForMerge,
config.getSpillableMapBasePath(),
- new DefaultSizeEstimator(), new
HoodieRecordSizeEstimator(writerSchema));
+ new DefaultSizeEstimator(), new
HoodieRecordSizeEstimator(inputSchema));
Review comment:
> On the inputSchema, is this always equal to the table's schema?
The `write
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2822: [Hotfix][hudi-sync] Refactor method up to parent-class
codecov-commenter edited a comment on pull request #2822: URL: https://github.com/apache/hudi/pull/2822#issuecomment-821787385 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2822?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2822](https://codecov.io/gh/apache/hudi/pull/2822?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3693571) into [master](https://codecov.io/gh/apache/hudi/commit/ab4a7b0b4afc66d2123c7f63fdab77b925a8a7f1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ab4a7b0) will **decrease** coverage by `52.59%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2822?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2822 +/- ## - Coverage 61.96% 9.36% -52.60% + Complexity 335 48 -287 Files54 54 Lines 19931996+3 Branches235 236+1 - Hits 1235 187 -1048 - Misses 6381796 +1158 + Partials120 13 -107 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiclient | `∅ <ø> (?)` | `0.00 <ø> (?)` | | | hudiutilities | `9.36% <ø> (-52.60%)` | `48.00 <ø> (-287.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2822?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_c
[jira] [Created] (HUDI-1840) Add Commit Time Prefix To The UuidKeyGenerator For Insert Only
pengzhiwei created HUDI-1840: Summary: Add Commit Time Prefix To The UuidKeyGenerator For Insert Only Key: HUDI-1840 URL: https://issues.apache.org/jira/browse/HUDI-1840 Project: Apache Hudi Issue Type: Sub-task Components: Spark Integration Reporter: pengzhiwei Assignee: pengzhiwei Fix For: 0.9.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2822: [Hotfix][hudi-sync] Refactor method up to parent-class
codecov-commenter edited a comment on pull request #2822: URL: https://github.com/apache/hudi/pull/2822#issuecomment-821787385 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2822?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2822](https://codecov.io/gh/apache/hudi/pull/2822?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3693571) into [master](https://codecov.io/gh/apache/hudi/commit/ab4a7b0b4afc66d2123c7f63fdab77b925a8a7f1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ab4a7b0) will **decrease** coverage by `52.59%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2822?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2822 +/- ## - Coverage 61.96% 9.36% -52.60% + Complexity 335 48 -287 Files54 54 Lines 19931996+3 Branches235 236+1 - Hits 1235 187 -1048 - Misses 6381796 +1158 + Partials120 13 -107 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiutilities | `9.36% <ø> (-52.60%)` | `48.00 <ø> (-287.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2822?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2822/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Th
[jira] [Resolved] (HUDI-1591) Implement Spark's FileIndex for Hudi to support queries via Hudi DataSource using non-globbed table path and partition pruning
[ https://issues.apache.org/jira/browse/HUDI-1591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengzhiwei resolved HUDI-1591. -- Resolution: Fixed > Implement Spark's FileIndex for Hudi to support queries via Hudi DataSource > using non-globbed table path and partition pruning > -- > > Key: HUDI-1591 > URL: https://issues.apache.org/jira/browse/HUDI-1591 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Affects Versions: 0.9.0 >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > We have found same problems on query hoodie table on spark: > 1、Users must specify "*" to tell the partition level to spark for the query. > 2、Cannot support partition prune for COW table. > This issue wants to achieve the following goals： > 1、Support No Stars query for hoodie table. > 2、Support partition prune for COW table. > Refer to the documentation for more details about this: [Optimization For > Hudi COW > Query|https://docs.google.com/document/d/1qG014M3VZg3lMswsZv7cYB9Tb0vz8yXgqvlI_Jlnnsc/edit#heading=h.k6ro6dhgwh8y] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support
pengzhiwei2018 commented on a change in pull request #2645:
URL: https://github.com/apache/hudi/pull/2645#discussion_r619739911
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/HoodieLazyInsertIterable.java
##
@@ -42,6 +43,8 @@
public abstract class HoodieLazyInsertIterable
extends LazyIterableIterator, List> {
+ private static final Properties EMPTY_PROPERTIES = new Properties();
Review comment:
The HoodieRecordPayload#getInsertValue required a `Properties`
parameter, so here we must pass a properties object. Move the EMPTY_PROPERTIES
to `CollectionUtils` make sense to me !
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/HoodieLazyInsertIterable.java
##
@@ -42,6 +43,8 @@
public abstract class HoodieLazyInsertIterable
extends LazyIterableIterator, List> {
+ private static final Properties EMPTY_PROPERTIES = new Properties();
Review comment:
The HoodieRecordPayload#getInsertValue required a `Properties`
parameter, so here we must pass a properties object. Moving the
EMPTY_PROPERTIES to `CollectionUtils` make sense to me !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #2719: [HUDI-1721] run_sync_tool support hive3
danny0405 commented on pull request #2719: URL: https://github.com/apache/hudi/pull/2719#issuecomment-826205194 > @vinothchandar I test with hive2.3.8. it's not working. can we break down the one shell to hive2 or hive3 ? > > ls: cannot access /opt/module/apache-hive-2.3.8-bin/lib/calcite-*.jar: No such file or directory Maybe we can add a cmd config option to switch hive2 and hive3 jars loading. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
yanghua commented on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826203297 > @yanghua - I have added the unit tests, Can you please review and merge? thanks for addressing my concerns. @vinothchandar will take over this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1811) Azure CI connection refused error when init HoodieRealtimeRecordReader
[ https://issues.apache.org/jira/browse/HUDI-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1811: - Status: In Progress (was: Open) > Azure CI connection refused error when init HoodieRealtimeRecordReader > -- > > Key: HUDI-1811 > URL: https://issues.apache.org/jira/browse/HUDI-1811 > Project: Apache Hudi > Issue Type: Sub-task > Components: Testing >Reporter: Raymond Xu >Assignee: Raymond Xu >Priority: Blocker > Labels: pull-request-available > Fix For: 0.9.0 > > > Failed job > https://dev.azure.com/xushiyan/apache-hudi-ci/_build/results?buildId=35&view=logs&j=d3721143-1417-5e3d-cf04-c39c0756eab9&t=a7783b9f-edd1-5bb0-5301-1955fdbfb2c4 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
hudi-bot edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826138628 ## CI report: * bb3bde912af4cb68827a7d7d037e8e79d6e32036 Travis: [FAILURE](https://travis-ci.com/github/apachehudi-ci/hudi-branch-ci/builds/224074717) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
codecov-commenter edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826139460 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
codecov-commenter edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826139460 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2873](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb3bde9) into [master](https://codecov.io/gh/apache/hudi/commit/a5789c40673b36d40adab696706acc1446a286f8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a5789c4) will **increase** coverage by `16.68%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2873 +/- ## = + Coverage 53.00% 69.68% +16.68% + Complexity 3743 373 -3370 = Files 488 54 -434 Lines 23435 1996-21439 Branches 2500 236 -2264 = - Hits 12422 1391-11031 + Misses 9913 473 -9440 + Partials 1100 132 -968 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.68% <ø> (ø)` | `373.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...pache/hudi/common/fs/FailSafeConsistencyGuard.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0ZhaWxTYWZlQ29uc2lzdGVuY3lHdWFyZC5qYXZh) | | | | | [.../common/util/queue/FunctionBasedQueueProducer.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvcXVldWUvRnVuY3Rpb25CYXNlZFF1ZXVlUHJvZHVjZXIuamF2YQ==) | | | | | [.../common/table/log/block/HoodieLogBlockVersion.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9ibG9jay9Ib29kaWVMb2dCbG9ja1ZlcnNpb24uamF2YQ==) | | | | | [...java/org/apache/hudi/common/util/NumericUtils.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvTnVtZXJpY1V0aWxzLmphdmE=) | | | | | [...rc/main/java/org/apache/hudi/ApiMaturityLevel.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvQXBpTWF0dXJpdHlMZXZlbC5qYXZh) | | | | | [.../versioning/clean/CleanPlanV2MigrationHandler.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_cam
[GitHub] [hudi] vingov commented on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
vingov commented on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826157848 @yanghua - I have added the unit tests, Can you please review and merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
codecov-commenter edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826139460 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2873](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb3bde9) into [master](https://codecov.io/gh/apache/hudi/commit/a5789c40673b36d40adab696706acc1446a286f8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a5789c4) will **decrease** coverage by `43.63%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2873 +/- ## - Coverage 53.00% 9.36% -43.64% + Complexity 3743 48 -3695 Files 488 54 -434 Lines 234351996-21439 Branches 2500 236 -2264 - Hits 12422 187-12235 + Misses 99131796 -8117 + Partials 1100 13 -1087 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.36% <ø> (-60.33%)` | `48.00 <ø> (-325.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <
[GitHub] [hudi] hudi-bot edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
hudi-bot edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826138628 ## CI report: * fb7b1ee06b79fea7215b5fa4bf8117e69ad989a1 Travis: [FAILURE](https://travis-ci.com/github/apachehudi-ci/hudi-branch-ci/builds/224071689) * bb3bde912af4cb68827a7d7d037e8e79d6e32036 Travis: [PENDING](https://travis-ci.com/github/apachehudi-ci/hudi-branch-ci/builds/224074717) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
hudi-bot edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826138628 ## CI report: * fb7b1ee06b79fea7215b5fa4bf8117e69ad989a1 Travis: [FAILURE](https://travis-ci.com/github/apachehudi-ci/hudi-branch-ci/builds/224071689) * bb3bde912af4cb68827a7d7d037e8e79d6e32036 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
hudi-bot edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826138628 ## CI report: * fb7b1ee06b79fea7215b5fa4bf8117e69ad989a1 Travis: [FAILURE](https://travis-ci.com/github/apachehudi-ci/hudi-branch-ci/builds/224071689) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
xushiyan commented on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826155811 Azure test passed (skip integ test) https://dev.azure.com/xushiyan/apache-hudi-ci/_build/results?buildId=52&view=results -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan removed a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
xushiyan removed a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826155635 @hudi-bot run travis -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
hudi-bot edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826138628 ## CI report: * fb7b1ee06b79fea7215b5fa4bf8117e69ad989a1 Travis: [PENDING](https://travis-ci.com/github/apachehudi-ci/hudi-branch-ci/builds/224071689) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
xushiyan commented on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826155635 @hudi-bot run travis -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
codecov-commenter edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826139460 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2873](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb7b1ee) into [master](https://codecov.io/gh/apache/hudi/commit/a5789c40673b36d40adab696706acc1446a286f8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a5789c4) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@Coverage Diff@@ ## master#2873 +/- ## = Coverage 53.00% 53.00% Complexity 3743 3743 = Files 488 488 Lines 2343523435 Branches 2500 2500 = Hits 1242212422 Misses 9913 9913 Partials 1100 1100 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (ø)` | `220.00 <ø> (ø)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.65% <ø> (ø)` | `1976.00 <ø> (ø)` | | | hudiflink | `59.11% <ø> (ø)` | `535.00 <ø> (ø)` | | | hudihadoopmr | `33.33% <ø> (ø)` | `198.00 <ø> (ø)` | | | hudisparkdatasource | `73.33% <ø> (ø)` | `237.00 <ø> (ø)` | | | hudisync | `46.10% <ø> (ø)` | `142.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `69.68% <ø> (ø)` | `373.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: DO NOT MERGE (Azure CI) test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * 9831a6c50e9f49f8a71c02fc6ac50ae1446f7c1f UNKNOWN * a569dbe9409910fbb83b3764b300574c0e52612e Azure: [FAILURE](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=142) * e6e9f1f1554a1474dd6c20338215030cad23a2e0 UNKNOWN * 2a6690a256c8cd8efe9ed2b1984b896fb27ef077 UNKNOWN * d8b7cca55e057a52a2e229d81e8cb52b60dc275f UNKNOWN * 3bce301333cc78194d13a702598b46e04fe9f85f UNKNOWN * f07f345baa450f3fec7eab59caa76b0fbda1e132 UNKNOWN * 869d2ce3fad330af93c1bb3b576824f519c6e68b UNKNOWN * fa86907f7522bc8dbe512d48b5a87e4a6b13f035 UNKNOWN * 4ebe53016ce3e0648992dbe14d04f71a92f116e6 UNKNOWN * 682ae9985f591f6d0c30ee2ef9b159403c1e46de UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
codecov-commenter edited a comment on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826048129 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2747](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f089324) into [master](https://codecov.io/gh/apache/hudi/commit/aa0da72c59cb3764205f90b025b24d1640727795?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa0da72) will **increase** coverage by `0.99%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2747 +/- ## + Coverage 52.06% 53.05% +0.99% - Complexity 3625 3748 +123 Files 479 489 +10 Lines 2280423458 +654 Branches 2415 2503 +88 + Hits 1187212445 +573 - Misses 9907 9913 +6 - Partials 1025 1100 +75 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (+2.52%)` | `220.00 <ø> (+25.00)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.67% <ø> (-0.31%)` | `1976.00 <ø> (+1.00)` | :arrow_down: | | hudiflink | `59.11% <ø> (+3.09%)` | `535.00 <ø> (+33.00)` | | | hudihadoopmr | `33.33% <ø> (-0.12%)` | `198.00 <ø> (+1.00)` | :arrow_down: | | hudisparkdatasource | `73.33% <ø> (+2.46%)` | `237.00 <ø> (+42.00)` | | | hudisync | `46.10% <ø> (+0.63%)` | `142.00 <ø> (+14.00)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `69.93% <91.30%> (+0.20%)` | `378.00 <5.00> (+7.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...i/utilities/transform/SqlFileBasedTransformer.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9TcWxGaWxlQmFzZWRUcmFuc2Zvcm1lci5qYXZh) | `91.30% <91.30%> (ø)` | `5.00 <5.00> (?)` | | | [.../apache/hudi/sink/compact/CompactionPlanEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5FdmVudC5qYXZh) | `50.00% <0.00%> (-50.00%)` | `3.00% <0.00%> (ø%)` | | | [...pache/hudi/sink/compact/CompactionCommitEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvbkNvbW1pdEV2ZW50LmphdmE=) | `43.75% <0.00%> (-43.75%)` | `3.00% <0.00%> (ø%)` | | | [...ache/hudi/sink/compact/CompactionPlanOperator.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5PcGVyYXRvci5qYXZh) | `58.00% <0.00%> (-21.07%)` | `8.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/sink/partitioner/BucketAssigners.java](https:
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
codecov-commenter edited a comment on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826048129 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2747](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f089324) into [master](https://codecov.io/gh/apache/hudi/commit/aa0da72c59cb3764205f90b025b24d1640727795?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa0da72) will **increase** coverage by `0.99%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2747 +/- ## + Coverage 52.06% 53.05% +0.99% - Complexity 3625 3748 +123 Files 479 489 +10 Lines 2280423458 +654 Branches 2415 2503 +88 + Hits 1187212445 +573 - Misses 9907 9913 +6 - Partials 1025 1100 +75 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (+2.52%)` | `220.00 <ø> (+25.00)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.67% <ø> (-0.31%)` | `1976.00 <ø> (+1.00)` | :arrow_down: | | hudiflink | `59.11% <ø> (+3.09%)` | `535.00 <ø> (+33.00)` | | | hudihadoopmr | `33.33% <ø> (-0.12%)` | `198.00 <ø> (+1.00)` | :arrow_down: | | hudisparkdatasource | `73.33% <ø> (+2.46%)` | `237.00 <ø> (+42.00)` | | | hudisync | `46.10% <ø> (+0.63%)` | `142.00 <ø> (+14.00)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `69.93% <91.30%> (+0.20%)` | `378.00 <5.00> (+7.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...i/utilities/transform/SqlFileBasedTransformer.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9TcWxGaWxlQmFzZWRUcmFuc2Zvcm1lci5qYXZh) | `91.30% <91.30%> (ø)` | `5.00 <5.00> (?)` | | | [.../apache/hudi/sink/compact/CompactionPlanEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5FdmVudC5qYXZh) | `50.00% <0.00%> (-50.00%)` | `3.00% <0.00%> (ø%)` | | | [...pache/hudi/sink/compact/CompactionCommitEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvbkNvbW1pdEV2ZW50LmphdmE=) | `43.75% <0.00%> (-43.75%)` | `3.00% <0.00%> (ø%)` | | | [...ache/hudi/sink/compact/CompactionPlanOperator.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5PcGVyYXRvci5qYXZh) | `58.00% <0.00%> (-21.07%)` | `8.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/sink/partitioner/BucketAssigners.java](https:
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
codecov-commenter edited a comment on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826048129 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2747](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f089324) into [master](https://codecov.io/gh/apache/hudi/commit/aa0da72c59cb3764205f90b025b24d1640727795?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa0da72) will **decrease** coverage by `0.68%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2747 +/- ## - Coverage 52.06% 51.37% -0.69% + Complexity 3625 3307 -318 Files 479 426 -53 Lines 2280419969-2835 Branches 2415 2083 -332 - Hits 1187210259-1613 + Misses 9907 8856-1051 + Partials 1025 854 -171 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (+2.52%)` | `220.00 <ø> (+25.00)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.67% <ø> (-0.31%)` | `1976.00 <ø> (+1.00)` | :arrow_down: | | hudiflink | `59.11% <ø> (+3.09%)` | `535.00 <ø> (+33.00)` | | | hudihadoopmr | `33.33% <ø> (-0.12%)` | `198.00 <ø> (+1.00)` | :arrow_down: | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.93% <91.30%> (+0.20%)` | `378.00 <5.00> (+7.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...i/utilities/transform/SqlFileBasedTransformer.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9TcWxGaWxlQmFzZWRUcmFuc2Zvcm1lci5qYXZh) | `91.30% <91.30%> (ø)` | `5.00 <5.00> (?)` | | | [.../apache/hudi/sink/compact/CompactionPlanEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5FdmVudC5qYXZh) | `50.00% <0.00%> (-50.00%)` | `3.00% <0.00%> (ø%)` | | | [...pache/hudi/sink/compact/CompactionCommitEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvbkNvbW1pdEV2ZW50LmphdmE=) | `43.75% <0.00%> (-43.75%)` | `3.00% <0.00%> (ø%)` | | | [...ache/hudi/sink/compact/CompactionPlanOperator.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5PcGVyYXRvci5qYXZh) | `58.00% <0.00%> (-21.07%)` | `8.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/sink/partitioner/BucketAssigners.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&u
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
codecov-commenter edited a comment on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826048129 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2747](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f089324) into [master](https://codecov.io/gh/apache/hudi/commit/aa0da72c59cb3764205f90b025b24d1640727795?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa0da72) will **increase** coverage by `17.87%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2747 +/- ## = + Coverage 52.06% 69.93% +17.87% + Complexity 3625 378 -3247 = Files 479 55 -424 Lines 22804 2019-20785 Branches 2415 239 -2176 = - Hits 11872 1412-10460 + Misses 9907 474 -9433 + Partials 1025 133 -892 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.93% <91.30%> (+0.20%)` | `378.00 <5.00> (+7.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...i/utilities/transform/SqlFileBasedTransformer.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9TcWxGaWxlQmFzZWRUcmFuc2Zvcm1lci5qYXZh) | `91.30% <91.30%> (ø)` | `5.00 <5.00> (?)` | | | [...org/apache/hudi/utilities/HoodieClusteringJob.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZUNsdXN0ZXJpbmdKb2IuamF2YQ==) | `62.50% <0.00%> (-2.72%)` | `9.00% <0.00%> (ø%)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.08% <0.00%> (-0.30%)` | `55.00% <0.00%> (ø%)` | | | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <0.00%> (ø)` | `18.00% <0.00%> (ø%)` | | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3Jn
[jira] [Updated] (HUDI-1811) Azure CI connection refused error when init HoodieRealtimeRecordReader
[ https://issues.apache.org/jira/browse/HUDI-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1811: - Labels: pull-request-available (was: ) > Azure CI connection refused error when init HoodieRealtimeRecordReader > -- > > Key: HUDI-1811 > URL: https://issues.apache.org/jira/browse/HUDI-1811 > Project: Apache Hudi > Issue Type: Sub-task > Components: Testing >Reporter: Raymond Xu >Assignee: Raymond Xu >Priority: Blocker > Labels: pull-request-available > Fix For: 0.9.0 > > > Failed job > https://dev.azure.com/xushiyan/apache-hudi-ci/_build/results?buildId=35&view=logs&j=d3721143-1417-5e3d-cf04-c39c0756eab9&t=a7783b9f-edd1-5bb0-5301-1955fdbfb2c4 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
hudi-bot edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826138628 ## CI report: * fb7b1ee06b79fea7215b5fa4bf8117e69ad989a1 Travis: [FAILURE](https://travis-ci.com/github/apachehudi-ci/hudi-branch-ci/builds/224071689) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
codecov-commenter edited a comment on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826048129 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2747](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f089324) into [master](https://codecov.io/gh/apache/hudi/commit/aa0da72c59cb3764205f90b025b24d1640727795?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa0da72) will **decrease** coverage by `42.79%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2747 +/- ## - Coverage 52.06% 9.26% -42.80% + Complexity 3625 48 -3577 Files 479 55 -424 Lines 228042019-20785 Branches 2415 239 -2176 - Hits 11872 187-11685 + Misses 99071819 -8088 + Partials 1025 13 -1012 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.26% <0.00%> (-60.48%)` | `48.00 <0.00> (-323.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...i/utilities/transform/SqlFileBasedTransformer.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9TcWxGaWxlQmFzZWRUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYX
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
codecov-commenter edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826139460 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
codecov-commenter edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826139460 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2873](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb7b1ee) into [master](https://codecov.io/gh/apache/hudi/commit/a5789c40673b36d40adab696706acc1446a286f8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a5789c4) will **increase** coverage by `16.68%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2873 +/- ## = + Coverage 53.00% 69.68% +16.68% + Complexity 3743 373 -3370 = Files 488 54 -434 Lines 23435 1996-21439 Branches 2500 236 -2264 = - Hits 12422 1391-11031 + Misses 9913 473 -9440 + Partials 1100 132 -968 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.68% <ø> (ø)` | `373.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...apache/hudi/common/engine/HoodieEngineContext.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2VuZ2luZS9Ib29kaWVFbmdpbmVDb250ZXh0LmphdmE=) | | | | | [...udi/timeline/service/handlers/BaseFileHandler.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS10aW1lbGluZS1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RpbWVsaW5lL3NlcnZpY2UvaGFuZGxlcnMvQmFzZUZpbGVIYW5kbGVyLmphdmE=) | | | | | [...java/org/apache/hudi/hive/util/HiveSchemaUtil.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlU2NoZW1hVXRpbC5qYXZh) | | | | | [...on/table/timeline/versioning/MetadataMigrator.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL3ZlcnNpb25pbmcvTWV0YWRhdGFNaWdyYXRvci5qYXZh) | | | | | [...apache/hudi/common/util/collection/ArrayUtils.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvY29sbGVjdGlvbi9BcnJheVV0aWxzLmphdmE=) | | | | | [...i/common/table/view/HoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=re
[GitHub] [hudi] xushiyan commented on a change in pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
xushiyan commented on a change in pull request #2873:
URL: https://github.com/apache/hudi/pull/2873#discussion_r619699121
##
File path:
hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##
@@ -237,13 +231,13 @@ public void testUnMergedReader() throws Exception {
// create a split with baseFile (parquet file written earlier) and new log
file(s)
String logFilePath = writer.getLogFile().getPath().toString();
HoodieRealtimeFileSplit split = new HoodieRealtimeFileSplit(
-new FileSplit(new Path(partitionDir + "/fileid0_1-0-1_" + instantTime
+ ".parquet"), 0, 1, jobConf),
-basePath.toString(), Collections.singletonList(logFilePath),
newCommitTime);
+new FileSplit(new Path(partitionDir + "/fileid0_1-0-1_" + instantTime
+ ".parquet"), 0, 1, baseJobConf),
+basePath.toUri().toString(), Collections.singletonList(logFilePath),
newCommitTime);
Review comment:
i could spend more time digging into Azure env around this problem with
must having scheme, but given this is a blocker, shall we move on and keep a
note about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
xushiyan commented on a change in pull request #2873:
URL: https://github.com/apache/hudi/pull/2873#discussion_r619698467
##
File path:
hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##
@@ -237,13 +231,13 @@ public void testUnMergedReader() throws Exception {
// create a split with baseFile (parquet file written earlier) and new log
file(s)
String logFilePath = writer.getLogFile().getPath().toString();
HoodieRealtimeFileSplit split = new HoodieRealtimeFileSplit(
-new FileSplit(new Path(partitionDir + "/fileid0_1-0-1_" + instantTime
+ ".parquet"), 0, 1, jobConf),
-basePath.toString(), Collections.singletonList(logFilePath),
newCommitTime);
+new FileSplit(new Path(partitionDir + "/fileid0_1-0-1_" + instantTime
+ ".parquet"), 0, 1, baseJobConf),
+basePath.toUri().toString(), Collections.singletonList(logFilePath),
newCommitTime);
// create a RecordReader to be used by HoodieRealtimeRecordReader
RecordReader reader = new
MapredParquetInputFormat().getRecordReader(
-new FileSplit(split.getPath(), 0, fs.getLength(split.getPath()),
(String[]) null), jobConf, null);
-JobConf jobConf = new JobConf();
+new FileSplit(split.getPath(), 0, fs.getLength(split.getPath()),
(String[]) null), baseJobConf, null);
+JobConf jobConf = new JobConf(baseJobConf);
Review comment:
@vinothchandar this `jobConf` is needed for setting some hive col
properties via `setHiveColumnNameProps()`
##
File path:
hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java
##
@@ -237,13 +231,13 @@ public void testUnMergedReader() throws Exception {
// create a split with baseFile (parquet file written earlier) and new log
file(s)
String logFilePath = writer.getLogFile().getPath().toString();
HoodieRealtimeFileSplit split = new HoodieRealtimeFileSplit(
-new FileSplit(new Path(partitionDir + "/fileid0_1-0-1_" + instantTime
+ ".parquet"), 0, 1, jobConf),
-basePath.toString(), Collections.singletonList(logFilePath),
newCommitTime);
+new FileSplit(new Path(partitionDir + "/fileid0_1-0-1_" + instantTime
+ ".parquet"), 0, 1, baseJobConf),
+basePath.toUri().toString(), Collections.singletonList(logFilePath),
newCommitTime);
Review comment:
@vinothchandar Tests are passing now. the issue is isolated with this
test case. do you concern about this change masking the issue with Azure
environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
codecov-commenter commented on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826139460 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2873](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb7b1ee) into [master](https://codecov.io/gh/apache/hudi/commit/a5789c40673b36d40adab696706acc1446a286f8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a5789c4) will **decrease** coverage by `43.63%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #2873 +/- ## - Coverage 53.00% 9.36% -43.64% + Complexity 3743 48 -3695 Files 488 54 -434 Lines 234351996-21439 Branches 2500 236 -2264 - Hits 12422 187-12235 + Misses 99131796 -8117 + Partials 1100 13 -1087 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.36% <ø> (-60.33%)` | `48.00 <ø> (-325.00)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%>
[GitHub] [hudi] hudi-bot edited a comment on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
hudi-bot edited a comment on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826138628 ## CI report: * fb7b1ee06b79fea7215b5fa4bf8117e69ad989a1 Travis: [PENDING](https://travis-ci.com/github/apachehudi-ci/hudi-branch-ci/builds/224071689) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
hudi-bot commented on pull request #2873: URL: https://github.com/apache/hudi/pull/2873#issuecomment-826138628 ## CI report: * fb7b1ee06b79fea7215b5fa4bf8117e69ad989a1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan opened a new pull request #2873: [HUDI-1811] Fix TestHoodieRealtimeRecordReader (Azure CI)
xushiyan opened a new pull request #2873: URL: https://github.com/apache/hudi/pull/2873 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: DO NOT MERGE (Azure CI) test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * 9831a6c50e9f49f8a71c02fc6ac50ae1446f7c1f UNKNOWN * a569dbe9409910fbb83b3764b300574c0e52612e Azure: [FAILURE](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=142) * e6e9f1f1554a1474dd6c20338215030cad23a2e0 UNKNOWN * 2a6690a256c8cd8efe9ed2b1984b896fb27ef077 UNKNOWN * d8b7cca55e057a52a2e229d81e8cb52b60dc275f UNKNOWN * 3bce301333cc78194d13a702598b46e04fe9f85f UNKNOWN * f07f345baa450f3fec7eab59caa76b0fbda1e132 UNKNOWN * 869d2ce3fad330af93c1bb3b576824f519c6e68b UNKNOWN * fa86907f7522bc8dbe512d48b5a87e4a6b13f035 UNKNOWN * 4ebe53016ce3e0648992dbe14d04f71a92f116e6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
codecov-commenter edited a comment on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826048129 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2747](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b47c0bd) into [master](https://codecov.io/gh/apache/hudi/commit/aa0da72c59cb3764205f90b025b24d1640727795?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa0da72) will **decrease** coverage by `0.58%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2747 +/- ## - Coverage 52.06% 51.47% -0.59% + Complexity 3625 3371 -254 Files 479 434 -45 Lines 2280421439-1365 Branches 2415 2264 -151 - Hits 1187211036 -836 + Misses 9907 9436 -471 + Partials 1025 967 -58 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (+2.52%)` | `220.00 <ø> (+25.00)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.70% <ø> (-0.28%)` | `1977.00 <ø> (+2.00)` | :arrow_down: | | hudiflink | `59.11% <ø> (+3.09%)` | `535.00 <ø> (+33.00)` | | | hudihadoopmr | `33.33% <ø> (-0.12%)` | `198.00 <ø> (+1.00)` | :arrow_down: | | hudisparkdatasource | `73.33% <ø> (+2.46%)` | `237.00 <ø> (+42.00)` | | | hudisync | `46.10% <ø> (+0.63%)` | `142.00 <ø> (+14.00)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `?` | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../apache/hudi/sink/compact/CompactionPlanEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5FdmVudC5qYXZh) | `50.00% <0.00%> (-50.00%)` | `3.00% <0.00%> (ø%)` | | | [...pache/hudi/sink/compact/CompactionCommitEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvbkNvbW1pdEV2ZW50LmphdmE=) | `43.75% <0.00%> (-43.75%)` | `3.00% <0.00%> (ø%)` | | | [...ache/hudi/sink/compact/CompactionPlanOperator.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5PcGVyYXRvci5qYXZh) | `58.00% <0.00%> (-21.07%)` | `8.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/sink/partitioner/BucketAssigners.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL0J1Y2tldEFzc2lnbmVycy5qYXZh) | `33.33% <0.00%> (-16.67%)` | `2.00% <0.00%> (ø%)` | | | [.../org/apache/hudi/sink/utils/NonThrownExecutor.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
codecov-commenter edited a comment on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826048129 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2747](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b47c0bd) into [master](https://codecov.io/gh/apache/hudi/commit/aa0da72c59cb3764205f90b025b24d1640727795?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa0da72) will **decrease** coverage by `0.54%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2747 +/- ## - Coverage 52.06% 51.51% -0.55% + Complexity 3625 3167 -458 Files 479 410 -69 Lines 2280419769-3035 Branches 2415 2097 -318 - Hits 1187210184-1688 + Misses 9907 8700-1207 + Partials 1025 885 -140 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (+2.52%)` | `220.00 <ø> (+25.00)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.70% <ø> (-0.28%)` | `1977.00 <ø> (+2.00)` | :arrow_down: | | hudiflink | `59.11% <ø> (+3.09%)` | `535.00 <ø> (+33.00)` | | | hudihadoopmr | `33.33% <ø> (-0.12%)` | `198.00 <ø> (+1.00)` | :arrow_down: | | hudisparkdatasource | `73.33% <ø> (+2.46%)` | `237.00 <ø> (+42.00)` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `?` | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../apache/hudi/sink/compact/CompactionPlanEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5FdmVudC5qYXZh) | `50.00% <0.00%> (-50.00%)` | `3.00% <0.00%> (ø%)` | | | [...pache/hudi/sink/compact/CompactionCommitEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvbkNvbW1pdEV2ZW50LmphdmE=) | `43.75% <0.00%> (-43.75%)` | `3.00% <0.00%> (ø%)` | | | [...ache/hudi/sink/compact/CompactionPlanOperator.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5PcGVyYXRvci5qYXZh) | `58.00% <0.00%> (-21.07%)` | `8.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/sink/partitioner/BucketAssigners.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL0J1Y2tldEFzc2lnbmVycy5qYXZh) | `33.33% <0.00%> (-16.67%)` | `2.00% <0.00%> (ø%)` | | | [.../org/apache/hudi/sink/utils/NonThrownExecutor.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm
[GitHub] [hudi] vinothchandar commented on a change in pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support
vinothchandar commented on a change in pull request #2645:
URL: https://github.com/apache/hudi/pull/2645#discussion_r619602717
##
File path:
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCreateTable.scala
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import scala.collection.JavaConverters._
+import org.apache.hudi.common.model.HoodieRecord
+import org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+import org.apache.spark.sql.types.{DoubleType, IntegerType, LongType,
StringType, StructField}
+
+class TestCreateTable extends TestHoodieSqlBase {
+
+ test("Test Create Managed Hoodie Table") {
+val tableName = generateTableName
+// Create a managed table
+spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ | ) using hudi
+ | options (
+ | primaryKey = 'id',
+ | versionColumn = 'ts'
Review comment:
this is the precombine field right?
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/payload/ExpressionPayload.scala
##
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.payload
+
+import java.util.{Base64, Properties}
+import java.util.concurrent.Callable
+
+import scala.collection.JavaConverters._
+import com.google.common.cache.CacheBuilder
+import org.apache.avro.Schema
+import org.apache.avro.generic.{GenericData, GenericRecord, IndexedRecord}
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.avro.HoodieAvroUtils
+import org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro
+import org.apache.hudi.common.model.{DefaultHoodieRecordPayload, HoodieRecord}
+import org.apache.hudi.common.util.{ValidationUtils, Option => HOption}
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.io.HoodieWriteHandle
+import org.apache.hudi.sql.IExpressionEvaluator
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.hudi.SerDeUtils
+import org.apache.spark.sql.hudi.command.payload.ExpressionPayload.getEvaluator
+import org.apache.spark.unsafe.types.UTF8String
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * A HoodieRecordPayload for MergeIntoHoodieTableCommand.
+ * It will execute the condition and assignments expression in the
+ * match and not-match actions and compute the final record to write.
+ *
+ * If there is no condition match the record, ExpressionPayload will return
+ * a HoodieWriteHandle.IGNORE_RECORD, and the write handles will ignore this
record.
+ */
+class ExpressionPayload(record: GenericRecord,
+orderingVal: Comparable[_])
+ extends DefaultHoodieRecordPayload(record, orderingVal) {
+
+ def this(recordOpt: HOption[GenericRecord]) {
+this(recordOpt.orElse(null), 0)
+ }
+
+ /**
+ * The schema of this table.
+ */
+ private var writeSchema: Schema = _
+
+ override def combineAndGetUpdateValue(currentValue: IndexedRecord,
+schema: Schema):
HOption[IndexedRe
[GitHub] [hudi] codecov-commenter edited a comment on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
codecov-commenter edited a comment on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826048129 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2747](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ef55cd7) into [master](https://codecov.io/gh/apache/hudi/commit/aa0da72c59cb3764205f90b025b24d1640727795?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa0da72) will **decrease** coverage by `0.59%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2747 +/- ## - Coverage 52.06% 51.46% -0.60% + Complexity 3625 3370 -255 Files 479 434 -45 Lines 2280421439-1365 Branches 2415 2264 -151 - Hits 1187211033 -839 + Misses 9907 9439 -468 + Partials 1025 967 -58 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (+2.52%)` | `220.00 <ø> (+25.00)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.67% <ø> (-0.31%)` | `1976.00 <ø> (+1.00)` | :arrow_down: | | hudiflink | `59.11% <ø> (+3.09%)` | `535.00 <ø> (+33.00)` | | | hudihadoopmr | `33.33% <ø> (-0.12%)` | `198.00 <ø> (+1.00)` | :arrow_down: | | hudisparkdatasource | `73.33% <ø> (+2.46%)` | `237.00 <ø> (+42.00)` | | | hudisync | `46.10% <ø> (+0.63%)` | `142.00 <ø> (+14.00)` | | | huditimelineservice | `64.36% <ø> (ø)` | `62.00 <ø> (ø)` | | | hudiutilities | `?` | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../apache/hudi/sink/compact/CompactionPlanEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5FdmVudC5qYXZh) | `50.00% <0.00%> (-50.00%)` | `3.00% <0.00%> (ø%)` | | | [...pache/hudi/sink/compact/CompactionCommitEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvbkNvbW1pdEV2ZW50LmphdmE=) | `43.75% <0.00%> (-43.75%)` | `3.00% <0.00%> (ø%)` | | | [...ache/hudi/sink/compact/CompactionPlanOperator.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5PcGVyYXRvci5qYXZh) | `58.00% <0.00%> (-21.07%)` | `8.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/sink/partitioner/BucketAssigners.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL0J1Y2tldEFzc2lnbmVycy5qYXZh) | `33.33% <0.00%> (-16.67%)` | `2.00% <0.00%> (ø%)` | | | [.../org/apache/hudi/sink/utils/NonThrownExecutor.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&
[GitHub] [hudi] codecov-commenter commented on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer
codecov-commenter commented on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-826048129 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#2747](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ef55cd7) into [master](https://codecov.io/gh/apache/hudi/commit/aa0da72c59cb3764205f90b025b24d1640727795?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (aa0da72) will **decrease** coverage by `0.87%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#2747 +/- ## - Coverage 52.06% 51.18% -0.88% + Complexity 3625 3308 -317 Files 479 428 -51 Lines 2280420990-1814 Branches 2415 2240 -175 - Hits 1187210744-1128 + Misses 9907 9298 -609 + Partials 1025 948 -77 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `39.53% <ø> (+2.52%)` | `220.00 <ø> (+25.00)` | | | hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | | | hudicommon | `50.67% <ø> (-0.31%)` | `1976.00 <ø> (+1.00)` | :arrow_down: | | hudiflink | `59.11% <ø> (+3.09%)` | `535.00 <ø> (+33.00)` | | | hudihadoopmr | `33.33% <ø> (-0.12%)` | `198.00 <ø> (+1.00)` | :arrow_down: | | hudisparkdatasource | `73.33% <ø> (+2.46%)` | `237.00 <ø> (+42.00)` | | | hudisync | `46.10% <ø> (+0.63%)` | `142.00 <ø> (+14.00)` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `?` | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2747?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../apache/hudi/sink/compact/CompactionPlanEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5FdmVudC5qYXZh) | `50.00% <0.00%> (-50.00%)` | `3.00% <0.00%> (ø%)` | | | [...pache/hudi/sink/compact/CompactionCommitEvent.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvbkNvbW1pdEV2ZW50LmphdmE=) | `43.75% <0.00%> (-43.75%)` | `3.00% <0.00%> (ø%)` | | | [...ache/hudi/sink/compact/CompactionPlanOperator.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL2NvbXBhY3QvQ29tcGFjdGlvblBsYW5PcGVyYXRvci5qYXZh) | `58.00% <0.00%> (-21.07%)` | `8.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/sink/partitioner/BucketAssigners.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zaW5rL3BhcnRpdGlvbmVyL0J1Y2tldEFzc2lnbmVycy5qYXZh) | `33.33% <0.00%> (-16.67%)` | `2.00% <0.00%> (ø%)` | | | [.../org/apache/hudi/sink/utils/NonThrownExecutor.java](https://codecov.io/gh/apache/hudi/pull/2747/diff?src=pr&el=tree&utm_medium=referral&utm_source=g
[GitHub] [hudi] vinothchandar commented on a change in pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support
vinothchandar commented on a change in pull request #2645:
URL: https://github.com/apache/hudi/pull/2645#discussion_r619592740
##
File path: pom.xml
##
@@ -112,6 +112,7 @@
3.0.0
3
+hudi-spark2
Review comment:
hudi.spark.module? is this change really essential for this PR or can
it go in a separate one?
##
File path: packaging/hudi-integ-test-bundle/pom.xml
##
@@ -73,8 +73,7 @@
org.apache.hudi:hudi-spark-common
org.apache.hudi:hudi-utilities_${scala.binary.version}
org.apache.hudi:hudi-spark_${scala.binary.version}
-
org.apache.hudi:hudi-spark2_${scala.binary.version}
- org.apache.hudi:hudi-spark3_2.12
+
org.apache.hudi:${hudi.spark.moudle}_${scala.binary.version}
Review comment:
So you want to move towards an approach of picking the actual module to
bundle? I think we have too many properties for spark 2 vs 3 already. We may
need to think about re-simplifying everything actually, if this change is
indeed needs for this PR
##
File path:
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/MultiPartKeysValueExtractor.java
##
@@ -31,6 +32,11 @@
@Override
public List extractPartitionValuesInPath(String partitionPath) {
+// If the partitionPath is empty string( which means none-partition
table), the partition values
Review comment:
again something that can go in its own PR?
##
File path:
hudi-spark-datasource/hudi-spark2/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/mergeInto.scala
##
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.analysis.UnresolvedException
+import org.apache.spark.sql.catalyst.expressions.{Expression, Unevaluable}
+import org.apache.spark.sql.types.DataType
+
+// This code is just copy from v2Commands.scala in spark 3.0
+
+/**
+ * The logical plan of the MERGE INTO command that works for v2 tables.
+ */
+case class MergeIntoTable(
Review comment:
name the file as `MergeInto.scala`?
##
File path: hudi-spark-datasource/hudi-spark2/src/main/antlr4/imports/SqlBase.g4
##
@@ -0,0 +1,1099 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ * This file is an adaptation of Presto's
presto-parser/src/main/antlr4/com/facebook/presto/sql/parser/SqlBase.g4 grammar.
Review comment:
can we add this to NOTICE. there are examples there. whats the license
of this file?
##
File path: hudi-spark-datasource/hudi-spark2/pom.xml
##
@@ -29,6 +29,8 @@
${project.parent.parent.basedir}
+2.11
Review comment:
can you please explain the reason for adding this?
##
File path:
hudi-spark-datasource/hudi-spark2/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UpdateTable.scala
##
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
