[GitHub] [hudi] pj001 commented on issue #2644: Hudi cow table incremental data error
pj001 commented on issue #2644: URL: https://github.com/apache/hudi/issues/2644#issuecomment-792544242 It seems that the commit under Hoodie will clean up periodically. Can't incremental read read all the data? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #2638: [MINOR] Fix import in StreamerUtil.java
danny0405 commented on a change in pull request #2638: URL: https://github.com/apache/hudi/pull/2638#discussion_r589215221 ## File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java ## @@ -19,6 +19,7 @@ package org.apache.hudi.util; import org.apache.hudi.common.model.HoodieRecordLocation; +import org.apache.hudi.common.model.HoodieTableType; Review comment: That's weird ~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #2638: [MINOR] Fix import in StreamerUtil.java
danny0405 commented on a change in pull request #2638: URL: https://github.com/apache/hudi/pull/2638#discussion_r589211312 ## File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java ## @@ -19,6 +19,7 @@ package org.apache.hudi.util; import org.apache.hudi.common.model.HoodieRecordLocation; +import org.apache.hudi.common.model.HoodieTableType; Review comment: Before you merge the code, at lease you need to see if the `HoodieTableType` is used in the master code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #2638: [MINOR] Fix import in StreamerUtil.java
danny0405 commented on a change in pull request #2638: URL: https://github.com/apache/hudi/pull/2638#discussion_r589211312 ## File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java ## @@ -19,6 +19,7 @@ package org.apache.hudi.util; import org.apache.hudi.common.model.HoodieRecordLocation; +import org.apache.hudi.common.model.HoodieTableType; Review comment: Before you merge the code, at lease you need to see if the `HoodieTableType` is used in the master code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2640: [HUDI-1663] Streaming read for Flink MOR table
codecov-io edited a comment on pull request #2640: URL: https://github.com/apache/hudi/pull/2640#issuecomment-791887408 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2640?src=pr=h1) Report > Merging [#2640](https://codecov.io/gh/apache/hudi/pull/2640?src=pr=desc) (eacb026) into [master](https://codecov.io/gh/apache/hudi/commit/02073235c3bbe72db32ed282b3e318e7daa0800a?el=desc) (0207323) will **increase** coverage by `9.94%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2640?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2640 +/- ## + Coverage 51.53% 61.48% +9.94% + Complexity 3491 324-3167 Files 462 53 -409 Lines 21881 1963 -19918 Branches 2327 235-2092 - Hits 11277 1207 -10070 + Misses 9624 632-8992 + Partials980 124 -856 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `61.48% <ø> (-8.00%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2640?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.63%)` | `0.00% <0.00%> (-28.00%)` | | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | `0.00% <0.00%> (-2.00%)` | | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | `0.00% <0.00%> (-2.00%)` | | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=) | `40.69% <0.00%> (-23.84%)` | `27.00% <0.00%> (-6.00%)` | | | [...apache/hudi/common/engine/HoodieEngineContext.java](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2VuZ2luZS9Ib29kaWVFbmdpbmVDb250ZXh0LmphdmE=) | | | | | [...e/hudi/common/table/log/HoodieFileSliceReader.java](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVGaWxlU2xpY2VSZWFkZXIuamF2YQ==) | | | | | [.../org/apache/hudi/MergeOnReadSnapshotRelation.scala](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL01lcmdlT25SZWFkU25hcHNob3RSZWxhdGlvbi5zY2FsYQ==) | | | | | [...main/scala/org/apache/hudi/HoodieWriterUtils.scala](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVdyaXRlclV0aWxzLnNjYWxh) | | | | | [...rc/main/java/org/apache/hudi/ApiMaturityLevel.java](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvQXBpTWF0dXJpdHlMZXZlbC5qYXZh) | | | | | ... and [395 more](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree-more) | |
[GitHub] [hudi] danny0405 commented on pull request #2642: [HUDI-1673] Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex
danny0405 commented on pull request #2642: URL: https://github.com/apache/hudi/pull/2642#issuecomment-792523708 `FlinkHoodieBloomIndex` is useless now for both the old the new write pipeline since we introduce the bucket assigner. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic implementation Of Spark Sql Support For Hoodie
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (43e5520) into [master](https://codecov.io/gh/apache/hudi/commit/11ad4ed26b6046201945f0e14449e1cbc5b6f1f2?el=desc) (11ad4ed) will **increase** coverage by `7.99%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2645 +/- ## + Coverage 61.48% 69.48% +7.99% - Complexity 324 363 +39 Files53 53 Lines 1963 1963 Branches235 235 + Hits 1207 1364 +157 + Misses 632 465 -167 - Partials124 134 +10 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiclient | `?` | `?` | | | hudiutilities | `69.48% <ø> (+7.99%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <0.00%> (ø)` | `18.00% <0.00%> (ø%)` | | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=) | `64.53% <0.00%> (+23.83%)` | `33.00% <0.00%> (+6.00%)` | | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `66.66% <0.00%> (+66.66%)` | `2.00% <0.00%> (+2.00%)` | | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `72.22% <0.00%> (+72.22%)` | `2.00% <0.00%> (+2.00%)` | | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `88.79% <0.00%> (+83.62%)` | `28.00% <0.00%> (+28.00%)` | | | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `100.00% <0.00%> (+100.00%)` | `1.00% <0.00%> (+1.00%)` | | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] garyli1019 merged pull request #2642: [HUDI-1673] Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex
garyli1019 merged pull request #2642: URL: https://github.com/apache/hudi/pull/2642 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated: [HUDI-1673] Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex (#2642)
This is an automated email from the ASF dual-hosted git repository.
garyli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 8b9dea4 [HUDI-1673] Replace scala.Tule2 to Pair in
FlinkHoodieBloomIndex (#2642)
8b9dea4 is described below
commit 8b9dea4ad9012d84203a888b601cca03bf857aeb
Author: Shen Hong
AuthorDate: Mon Mar 8 14:30:34 2021 +0800
[HUDI-1673] Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex (#2642)
---
.../hudi/index/bloom/FlinkHoodieBloomIndex.java| 36 +--
.../bloom/HoodieFlinkBloomIndexCheckFunction.java | 22 ++--
.../index/bloom/TestFlinkHoodieBloomIndex.java | 40 ++
3 files changed, 46 insertions(+), 52 deletions(-)
diff --git
a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/index/bloom/FlinkHoodieBloomIndex.java
b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/index/bloom/FlinkHoodieBloomIndex.java
index 6a3edc7..255a66b 100644
---
a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/index/bloom/FlinkHoodieBloomIndex.java
+++
b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/index/bloom/FlinkHoodieBloomIndex.java
@@ -44,8 +44,6 @@ import java.util.Iterator;
import java.util.List;
import java.util.Map;
-import scala.Tuple2;
-
import static java.util.stream.Collectors.mapping;
import static java.util.stream.Collectors.groupingBy;
import static java.util.stream.Collectors.toList;
@@ -106,14 +104,14 @@ public class FlinkHoodieBloomIndex extends FlinkH
List affectedPartitionPathList = new
ArrayList<>(recordsPerPartition.keySet());
// Step 2: Load all involved files as pairs
-List> fileInfoList =
+List> fileInfoList =
loadInvolvedFiles(affectedPartitionPathList, context, hoodieTable);
final Map> partitionToFileInfo =
-fileInfoList.stream().collect(groupingBy(Tuple2::_1,
mapping(Tuple2::_2, toList(;
+fileInfoList.stream().collect(groupingBy(Pair::getLeft,
mapping(Pair::getRight, toList(;
// Step 3: Obtain a List, for each incoming record, that already exists,
with the file id,
// that contains it.
-List> fileComparisons =
+List> fileComparisons =
explodeRecordsWithFileComparisons(partitionToFileInfo,
partitionRecordKeyMap);
return findMatchingFilesForRecordKeys(fileComparisons, hoodieTable);
}
@@ -122,7 +120,7 @@ public class FlinkHoodieBloomIndex extends FlinkH
* Load all involved files as pair List.
*/
//TODO duplicate code with spark, we can optimize this method later
- List> loadInvolvedFiles(List
partitions, final HoodieEngineContext context,
+ List> loadInvolvedFiles(List
partitions, final HoodieEngineContext context,
final HoodieTable
hoodieTable) {
// Obtain the latest data files from all the partitions.
List> partitionPathFileIDList =
getLatestBaseFilesForAllPartitions(partitions, context, hoodieTable).stream()
@@ -136,15 +134,15 @@ public class FlinkHoodieBloomIndex extends FlinkH
try {
HoodieRangeInfoHandle rangeInfoHandle = new
HoodieRangeInfoHandle(config, hoodieTable, pf);
String[] minMaxKeys = rangeInfoHandle.getMinMaxKeys();
- return new Tuple2<>(pf.getKey(), new
BloomIndexFileInfo(pf.getValue(), minMaxKeys[0], minMaxKeys[1]));
+ return Pair.of(pf.getKey(), new BloomIndexFileInfo(pf.getValue(),
minMaxKeys[0], minMaxKeys[1]));
} catch (MetadataNotFoundException me) {
LOG.warn("Unable to find range metadata in file :" + pf);
- return new Tuple2<>(pf.getKey(), new
BloomIndexFileInfo(pf.getValue()));
+ return Pair.of(pf.getKey(), new BloomIndexFileInfo(pf.getValue()));
}
}, Math.max(partitionPathFileIDList.size(), 1));
} else {
return partitionPathFileIDList.stream()
- .map(pf -> new Tuple2<>(pf.getKey(), new
BloomIndexFileInfo(pf.getValue(.collect(toList());
+ .map(pf -> Pair.of(pf.getKey(), new
BloomIndexFileInfo(pf.getValue(.collect(toList());
}
}
@@ -186,19 +184,19 @@ public class FlinkHoodieBloomIndex extends FlinkH
* Sub-partition to ensure the records can be looked up against files & also
prune file<=>record comparisons based on
* recordKey ranges in the index info.
*/
- List> explodeRecordsWithFileComparisons(
+ List> explodeRecordsWithFileComparisons(
final Map> partitionToFileIndexInfo,
Map> partitionRecordKeyMap) {
IndexFileFilter indexFileFilter =
config.useBloomIndexTreebasedFilter() ? new
IntervalTreeBasedIndexFileFilter(partitionToFileIndexInfo)
: new ListBasedIndexFileFilter(partitionToFileIndexInfo);
-List> fileRecordPairs = new ArrayList<>();
+List> fileRecordPairs = new ArrayList<>();
[GitHub] [hudi] yanghua commented on pull request #2642: [HUDI-1673] Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex
yanghua commented on pull request #2642: URL: https://github.com/apache/hudi/pull/2642#issuecomment-792497493 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic implementation Of Spark Sql Support For Hoodie
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (861806f) into [master](https://codecov.io/gh/apache/hudi/commit/11ad4ed26b6046201945f0e14449e1cbc5b6f1f2?el=desc) (11ad4ed) will **increase** coverage by `8.04%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2645 +/- ## + Coverage 61.48% 69.53% +8.04% - Complexity 324 364 +40 Files53 53 Lines 1963 1963 Branches235 235 + Hits 1207 1365 +158 + Misses 632 465 -167 - Partials124 133 +9 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiclient | `?` | `?` | | | hudiutilities | `69.53% <ø> (+8.04%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <0.00%> (ø)` | `18.00% <0.00%> (ø%)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.34% <0.00%> (+0.34%)` | `53.00% <0.00%> (+1.00%)` | | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=) | `64.53% <0.00%> (+23.83%)` | `33.00% <0.00%> (+6.00%)` | | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `66.66% <0.00%> (+66.66%)` | `2.00% <0.00%> (+2.00%)` | | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `72.22% <0.00%> (+72.22%)` | `2.00% <0.00%> (+2.00%)` | | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `88.79% <0.00%> (+83.62%)` | `28.00% <0.00%> (+28.00%)` | | | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `100.00% <0.00%> (+100.00%)` | `1.00% <0.00%> (+1.00%)` | | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1672) packaging fails w/ scala 12 (due to flink)
[ https://issues.apache.org/jira/browse/HUDI-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297085#comment-17297085 ] vinoyang commented on HUDI-1672: [~shivnarayan] It's strange. I did not find the compile error. It's OK, in my local. > packaging fails w/ scala 12 (due to flink) > -- > > Key: HUDI-1672 > URL: https://issues.apache.org/jira/browse/HUDI-1672 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: vinoyang >Priority: Major > Labels: sev:critical, user-support-issues > > mvn clean package -DskipTests -Dspark3 -Dscala-2.12 > . > . > [*INFO*] > /Users/sivabala/Documents/personal/projects/siva_hudi/hudi_march2021/hudi/hudi-flink/src/main/java/org/apache/hudi/util/AvroSchemaConverter.java: > Some input files use or override a deprecated API. > [*INFO*] > /Users/sivabala/Documents/personal/projects/siva_hudi/hudi_march2021/hudi/hudi-flink/src/main/java/org/apache/hudi/util/AvroSchemaConverter.java: > Recompile with -Xlint:deprecation for details. > [*INFO*] > /Users/sivabala/Documents/personal/projects/siva_hudi/hudi_march2021/hudi/hudi-flink/src/main/java/org/apache/hudi/operator/StreamWriteOperatorFactory.java: > Some input files use unchecked or unsafe operations. > [*INFO*] > /Users/sivabala/Documents/personal/projects/siva_hudi/hudi_march2021/hudi/hudi-flink/src/main/java/org/apache/hudi/operator/StreamWriteOperatorFactory.java: > Recompile with -Xlint:unchecked for details. > [*INFO*] - > [*ERROR*] COMPILATION ERROR : > [*INFO*] - > [*ERROR*] > /Users/sivabala/Documents/personal/projects/siva_hudi/hudi_march2021/hudi/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:[334,32] > package HoodieTableType does not exist > [*INFO*] 1 error > [*INFO*] - > [*INFO*] > ** > [*INFO*] *Reactor Summary for Hudi 0.8.0-SNAPSHOT:* > [*INFO*] > [*INFO*] Hudi ... *SUCCESS* [ > 2.736 s] > [*INFO*] hudi-common *SUCCESS* [ > 38.206 s] > [*INFO*] hudi-timeline-service .. *SUCCESS* [ > 2.579 s] > [*INFO*] hudi-client *SUCCESS* [ > 0.083 s] > [*INFO*] hudi-client-common . *SUCCESS* [ > 10.578 s] > [*INFO*] hudi-hadoop-mr . *SUCCESS* [ > 6.155 s] > [*INFO*] hudi-spark-client .. *SUCCESS* [ > 18.130 s] > [*INFO*] hudi-sync-common ... *SUCCESS* [ > 0.549 s] > [*INFO*] hudi-hive-sync . *SUCCESS* [ > 3.376 s] > [*INFO*] hudi-spark-datasource .. *SUCCESS* [ > 0.077 s] > [*INFO*] hudi-spark-common .. *SUCCESS* [ > 10.523 s] > [*INFO*] hudi-spark2_2.12 ... *SUCCESS* [ > 9.411 s] > [*INFO*] hudi-spark3_2.12 ... *SUCCESS* [ > 12.286 s] > [*INFO*] hudi-spark_2.12 *SUCCESS* [ > 40.109 s] > [*INFO*] hudi-utilities_2.12 *SUCCESS* [ > 7.229 s] > [*INFO*] hudi-utilities-bundle_2.12 . *SUCCESS* [ > 36.710 s] > [*INFO*] hudi-cli ... *SUCCESS* [ > 13.804 s] > [*INFO*] hudi-java-client ... *SUCCESS* [ > 1.869 s] > [*INFO*] hudi-flink-client .. *SUCCESS* [ > 7.289 s] > [*INFO*] hudi-dla-sync .. *SUCCESS* [ > 1.627 s] > [*INFO*] hudi-sync .. *SUCCESS* [ > 0.058 s] > [*INFO*] hudi-hadoop-mr-bundle .. *SUCCESS* [ > 4.328 s] > [*INFO*] hudi-hive-sync-bundle .. *SUCCESS* [ > 1.307 s] > [*INFO*] hudi-spark-bundle_2.12 . *SUCCESS* [ > 8.829 s] > [*INFO*] hudi-presto-bundle . *SUCCESS* [ > 12.235 s] > [*INFO*] hudi-timeline-server-bundle *SUCCESS* [ > 4.582 s] > [*INFO*] hudi-hadoop-docker . *SUCCESS* [ > 0.543 s] > [*INFO*] hudi-hadoop-base-docker *SUCCESS* [ > 0.080 s] > [*INFO*] hudi-hadoop-namenode-docker *SUCCESS* [ > 0.071 s] > [*INFO*] hudi-hadoop-datanode-docker
[GitHub] [hudi] xushiyan edited a comment on pull request #2646: [MINOR] HoodieClientTestHarness close resources in AfterAll phase (Azure CI)
xushiyan edited a comment on pull request #2646: URL: https://github.com/apache/hudi/pull/2646#issuecomment-792472062 @yanghua @vinothchandar `(Azure CI)` is a special temporary token to trigger azure CI build for the PR at this current stage, until infra setup completes. When merging PR, that token can be removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: DO NOT MERGE (Azure CI) test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * 527fbda8fdc5348e87852f0141b3512c28ef96e9 Azure: [SUCCESS](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=108) * 6624747efb6b164a7d75430b3225897a67c7af7d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #2646: [MINOR] HoodieClientTestHarness close resources in AfterAll phase (Azure CI)
xushiyan commented on pull request #2646: URL: https://github.com/apache/hudi/pull/2646#issuecomment-792472062 @yanghua @vinothchandar `(Azure CI)` is a special temporary token to trigger azure CI build for the PR should be built at this current stage, until infra setup completes. When merging PR, that token can be removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2646: [MINOR] HoodieClientTestHarness close resources in AfterAll phase (Azure CI)
codecov-io edited a comment on pull request #2646: URL: https://github.com/apache/hudi/pull/2646#issuecomment-792445886 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2646?src=pr=h1) Report > Merging [#2646](https://codecov.io/gh/apache/hudi/pull/2646?src=pr=desc) (38b8302) into [master](https://codecov.io/gh/apache/hudi/commit/5cf2f2618b6a59a831543b588fb3bb85bdf5f1e8?el=desc) (5cf2f26) will **increase** coverage by `0.00%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2646?src=pr=tree) ```diff @@Coverage Diff@@ ## master#2646 +/- ## = Coverage 51.54% 51.54% - Complexity 3491 3492+1 = Files 462 462 Lines 2187921879 Branches 2326 2326 = + Hits 1127711278+1 Misses 9623 9623 + Partials979 978-1 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudicommon | `51.48% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiflink | `50.34% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudihadoopmr | `33.48% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisparkdatasource | `69.84% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisync | `49.62% <ø> (ø)` | `0.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiutilities | `69.53% <ø> (+0.05%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2646?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.34% <0.00%> (+0.34%)` | `53.00% <0.00%> (+1.00%)` | | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2646: [MINOR] HoodieClientTestHarness close resources in AfterAll phase (Azure CI)
hudi-bot edited a comment on pull request #2646: URL: https://github.com/apache/hudi/pull/2646#issuecomment-792440946 ## CI report: * 38b83029d244cceba7b2f2753ea0f2780580ffb6 Azure: [SUCCESS](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=109) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: DO NOT MERGE (Azure CI) test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * 527fbda8fdc5348e87852f0141b3512c28ef96e9 Azure: [SUCCESS](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=108) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] garyli1019 commented on pull request #2642: [HUDI-1673] Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex
garyli1019 commented on pull request #2642: URL: https://github.com/apache/hudi/pull/2642#issuecomment-792454545 cc: @danny0405 @yanghua , I gonna merge this if no one disagree This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic implementation Of Spark Sql Support For Hoodie
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (d42690e) into [master](https://codecov.io/gh/apache/hudi/commit/11ad4ed26b6046201945f0e14449e1cbc5b6f1f2?el=desc) (11ad4ed) will **increase** coverage by `8.04%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2645 +/- ## + Coverage 61.48% 69.53% +8.04% - Complexity 324 364 +40 Files53 53 Lines 1963 1963 Branches235 235 + Hits 1207 1365 +158 + Misses 632 465 -167 - Partials124 133 +9 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiclient | `?` | `?` | | | hudiutilities | `69.53% <ø> (+8.04%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <0.00%> (ø)` | `18.00% <0.00%> (ø%)` | | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.34% <0.00%> (+0.34%)` | `53.00% <0.00%> (+1.00%)` | | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=) | `64.53% <0.00%> (+23.83%)` | `33.00% <0.00%> (+6.00%)` | | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `66.66% <0.00%> (+66.66%)` | `2.00% <0.00%> (+2.00%)` | | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `72.22% <0.00%> (+72.22%)` | `2.00% <0.00%> (+2.00%)` | | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `88.79% <0.00%> (+83.62%)` | `28.00% <0.00%> (+28.00%)` | | | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `100.00% <0.00%> (+100.00%)` | `1.00% <0.00%> (+1.00%)` | | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic implementation Of Spark Sql Support For Hoodie
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (8a3f016) into [master](https://codecov.io/gh/apache/hudi/commit/11ad4ed26b6046201945f0e14449e1cbc5b6f1f2?el=desc) (11ad4ed) will **increase** coverage by `7.99%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2645 +/- ## + Coverage 61.48% 69.48% +7.99% - Complexity 324 363 +39 Files53 53 Lines 1963 1963 Branches235 235 + Hits 1207 1364 +157 + Misses 632 465 -167 - Partials124 134 +10 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiclient | `?` | `?` | | | hudiutilities | `69.48% <ø> (+7.99%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <0.00%> (ø)` | `18.00% <0.00%> (ø%)` | | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=) | `64.53% <0.00%> (+23.83%)` | `33.00% <0.00%> (+6.00%)` | | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `66.66% <0.00%> (+66.66%)` | `2.00% <0.00%> (+2.00%)` | | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `72.22% <0.00%> (+72.22%)` | `2.00% <0.00%> (+2.00%)` | | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `88.79% <0.00%> (+83.62%)` | `28.00% <0.00%> (+28.00%)` | | | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `100.00% <0.00%> (+100.00%)` | `1.00% <0.00%> (+1.00%)` | | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2646: [MINOR] HoodieClientTestHarness close resources in AfterAll phase (Azure CI)
codecov-io edited a comment on pull request #2646: URL: https://github.com/apache/hudi/pull/2646#issuecomment-792445886 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] garyli1019 merged pull request #2634: [HUDI-1662] Failed to query real-time view use hive/spark-sql when hudi mor table contains dateType
garyli1019 merged pull request #2634: URL: https://github.com/apache/hudi/pull/2634 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated (5cf2f26 -> 0207323)
This is an automated email from the ASF dual-hosted git repository. garyli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 5cf2f26 [HUDI-1618] Fixing NPE with Parquet src in multi table delta streamer (#2577) add 0207323 [HUDI-1662] Fix hive date type conversion for mor table (#2634) No new revisions were added by this update. Summary of changes: .../org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java | 4 1 file changed, 4 insertions(+)
[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic implementation Of Spark Sql Support For Hoodie
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (d42690e) into [master](https://codecov.io/gh/apache/hudi/commit/11ad4ed26b6046201945f0e14449e1cbc5b6f1f2?el=desc) (11ad4ed) will **decrease** coverage by `51.96%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2645 +/- ## - Coverage 61.48% 9.52% -51.97% + Complexity 324 48 -276 Files53 53 Lines 19631963 Branches235 235 - Hits 1207 187 -1020 - Misses 6321763 +1131 + Partials124 13 -111 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiclient | `?` | `?` | | | hudiutilities | `9.52% <ø> (-51.97%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | | | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | | | [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | ... and [33 more](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree-more) | | This is an automated message
[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic implementation Of Spark Sql Support For Hoodie
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (8a3f016) into [master](https://codecov.io/gh/apache/hudi/commit/11ad4ed26b6046201945f0e14449e1cbc5b6f1f2?el=desc) (11ad4ed) will **decrease** coverage by `51.96%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2645 +/- ## - Coverage 61.48% 9.52% -51.97% + Complexity 324 48 -276 Files53 53 Lines 19631963 Branches235 235 - Hits 1207 187 -1020 - Misses 6321763 +1131 + Partials124 13 -111 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiclient | `?` | `?` | | | hudiutilities | `9.52% <ø> (-51.97%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | | | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | | | [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | ... and [33 more](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree-more) | | This is an automated message
[GitHub] [hudi] codecov-io edited a comment on pull request #2640: [HUDI-1663] Streaming read for Flink MOR table
codecov-io edited a comment on pull request #2640: URL: https://github.com/apache/hudi/pull/2640#issuecomment-791887408 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2640?src=pr=h1) Report > Merging [#2640](https://codecov.io/gh/apache/hudi/pull/2640?src=pr=desc) (206d070) into [master](https://codecov.io/gh/apache/hudi/commit/11ad4ed26b6046201945f0e14449e1cbc5b6f1f2?el=desc) (11ad4ed) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2640?src=pr=tree) ```diff @@Coverage Diff@@ ## master#2640 +/- ## = Coverage 61.48% 61.48% Complexity 324 324 = Files53 53 Lines 1963 1963 Branches235 235 = Hits 1207 1207 Misses 632 632 Partials124 124 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiutilities | `61.48% <ø> (ø)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2640?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2640/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <0.00%> (ø)` | `18.00% <0.00%> (ø%)` | | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2646: [MINOR] HoodieClientTestHarness close resources in AfterAll phase (Azure CI)
codecov-io commented on pull request #2646: URL: https://github.com/apache/hudi/pull/2646#issuecomment-792445886 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2646?src=pr=h1) Report > Merging [#2646](https://codecov.io/gh/apache/hudi/pull/2646?src=pr=desc) (38b8302) into [master](https://codecov.io/gh/apache/hudi/commit/5cf2f2618b6a59a831543b588fb3bb85bdf5f1e8?el=desc) (5cf2f26) will **increase** coverage by `17.99%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2646?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2646 +/- ## = + Coverage 51.54% 69.53% +17.99% + Complexity 3491 364 -3127 = Files 462 53 -409 Lines 21879 1963-19916 Branches 2326 235 -2091 = - Hits 11277 1365 -9912 + Misses 9623 465 -9158 + Partials979 133 -846 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.53% <ø> (+0.05%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2646?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...src/main/scala/org/apache/hudi/DefaultSource.scala](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0RlZmF1bHRTb3VyY2Uuc2NhbGE=) | | | | | [.../hive/SlashEncodedHourPartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvU2xhc2hFbmNvZGVkSG91clBhcnRpdGlvblZhbHVlRXh0cmFjdG9yLmphdmE=) | | | | | [...ava/org/apache/hudi/cli/commands/StatsCommand.java](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1N0YXRzQ29tbWFuZC5qYXZh) | | | | | [...apache/hudi/common/fs/HoodieWrapperFileSystem.java](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL0hvb2RpZVdyYXBwZXJGaWxlU3lzdGVtLmphdmE=) | | | | | [.../hudi/common/bloom/InternalDynamicBloomFilter.java](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2Jsb29tL0ludGVybmFsRHluYW1pY0Jsb29tRmlsdGVyLmphdmE=) | | | | | [...udi/common/table/log/block/HoodieCorruptBlock.java](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9ibG9jay9Ib29kaWVDb3JydXB0QmxvY2suamF2YQ==) | | | | | [.../spark/sql/hudi/streaming/HoodieStreamSource.scala](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvaHVkaS9zdHJlYW1pbmcvSG9vZGllU3RyZWFtU291cmNlLnNjYWxh) | | | | | [...rc/main/scala/org/apache/hudi/cli/DeDupeType.scala](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS1jbGkvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL2NsaS9EZUR1cGVUeXBlLnNjYWxh) | | | | | [...n/java/org/apache/hudi/common/HoodieCleanStat.java](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL0hvb2RpZUNsZWFuU3RhdC5qYXZh) | | | | | [...he/hudi/common/fs/SizeAwareFSDataOutputStream.java](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL1NpemVBd2FyZUZTRGF0YU91dHB1dFN0cmVhbS5qYXZh) | | | | | ... and [400 more](https://codecov.io/gh/apache/hudi/pull/2646/diff?src=pr=tree-more) | | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: DO NOT MERGE (Azure CI) test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * fbc64a121f388296a5c4478707ea3d76994f0b3f Azure: [SUCCESS](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=107) * 527fbda8fdc5348e87852f0141b3512c28ef96e9 Azure: [PENDING](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=108) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2646: [MINOR] HoodieClientTestHarness close resources in AfterAll phase (Azure CI)
hudi-bot edited a comment on pull request #2646: URL: https://github.com/apache/hudi/pull/2646#issuecomment-792440946 ## CI report: * 38b83029d244cceba7b2f2753ea0f2780580ffb6 Azure: [PENDING](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=109) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #2646: [MINOR] HoodieClientTestHarness close resources in AfterAll phase (Azure CI)
hudi-bot commented on pull request #2646: URL: https://github.com/apache/hudi/pull/2646#issuecomment-792440946 ## CI report: * 38b83029d244cceba7b2f2753ea0f2780580ffb6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: DO NOT MERGE (Azure CI) test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * fbc64a121f388296a5c4478707ea3d76994f0b3f Azure: [SUCCESS](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=107) * 527fbda8fdc5348e87852f0141b3512c28ef96e9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan opened a new pull request #2646: [MINOR] HoodieClientTestHarness close resources in AfterAll phase (Azure CI)
xushiyan opened a new pull request #2646: URL: https://github.com/apache/hudi/pull/2646 Parameterized test case like `org.apache.hudi.table.upgrade.TestUpgradeDowngrade#testUpgrade` incurs flakiness when org.apache.hadoop.fs.FileSystem#closeAll is invoked at BeforeEach; it should be invoked in AfterAll instead. ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #2634: [HUDI-1662] Failed to query real-time view use hive/spark-sql when hudi mor table contains dateType
xiarixiaoyao commented on pull request #2634: URL: https://github.com/apache/hudi/pull/2634#issuecomment-792440363 cc @garyli1019 . Sorry for the late reply。 now the ci is pass, could you check and merge This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: [Azure test] test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * fbc64a121f388296a5c4478707ea3d76994f0b3f Azure: [SUCCESS](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=107) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2645: [HUDI-1659] Basic implementation Of Spark Sql Support For Hoodie
codecov-io commented on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (1578923) into [master](https://codecov.io/gh/apache/hudi/commit/11ad4ed26b6046201945f0e14449e1cbc5b6f1f2?el=desc) (11ad4ed) will **decrease** coverage by `51.96%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2645 +/- ## - Coverage 61.48% 9.52% -51.97% + Complexity 324 48 -276 Files53 53 Lines 19631963 Branches235 235 - Hits 1207 187 -1020 - Misses 6321763 +1131 + Partials124 13 -111 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudiclient | `?` | `?` | | | hudiutilities | `9.52% <ø> (-51.97%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | | | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | | | [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | ... and [33 more](https://codecov.io/gh/apache/hudi/pull/2645/diff?src=pr=tree-more) | | This is an automated message from
[jira] [Updated] (HUDI-1659) Basic implementation Of Spark Sql Support For Hoodie
[ https://issues.apache.org/jira/browse/HUDI-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1659: - Labels: pull-request-available (was: ) > Basic implementation Of Spark Sql Support For Hoodie > > > Key: HUDI-1659 > URL: https://issues.apache.org/jira/browse/HUDI-1659 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Major > Labels: pull-request-available > > The Basic Implement include the follow things based on DataSource V1: > 1、CREATE TABLE FOR HOODIE > 2、CTAS > 3、INSERT Hoodie Table > 4、MergeInto with the RowKey constraint. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-io edited a comment on pull request #2634: [HUDI-1662] Failed to query real-time view use hive/spark-sql when hudi mor table contains dateType
codecov-io edited a comment on pull request #2634: URL: https://github.com/apache/hudi/pull/2634#issuecomment-791203944 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2634?src=pr=h1) Report > Merging [#2634](https://codecov.io/gh/apache/hudi/pull/2634?src=pr=desc) (f57ce9c) into [master](https://codecov.io/gh/apache/hudi/commit/899ae70fdb70c1511c099a64230fd91b2fe8d4ee?el=desc) (899ae70) will **increase** coverage by `17.90%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2634?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2634 +/- ## = + Coverage 51.58% 69.48% +17.90% + Complexity 3285 363 -2922 = Files 446 53 -393 Lines 20409 1963-18446 Branches 2116 235 -1881 = - Hits 10528 1364 -9164 + Misses 9003 465 -8538 + Partials878 134 -744 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.48% <ø> (+0.04%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2634?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.00% <0.00%> (-0.72%)` | `52.00% <0.00%> (ø%)` | | | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <0.00%> (ø)` | `18.00% <0.00%> (ø%)` | | | [...src/main/java/org/apache/hudi/cli/TableHeader.java](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL1RhYmxlSGVhZGVyLmphdmE=) | | | | | [.../common/table/view/RocksDbBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUm9ja3NEYkJhc2VkRmlsZVN5c3RlbVZpZXcuamF2YQ==) | | | | | [...penJ9MemoryLayoutSpecification64bitCompressed.java](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvanZtL09wZW5KOU1lbW9yeUxheW91dFNwZWNpZmljYXRpb242NGJpdENvbXByZXNzZWQuamF2YQ==) | | | | | [.../apache/hudi/common/config/SerializableSchema.java](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9TZXJpYWxpemFibGVTY2hlbWEuamF2YQ==) | | | | | [...pache/hudi/hadoop/HoodieColumnProjectionUtils.java](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZUNvbHVtblByb2plY3Rpb25VdGlscy5qYXZh) | | | | | [...che/hudi/operator/partitioner/BucketAssigners.java](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9vcGVyYXRvci9wYXJ0aXRpb25lci9CdWNrZXRBc3NpZ25lcnMuamF2YQ==) | | | | | [...pache/hudi/io/storage/HoodieFileReaderFactory.java](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW8vc3RvcmFnZS9Ib29kaWVGaWxlUmVhZGVyRmFjdG9yeS5qYXZh) | | | | | [...rg/apache/hudi/cli/commands/CompactionCommand.java](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL0NvbXBhY3Rpb25Db21tYW5kLmphdmE=) | | | | | ... and [386 more](https://codecov.io/gh/apache/hudi/pull/2634/diff?src=pr=tree-more) | | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
[GitHub] [hudi] pengzhiwei2018 opened a new pull request #2645: [HUDI-1659] Basic implementation Of Spark Sql Support For Hoodie
pengzhiwei2018 opened a new pull request #2645: URL: https://github.com/apache/hudi/pull/2645 ## What is the purpose of the pull request Basic implement for spark sql integrates with Hoodie. ## Brief change log Main functions: - Support create table for hoodie. - Support CTAS. - Support Insert for hoodie. Including dynamic partition and static partition insert. - Support MergeInto for hoodie. Main changes: - Add sql parser for spark2. - Add HoodieAnalysis for sql resolve and logical plan rewrite. - Add commands implementation for CREATE TABLE、INSERT、MERGE INTO & CTAS. - Add `WRITE_SCHEMA` config to `HoodieWriteConfig` to pass the write schema for `MergeIntoHoodieTableCommand`. We made some modifications for HoodieWriteHandler to support specify write schema. ## Verify this pull request - Add CreateTableTest for test create hoodie tables. - Add InsertTableTest for test insert hoodie tables. - Add MergeIntoTest for test merge hoodie tables. ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-1659) Basic implementation Of Spark Sql Support For Hoodie
[ https://issues.apache.org/jira/browse/HUDI-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengzhiwei updated HUDI-1659: - Summary: Basic implementation Of Spark Sql Support For Hoodie (was: Basic Implement Of Spark Sql Support For Hoodie) > Basic implementation Of Spark Sql Support For Hoodie > > > Key: HUDI-1659 > URL: https://issues.apache.org/jira/browse/HUDI-1659 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Major > > The Basic Implement include the follow things based on DataSource V1: > 1、CREATE TABLE FOR HOODIE > 2、CTAS > 3、INSERT Hoodie Table > 4、MergeInto with the RowKey constraint. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-io edited a comment on pull request #2634: [HUDI-1662] Failed to query real-time view use hive/spark-sql when hudi mor table contains dateType
codecov-io edited a comment on pull request #2634: URL: https://github.com/apache/hudi/pull/2634#issuecomment-791203944 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2643: [Azure test] test branch ci
codecov-io edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792364978 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=h1) Report > Merging [#2643](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=desc) (fbc64a1) into [master](https://codecov.io/gh/apache/hudi/commit/5cf2f2618b6a59a831543b588fb3bb85bdf5f1e8?el=desc) (5cf2f26) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=tree) ```diff @@Coverage Diff@@ ## master#2643 +/- ## = Coverage 51.54% 51.54% Complexity 3491 3491 = Files 462 462 Lines 2187921879 Branches 2326 2326 = Hits 1127711277 Misses 9623 9623 Partials979 979 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudicommon | `51.48% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiflink | `50.34% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudihadoopmr | `33.48% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisparkdatasource | `69.84% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisync | `49.62% <ø> (ø)` | `0.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiutilities | `69.48% <ø> (ø)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: [Azure test] test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * 0e1da8eb5c7a0b033bfe2667bb76e6cf42cd5cf3 Azure: [CANCELED](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=106) * fbc64a121f388296a5c4478707ea3d76994f0b3f Azure: [PENDING](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=107) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] garyli1019 commented on a change in pull request #2636: [HUDI-1667]: Fix bug when HoodieMergeOnReadRDD read record from base …
garyli1019 commented on a change in pull request #2636:
URL: https://github.com/apache/hudi/pull/2636#discussion_r589133673
##
File path:
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDD.scala
##
@@ -281,7 +281,7 @@ class HoodieMergeOnReadRDD(@transient sc: SparkContext,
tableState.requiredStructSchema.foreach(
f => {
val curPos = posIterator.next()
-val curField = row.get(curPos, f.dataType)
+val curField = if (row.isNullAt(curPos)) null else row.get(curPos,
f.dataType)
Review comment:
Hi @Liulietong , thanks for your contribution! Is that possible to add a
test case to verify this will fix the vectorized reader bug?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2643: [Azure test] test branch ci
codecov-io edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792364978 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=h1) Report > Merging [#2643](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=desc) (0e1da8e) into [master](https://codecov.io/gh/apache/hudi/commit/5cf2f2618b6a59a831543b588fb3bb85bdf5f1e8?el=desc) (5cf2f26) will **increase** coverage by `17.94%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2643 +/- ## = + Coverage 51.54% 69.48% +17.94% + Complexity 3491 363 -3128 = Files 462 53 -409 Lines 21879 1963-19916 Branches 2326 235 -2091 = - Hits 11277 1364 -9913 + Misses 9623 465 -9158 + Partials979 134 -845 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.48% <ø> (ø)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../versioning/clean/CleanPlanV2MigrationHandler.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL3ZlcnNpb25pbmcvY2xlYW4vQ2xlYW5QbGFuVjJNaWdyYXRpb25IYW5kbGVyLmphdmE=) | | | | | [...che/hudi/common/util/BufferedRandomAccessFile.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQnVmZmVyZWRSYW5kb21BY2Nlc3NGaWxlLmphdmE=) | | | | | [.../org/apache/hudi/util/RowDataToAvroConverters.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS91dGlsL1Jvd0RhdGFUb0F2cm9Db252ZXJ0ZXJzLmphdmE=) | | | | | [...va/org/apache/hudi/factory/HoodieTableFactory.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9mYWN0b3J5L0hvb2RpZVRhYmxlRmFjdG9yeS5qYXZh) | | | | | [...ain/java/org/apache/hudi/cli/utils/CommitUtil.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL3V0aWxzL0NvbW1pdFV0aWwuamF2YQ==) | | | | | [...ava/org/apache/hudi/common/HoodieRollbackStat.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL0hvb2RpZVJvbGxiYWNrU3RhdC5qYXZh) | | | | | [...a/org/apache/hudi/cli/commands/RepairsCommand.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1JlcGFpcnNDb21tYW5kLmphdmE=) | | | | | [...e/hudi/cli/commands/UpgradeOrDowngradeCommand.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL1VwZ3JhZGVPckRvd25ncmFkZUNvbW1hbmQuamF2YQ==) | | | | | [.../main/scala/org/apache/hudi/HoodieSparkUtils.scala](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVNwYXJrVXRpbHMuc2NhbGE=) | | | | | [...ache/hudi/hadoop/utils/HoodieInputFormatUtils.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZUlucHV0Rm9ybWF0VXRpbHMuamF2YQ==) | | | | | ... and [399 more](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree-more) | | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: [Azure test] test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * 0e1da8eb5c7a0b033bfe2667bb76e6cf42cd5cf3 Azure: [PENDING](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=106) * fbc64a121f388296a5c4478707ea3d76994f0b3f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] guanziyue closed issue #2630: [SUPPORT]Confuse about the strategy to evaluate average record size
guanziyue closed issue #2630: URL: https://github.com/apache/hudi/issues/2630 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2643: [Azure test] test branch ci
codecov-io edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792364978 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=h1) Report > Merging [#2643](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=desc) (0e1da8e) into [master](https://codecov.io/gh/apache/hudi/commit/5cf2f2618b6a59a831543b588fb3bb85bdf5f1e8?el=desc) (5cf2f26) will **decrease** coverage by `42.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2643 +/- ## - Coverage 51.54% 9.52% -42.02% + Complexity 3491 48 -3443 Files 462 53 -409 Lines 218791963-19916 Branches 2326 235 -2091 - Hits 11277 187-11090 + Misses 96231763 -7860 + Partials979 13 -966 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.52% <ø> (-59.96%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | | | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | | | [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%>
[GitHub] [hudi] codecov-io edited a comment on pull request #2642: [HUDI-1673] Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex
codecov-io edited a comment on pull request #2642: URL: https://github.com/apache/hudi/pull/2642#issuecomment-792276483 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2642?src=pr=h1) Report > Merging [#2642](https://codecov.io/gh/apache/hudi/pull/2642?src=pr=desc) (27dde93) into [master](https://codecov.io/gh/apache/hudi/commit/06dc7c7fd8a867a1e1da90f7dc19b0cc2da69bba?el=desc) (06dc7c7) will **increase** coverage by `18.31%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2642?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2642 +/- ## = + Coverage 51.22% 69.53% +18.31% + Complexity 3230 364 -2866 = Files 438 53 -385 Lines 20093 1963-18130 Branches 2069 235 -1834 = - Hits 10292 1365 -8927 + Misses 8954 465 -8489 + Partials847 133 -714 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `69.53% <ø> (+0.01%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2642?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `85.84% <0.00%> (-2.94%)` | `20.00% <0.00%> (+4.00%)` | :arrow_down: | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=) | `64.53% <0.00%> (-1.17%)` | `33.00% <0.00%> (+1.00%)` | :arrow_down: | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.34% <0.00%> (-0.02%)` | `53.00% <0.00%> (+2.00%)` | :arrow_down: | | [...s/deltastreamer/HoodieMultiTableDeltaStreamer.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvSG9vZGllTXVsdGlUYWJsZURlbHRhU3RyZWFtZXIuamF2YQ==) | `78.39% <0.00%> (ø)` | `18.00% <0.00%> (ø%)` | | | [...apache/hudi/common/engine/HoodieEngineContext.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2VuZ2luZS9Ib29kaWVFbmdpbmVDb250ZXh0LmphdmE=) | | | | | [.../org/apache/hudi/streamer/HoodieFlinkStreamer.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zdHJlYW1lci9Ib29kaWVGbGlua1N0cmVhbWVyLmphdmE=) | | | | | [...e/hudi/common/table/log/HoodieFileSliceReader.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVGaWxlU2xpY2VSZWFkZXIuamF2YQ==) | | | | | [.../org/apache/hudi/MergeOnReadSnapshotRelation.scala](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL01lcmdlT25SZWFkU25hcHNob3RSZWxhdGlvbi5zY2FsYQ==) | | | | | [...main/scala/org/apache/hudi/HoodieWriterUtils.scala](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9odWRpL0hvb2RpZVdyaXRlclV0aWxzLnNjYWxh) | | | | | [...rc/main/java/org/apache/hudi/ApiMaturityLevel.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvQXBpTWF0dXJpdHlMZXZlbC5qYXZh) | | | | | ... and [381 more](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree-more) | | This is an automated message
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: [Azure test] test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * 0e1da8eb5c7a0b033bfe2667bb76e6cf42cd5cf3 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-1674) add partition level delete DOC or example
liwei created HUDI-1674: --- Summary: add partition level delete DOC or example Key: HUDI-1674 URL: https://issues.apache.org/jira/browse/HUDI-1674 Project: Apache Hudi Issue Type: Sub-task Reporter: liwei Attachments: image-2021-03-08-09-57-05-768.png !image-2021-03-08-09-57-05-768.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] pj001 opened a new issue #2644: Hudi cow table incremental data error
pj001 opened a new issue #2644: URL: https://github.com/apache/hudi/issues/2644 When using Hudi cow table incremental data, the result is inconsistent with the actual data. Is it caused by metadata merging? How can I solve it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: [Azure test] test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2643: [Azure test] test branch ci
codecov-io edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792364978 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=h1) Report > Merging [#2643](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=desc) (a39b69b) into [master](https://codecov.io/gh/apache/hudi/commit/5cf2f2618b6a59a831543b588fb3bb85bdf5f1e8?el=desc) (5cf2f26) will **increase** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2643 +/- ## + Coverage 51.54% 51.55% +0.01% - Complexity 3491 3492 +1 Files 462 462 Lines 2187921879 Branches 2326 2326 + Hits 1127711280 +3 + Misses 9623 9622 -1 + Partials979 977 -2 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudicommon | `51.50% <ø> (+0.01%)` | `0.00 <ø> (ø)` | | | hudiflink | `50.34% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudihadoopmr | `33.48% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisparkdatasource | `69.84% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudisync | `49.62% <ø> (ø)` | `0.00 <ø> (ø)` | | | huditimelineservice | `64.36% <ø> (ø)` | `0.00 <ø> (ø)` | | | hudiutilities | `69.53% <ø> (+0.05%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.34% <0.00%> (+0.34%)` | `53.00% <0.00%> (+1.00%)` | | | [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `79.68% <0.00%> (+1.56%)` | `26.00% <0.00%> (ø%)` | | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: [Azure test] test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * Unknown: [CANCELED](TBD) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: [Azure test] test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * Unknown: [CANCELED](TBD) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #2643: [Azure test] test branch ci
xushiyan commented on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792369210 @hudi-bot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #2643: [Azure test] test branch ci
xushiyan commented on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792369102 @hudi-bot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #2643: [Azure test] test branch ci
hudi-bot commented on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792368481 ## CI report: * a39b69b9761399577836ea2677d1ddff15e8a7e0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #2643: [Azure test] test branch ci
hudi-bot removed a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792364157 ## CI report: * a39b69b9761399577836ea2677d1ddff15e8a7e0 Azure: [PENDING](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=104) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2643: [Azure test] test branch ci
codecov-io commented on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792364978 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=h1) Report > Merging [#2643](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=desc) (a39b69b) into [master](https://codecov.io/gh/apache/hudi/commit/5cf2f2618b6a59a831543b588fb3bb85bdf5f1e8?el=desc) (5cf2f26) will **decrease** coverage by `42.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2643 +/- ## - Coverage 51.54% 9.52% -42.02% + Complexity 3491 48 -3443 Files 462 53 -409 Lines 218791963-19916 Branches 2326 235 -2091 - Hits 11277 187-11090 + Misses 96231763 -7860 + Partials979 13 -966 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `9.52% <ø> (-59.96%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2643?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | | | [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | | | [...lities/schema/SchemaProviderWithPostProcessor.java](https://codecov.io/gh/apache/hudi/pull/2643/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFQcm92aWRlcldpdGhQb3N0UHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%>
[GitHub] [hudi] hudi-bot edited a comment on pull request #2643: [Azure test] test branch ci
hudi-bot edited a comment on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792364157 ## CI report: * a39b69b9761399577836ea2677d1ddff15e8a7e0 Azure: [PENDING](https://dev.azure.com/XUSH0012/0ef433cc-d4b4-47cc-b6a1-03d032ef546c/_build/results?buildId=104) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #2643: [Azure test] test branch ci
hudi-bot commented on pull request #2643: URL: https://github.com/apache/hudi/pull/2643#issuecomment-792364157 ## CI report: * a39b69b9761399577836ea2677d1ddff15e8a7e0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan opened a new pull request #2643: [Azure test] test branch ci
xushiyan opened a new pull request #2643: URL: https://github.com/apache/hudi/pull/2643 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated: [HUDI-1618] Fixing NPE with Parquet src in multi table delta streamer (#2577)
This is an automated email from the ASF dual-hosted git repository.
sivabalan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 5cf2f26 [HUDI-1618] Fixing NPE with Parquet src in multi table delta
streamer (#2577)
5cf2f26 is described below
commit 5cf2f2618b6a59a831543b588fb3bb85bdf5f1e8
Author: Sivabalan Narayanan
AuthorDate: Sun Mar 7 16:40:40 2021 -0500
[HUDI-1618] Fixing NPE with Parquet src in multi table delta streamer
(#2577)
---
.../HoodieMultiTableDeltaStreamer.java | 7 +-
.../functional/TestHoodieDeltaStreamer.java| 44 +++---
.../TestHoodieMultiTableDeltaStreamer.java | 95 +-
3 files changed, 127 insertions(+), 19 deletions(-)
diff --git
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
index 9d5ca3c..be2fe54 100644
---
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
+++
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
@@ -47,6 +47,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
+import java.util.Objects;
import java.util.Set;
/**
@@ -75,9 +76,9 @@ public class HoodieMultiTableDeltaStreamer {
FileSystem fs = FSUtils.getFs(commonPropsFile, jssc.hadoopConfiguration());
configFolder = configFolder.charAt(configFolder.length() - 1) == '/' ?
configFolder.substring(0, configFolder.length() - 1) : configFolder;
checkIfPropsFileAndConfigFolderExist(commonPropsFile, configFolder, fs);
-TypedProperties properties = UtilHelpers.readConfig(fs, new
Path(commonPropsFile), new ArrayList<>()).getConfig();
+TypedProperties commonProperties = UtilHelpers.readConfig(fs, new
Path(commonPropsFile), new ArrayList<>()).getConfig();
//get the tables to be ingested and their corresponding config files from
this properties instance
-populateTableExecutionContextList(properties, configFolder, fs, config);
+populateTableExecutionContextList(commonProperties, configFolder, fs,
config);
}
private void checkIfPropsFileAndConfigFolderExist(String commonPropsFile,
String configFolder, FileSystem fs) throws IOException {
@@ -147,7 +148,7 @@ public class HoodieMultiTableDeltaStreamer {
}
private void populateSchemaProviderProps(HoodieDeltaStreamer.Config cfg,
TypedProperties typedProperties) {
-if
(cfg.schemaProviderClassName.equals(SchemaRegistryProvider.class.getName())) {
+if (Objects.equals(cfg.schemaProviderClassName,
SchemaRegistryProvider.class.getName())) {
String schemaRegistryBaseUrl =
typedProperties.getString(Constants.SCHEMA_REGISTRY_BASE_URL_PROP);
String schemaRegistrySuffix =
typedProperties.getString(Constants.SCHEMA_REGISTRY_URL_SUFFIX_PROP);
typedProperties.setProperty(Constants.SOURCE_SCHEMA_REGISTRY_URL_PROP,
schemaRegistryBaseUrl + typedProperties.getString(Constants.KAFKA_TOPIC_PROP) +
schemaRegistrySuffix);
diff --git
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
index 7fb5b18..7522c2d 100644
---
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
+++
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
@@ -118,8 +118,9 @@ public class TestHoodieDeltaStreamer extends
UtilitiesTestBase {
public static final String PROPS_INVALID_TABLE_CONFIG_FILE =
"test-invalid-table-config.properties";
private static final String PROPS_FILENAME_TEST_INVALID =
"test-invalid.properties";
private static final String PROPS_FILENAME_TEST_CSV =
"test-csv-dfs-source.properties";
- private static final String PROPS_FILENAME_TEST_PARQUET =
"test-parquet-dfs-source.properties";
+ protected static final String PROPS_FILENAME_TEST_PARQUET =
"test-parquet-dfs-source.properties";
private static final String PROPS_FILENAME_TEST_JSON_KAFKA =
"test-json-kafka-dfs-source.properties";
+ private static final String FIRST_PARQUET_FILE_NAME = "1.parquet";
private static String PARQUET_SOURCE_ROOT;
private static String JSON_KAFKA_SOURCE_ROOT;
private static final int PARQUET_NUM_RECORDS = 5;
@@ -214,7 +215,7 @@ public class TestHoodieDeltaStreamer extends
UtilitiesTestBase {
UtilitiesTestBase.Helpers.savePropsToDFS(invalidProps, dfs, dfsBasePath +
"/" + PROPS_FILENAME_TEST_INVALID);
TypedProperties props1 = new TypedProperties();
-populateCommonProps(props1);
+populateAllCommonProps(props1);
[GitHub] [hudi] nsivabalan merged pull request #2577: [HUDI-1618] Fixing NPE with Parquet src in multi table delta streamer
nsivabalan merged pull request #2577: URL: https://github.com/apache/hudi/pull/2577 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (HUDI-1655) Support custom date format and fix unsupported exception in DatePartitionPathSelector
[
https://issues.apache.org/jira/browse/HUDI-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu closed HUDI-1655.
> Support custom date format and fix unsupported exception in
> DatePartitionPathSelector
> -
>
> Key: HUDI-1655
> URL: https://issues.apache.org/jira/browse/HUDI-1655
> Project: Apache Hudi
> Issue Type: Improvement
> Components: DeltaStreamer
>Reporter: Raymond Xu
>Assignee: Raymond Xu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> Add a config to allow parsing custom date format in
> {{DatePartitionPathSelector}}. Currently it assumes date partition string in
> the format of {{-MM-dd}}.
>
> Also eligibleFiles.sort() throws this exception
> {quote}java.lang.UnsupportedOperationException at
> java.util.AbstractList.set(AbstractList.java:132) at
> java.util.AbstractList$ListItr.set(AbstractList.java:426) at
> java.util.List.sort(List.java:482) at
> org.apache.hudi.utilities.sources.helpers.DatePartitionPathSelector.getNextFilePathsAndMaxModificationTime(DatePartitionPathSelector.java:141)
> at
> org.apache.hudi.utilities.sources.ParquetDFSSource.fetchNextBatch(ParquetDFSSource.java:48)
> at
> org.apache.hudi.utilities.sources.RowSource.fetchNewData(RowSource.java:43)
> at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:75) at
> org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter.fetchNewDataInRowFormat(SourceFormatAdapter.java:94)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:338)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:255)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:587)
> {quote}
>
> {{org.apache.hudi.client.common.HoodieSparkEngineContext#flatMap}} returns a
> list that can't be sorted in-place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HUDI-1655) Support custom date format and fix unsupported exception in DatePartitionPathSelector
[
https://issues.apache.org/jira/browse/HUDI-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu resolved HUDI-1655.
--
Resolution: Fixed
> Support custom date format and fix unsupported exception in
> DatePartitionPathSelector
> -
>
> Key: HUDI-1655
> URL: https://issues.apache.org/jira/browse/HUDI-1655
> Project: Apache Hudi
> Issue Type: Improvement
> Components: DeltaStreamer
>Reporter: Raymond Xu
>Assignee: Raymond Xu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> Add a config to allow parsing custom date format in
> {{DatePartitionPathSelector}}. Currently it assumes date partition string in
> the format of {{-MM-dd}}.
>
> Also eligibleFiles.sort() throws this exception
> {quote}java.lang.UnsupportedOperationException at
> java.util.AbstractList.set(AbstractList.java:132) at
> java.util.AbstractList$ListItr.set(AbstractList.java:426) at
> java.util.List.sort(List.java:482) at
> org.apache.hudi.utilities.sources.helpers.DatePartitionPathSelector.getNextFilePathsAndMaxModificationTime(DatePartitionPathSelector.java:141)
> at
> org.apache.hudi.utilities.sources.ParquetDFSSource.fetchNextBatch(ParquetDFSSource.java:48)
> at
> org.apache.hudi.utilities.sources.RowSource.fetchNewData(RowSource.java:43)
> at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:75) at
> org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter.fetchNewDataInRowFormat(SourceFormatAdapter.java:94)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:338)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:255)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:587)
> {quote}
>
> {{org.apache.hudi.client.common.HoodieSparkEngineContext#flatMap}} returns a
> list that can't be sorted in-place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] xushiyan merged pull request #2638: [MINOR] Fix import in StreamerUtil.java
xushiyan merged pull request #2638: URL: https://github.com/apache/hudi/pull/2638 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch master updated: [MINOR] Fix import in StreamerUtil.java (#2638)
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 9437e0d [MINOR] Fix import in StreamerUtil.java (#2638) 9437e0d is described below commit 9437e0ddef9821f728a4edf24ab90506532b6d0d Author: Raymond Xu <2701446+xushi...@users.noreply.github.com> AuthorDate: Sun Mar 7 12:37:15 2021 -0800 [MINOR] Fix import in StreamerUtil.java (#2638) --- hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java | 1 + 1 file changed, 1 insertion(+) diff --git a/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java b/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java index cc161ce..fdab92b 100644 --- a/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java +++ b/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java @@ -19,6 +19,7 @@ package org.apache.hudi.util; import org.apache.hudi.common.model.HoodieRecordLocation; +import org.apache.hudi.common.model.HoodieTableType; import org.apache.hudi.common.table.HoodieTableMetaClient; import org.apache.hudi.common.util.TablePathUtils; import org.apache.hudi.exception.HoodieException;
[GitHub] [hudi] xushiyan commented on pull request #2638: [MINOR] Fix import in StreamerUtil.java
xushiyan commented on pull request #2638: URL: https://github.com/apache/hudi/pull/2638#issuecomment-792348096 @yanghua merging to make master build pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #2638: [MINOR] Fix import in StreamerUtil.java
xushiyan commented on a change in pull request #2638: URL: https://github.com/apache/hudi/pull/2638#discussion_r589084401 ## File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java ## @@ -19,6 +19,7 @@ package org.apache.hudi.util; import org.apache.hudi.common.model.HoodieRecordLocation; +import org.apache.hudi.common.model.HoodieTableType; Review comment: @yanghua indeed strange. CI should have failed for [this PR](https://github.com/apache/hudi/commit/bc883db5de5832fa429bbb04a35d3606fdacdb2a#diff-819d51e0aa6f24c56d99119ad4888e85db6ad51a33545ac5062ba86136fd0d92). Looks like Travis went blind for a while This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #2500: [HUDI-1496] Fixing detection of GCS FileSystem
nsivabalan edited a comment on pull request #2500:
URL: https://github.com/apache/hudi/pull/2500#issuecomment-792339635
If I am not wrong, there was a bug in the code version that you ran. I
tested 4 different variants of code to arrive at the latest proposal. Let me
walk through them :) Sorry about the lengthy response. Hopefully we get a
closure.
1st variant. Current master branch:
```
FSDataInputStream fsDataInputStream = fs.open(logFile.getPath(), bufferSize);
LOG.warn("HoodieLogFileReader :: canonical name :: " +
fsDataInputStream.getClass().getCanonicalName() + ", name "
+ fsDataInputStream.getClass().getName());
if (FSUtils.isGCSInputStream(fsDataInputStream)) {
LOG.warn("HoodieLogFileReader :: 111 start GCSFileSystem " +
fsDataInputStream.getWrappedStream().getClass().getCanonicalName());
this.inputStream = new TimedFSDataInputStream(logFile.getPath(), new
FSDataInputStream(
new BufferedFSInputStream((FSInputStream) ((
(FSDataInputStream)
fsDataInputStream.getWrappedStream()).getWrappedStream()), bufferSize)));
LOG.warn("HoodieLogFileReader :: 111 completed ");
} else if (fsDataInputStream.getWrappedStream() instanceof
FSInputStream) {
LOG.warn("HoodieLogFileReader :: 222 start " +
fsDataInputStream.getWrappedStream().getClass().getCanonicalName());
this.inputStream = new TimedFSDataInputStream(logFile.getPath(), new
FSDataInputStream(
new BufferedFSInputStream((FSInputStream)
fsDataInputStream.getWrappedStream(), bufferSize)));
LOG.warn("HoodieLogFileReader :: 222 complete");
} else {
LOG.warn("HoodieLogFileReader :: 333 ");
// fsDataInputStream.getWrappedStream() maybe a BufferedFSInputStream
// need to wrap in another BufferedFSInputStream the make bufferSize
work?
this.inputStream = fsDataInputStream;
}
```
Output from my run:
"HoodieLogFileReader :: canonical name ::
org.apache.hadoop.fs.FSDataInputStream, name
org.apache.hadoop.fs.FSDataInputStream"
"HoodieLogFileReader :: 111 start GCSFileSystem
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream"
Caused by: java.lang.ClassCastException:
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream cannot be cast to
org.apache.hadoop
.fs.FSDataInputStream
at
org.apache.hudi.common.table.log.HoodieLogFileReader.(HoodieLogFileReader.java:84)
at
org.apache.hudi.common.table.log.HoodieLogFormatReader.(HoodieLogFormatReader.java:62)
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.scan(AbstractHoodieLogRecordScanner.java:131)
... 24 more
2nd variant:
This PR just before my last commit.
```
if (fsDataInputStream.getWrappedStream() instanceof FSInputStream) {
LOG.warn("HoodieLogFileReader " + logFile.getFileName() + " " +
fsDataInputStream.getWrappedStream().getClass().getCanonicalName());
inputStreamLocal = new TimedFSDataInputStream(logFile.getPath(), new
FSDataInputStream(
new BufferedFSInputStream((FSInputStream)
fsDataInputStream.getWrappedStream(), bufferSize)));
} else if (FSUtils.isGCSFileSystem(fs)) {
LOG.warn("HoodieLogFileReader aaa " + logFile.getFileName() + " "
+ fsDataInputStream.getWrappedStream().getClass().getCanonicalName());
try {
FSInputStream localFSInputStream =
(FSInputStream)(((FSDataInputStream)fsDataInputStream.getWrappedStream()).getWrappedStream());
inputStreamLocal = new SchemeAwareFSDataInputStream(new
TimedFSDataInputStream(logFile.getPath(), new FSDataInputStream(
new BufferedFSInputStream(localFSInputStream,bufferSize))),
true);
LOG.warn("HoodieLogFileReader aaa succeeded " +
logFile.getFileName());
} catch (ClassCastException e) {
Log.warn("HoodieLogFileReader bbb (aaa failed) " +
logFile.getFileName() + " " + e.getCause()
+ ", msg " + e.getMessage());
// if we cannot cast
fsDataInputStream.getWrappedStream().getWrappedStream() to FSInputStream,
fallback to using as is
LOG.warn("Cannot cast
fsDataInputStream.getWrappedStream().getWrappedStream() to FSInputStream with
GCSFileSystem, falling back to original "
+ "fsDataInputStream");
inputStreamLocal = fsDataInputStream;
}
} else {
// fsDataInputStream.getWrappedStream() maybe a BufferedFSInputStream
// need to wrap in another BufferedFSInputStream the make bufferSize
work?
LOG.warn("HoodieLogFileReader " + logFile.getFileName());
inputStreamLocal = fsDataInputStream;
}
```
Output from the run:
"HoodieLogFileReader
.0d7ba334-2847-4b24-997e-1dbecfd12e3b-0_20210306132835.log.1_0-55-75
[GitHub] [hudi] nsivabalan commented on pull request #2500: [HUDI-1496] Fixing detection of GCS FileSystem
nsivabalan commented on pull request #2500:
URL: https://github.com/apache/hudi/pull/2500#issuecomment-792339635
If I am not wrong, there was a bug in the code version that you ran. I
tested 4 different variants of code to arrive at the latest proposal. Let me
walk through them :) Sorry about the lengthy response. Hopefully we get a
closure.
1st variant. Current master branch:
```
FSDataInputStream fsDataInputStream = fs.open(logFile.getPath(), bufferSize);
LOG.warn("HoodieLogFileReader :: canonical name :: " +
fsDataInputStream.getClass().getCanonicalName() + ", name "
+ fsDataInputStream.getClass().getName());
if (FSUtils.isGCSInputStream(fsDataInputStream)) {
LOG.warn("HoodieLogFileReader :: 111 start GCSFileSystem " +
fsDataInputStream.getWrappedStream().getClass().getCanonicalName());
this.inputStream = new TimedFSDataInputStream(logFile.getPath(), new
FSDataInputStream(
new BufferedFSInputStream((FSInputStream) ((
(FSDataInputStream)
fsDataInputStream.getWrappedStream()).getWrappedStream()), bufferSize)));
LOG.warn("HoodieLogFileReader :: 111 completed ");
} else if (fsDataInputStream.getWrappedStream() instanceof
FSInputStream) {
LOG.warn("HoodieLogFileReader :: 222 start " +
fsDataInputStream.getWrappedStream().getClass().getCanonicalName());
this.inputStream = new TimedFSDataInputStream(logFile.getPath(), new
FSDataInputStream(
new BufferedFSInputStream((FSInputStream)
fsDataInputStream.getWrappedStream(), bufferSize)));
LOG.warn("HoodieLogFileReader :: 222 complete");
} else {
LOG.warn("HoodieLogFileReader :: 333 ");
// fsDataInputStream.getWrappedStream() maybe a BufferedFSInputStream
// need to wrap in another BufferedFSInputStream the make bufferSize
work?
this.inputStream = fsDataInputStream;
}
```
Output from my run:
"HoodieLogFileReader :: canonical name ::
org.apache.hadoop.fs.FSDataInputStream, name
org.apache.hadoop.fs.FSDataInputStream"
"HoodieLogFileReader :: 111 start GCSFileSystem
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream"
Caused by: java.lang.ClassCastException:
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream cannot be cast to
org.apache.hadoop
.fs.FSDataInputStream
at
org.apache.hudi.common.table.log.HoodieLogFileReader.(HoodieLogFileReader.java:84)
at
org.apache.hudi.common.table.log.HoodieLogFormatReader.(HoodieLogFormatReader.java:62)
at
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.scan(AbstractHoodieLogRecordScanner.java:131)
... 24 more
2nd variant:
This PR just before my last commit.
```
if (fsDataInputStream.getWrappedStream() instanceof FSInputStream) {
LOG.warn("HoodieLogFileReader " + logFile.getFileName() + " " +
fsDataInputStream.getWrappedStream().getClass().getCanonicalName());
inputStreamLocal = new TimedFSDataInputStream(logFile.getPath(), new
FSDataInputStream(
new BufferedFSInputStream((FSInputStream)
fsDataInputStream.getWrappedStream(), bufferSize)));
} else if (FSUtils.isGCSFileSystem(fs)) {
LOG.warn("HoodieLogFileReader aaa " + logFile.getFileName() + " "
+ fsDataInputStream.getWrappedStream().getClass().getCanonicalName());
try {
FSInputStream localFSInputStream =
(FSInputStream)(((FSDataInputStream)fsDataInputStream.getWrappedStream()).getWrappedStream());
inputStreamLocal = new SchemeAwareFSDataInputStream(new
TimedFSDataInputStream(logFile.getPath(), new FSDataInputStream(
new BufferedFSInputStream(localFSInputStream,bufferSize))),
true);
LOG.warn("HoodieLogFileReader aaa succeeded " +
logFile.getFileName());
} catch (ClassCastException e) {
Log.warn("HoodieLogFileReader bbb (aaa failed) " +
logFile.getFileName() + " " + e.getCause()
+ ", msg " + e.getMessage());
// if we cannot cast
fsDataInputStream.getWrappedStream().getWrappedStream() to FSInputStream,
fallback to using as is
LOG.warn("Cannot cast
fsDataInputStream.getWrappedStream().getWrappedStream() to FSInputStream with
GCSFileSystem, falling back to original "
+ "fsDataInputStream");
inputStreamLocal = fsDataInputStream;
}
} else {
// fsDataInputStream.getWrappedStream() maybe a BufferedFSInputStream
// need to wrap in another BufferedFSInputStream the make bufferSize
work?
LOG.warn("HoodieLogFileReader " + logFile.getFileName());
inputStreamLocal = fsDataInputStream;
}
```
Output from the run:
"HoodieLogFileReader
.0d7ba334-2847-4b24-997e-1dbecfd12e3b-0_20210306132835.log.1_0-55-75
[GitHub] [hudi] codecov-io commented on pull request #2642: [HUDI-1673] Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex
codecov-io commented on pull request #2642: URL: https://github.com/apache/hudi/pull/2642#issuecomment-792276483 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2642?src=pr=h1) Report > Merging [#2642](https://codecov.io/gh/apache/hudi/pull/2642?src=pr=desc) (2008494) into [master](https://codecov.io/gh/apache/hudi/commit/06dc7c7fd8a867a1e1da90f7dc19b0cc2da69bba?el=desc) (06dc7c7) will **increase** coverage by `10.26%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hudi/pull/2642?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2642 +/- ## = + Coverage 51.22% 61.48% +10.26% + Complexity 3230 324 -2906 = Files 438 53 -385 Lines 20093 1963-18130 Branches 2069 235 -1834 = - Hits 10292 1207 -9085 + Misses 8954 632 -8322 + Partials847 124 -723 ``` | Flag | Coverage Δ | Complexity Δ | | |---|---|---|---| | hudicli | `?` | `?` | | | hudiclient | `?` | `?` | | | hudicommon | `?` | `?` | | | hudiflink | `?` | `?` | | | hudihadoopmr | `?` | `?` | | | hudisparkdatasource | `?` | `?` | | | hudisync | `?` | `?` | | | huditimelineservice | `?` | `?` | | | hudiutilities | `61.48% <ø> (-8.04%)` | `0.00 <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2642?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ies/exception/HoodieSnapshotExporterException.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVTbmFwc2hvdEV4cG9ydGVyRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | | | [.../apache/hudi/utilities/HoodieSnapshotExporter.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVNuYXBzaG90RXhwb3J0ZXIuamF2YQ==) | `5.17% <0.00%> (-83.63%)` | `0.00% <0.00%> (-28.00%)` | | | [...hudi/utilities/schema/JdbcbasedSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9KZGJjYmFzZWRTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (-72.23%)` | `0.00% <0.00%> (-2.00%)` | | | [...he/hudi/utilities/transform/AWSDmsTransformer.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3RyYW5zZm9ybS9BV1NEbXNUcmFuc2Zvcm1lci5qYXZh) | `0.00% <0.00%> (-66.67%)` | `0.00% <0.00%> (-2.00%)` | | | [...in/java/org/apache/hudi/utilities/UtilHelpers.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL1V0aWxIZWxwZXJzLmphdmE=) | `40.69% <0.00%> (-25.00%)` | `27.00% <0.00%> (-5.00%)` | | | [...hudi/utilities/sources/helpers/KafkaOffsetGen.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9LYWZrYU9mZnNldEdlbi5qYXZh) | `85.84% <0.00%> (-2.94%)` | `20.00% <0.00%> (+4.00%)` | :arrow_down: | | [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.00% <0.00%> (-0.36%)` | `52.00% <0.00%> (+1.00%)` | :arrow_down: | | [...n/java/org/apache/hudi/common/metrics/Counter.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21ldHJpY3MvQ291bnRlci5qYXZh) | | | | | [...ache/hudi/common/table/timeline/TimelineUtils.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL1RpbWVsaW5lVXRpbHMuamF2YQ==) | | | | | [...i/bootstrap/SparkParquetBootstrapDataProvider.java](https://codecov.io/gh/apache/hudi/pull/2642/diff?src=pr=tree#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3Bhcmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvYm9vdHN0cmFwL1NwYXJrUGFycXVldEJvb3RzdHJhcERhdGFQcm92aWRlci5qYXZh) | | | | | ...
[jira] [Updated] (HUDI-1673) Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex
[ https://issues.apache.org/jira/browse/HUDI-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1673: - Labels: pull-request-available (was: ) > Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex > > > Key: HUDI-1673 > URL: https://issues.apache.org/jira/browse/HUDI-1673 > Project: Apache Hudi > Issue Type: Sub-task >Reporter: shenh062326 >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] shenh062326 opened a new pull request #2642: [HUDI-1673] Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex
shenh062326 opened a new pull request #2642: URL: https://github.com/apache/hudi/pull/2642 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-1673) Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex
shenh062326 created HUDI-1673: - Summary: Replace scala.Tule2 to Pair in FlinkHoodieBloomIndex Key: HUDI-1673 URL: https://issues.apache.org/jira/browse/HUDI-1673 Project: Apache Hudi Issue Type: Sub-task Reporter: shenh062326 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] shenh062326 commented on a change in pull request #2608: [HUDI-1478] Introduce HoodieBloomIndex to hudi-java-client
shenh062326 commented on a change in pull request #2608:
URL: https://github.com/apache/hudi/pull/2608#discussion_r589001773
##
File path:
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/index/bloom/FlinkHoodieBloomIndex.java
##
@@ -18,250 +18,14 @@
package org.apache.hudi.index.bloom;
-import org.apache.hudi.client.WriteStatus;
-import org.apache.hudi.common.engine.HoodieEngineContext;
-import org.apache.hudi.common.model.HoodieKey;
-import org.apache.hudi.common.model.HoodieRecord;
-import org.apache.hudi.common.model.HoodieRecordLocation;
import org.apache.hudi.common.model.HoodieRecordPayload;
-import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.collection.Pair;
import org.apache.hudi.config.HoodieWriteConfig;
-import org.apache.hudi.exception.MetadataNotFoundException;
-import org.apache.hudi.index.FlinkHoodieIndex;
-import org.apache.hudi.index.HoodieIndexUtils;
-import org.apache.hudi.io.HoodieKeyLookupHandle;
-import org.apache.hudi.io.HoodieRangeInfoHandle;
-import org.apache.hudi.table.HoodieTable;
-
-import org.apache.log4j.LogManager;
-import org.apache.log4j.Logger;
-import com.beust.jcommander.internal.Lists;
-
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-
-import scala.Tuple2;
-
-import static java.util.stream.Collectors.mapping;
-import static java.util.stream.Collectors.groupingBy;
-import static java.util.stream.Collectors.toList;
-import static
org.apache.hudi.index.HoodieIndexUtils.getLatestBaseFilesForAllPartitions;
/**
* Indexing mechanism based on bloom filter. Each parquet file includes its
row_key bloom filter in its metadata.
*/
-@SuppressWarnings("checkstyle:LineLength")
-public class FlinkHoodieBloomIndex extends
FlinkHoodieIndex {
Review comment:
Same as mentioned above, It seems better to move most implements from
FlinkHoodieBloomIndex.java to HoodieBaseBloomIndex.java, then it can be shared
with java client.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] shenh062326 commented on a change in pull request #2608: [HUDI-1478] Introduce HoodieBloomIndex to hudi-java-client
shenh062326 commented on a change in pull request #2608:
URL: https://github.com/apache/hudi/pull/2608#discussion_r589001564
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/HoodieBaseBloomIndexCheckFunction.java
##
@@ -33,44 +33,42 @@
import java.util.Iterator;
import java.util.List;
-import scala.Tuple2;
-
/**
* Function performing actual checking of list containing (fileId, hoodieKeys)
against the actual files.
*/
//TODO we can move this class into the hudi-client-common and reuse it for
spark client
-public class HoodieFlinkBloomIndexCheckFunction
-implements Function>,
Iterator>> {
+public class HoodieBaseBloomIndexCheckFunction
+implements Function>,
Iterator>> {
private final HoodieTable hoodieTable;
private final HoodieWriteConfig config;
- public HoodieFlinkBloomIndexCheckFunction(HoodieTable hoodieTable,
HoodieWriteConfig config) {
+ public HoodieBaseBloomIndexCheckFunction(HoodieTable hoodieTable,
HoodieWriteConfig config) {
this.hoodieTable = hoodieTable;
this.config = config;
}
@Override
- public Iterator> apply(Iterator> fileParitionRecordKeyTripletItr) {
+ public Iterator> apply(Iterator> fileParitionRecordKeyTripletItr) {
Review comment:
It seems better to move most implements from FlinkHoodieBloomIndex.java
to HoodieBaseBloomIndex.java, then it can be shared with java client. Since
java client not dependent on scala, so we need to replace scala.Tuple2 with
Pair.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #2608: [HUDI-1478] Introduce HoodieBloomIndex to hudi-java-client
leesf commented on a change in pull request #2608:
URL: https://github.com/apache/hudi/pull/2608#discussion_r588993688
##
File path:
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/index/bloom/FlinkHoodieBloomIndex.java
##
@@ -18,250 +18,14 @@
package org.apache.hudi.index.bloom;
-import org.apache.hudi.client.WriteStatus;
-import org.apache.hudi.common.engine.HoodieEngineContext;
-import org.apache.hudi.common.model.HoodieKey;
-import org.apache.hudi.common.model.HoodieRecord;
-import org.apache.hudi.common.model.HoodieRecordLocation;
import org.apache.hudi.common.model.HoodieRecordPayload;
-import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.collection.Pair;
import org.apache.hudi.config.HoodieWriteConfig;
-import org.apache.hudi.exception.MetadataNotFoundException;
-import org.apache.hudi.index.FlinkHoodieIndex;
-import org.apache.hudi.index.HoodieIndexUtils;
-import org.apache.hudi.io.HoodieKeyLookupHandle;
-import org.apache.hudi.io.HoodieRangeInfoHandle;
-import org.apache.hudi.table.HoodieTable;
-
-import org.apache.log4j.LogManager;
-import org.apache.log4j.Logger;
-import com.beust.jcommander.internal.Lists;
-
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-
-import scala.Tuple2;
-
-import static java.util.stream.Collectors.mapping;
-import static java.util.stream.Collectors.groupingBy;
-import static java.util.stream.Collectors.toList;
-import static
org.apache.hudi.index.HoodieIndexUtils.getLatestBaseFilesForAllPartitions;
/**
* Indexing mechanism based on bloom filter. Each parquet file includes its
row_key bloom filter in its metadata.
*/
-@SuppressWarnings("checkstyle:LineLength")
-public class FlinkHoodieBloomIndex extends
FlinkHoodieIndex {
Review comment:
why change flinkhoodiebloomindex since it only introduce hoodieblooindex
to java client?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #2608: [HUDI-1478] Introduce HoodieBloomIndex to hudi-java-client
leesf commented on a change in pull request #2608:
URL: https://github.com/apache/hudi/pull/2608#discussion_r588993489
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/HoodieBaseBloomIndexCheckFunction.java
##
@@ -33,44 +33,42 @@
import java.util.Iterator;
import java.util.List;
-import scala.Tuple2;
-
/**
* Function performing actual checking of list containing (fileId, hoodieKeys)
against the actual files.
*/
//TODO we can move this class into the hudi-client-common and reuse it for
spark client
-public class HoodieFlinkBloomIndexCheckFunction
-implements Function>,
Iterator>> {
+public class HoodieBaseBloomIndexCheckFunction
+implements Function>,
Iterator>> {
private final HoodieTable hoodieTable;
private final HoodieWriteConfig config;
- public HoodieFlinkBloomIndexCheckFunction(HoodieTable hoodieTable,
HoodieWriteConfig config) {
+ public HoodieBaseBloomIndexCheckFunction(HoodieTable hoodieTable,
HoodieWriteConfig config) {
this.hoodieTable = hoodieTable;
this.config = config;
}
@Override
- public Iterator> apply(Iterator> fileParitionRecordKeyTripletItr) {
+ public Iterator> apply(Iterator> fileParitionRecordKeyTripletItr) {
Review comment:
any reason to replace Tuple2 with Pair?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
