xushiyan commented on a change in pull request #2426:
URL: https://github.com/apache/hudi/pull/2426#discussion_r554531429
##
File path: pom.xml
##
@@ -198,34 +200,36 @@
-
xushiyan removed a comment on pull request #2426:
URL: https://github.com/apache/hudi/pull/2426#issuecomment-757438915
## TODO
- [ ] Manually verify some diffs after spotless apply won't conflict with
IDE formatter and checkstyle
xushiyan commented on pull request #2426:
URL: https://github.com/apache/hudi/pull/2426#issuecomment-760076561
@vinothchandar The style can be sync'ed by
- using google-java-format in spotless config and `spotless:apply` enforces
the style that is also compatible with existing checkstyl
quitozang opened a new issue #2446:
URL: https://github.com/apache/hudi/issues/2446
Why does this parameter "hoodie.bloom.index.filter.type" not take effect in
deltaStreamer, the bloom filter type is always in SIMPLE.
This i
wangxianghu commented on a change in pull request #2434:
URL: https://github.com/apache/hudi/pull/2434#discussion_r557021821
##
File path:
hudi-flink/src/main/java/org/apache/hudi/operator/InstantGenerateOperator.java
##
@@ -71,16 +72,18 @@
private String latestInstant = ""
wangxianghu commented on a change in pull request #2434:
URL: https://github.com/apache/hudi/pull/2434#discussion_r557021821
##
File path:
hudi-flink/src/main/java/org/apache/hudi/operator/InstantGenerateOperator.java
##
@@ -71,16 +72,18 @@
private String latestInstant = ""
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=h1) Report
> Merging
[#2334](https://codecov.io/gh/apache/hudi/pull/2334?src=pr&el=desc) (bbb604a)
in
wangxianghu commented on a change in pull request #2434:
URL: https://github.com/apache/hudi/pull/2434#discussion_r557021821
##
File path:
hudi-flink/src/main/java/org/apache/hudi/operator/InstantGenerateOperator.java
##
@@ -71,16 +72,18 @@
private String latestInstant = ""
yanghua commented on a change in pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#discussion_r557306549
##
File path: hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
##
@@ -81,16 +103,50 @@ public static DFSPropertiesConfiguration
readConfig(Fi
teeyog created HUDI-1527:
Summary: Automatically infer the data directory, users only need
to specify the table directory
Key: HUDI-1527
URL: https://issues.apache.org/jira/browse/HUDI-1527
Project: Apache Hu
[
https://issues.apache.org/jira/browse/HUDI-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
teeyog updated HUDI-1527:
-
Description:
To read the hudi table, you need to specify the path, but the path is not only
the tablePath corresp
teeyog opened a new pull request #2447:
URL: https://github.com/apache/hudi/pull/2447
…o specify the table directory
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a
pull request.*
[
https://issues.apache.org/jira/browse/HUDI-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-1527:
-
Labels: pull-request-available (was: )
> Automatically infer the data directory, users only need
codecov-io edited a comment on pull request #2334:
URL: https://github.com/apache/hudi/pull/2334#issuecomment-745334158
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitH
yanghua commented on a change in pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#discussion_r557329799
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestAvroPayload.java
##
@@ -79,6 +79,11 @@ public OverwriteWithLatestAvr
yanghua commented on a change in pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#discussion_r557331716
##
File path: hudi-flink/pom.xml
##
@@ -124,28 +124,77 @@
kafka-clients
${kafka.version}
+
+ org.apache.flink
+
flink-hado
yanghua commented on a change in pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#discussion_r557331716
##
File path: hudi-flink/pom.xml
##
@@ -124,28 +124,77 @@
kafka-clients
${kafka.version}
+
+ org.apache.flink
+
flink-hado
yanghua commented on a change in pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#discussion_r557337056
##
File path: hudi-flink/src/main/java/org/apache/hudi/operator/HoodieOptions.java
##
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (A
codecov-io commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760147630
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=h1) Report
> Merging
[#2443](https://codecov.io/gh/apache/hudi/pull/2443?src=pr&el=desc) (0d98db7)
into
[ma
yanghua commented on a change in pull request #2434:
URL: https://github.com/apache/hudi/pull/2434#discussion_r557347825
##
File path:
hudi-flink/src/main/java/org/apache/hudi/operator/InstantGenerateOperator.java
##
@@ -71,16 +72,18 @@
private String latestInstant = "";
Trevorzhang created HUDI-1528:
-
Summary: hudi-sync-tool error
Key: HUDI-1528
URL: https://issues.apache.org/jira/browse/HUDI-1528
Project: Apache Hudi
Issue Type: Bug
Components: Hive I
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264859#comment-17264859
]
Trevorzhang edited comment on HUDI-1528 at 1/14/21, 12:25 PM:
--
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264859#comment-17264859
]
Trevorzhang commented on HUDI-1528:
---
{code:java}
//代码占位符[lingqu@xx-dev-cq-ecs-dtpbu-
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264859#comment-17264859
]
Trevorzhang edited comment on HUDI-1528 at 1/14/21, 12:25 PM:
--
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trevorzhang updated HUDI-1528:
--
Comment: was deleted
(was:
{code:java}
//[lingqu@xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01 jars]$ sh
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264862#comment-17264862
]
Trevorzhang edited comment on HUDI-1528 at 1/14/21, 12:28 PM:
--
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264862#comment-17264862
]
Trevorzhang commented on HUDI-1528:
---
{panel:title=我的标题}
[lingqu@xx-dev-cq-ecs-dtpbu-data
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trevorzhang updated HUDI-1528:
--
Comment: was deleted
(was: {panel:title=log}
[lingqu@xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01 jars]$ sh
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264868#comment-17264868
]
Trevorzhang commented on HUDI-1528:
---
21/01/14 20:21:00 INFO hive.HoodieHiveClient: Creat
peng-xin opened a new issue #2448:
URL: https://github.com/apache/hudi/issues/2448
**Environment Description**
* Hudi version :
0.6.0
* Spark version :
spark-2.4.4-bin-hadoop2.7
* Hive version :
hive-2.3.4
* Hadoop version :
hadoop2.7.3
* Storage (HDFS/S3/GCS..) :
codecov-io edited a comment on pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#issuecomment-757736411
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2430?src=pr&el=h1) Report
> Merging
[#2430](https://codecov.io/gh/apache/hudi/pull/2430?src=pr&el=desc) (c4a04f9)
in
codecov-io edited a comment on pull request #2260:
URL: https://github.com/apache/hudi/pull/2260#issuecomment-729530724
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2260?src=pr&el=h1) Report
> Merging
[#2260](https://codecov.io/gh/apache/hudi/pull/2260?src=pr&el=desc) (7d0453e)
in
Trevor-zhang opened a new pull request #2449:
URL: https://github.com/apache/hudi/pull/2449
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a
pull request.*
## What is the purpose of th
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-1528:
-
Labels: pull-request-available (was: )
> hudi-sync-tool error
>
>
>
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trevorzhang updated HUDI-1528:
--
Summary: hudi-sync-tools error (was: hudi-sync-tool error)
> hudi-sync-tools error
> --
[
https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trevorzhang updated HUDI-1528:
--
Description:
When using hudi-sync-tools to synchronize to a remote hive, hivemetastore throw
exceptions
codecov-io edited a comment on pull request #2430:
URL: https://github.com/apache/hudi/pull/2430#issuecomment-757736411
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitH
codecov-io edited a comment on pull request #2260:
URL: https://github.com/apache/hudi/pull/2260#issuecomment-729530724
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitH
yanghua commented on a change in pull request #2434:
URL: https://github.com/apache/hudi/pull/2434#discussion_r557462779
##
File path:
hudi-flink/src/main/java/org/apache/hudi/operator/InstantGenerateOperator.java
##
@@ -71,16 +73,18 @@
private String latestInstant = "";
yanghua commented on pull request #2449:
URL: https://github.com/apache/hudi/pull/2449#issuecomment-760264278
@wangxianghu please help to review thanks.
This is an automated message from the Apache Git Service.
To respond to
yanghua commented on pull request #2443:
URL: https://github.com/apache/hudi/pull/2443#issuecomment-760270387
@liujinhui1994 Travis is red. @wangxianghu help to review firstly.
This is an automated message from the Apache Git
codecov-io commented on pull request #2444:
URL: https://github.com/apache/hudi/pull/2444#issuecomment-760286269
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2444?src=pr&el=h1) Report
> Merging
[#2444](https://codecov.io/gh/apache/hudi/pull/2444?src=pr&el=desc) (0b4eb5c)
into
[ma
vburenin commented on a change in pull request #2440:
URL: https://github.com/apache/hudi/pull/2440#discussion_r557508471
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
##
@@ -274,19 +275,27 @@ private boolean isBlockCorrupt(i
vburenin commented on pull request #2440:
URL: https://github.com/apache/hudi/pull/2440#issuecomment-760299104
> @vburenin Left a comment to restructure the code to support buffering, are
you going to look into improving the O(m*n) search ?
At this point of time I think it is not necessa
rakeshramakrishnan commented on pull request #2449:
URL: https://github.com/apache/hudi/pull/2449#issuecomment-760299334
Much thanks! This would address #2439 ?
This is an automated message from the Apache Git Service.
To res
rakeshramakrishnan edited a comment on pull request #2449:
URL: https://github.com/apache/hudi/pull/2449#issuecomment-760299334
Awesome! This would address #2439 ?
This is an automated message from the Apache Git Service.
To
codecov-io edited a comment on pull request #2444:
URL: https://github.com/apache/hudi/pull/2444#issuecomment-760286269
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2444?src=pr&el=h1) Report
> Merging
[#2444](https://codecov.io/gh/apache/hudi/pull/2444?src=pr&el=desc) (0b4eb5c)
in
codecov-io edited a comment on pull request #2444:
URL: https://github.com/apache/hudi/pull/2444#issuecomment-760286269
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitH
vburenin opened a new pull request #2450:
URL: https://github.com/apache/hudi/pull/2450
## What is the purpose of the pull request
UtilHelpers.createSource had a hardcoded way of checking which
constructor signature needs to be used to instantiate a class
which makes it impossible t
n3nash merged pull request #2424:
URL: https://github.com/apache/hudi/pull/2424
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
This is an automated email from the ASF dual-hosted git repository.
nagarwal pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 749f657 [HUDI-1509]: Reverting LinkedHashSet ch
codecov-io edited a comment on pull request #2431:
URL: https://github.com/apache/hudi/pull/2431#issuecomment-757929313
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2431?src=pr&el=h1) Report
> Merging
[#2431](https://codecov.io/gh/apache/hudi/pull/2431?src=pr&el=desc) (e63414d)
in
[
https://issues.apache.org/jira/browse/HUDI-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-1526:
-
Labels: pull-request-available (was: )
> Translate the spark api partitionBy to
> hoodie.datasou
codecov-io edited a comment on pull request #2434:
URL: https://github.com/apache/hudi/pull/2434#issuecomment-758857465
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2434?src=pr&el=h1) Report
> Merging
[#2434](https://codecov.io/gh/apache/hudi/pull/2434?src=pr&el=desc) (53d9942)
in
codecov-io edited a comment on pull request #2431:
URL: https://github.com/apache/hudi/pull/2431#issuecomment-757929313
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2431?src=pr&el=h1) Report
> Merging
[#2431](https://codecov.io/gh/apache/hudi/pull/2431?src=pr&el=desc) (e63414d)
in
[
https://issues.apache.org/jira/browse/HUDI-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udit Mehrotra reassigned HUDI-1529:
---
Assignee: Udit Mehrotra
> Spark-SQL drvier runs out of memory when metadata table is enabled
Udit Mehrotra created HUDI-1529:
---
Summary: Spark-SQL drvier runs out of memory when metadata table
is enabled
Key: HUDI-1529
URL: https://issues.apache.org/jira/browse/HUDI-1529
Project: Apache Hudi
[
https://issues.apache.org/jira/browse/HUDI-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udit Mehrotra updated HUDI-1529:
Description:
When testing a large dataset around 1.2TB data and around 20k files, we notice
an issu
vinothchandar commented on pull request #2442:
URL: https://github.com/apache/hudi/pull/2442#issuecomment-760571146
@nsivabalan can we just fix the configs first on the current version of the
site. its possible we will make more changes until we release? We can make the
0.7.0 specific page
vburenin edited a comment on pull request #2440:
URL: https://github.com/apache/hudi/pull/2440#issuecomment-760299104
> @vburenin Left a comment to restructure the code to support buffering, are
you going to look into improving the O(m*n) search ?
At this point of time I think it is
vinothchandar commented on pull request #2440:
URL: https://github.com/apache/hudi/pull/2440#issuecomment-760571612
@n3nash can we take a call on this and get it into the current release.
marking as blocker for now.
This is
vinothchandar commented on pull request #2440:
URL: https://github.com/apache/hudi/pull/2440#issuecomment-760571874
@vburenin do you mind creating a JIRA for this issue.? We can give you
perms if you can ping us your id from issue.apache.org/jira
umehrot2 opened a new pull request #2451:
URL: https://github.com/apache/hudi/pull/2451
## What is the purpose of the pull request
This PR fixes an issue we identified when enabling **metadata table** for
SparkSQL queries, which cause a huge number of file splits to be generate,
cau
[
https://issues.apache.org/jira/browse/HUDI-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-1529:
-
Labels: pull-request-available (was: )
> Spark-SQL drvier runs out of memory when metadata table
wangxianghu commented on a change in pull request #2449:
URL: https://github.com/apache/hudi/pull/2449#discussion_r557806942
##
File path:
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java
##
@@ -49,6 +49,9 @@
@Parameter(names = {"--jdbc-url"},
wangxianghu commented on a change in pull request #2449:
URL: https://github.com/apache/hudi/pull/2449#discussion_r557807516
##
File path:
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java
##
@@ -49,6 +49,9 @@
@Parameter(names = {"--jdbc-url"},
Trevor-zhang commented on pull request #2449:
URL: https://github.com/apache/hudi/pull/2449#issuecomment-760586002
> Awesome! This would address #2439 ?
I'm not sure if it can solve your problem, wait for me to test it.
---
Trevor-zhang commented on a change in pull request #2449:
URL: https://github.com/apache/hudi/pull/2449#discussion_r557807620
##
File path:
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfig.java
##
@@ -49,6 +49,9 @@
@Parameter(names = {"--jdbc-url"}
Trevorzhang created HUDI-1530:
-
Summary: make HoodieDeltaStreamer and SparkDataSource support
HiveMetaStore
Key: HUDI-1530
URL: https://issues.apache.org/jira/browse/HUDI-1530
Project: Apache Hudi
loukey-lj commented on a change in pull request #2434:
URL: https://github.com/apache/hudi/pull/2434#discussion_r557818230
##
File path:
hudi-flink/src/main/java/org/apache/hudi/operator/InstantGenerateOperator.java
##
@@ -222,4 +234,59 @@ public void close() throws Exception
bvaradar commented on issue #2439:
URL: https://github.com/apache/hudi/issues/2439#issuecomment-760616648
@satishkotha : Can you help with this ?
This is an automated message from the Apache Git Service.
To respond to the mes
bvaradar commented on issue #2446:
URL: https://github.com/apache/hudi/issues/2446#issuecomment-760617405
@quitozang : Which version of Hoodie are you using ? Are you passing the
configuration like "--hoodie-conf hoodie.bloom.index.filter.type=ABC" ?
@nsivabalan : Can you follow-up
loukey-lj commented on a change in pull request #2434:
URL: https://github.com/apache/hudi/pull/2434#discussion_r557820490
##
File path:
hudi-flink/src/main/java/org/apache/hudi/operator/InstantGenerateOperator.java
##
@@ -102,65 +105,76 @@ public void open() throws Exception
loukey-lj commented on a change in pull request #2434:
URL: https://github.com/apache/hudi/pull/2434#discussion_r557821706
##
File path:
hudi-flink/src/main/java/org/apache/hudi/operator/InstantGenerateOperator.java
##
@@ -71,16 +73,18 @@
private String latestInstant = "";
vinothchandar commented on a change in pull request #2451:
URL: https://github.com/apache/hudi/pull/2451#discussion_r557874678
##
File path:
hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java
##
@@ -202,7 +202,7 @@ protected BaseTableMetadata(HoodieEngin
so-lazy commented on issue #2338:
URL: https://github.com/apache/hudi/issues/2338#issuecomment-760708042
@bvaradar sir, now i used global simple index, but for some satages
**Getting small files from partitions**
**Compacting file slices**
they cost so long mintues, and i attach m
76 matches
Mail list logo
