liujinhui1994 commented on issue #2162:
URL: https://github.com/apache/hudi/issues/2162#issuecomment-707525990
This is the data I printed in the transform
1. This is before adding the ds field
|-- collectTime: strin
lw309637554 commented on pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#issuecomment-707514223
> > > > @leesf #2048 is landed. is it possible to merge this and address
Balaji's comments? (I can help if needed)
> > >
> > >
> > > Sure, considering I am a little b
bvaradar commented on issue #2162:
URL: https://github.com/apache/hudi/issues/2162#issuecomment-707502829
1. I am not able to pinpoint the issue rightaway but let me engage in
debugging this with you. Couple of things :
1. Can you make ds field and any additional fields you are addi
satishkotha commented on pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#issuecomment-707476925
> > > @leesf #2048 is landed. is it possible to merge this and address
Balaji's comments? (I can help if needed)
> >
> >
> > Sure, considering I am a little busy thes
[
https://issues.apache.org/jira/browse/HUDI-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
satish resolved HUDI-1304.
--
Resolution: Fixed
> test compaction workflow with replacecommit action
> ---
[
https://issues.apache.org/jira/browse/HUDI-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
satish updated HUDI-1260:
-
Status: In Progress (was: Open)
> Reader changes to supportinsert overwrite
> ---
[
https://issues.apache.org/jira/browse/HUDI-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
satish resolved HUDI-1260.
--
Resolution: Fixed
> Reader changes to supportinsert overwrite
> -
>
>
[
https://issues.apache.org/jira/browse/HUDI-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
satish updated HUDI-1260:
-
Status: Open (was: New)
> Reader changes to supportinsert overwrite
> -
>
[
https://issues.apache.org/jira/browse/HUDI-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
satish updated HUDI-1304:
-
Status: Open (was: New)
> test compaction workflow with replacecommit action
> --
bvaradar commented on issue #2165:
URL: https://github.com/apache/hudi/issues/2165#issuecomment-707475775
@tandonraghav : Yes, you need to shade the jar containing the custom record
payload. Here is some context
http://hudi.apache.org/releases.html#release-highlights-1
Look for s
[
https://issues.apache.org/jira/browse/HUDI-1304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
satish updated HUDI-1304:
-
Status: In Progress (was: Open)
> test compaction workflow with replacecommit action
> --
lw309637554 edited a comment on pull request #2127:
URL: https://github.com/apache/hudi/pull/2127#issuecomment-706813674
> lagging a bit. Will take a pass today and circle back.
@pratyakshsharma thanks，please help to review
---
[
https://issues.apache.org/jira/browse/HUDI-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-781:
Fix Version/s: 0.6.1
> Re-design test utilities
>
>
> Key: HUDI-781
[
https://issues.apache.org/jira/browse/HUDI-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-995:
Fix Version/s: (was: 0.6.1)
> Organize test utils methods and classes
> -
[
https://issues.apache.org/jira/browse/HUDI-779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu resolved HUDI-779.
-
Assignee: Raymond Xu
Resolution: Done
> [Umbrella] Unit tests improvements
>
[
https://issues.apache.org/jira/browse/HUDI-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu resolved HUDI-781.
-
Resolution: Implemented
> Re-design test utilities
>
>
> Key: HUDI
[
https://issues.apache.org/jira/browse/HUDI-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu closed HUDI-781.
---
> Re-design test utilities
>
>
> Key: HUDI-781
> URL: http
[
https://issues.apache.org/jira/browse/HUDI-779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-779:
Status: Open (was: New)
> [Umbrella] Unit tests improvements
> --
>
>
[
https://issues.apache.org/jira/browse/HUDI-779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu closed HUDI-779.
---
> [Umbrella] Unit tests improvements
> --
>
> Key: HUDI-779
>
[
https://issues.apache.org/jira/browse/HUDI-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-1010:
-
Parent: (was: HUDI-781)
Issue Type: Bug (was: Sub-task)
> Fix the memory leak for hudi-client
[
https://issues.apache.org/jira/browse/HUDI-994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu updated HUDI-994:
Status: Open (was: New)
> Identify functional tests that are convertible to unit tests with mocks
>
[
https://issues.apache.org/jira/browse/HUDI-994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu resolved HUDI-994.
-
Resolution: Done
> Identify functional tests that are convertible to unit tests with mocks
> --
[
https://issues.apache.org/jira/browse/HUDI-994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu reassigned HUDI-994:
---
Assignee: Raymond Xu (was: Prashant Wason)
> Identify functional tests that are convertible to unit t
[
https://issues.apache.org/jira/browse/HUDI-994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu closed HUDI-994.
---
> Identify functional tests that are convertible to unit tests with mocks
> ---
[
https://issues.apache.org/jira/browse/HUDI-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu resolved HUDI-996.
-
Resolution: Done
Closing this as the functional test utilities are implemented. The future work
is to deci
[
https://issues.apache.org/jira/browse/HUDI-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu closed HUDI-996.
---
Assignee: Raymond Xu
> Use shared spark session provider
> --
>
>
[
https://issues.apache.org/jira/browse/HUDI-896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu closed HUDI-896.
---
> Parallelize CI testing to reduce CI wait time
> -
>
>
[
https://issues.apache.org/jira/browse/HUDI-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu closed HUDI-995.
---
> Organize test utils methods and classes
> ---
>
> Key: HUDI-9
[
https://issues.apache.org/jira/browse/HUDI-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymond Xu resolved HUDI-995.
-
Fix Version/s: 0.6.1
Resolution: Done
> Organize test utils methods and classes
> --
[
https://issues.apache.org/jira/browse/HUDI-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar reassigned HUDI-1323:
Assignee: Vinoth Chandar
> Fence metadata reads using latest data timeline commit times!
>
[
https://issues.apache.org/jira/browse/HUDI-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-1323:
-
Status: Open (was: New)
> Fence metadata reads using latest data timeline commit times!
> ---
[
https://issues.apache.org/jira/browse/HUDI-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-1323:
-
Status: In Progress (was: Open)
> Fence metadata reads using latest data timeline commit times!
>
[
https://issues.apache.org/jira/browse/HUDI-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212745#comment-17212745
]
Vinoth Chandar commented on HUDI-1312:
--
[~uditme] are you interested in taking this u
vinothchandar merged pull request #2150:
URL: https://github.com/apache/hudi/pull/2150
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
This is an automated email from the ASF dual-hosted git repository.
vinoth pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git.
from c5e10d6 [HUDI-995] Migrate HoodieTestUtils APIs to HoodieTestTable
(#2167)
add 0d40734 [HUDI-1304] Add unit tes
codecov-io edited a comment on pull request #2150:
URL: https://github.com/apache/hudi/pull/2150#issuecomment-704505827
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=h1) Report
> Merging
[#2150](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=desc) into
[master
codecov-io edited a comment on pull request #2150:
URL: https://github.com/apache/hudi/pull/2150#issuecomment-704505827
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=h1) Report
> Merging
[#2150](https://codecov.io/gh/apache/hudi/pull/2150?src=pr&el=desc) into
[master
satishkotha commented on a change in pull request #2150:
URL: https://github.com/apache/hudi/pull/2150#discussion_r503565568
##
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/compact/TestAsyncCompaction.java
##
@@ -332,4 +336,51 @@ public v
umehrot2 commented on pull request #2147:
URL: https://github.com/apache/hudi/pull/2147#issuecomment-707334174
@rmpifer A couple of points:
- As @vinothchandar mentioned, it would be worth exploring if by just
removing the dependency relocation and still continuing to shade, helps avoid
[
https://issues.apache.org/jira/browse/HUDI-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Wason updated HUDI-1320:
-
Status: Open (was: New)
> Move static invocations of HoodieMetadata.xxx to HoodieTable
>
[
https://issues.apache.org/jira/browse/HUDI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Wason updated HUDI-1322:
-
Status: Open (was: New)
> Refactor into Reader & Writer side for Metadata
> -
[
https://issues.apache.org/jira/browse/HUDI-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Wason updated HUDI-1320:
-
Status: In Progress (was: Open)
> Move static invocations of HoodieMetadata.xxx to HoodieTable
>
[
https://issues.apache.org/jira/browse/HUDI-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Wason reassigned HUDI-1322:
Assignee: Prashant Wason
> Refactor into Reader & Writer side for Metadata
> --
[
https://issues.apache.org/jira/browse/HUDI-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Wason reassigned HUDI-1320:
Assignee: Prashant Wason
> Move static invocations of HoodieMetadata.xxx to HoodieTable
> -
tandonraghav commented on issue #2165:
URL: https://github.com/apache/hudi/issues/2165#issuecomment-707231701
@bvaradar I was trying on Presto with Glue on AWS EMR. presto-bundle is
present inside /plugins/hive-hadoop2/.
But my problem is why this error - `Caused by: java.lang.ClassC
bvaradar commented on issue #2165:
URL: https://github.com/apache/hudi/issues/2165#issuecomment-707226897
@tandonraghav : It was not clear from your original description of the
issue whether you are making a spark or presto query. Looking at the previous
comments, it looks like you are m
liwei created HUDI-1342:
---
Summary: hudi-dla-sync support modify table properties
Key: HUDI-1342
URL: https://issues.apache.org/jira/browse/HUDI-1342
Project: Apache Hudi
Issue Type: Improvement
lw309637554 commented on a change in pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#discussion_r503376335
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieTableFileSystemView.java
##
@@ -110,6 +114,11 @@ protected void resetVie
lw309637554 commented on a change in pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#discussion_r503376042
##
File path: hudi-common/src/main/avro/HoodieClusteringPlan.avsc
##
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
lw309637554 commented on a change in pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#discussion_r503375885
##
File path:
hudi-client/src/main/java/org/apache/hudi/table/action/clustering/updates/UpdateStrategy.java
##
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the
lw309637554 commented on a change in pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#discussion_r503375808
##
File path:
hudi-client/src/main/java/org/apache/hudi/table/action/clustering/updates/RejectUpdateStrategy.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed t
lw309637554 commented on a change in pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#discussion_r503375182
##
File path:
hudi-client/src/main/java/org/apache/hudi/table/action/clustering/strategy/BaseFileSizeBasedClusteringStrategy.java
##
@@ -0,0 +1,73 @@
+/
lw309637554 commented on a change in pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#discussion_r503375018
##
File path:
hudi-client/src/main/java/org/apache/hudi/table/action/clustering/HoodieCopyOnWriteTableCluster.java
##
@@ -0,0 +1,243 @@
+/*
+ * Licensed
lw309637554 commented on a change in pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#discussion_r503374310
##
File path:
hudi-client/src/main/java/org/apache/hudi/table/HoodieCopyOnWriteTable.java
##
@@ -125,6 +128,16 @@ public HoodieWriteMetadata
bulkInsert
tandonraghav edited a comment on issue #2165:
URL: https://github.com/apache/hudi/issues/2165#issuecomment-707163257
Attaching the presto logs-
2020-10-12T14:41:49.229Z INFO20201012_144143_00011_zymbu.1.0.0-0-44
org.apache.hudi.common.table.log.AbstractHoodieLogRecor
tandonraghav commented on issue #2165:
URL: https://github.com/apache/hudi/issues/2165#issuecomment-707163257
Attaching the presto logs-
2020-10-12T14:41:49.229Z INFO20201012_144143_00011_zymbu.1.0.0-0-44
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanne
ashishmgofficial commented on issue #2149:
URL: https://github.com/apache/hudi/issues/2149#issuecomment-707061282
@bvaradar PFA below the files
[Downloads.zip](https://github.com/apache/hudi/files/5364821/Downloads.zip)
-
tandonraghav closed issue #2151:
URL: https://github.com/apache/hudi/issues/2151
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
tandonraghav commented on issue #2151:
URL: https://github.com/apache/hudi/issues/2151#issuecomment-707068148
@bvaradar Thanks for the update.
This is an automated message from the Apache Git Service.
To respond to the messag
tandonraghav commented on issue #2165:
URL: https://github.com/apache/hudi/issues/2165#issuecomment-707077681
@bvaradar There is a clear issue between hudi-hadoop-mr-bundle jar and
hudi-spark-bundle_2.11.jar
Can you please check and clarify once. I dont think it is related to any
classp
bvaradar commented on issue #2149:
URL: https://github.com/apache/hudi/issues/2149#issuecomment-707016759
@ashishmgofficial : Would it be possible to dump the avro records (value)
as-is in a file and attach ?
This is an auto
hddong commented on pull request #1946:
URL: https://github.com/apache/hudi/pull/1946#issuecomment-707016010
@vinothchandar : yes, +1 for move to log4j2. I will do it if necessary.
This is an automated message from the Apache
[
https://issues.apache.org/jira/browse/HUDI-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212267#comment-17212267
]
liwei commented on HUDI-1341:
-
[~vinoth] Do we already have relevant plans about this ? And w
liwei created HUDI-1341:
---
Summary: hudi cli command such as rollback 、bootstrap support
spark sql implement
Key: HUDI-1341
URL: https://issues.apache.org/jira/browse/HUDI-1341
Project: Apache Hudi
Is
bvaradar commented on issue #2165:
URL: https://github.com/apache/hudi/issues/2165#issuecomment-706999132
It looks like there are more than 1 fat bundles in the class path
(hudi-hadoop-mr-bundle) and hudi-spark-bundle ?
If this is the case, You need to just use hudi-spark-bundle.
liujinhui1994 commented on issue #2162:
URL: https://github.com/apache/hudi/issues/2162#issuecomment-706989196
Please help me to find out what went wrong, it has troubled me for a long
time. Thank you very much for your help @bvaradar
bvaradar commented on issue #2151:
URL: https://github.com/apache/hudi/issues/2151#issuecomment-706987817
@tandonraghav : It is by design that a "file" which is pending compaction is
not scheduled for compaction till the compaction is done.
One another knob is the strategy for selec
liujinhui1994 commented on issue #2162:
URL: https://github.com/apache/hudi/issues/2162#issuecomment-706986244
---
liujinhui1994 commented on issue #2162:
URL: https://github.com/apache/hudi/issues/2162#issuecomment-706984549
@bvaradar I am using transform and want to add a new ds field. I now
re-create a new hudi table, and add the ds field to the end of target.avsc
according to your suggestion, but
liujinhui1994 removed a comment on issue #2162:
URL: https://github.com/apache/hudi/issues/2162#issuecomment-706983114
![Uploading 1602492991(1).jpg…]()
This is an automated message from the Apache Git Service.
To r
liujinhui1994 commented on issue #2162:
URL: https://github.com/apache/hudi/issues/2162#issuecomment-706983352
---
liujinhui1994 commented on issue #2162:
URL: https://github.com/apache/hudi/issues/2162#issuecomment-706983114
![Uploading 1602492991(1).jpg…]()
This is an automated message from the Apache Git Service.
To respond t
liujinhui1994 commented on issue #2162:
URL: https://github.com/apache/hudi/issues/2162#issuecomment-706981577
source.avsc
{
"type": "record",
"name": "t3_app_td_ad_info",
"fields": [{
"name": "dataId",
"type": "strin
bvaradar commented on issue #2162:
URL: https://github.com/apache/hudi/issues/2162#issuecomment-706977420
@liujinhui1994 : I see that the new column column is added to the middle of
the schema (not at the end). Are you doing the same thing with transformer ?
You need to make sure the fiel
lw309637554 edited a comment on pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#issuecomment-706971139
> > @leesf #2048 is landed. is it possible to merge this and address
Balaji's comments? (I can help if needed)
>
> Sure, considering I am a little busy these days,
lw309637554 edited a comment on pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#issuecomment-706971139
> > @leesf #2048 is landed. is it possible to merge this and address
Balaji's comments? (I can help if needed)
>
> Sure, considering I am a little busy these days,
lw309637554 commented on pull request #2082:
URL: https://github.com/apache/hudi/pull/2082#issuecomment-706971139
> > @leesf #2048 is landed. is it possible to merge this and address
Balaji's comments? (I can help if needed)
>
> Sure, considering I am a little busy these days, it is
bvaradar commented on issue #2166:
URL: https://github.com/apache/hudi/issues/2166#issuecomment-706965836
@somebol : Its hard to figure out if all 4 rows you are seeing in "Query in
hue/hive" have the same record key due to masking. But assuming that is the
case, you should not be seeing d
hotienvu commented on a change in pull request #2157:
URL: https://github.com/apache/hudi/pull/2157#discussion_r503108344
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java
##
@@ -119,4 +103,18 @@ public DFSPathSelector(Ty
lw309637554 edited a comment on pull request #2173:
URL: https://github.com/apache/hudi/pull/2173#issuecomment-706890444
> LGTM. looks like we should prioritize checkstyle for scala.
check useless import isn't generally possible in Scalastyle. It doesn't
know about types.
83 matches
Mail list logo
