[jira] [Updated] (HUDI-5643) Async compaction conflict with consistent hashing bucket resizing (i.e., Clustering)

2023-01-29 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-5643: - Issue Type: Bug (was: Improvement) > Async compaction conflict with consistent hashing bucket resizing

[jira] [Created] (HUDI-5644) ConcurrentWriteConflictResolver handle consistent hashing bucket resizing & write

2023-01-29 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-5644: Summary: ConcurrentWriteConflictResolver handle consistent hashing bucket resizing & write Key: HUDI-5644 URL: https://issues.apache.org/jira/browse/HUDI-5644 Project:

[jira] [Created] (HUDI-5643) Async compaction conflict with consistent hashing bucket resizing (i.e., Clustering)

2023-01-29 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-5643: Summary: Async compaction conflict with consistent hashing bucket resizing (i.e., Clustering) Key: HUDI-5643 URL: https://issues.apache.org/jira/browse/HUDI-5643 Project:

[jira] [Closed] (HUDI-4895) Object Store based lock provider

2022-10-20 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao closed HUDI-4895. Resolution: Won't Do > Object Store based lock provider > > >

[jira] [Updated] (HUDI-4895) Object Store based lock provider

2022-09-21 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-4895: - Component/s: multi-writer > Object Store based lock provider > > >

[jira] [Updated] (HUDI-4812) Lazy partition listing and file groups fetching in Spark Query

2022-09-21 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-4812: - Component/s: spark > Lazy partition listing and file groups fetching in Spark Query >

[jira] [Created] (HUDI-4896) Consistent hashing index resizing for Flink Engine

2022-09-21 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4896: Summary: Consistent hashing index resizing for Flink Engine Key: HUDI-4896 URL: https://issues.apache.org/jira/browse/HUDI-4896 Project: Apache Hudi Issue Type:

[jira] [Assigned] (HUDI-4895) Object Store based lock provider

2022-09-21 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao reassigned HUDI-4895: Assignee: Yuwei Xiao > Object Store based lock provider > > >

[jira] [Created] (HUDI-4895) Object Store based lock provider

2022-09-21 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4895: Summary: Object Store based lock provider Key: HUDI-4895 URL: https://issues.apache.org/jira/browse/HUDI-4895 Project: Apache Hudi Issue Type: Improvement

[jira] [Created] (HUDI-4888) Add validation to block COW table to use consistent hashing bucket index

2022-09-21 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4888: Summary: Add validation to block COW table to use consistent hashing bucket index Key: HUDI-4888 URL: https://issues.apache.org/jira/browse/HUDI-4888 Project: Apache Hudi

[jira] [Updated] (HUDI-4812) Lazy partition listing and file groups fetching in Spark Query

2022-09-12 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-4812: - Summary: Lazy partition listing and file groups fetching in Spark Query (was: Delay file groups fetching

[jira] [Created] (HUDI-4812) Delay file groups fetching after partition prune in Spark Query

2022-09-08 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4812: Summary: Delay file groups fetching after partition prune in Spark Query Key: HUDI-4812 URL: https://issues.apache.org/jira/browse/HUDI-4812 Project: Apache Hudi

[jira] [Assigned] (HUDI-4753) More accurate evaluation of log record during log writing or compaction

2022-09-08 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao reassigned HUDI-4753: Assignee: Yuwei Xiao (was: Ethan Guo) > More accurate evaluation of log record during log writing

[jira] [Created] (HUDI-4807) Use correct instant in metadata initialization

2022-09-07 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4807: Summary: Use correct instant in metadata initialization Key: HUDI-4807 URL: https://issues.apache.org/jira/browse/HUDI-4807 Project: Apache Hudi Issue Type: Bug

[jira] [Updated] (HUDI-4753) More accurate evaluation of log record during log writing or compaction

2022-08-30 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-4753: - Description: In current log writing, the avgRecordSize is taken from the first incoming log record,

[jira] [Created] (HUDI-4753) More accurate evaluation of log record during log writing or compaction

2022-08-30 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4753: Summary: More accurate evaluation of log record during log writing or compaction Key: HUDI-4753 URL: https://issues.apache.org/jira/browse/HUDI-4753 Project: Apache Hudi

[jira] [Updated] (HUDI-4521) Fix lost re-commit in rare restart case (changing write.tasks)

2022-08-01 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-4521: - Description: The current _*StreamWriteOperatorCoordinator*_ in Flink will try to re-commit the last

[jira] [Created] (HUDI-4521) Fix lost re-commit in rare restart case (changing write.tasks)

2022-08-01 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4521: Summary: Fix lost re-commit in rare restart case (changing write.tasks) Key: HUDI-4521 URL: https://issues.apache.org/jira/browse/HUDI-4521 Project: Apache Hudi

[jira] [Created] (HUDI-4419) Move to apache.logging.log4j and cleanup log dependency

2022-07-18 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4419: Summary: Move to apache.logging.log4j and cleanup log dependency Key: HUDI-4419 URL: https://issues.apache.org/jira/browse/HUDI-4419 Project: Apache Hudi Issue

[jira] [Created] (HUDI-4378) Refactor SIMPLE bucket index partitioner to enable bulk_insert

2022-07-10 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4378: Summary: Refactor SIMPLE bucket index partitioner to enable bulk_insert Key: HUDI-4378 URL: https://issues.apache.org/jira/browse/HUDI-4378 Project: Apache Hudi

[jira] [Created] (HUDI-4377) Support different split criteria for consistent hashing index resizing

2022-07-10 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4377: Summary: Support different split criteria for consistent hashing index resizing Key: HUDI-4377 URL: https://issues.apache.org/jira/browse/HUDI-4377 Project: Apache Hudi

[jira] [Comment Edited] (HUDI-4318) IndexOutOfBoundException when recordKey has List values for Bucket index table

2022-07-10 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564649#comment-17564649 ] Yuwei Xiao edited comment on HUDI-4318 at 7/10/22 7:37 AM: --- Failed to re-produce

[jira] [Commented] (HUDI-4318) IndexOutOfBoundException when recordKey has List values for Bucket index table

2022-07-10 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564649#comment-17564649 ] Yuwei Xiao commented on HUDI-4318: -- Failed to re-produce the exception. My test code:   {code:java} val

[jira] [Updated] (HUDI-4373) Consistent bucket index write path for Flink engine

2022-07-08 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-4373: - Status: Open (was: In Progress) > Consistent bucket index write path for Flink engine >

[jira] [Updated] (HUDI-4373) Consistent bucket index write path for Flink engine

2022-07-08 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-4373: - Epic Link: HUDI-3000 > Consistent bucket index write path for Flink engine >

[jira] [Updated] (HUDI-4373) Consistent bucket index write path for Flink engine

2022-07-08 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-4373: - Status: In Progress (was: Open) > Consistent bucket index write path for Flink engine >

[jira] [Updated] (HUDI-4373) Consistent bucket index write path for Flink engine

2022-07-08 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-4373: - Parent: (was: HUDI-3000) Issue Type: New Feature (was: Sub-task) > Consistent bucket index

[jira] [Updated] (HUDI-4373) Consistent bucket index write path for Flink engine

2022-07-08 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-4373: - Parent: HUDI-3000 Issue Type: Sub-task (was: New Feature) > Consistent bucket index write path

[jira] [Created] (HUDI-4373) Consistent bucket index write path for Flink engine

2022-07-08 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-4373: Summary: Consistent bucket index write path for Flink engine Key: HUDI-4373 URL: https://issues.apache.org/jira/browse/HUDI-4373 Project: Apache Hudi Issue Type:

[jira] [Assigned] (HUDI-4318) IndexOutOfBoundException when recordKey has List values for Bucket index table

2022-06-27 Thread Yuwei Xiao (Jira)
Title: Message Title Yuwei Xiao assigned

[jira] [Resolved] (HUDI-3085) Refactor fileId & writeHandler logic into partitioner for bulk_insert

2022-05-16 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao resolved HUDI-3085. -- > Refactor fileId & writeHandler logic into partitioner for bulk_insert >

[jira] [Resolved] (HUDI-3692) Write cannot see inflight compaction when using metadata table

2022-05-12 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao resolved HUDI-3692. -- > Write cannot see inflight compaction when using metadata table >

[jira] [Assigned] (HUDI-3095) Abstract partition filter logic of clustering plan to enable code reuse

2022-03-23 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao reassigned HUDI-3095: Assignee: Yuwei Xiao > Abstract partition filter logic of clustering plan to enable code reuse >

[jira] [Assigned] (HUDI-3085) Refactor fileId & writeHandler logic into partitioner for bulk_insert

2022-03-23 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao reassigned HUDI-3085: Assignee: Yuwei Xiao > Refactor fileId & writeHandler logic into partitioner for bulk_insert >

[jira] [Assigned] (HUDI-3123) Consistent hashing index for upsert/insert write path

2022-03-23 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao reassigned HUDI-3123: Assignee: Yuwei Xiao > Consistent hashing index for upsert/insert write path >

[jira] [Assigned] (HUDI-3558) Consistent hashing index resizing (bucket split)

2022-03-23 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao reassigned HUDI-3558: Assignee: Yuwei Xiao > Consistent hashing index resizing (bucket split) >

[jira] [Assigned] (HUDI-3692) Write cannot see inflight compaction when using metadata table

2022-03-23 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao reassigned HUDI-3692: Assignee: Yuwei Xiao > Write cannot see inflight compaction when using metadata table >

[jira] [Created] (HUDI-3692) Write cannot see inflight compaction when using metadata table

2022-03-23 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-3692: Summary: Write cannot see inflight compaction when using metadata table Key: HUDI-3692 URL: https://issues.apache.org/jira/browse/HUDI-3692 Project: Apache Hudi

[jira] [Created] (HUDI-3597) Use metadata table to manage hashing metadata

2022-03-09 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-3597: Summary: Use metadata table to manage hashing metadata Key: HUDI-3597 URL: https://issues.apache.org/jira/browse/HUDI-3597 Project: Apache Hudi Issue Type: Sub-task

[jira] [Created] (HUDI-3585) Docs for (consistent) hashing index

2022-03-08 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-3585: Summary: Docs for (consistent) hashing index Key: HUDI-3585 URL: https://issues.apache.org/jira/browse/HUDI-3585 Project: Apache Hudi Issue Type: Sub-task

[jira] [Updated] (HUDI-3123) Consistent hashing index for upsert/insert write path

2022-03-08 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-3123: - Epic Link: (was: HUDI-3000) > Consistent hashing index for upsert/insert write path >

[jira] [Updated] (HUDI-3123) Consistent hashing index for upsert/insert write path

2022-03-08 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-3123: - Parent: HUDI-3000 Issue Type: Sub-task (was: Task) > Consistent hashing index for upsert/insert

[jira] [Updated] (HUDI-3558) Consistent hashing index resizing (bucket split)

2022-03-03 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-3558: - Parent: HUDI-3000 Issue Type: Sub-task (was: New Feature) > Consistent hashing index resizing

[jira] [Created] (HUDI-3558) Consistent hashing index resizing (bucket split)

2022-03-03 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-3558: Summary: Consistent hashing index resizing (bucket split) Key: HUDI-3558 URL: https://issues.apache.org/jira/browse/HUDI-3558 Project: Apache Hudi Issue Type: New

[jira] [Created] (HUDI-3194) Fix invisible writes(commits) during compaction (HoodieParquetRealtimeInputFormat)

2022-01-07 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-3194: Summary: Fix invisible writes(commits) during compaction (HoodieParquetRealtimeInputFormat) Key: HUDI-3194 URL: https://issues.apache.org/jira/browse/HUDI-3194 Project:

[jira] [Resolved] (HUDI-3095) Abstract partition filter logic of clustering plan to enable code reuse

2021-12-30 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao resolved HUDI-3095. -- > Abstract partition filter logic of clustering plan to enable code reuse >

[jira] [Updated] (HUDI-3123) Consistent hashing index for upsert/insert write path

2021-12-28 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-3123: - Parent: HUDI-3000 Issue Type: Sub-task (was: Improvement) > Consistent hashing index for

[jira] [Created] (HUDI-3123) Consistent hashing index for upsert/insert write path

2021-12-28 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-3123: Summary: Consistent hashing index for upsert/insert write path Key: HUDI-3123 URL: https://issues.apache.org/jira/browse/HUDI-3123 Project: Apache Hudi Issue Type:

[jira] [Created] (HUDI-3095) Abstract partition filter logic of clustering plan to enable code reuse

2021-12-21 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-3095: Summary: Abstract partition filter logic of clustering plan to enable code reuse Key: HUDI-3095 URL: https://issues.apache.org/jira/browse/HUDI-3095 Project: Apache Hudi

[jira] [Created] (HUDI-3085) Refactor fileId & writeHandler logic into partitioner for bulk_insert

2021-12-20 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-3085: Summary: Refactor fileId & writeHandler logic into partitioner for bulk_insert Key: HUDI-3085 URL: https://issues.apache.org/jira/browse/HUDI-3085 Project: Apache Hudi

[jira] [Resolved] (HUDI-2998) Claim RFC number for RFC for Consistent Hashing Index

2021-12-20 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao resolved HUDI-2998. -- > Claim RFC number for RFC for Consistent Hashing Index >

[jira] [Updated] (HUDI-2998) Claim RFC number for RFC for Consistent Hashing Index

2021-12-13 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-2998: - Parent: HUDI-3000 Issue Type: Sub-task (was: Task) > Claim RFC number for RFC for Consistent

[jira] [Updated] (HUDI-2999) Consistent Hashing Index RFC

2021-12-13 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao updated HUDI-2999: - Parent: HUDI-3000 Issue Type: Sub-task (was: Task) > Consistent Hashing Index RFC >

[jira] [Created] (HUDI-3000) [UMBRELLA] Consistent Hashing Index

2021-12-13 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-3000: Summary: [UMBRELLA] Consistent Hashing Index Key: HUDI-3000 URL: https://issues.apache.org/jira/browse/HUDI-3000 Project: Apache Hudi Issue Type: New Feature

[jira] [Created] (HUDI-2999) Consistent Hashing Index RFC

2021-12-13 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-2999: Summary: Consistent Hashing Index RFC Key: HUDI-2999 URL: https://issues.apache.org/jira/browse/HUDI-2999 Project: Apache Hudi Issue Type: Task

[jira] [Created] (HUDI-2998) Claim RFC number for RFC for Consistent Hashing Index

2021-12-13 Thread Yuwei Xiao (Jira)
Yuwei Xiao created HUDI-2998: Summary: Claim RFC number for RFC for Consistent Hashing Index Key: HUDI-2998 URL: https://issues.apache.org/jira/browse/HUDI-2998 Project: Apache Hudi Issue Type:

[jira] [Resolved] (HUDI-2849) Improve job/stage description in spark UI for hudi write

2021-12-11 Thread Yuwei Xiao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuwei Xiao resolved HUDI-2849. -- > Improve job/stage description in spark UI for hudi write >