[jira] [Updated] (HUDI-1381) Schedule compaction based on time elapsed
[ https://issues.apache.org/jira/browse/HUDI-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1381: Status: In Progress (was: Open) > Schedule compaction based on time elapsed > -- > > Key: HUDI-1381 > URL: https://issues.apache.org/jira/browse/HUDI-1381 > Project: Apache Hudi > Issue Type: New Feature > Components: Compaction >Reporter: Balaji Varadarajan >Assignee: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.8.0 > > > GH : [https://github.com/apache/hudi/issues/2229] > It would be helpful to introduce configuration to schedule compaction based > on time elapsed since last scheduled compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1381) Schedule compaction based on time elapsed
[ https://issues.apache.org/jira/browse/HUDI-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang resolved HUDI-1381. - Resolution: Fixed > Schedule compaction based on time elapsed > -- > > Key: HUDI-1381 > URL: https://issues.apache.org/jira/browse/HUDI-1381 > Project: Apache Hudi > Issue Type: New Feature > Components: Compaction >Reporter: Balaji Varadarajan >Assignee: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.8.0 > > > GH : [https://github.com/apache/hudi/issues/2229] > It would be helpful to introduce configuration to schedule compaction based > on time elapsed since last scheduled compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1255) Combine and get updateValue in multiFields
[ https://issues.apache.org/jira/browse/HUDI-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang reassigned HUDI-1255: --- Assignee: karl wang > Combine and get updateValue in multiFields > -- > > Key: HUDI-1255 > URL: https://issues.apache.org/jira/browse/HUDI-1255 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: karl wang >Assignee: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.1 > > > update current value for several fields that you want to change. > The default payload OverwriteWithLatestAvroPayload overwrite the whole record > when > compare to orderingVal.This doesn't meet our need when we just want to change > specified fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1377) clean duplicate code in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang reassigned HUDI-1377: --- Assignee: karl wang > clean duplicate code in HoodieSparkSqlWriter > > > Key: HUDI-1377 > URL: https://issues.apache.org/jira/browse/HUDI-1377 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: karl wang >Assignee: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.1 > > > clean duplicate code in HoodieSparkSqlWriter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HUDI-1381) Schedule compaction based on time elapsed
[ https://issues.apache.org/jira/browse/HUDI-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang reassigned HUDI-1381: --- Assignee: karl wang > Schedule compaction based on time elapsed > -- > > Key: HUDI-1381 > URL: https://issues.apache.org/jira/browse/HUDI-1381 > Project: Apache Hudi > Issue Type: New Feature > Components: Compaction >Reporter: Balaji Varadarajan >Assignee: karl wang >Priority: Major > Fix For: 0.7.0 > > > GH : [https://github.com/apache/hudi/issues/2229] > It would be helpful to introduce configuration to schedule compaction based > on time elapsed since last scheduled compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HUDI-1381) Schedule compaction based on time elapsed
[ https://issues.apache.org/jira/browse/HUDI-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232036#comment-17232036 ] karl wang edited comment on HUDI-1381 at 11/16/20, 3:08 AM: [~vbalaji] I want to try it. Can u assign to me was (Author: wangshikai): I want to try it. Can u assign to me > Schedule compaction based on time elapsed > -- > > Key: HUDI-1381 > URL: https://issues.apache.org/jira/browse/HUDI-1381 > Project: Apache Hudi > Issue Type: New Feature > Components: Compaction >Reporter: Balaji Varadarajan >Priority: Major > Fix For: 0.7.0 > > > GH : [https://github.com/apache/hudi/issues/2229] > It would be helpful to introduce configuration to schedule compaction based > on time elapsed since last scheduled compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-1377) clean duplicate code in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang closed HUDI-1377. --- > clean duplicate code in HoodieSparkSqlWriter > > > Key: HUDI-1377 > URL: https://issues.apache.org/jira/browse/HUDI-1377 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.1 > > > clean duplicate code in HoodieSparkSqlWriter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HUDI-1381) Schedule compaction based on time elapsed
[ https://issues.apache.org/jira/browse/HUDI-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232036#comment-17232036 ] karl wang edited comment on HUDI-1381 at 11/15/20, 4:28 AM: I want to try it. Can u assign to me was (Author: wangshikai): I want to try it. > Schedule compaction based on time elapsed > -- > > Key: HUDI-1381 > URL: https://issues.apache.org/jira/browse/HUDI-1381 > Project: Apache Hudi > Issue Type: New Feature > Components: Compaction >Reporter: Balaji Varadarajan >Priority: Major > Fix For: 0.7.0 > > > GH : [https://github.com/apache/hudi/issues/2229] > It would be helpful to introduce configuration to schedule compaction based > on time elapsed since last scheduled compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HUDI-1381) Schedule compaction based on time elapsed
[ https://issues.apache.org/jira/browse/HUDI-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232036#comment-17232036 ] karl wang edited comment on HUDI-1381 at 11/15/20, 4:28 AM: I want to try it. was (Author: wangshikai): I am not sure if I can do it . But I want try it. > Schedule compaction based on time elapsed > -- > > Key: HUDI-1381 > URL: https://issues.apache.org/jira/browse/HUDI-1381 > Project: Apache Hudi > Issue Type: New Feature > Components: Compaction >Reporter: Balaji Varadarajan >Priority: Major > Fix For: 0.7.0 > > > GH : [https://github.com/apache/hudi/issues/2229] > It would be helpful to introduce configuration to schedule compaction based > on time elapsed since last scheduled compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1381) Schedule compaction based on time elapsed
[ https://issues.apache.org/jira/browse/HUDI-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232036#comment-17232036 ] karl wang commented on HUDI-1381: - I am not sure if I can do it . But I want try it. > Schedule compaction based on time elapsed > -- > > Key: HUDI-1381 > URL: https://issues.apache.org/jira/browse/HUDI-1381 > Project: Apache Hudi > Issue Type: New Feature > Components: Compaction >Reporter: Balaji Varadarajan >Priority: Major > Fix For: 0.7.0 > > > GH : [https://github.com/apache/hudi/issues/2229] > It would be helpful to introduce configuration to schedule compaction based > on time elapsed since last scheduled compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-1377) clean duplicate code in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231918#comment-17231918 ] karl wang commented on HUDI-1377: - [~wangxianghu] I don't have right to assign task to myself. > clean duplicate code in HoodieSparkSqlWriter > > > Key: HUDI-1377 > URL: https://issues.apache.org/jira/browse/HUDI-1377 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.1 > > > clean duplicate code in HoodieSparkSqlWriter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1377) clean duplicate code in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang resolved HUDI-1377. - Fix Version/s: 0.6.1 Resolution: Fixed > clean duplicate code in HoodieSparkSqlWriter > > > Key: HUDI-1377 > URL: https://issues.apache.org/jira/browse/HUDI-1377 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.1 > > > clean duplicate code in HoodieSparkSqlWriter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HUDI-1377) clean duplicate code in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang reopened HUDI-1377: - > clean duplicate code in HoodieSparkSqlWriter > > > Key: HUDI-1377 > URL: https://issues.apache.org/jira/browse/HUDI-1377 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: karl wang >Priority: Major > Labels: pull-request-available > > clean duplicate code in HoodieSparkSqlWriter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1377) clean duplicate code in HoodieSparkSqlWriter
[ https://issues.apache.org/jira/browse/HUDI-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang resolved HUDI-1377. - Resolution: Fixed > clean duplicate code in HoodieSparkSqlWriter > > > Key: HUDI-1377 > URL: https://issues.apache.org/jira/browse/HUDI-1377 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: karl wang >Priority: Major > Labels: pull-request-available > > clean duplicate code in HoodieSparkSqlWriter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1377) clean duplicate code in HoodieSparkSqlWriter
karl wang created HUDI-1377: --- Summary: clean duplicate code in HoodieSparkSqlWriter Key: HUDI-1377 URL: https://issues.apache.org/jira/browse/HUDI-1377 Project: Apache Hudi Issue Type: Improvement Components: Spark Integration Reporter: karl wang clean duplicate code in HoodieSparkSqlWriter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1284) preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal
[ https://issues.apache.org/jira/browse/HUDI-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1284: Description: When more than one HoodieRecord have the same HoodieKey, this function combines all fields(which is not DefaultValue) before attempting to insert/upsert (if combining turned on in HoodieClientConfig). eg: 1) Before: id name age ts 1 Karl null 0.0 1 null 18 0.0 After: id name age ts 1 Karl 18 0.0 was: When more than one HoodieRecord have the same HoodieKey, this function combines all fields(which is not null) before attempting to insert/upsert (if combining turned on in HoodieClientConfig). eg: 1) Before: id name age ts 1 Karl null 0.0 1 null 18 0.0 After: id name age ts 1 Karl 18 0.0 > preCombine all HoodieRecords and update all fields(which is not DefaultValue) > according to orderingVal > --- > > Key: HUDI-1284 > URL: https://issues.apache.org/jira/browse/HUDI-1284 > Project: Apache Hudi > Issue Type: New Feature > Components: Writer Core >Reporter: karl wang >Priority: Major > Fix For: 0.6.1 > > > When more than one HoodieRecord have the same HoodieKey, this function > combines all fields(which is not DefaultValue) > before attempting to insert/upsert (if combining turned on in > HoodieClientConfig). > eg: 1) > Before: > id name age ts > 1 Karl null 0.0 > 1 null 18 0.0 > After: > id name age ts > 1 Karl 18 0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1284) preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal
[ https://issues.apache.org/jira/browse/HUDI-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1284: Summary: preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal (was: preCombine all HoodieRecords and update all fields(which is not null) according to orderingVal) > preCombine all HoodieRecords and update all fields(which is not DefaultValue) > according to orderingVal > --- > > Key: HUDI-1284 > URL: https://issues.apache.org/jira/browse/HUDI-1284 > Project: Apache Hudi > Issue Type: New Feature > Components: Writer Core >Reporter: karl wang >Priority: Major > Fix For: 0.6.1 > > > When more than one HoodieRecord have the same HoodieKey, this function > combines all fields(which is not null) > before attempting to insert/upsert (if combining turned on in > HoodieClientConfig). > eg: 1) > Before: > id name age ts > 1 Karl null 0.0 > 1 null 18 0.0 > After: > id name age ts > 1 Karl 18 0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1284) preCombine all HoodieRecords and update all fields(which is not null) according to orderingVal
[ https://issues.apache.org/jira/browse/HUDI-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1284: Status: In Progress (was: Open) > preCombine all HoodieRecords and update all fields(which is not null) > according to orderingVal > --- > > Key: HUDI-1284 > URL: https://issues.apache.org/jira/browse/HUDI-1284 > Project: Apache Hudi > Issue Type: New Feature > Components: Writer Core >Reporter: karl wang >Priority: Major > Fix For: 0.6.1 > > > When more than one HoodieRecord have the same HoodieKey, this function > combines all fields(which is not null) > before attempting to insert/upsert (if combining turned on in > HoodieClientConfig). > eg: 1) > Before: > id name age ts > 1 Karl null 0.0 > 1 null 18 0.0 > After: > id name age ts > 1 Karl 18 0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1284) preCombine all HoodieRecords and update all fields(which is not null) according to orderingVal
karl wang created HUDI-1284: --- Summary: preCombine all HoodieRecords and update all fields(which is not null) according to orderingVal Key: HUDI-1284 URL: https://issues.apache.org/jira/browse/HUDI-1284 Project: Apache Hudi Issue Type: New Feature Components: Writer Core Reporter: karl wang Fix For: 0.6.1 When more than one HoodieRecord have the same HoodieKey, this function combines all fields(which is not null) before attempting to insert/upsert (if combining turned on in HoodieClientConfig). eg: 1) Before: id name age ts 1 Karl null 0.0 1 null 18 0.0 After: id name age ts 1 Karl 18 0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1284) preCombine all HoodieRecords and update all fields(which is not null) according to orderingVal
[ https://issues.apache.org/jira/browse/HUDI-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1284: Status: Open (was: New) > preCombine all HoodieRecords and update all fields(which is not null) > according to orderingVal > --- > > Key: HUDI-1284 > URL: https://issues.apache.org/jira/browse/HUDI-1284 > Project: Apache Hudi > Issue Type: New Feature > Components: Writer Core >Reporter: karl wang >Priority: Major > Fix For: 0.6.1 > > > When more than one HoodieRecord have the same HoodieKey, this function > combines all fields(which is not null) > before attempting to insert/upsert (if combining turned on in > HoodieClientConfig). > eg: 1) > Before: > id name age ts > 1 Karl null 0.0 > 1 null 18 0.0 > After: > id name age ts > 1 Karl 18 0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HUDI-1255) Combine and get updateValue in multiFields
[ https://issues.apache.org/jira/browse/HUDI-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang resolved HUDI-1255. - Resolution: Fixed > Combine and get updateValue in multiFields > -- > > Key: HUDI-1255 > URL: https://issues.apache.org/jira/browse/HUDI-1255 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.1 > > > update current value for several fields that you want to change. > The default payload OverwriteWithLatestAvroPayload overwrite the whole record > when > compare to orderingVal.This doesn't meet our need when we just want to change > specified fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1255) Combine and get updateValue in multiFields
[ https://issues.apache.org/jira/browse/HUDI-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1255: Fix Version/s: 0.6.1 > Combine and get updateValue in multiFields > -- > > Key: HUDI-1255 > URL: https://issues.apache.org/jira/browse/HUDI-1255 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0, 0.6.1 > > > update current value for several fields that you want to change. > The default payload OverwriteWithLatestAvroPayload overwrite the whole record > when > compare to orderingVal.This doesn't meet our need when we just want to change > specified fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1255) Combine and get updateValue in multiFields
[ https://issues.apache.org/jira/browse/HUDI-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1255: Fix Version/s: (was: 0.6.0) > Combine and get updateValue in multiFields > -- > > Key: HUDI-1255 > URL: https://issues.apache.org/jira/browse/HUDI-1255 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.1 > > > update current value for several fields that you want to change. > The default payload OverwriteWithLatestAvroPayload overwrite the whole record > when > compare to orderingVal.This doesn't meet our need when we just want to change > specified fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1255) Combine and get updateValue in multiFields
[ https://issues.apache.org/jira/browse/HUDI-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1255: Fix Version/s: (was: 0.6.1) > Combine and get updateValue in multiFields > -- > > Key: HUDI-1255 > URL: https://issues.apache.org/jira/browse/HUDI-1255 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > update current value for several fields that you want to change. > The default payload OverwriteWithLatestAvroPayload overwrite the whole record > when > compare to orderingVal.This doesn't meet our need when we just want to change > specified fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1255) Combine and get updateValue in multiFields
[ https://issues.apache.org/jira/browse/HUDI-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1255: Fix Version/s: 0.6.0 0.6.1 > Combine and get updateValue in multiFields > -- > > Key: HUDI-1255 > URL: https://issues.apache.org/jira/browse/HUDI-1255 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: karl wang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0, 0.6.1 > > > update current value for several fields that you want to change. > The default payload OverwriteWithLatestAvroPayload overwrite the whole record > when > compare to orderingVal.This doesn't meet our need when we just want to change > specified fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1255) Combine and get updateValue in multiFields
[ https://issues.apache.org/jira/browse/HUDI-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1255: Status: In Progress (was: Open) > Combine and get updateValue in multiFields > -- > > Key: HUDI-1255 > URL: https://issues.apache.org/jira/browse/HUDI-1255 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: karl wang >Priority: Major > > update current value for several fields that you want to change. > The default payload OverwriteWithLatestAvroPayload overwrite the whole record > when > compare to orderingVal.This doesn't meet our need when we just want to change > specified fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-1255) Combine and get updateValue in multiFields
karl wang created HUDI-1255: --- Summary: Combine and get updateValue in multiFields Key: HUDI-1255 URL: https://issues.apache.org/jira/browse/HUDI-1255 Project: Apache Hudi Issue Type: Improvement Components: Writer Core Reporter: karl wang update current value for several fields that you want to change. The default payload OverwriteWithLatestAvroPayload overwrite the whole record when compare to orderingVal.This doesn't meet our need when we just want to change specified fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1255) Combine and get updateValue in multiFields
[ https://issues.apache.org/jira/browse/HUDI-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated HUDI-1255: Status: Open (was: New) > Combine and get updateValue in multiFields > -- > > Key: HUDI-1255 > URL: https://issues.apache.org/jira/browse/HUDI-1255 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: karl wang >Priority: Major > > update current value for several fields that you want to change. > The default payload OverwriteWithLatestAvroPayload overwrite the whole record > when > compare to orderingVal.This doesn't meet our need when we just want to change > specified fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
