[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-1763: - Fix Version/s: 0.11.0 (was: 0.10.0) > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available, release-blocker, sev:critical > Fix For: 0.11.0 > > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1763: -- Fix Version/s: (was: 0.9.0) 0.10.0 > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available, release-blocker, sev:critical > Fix For: 0.10.0 > > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-1763: - Fixed via master branch: 5448cdde7e45c39b8bf8275d9f3ee425c9bfa90e > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available, release-blocker, sev:critical > Fix For: 0.9.0 > > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-1763: - Status: Closed (was: Patch Available) > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available, release-blocker, sev:critical > Fix For: 0.9.0 > > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udit Mehrotra updated HUDI-1763: Labels: pull-request-available release-blocker sev:critical (was: pull-request-available sev:critical) > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available, release-blocker, sev:critical > Fix For: 0.9.0 > > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udit Mehrotra updated HUDI-1763: Priority: Blocker (was: Major) > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Assignee: sivabalan narayanan >Priority: Blocker > Labels: pull-request-available, sev:critical > Fix For: 0.9.0 > > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udit Mehrotra updated HUDI-1763: Fix Version/s: 0.9.0 > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available, sev:critical > Fix For: 0.9.0 > > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1763: -- Status: Patch Available (was: In Progress) > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available, sev:critical > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1763: -- Status: In Progress (was: Open) > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available, sev:critical > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1763: - Labels: pull-request-available sev:critical (was: sev:critical) > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available, sev:critical > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1763: -- Labels: sev:critical (was: sev:high) > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Priority: Major > Labels: sev:critical > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1763) DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged
[ https://issues.apache.org/jira/browse/HUDI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1763: -- Labels: sev:high (was: ) > DefaultHoodieRecordPayload does not honor ordering value when records within > multiple log files are merged > -- > > Key: HUDI-1763 > URL: https://issues.apache.org/jira/browse/HUDI-1763 > Project: Apache Hudi > Issue Type: Bug > Components: Writer Core >Affects Versions: 0.8.0 >Reporter: sivabalan narayanan >Priority: Major > Labels: sev:high > > While creating HoodieRecordPayloads from log files in case of MOR tables, the > payloads are created without any orderingVal (even if specified while writing > data). Due to this the precombine function could result in any payload > irrespective of its orderingVal. > Attaching a sample script to reproduce the issue. > In this example, for key "key1", 1st insert is with ts=1000. Then we update > with ts=2000. Thenn we updated with ts=500. Ideally after last update if we > snnapshot query the table, we must get key1 with ts=2000 (since our ordering > field is ts). However it shows entry of ts=1000 because from logs it ignores > ts=2000 and only picks up ts=500. > Also AFAIU, the same flow will be used while compaction and then we might > lose data forever. > > More info: https://github.com/apache/hudi/issues/2756 -- This message was sent by Atlassian Jira (v8.3.4#803005)
