[jira] [Commented] (AVRO-3821) Rust: Record (de?)serialization is sensitive to order of fields in struct
[
https://issues.apache.org/jira/browse/AVRO-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751091#comment-17751091
]
Martin Tzvetanov Grigorov commented on AVRO-3821:
-
The problem here is in encode.rs -> encode_internal().
It encodes the Avro Value without taking into account the fields' order in the
schema. Later the decoding uses the schema to read the bytes but they are plain
wrong in this case !
> Rust: Record (de?)serialization is sensitive to order of fields in struct
> -
>
> Key: AVRO-3821
> URL: https://issues.apache.org/jira/browse/AVRO-3821
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Renar Narubin
>Priority: Major
>
> A previous issue describes a bug where record validation failed due to a
> struct's field declaration order not matching its schema:
> https://issues.apache.org/jira/browse/AVRO-3495
>
> That issue's fix appears to be incomplete. Validation during writing works,
> however the written records cannot be read back - it appears that the fields
> are mismatched during deserialization. I've reproduced using the existing
> `avro_old_issue_47` test case with a modification to read the data back:
>
> {code:java}
> diff --git a/lang/rust/avro/tests/schema.rs b/lang/rust/avro/tests/schema.rs
> index 00be0ab01..d5b1ee904 100644
> --- a/lang/rust/avro/tests/schema.rs
> +++ b/lang/rust/avro/tests/schema.rs
> @@ -18,6 +18,7 @@
> use std::io::{Cursor, Read};
>
> use apache_avro::{
> + from_avro_datum, from_value,
> schema::{EnumSchema, FixedSchema, Name, RecordField, RecordSchema},
> to_avro_datum, to_value,
> types::{Record, Value},
> @@ -1452,7 +1453,7 @@ fn avro_old_issue_47() -> TestResult {
>
> use serde::{Deserialize, Serialize};
>
> - #[derive(Deserialize, Serialize)]
> + #[derive(Deserialize, Serialize, Debug, Clone, PartialEq)]
> pub struct MyRecord {
> b: String,
> a: i64,
> @@ -1463,7 +1464,15 @@ fn avro_old_issue_47() -> TestResult {
> a: 1,
> };
>
> - let _ = to_avro_datum(&schema, to_value(record)?)?;
> + let serialized_bytes = to_avro_datum(&schema,
> to_value(record.clone())?)?;
> +
> + let deserialized_record = from_value::(&from_avro_datum(
> + &schema,
> + &mut Cursor::new(serialized_bytes),
> + None,
> + )?)?;
> +
> + assert_eq!(record, deserialized_record);
> Ok(())
> }
>
> {code}
> This fails as written, and passes if `a` and `b` are swapped in the MyRecord
> definition.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AVRO-3821) Rust: Record (de?)serialization is sensitive to order of fields in struct
[
https://issues.apache.org/jira/browse/AVRO-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751104#comment-17751104
]
ASF subversion and git services commented on AVRO-3821:
---
Commit ccde2e5d76b5091a0dcd5ffdba3b5427ab4c2d60 in avro's branch
refs/heads/avro-3821-fix-encoding-of-records from Martin Tzvetanov Grigorov
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=ccde2e5d7 ]
AVRO-3821: [Rust] Encoding records should follow the schema
When encoding Value::Record the logic should use the order of the fields
in the schema instead of the order in the Value::Record
Signed-off-by: Martin Tzvetanov Grigorov
> Rust: Record (de?)serialization is sensitive to order of fields in struct
> -
>
> Key: AVRO-3821
> URL: https://issues.apache.org/jira/browse/AVRO-3821
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Renar Narubin
>Assignee: Martin Tzvetanov Grigorov
>Priority: Major
>
> A previous issue describes a bug where record validation failed due to a
> struct's field declaration order not matching its schema:
> https://issues.apache.org/jira/browse/AVRO-3495
>
> That issue's fix appears to be incomplete. Validation during writing works,
> however the written records cannot be read back - it appears that the fields
> are mismatched during deserialization. I've reproduced using the existing
> `avro_old_issue_47` test case with a modification to read the data back:
>
> {code:java}
> diff --git a/lang/rust/avro/tests/schema.rs b/lang/rust/avro/tests/schema.rs
> index 00be0ab01..d5b1ee904 100644
> --- a/lang/rust/avro/tests/schema.rs
> +++ b/lang/rust/avro/tests/schema.rs
> @@ -18,6 +18,7 @@
> use std::io::{Cursor, Read};
>
> use apache_avro::{
> + from_avro_datum, from_value,
> schema::{EnumSchema, FixedSchema, Name, RecordField, RecordSchema},
> to_avro_datum, to_value,
> types::{Record, Value},
> @@ -1452,7 +1453,7 @@ fn avro_old_issue_47() -> TestResult {
>
> use serde::{Deserialize, Serialize};
>
> - #[derive(Deserialize, Serialize)]
> + #[derive(Deserialize, Serialize, Debug, Clone, PartialEq)]
> pub struct MyRecord {
> b: String,
> a: i64,
> @@ -1463,7 +1464,15 @@ fn avro_old_issue_47() -> TestResult {
> a: 1,
> };
>
> - let _ = to_avro_datum(&schema, to_value(record)?)?;
> + let serialized_bytes = to_avro_datum(&schema,
> to_value(record.clone())?)?;
> +
> + let deserialized_record = from_value::(&from_avro_datum(
> + &schema,
> + &mut Cursor::new(serialized_bytes),
> + None,
> + )?)?;
> +
> + assert_eq!(record, deserialized_record);
> Ok(())
> }
>
> {code}
> This fails as written, and passes if `a` and `b` are swapped in the MyRecord
> definition.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AVRO-3821) Rust: Record (de?)serialization is sensitive to order of fields in struct
[
https://issues.apache.org/jira/browse/AVRO-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751109#comment-17751109
]
ASF subversion and git services commented on AVRO-3821:
---
Commit d05ca14e7de4281f5db464546dbc89537db1bcb1 in avro's branch
refs/heads/master from Martin Grigorov
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=d05ca14e7 ]
AVRO-3821: [Rust] Encoding records should follow the schema (#2417)
When encoding Value::Record the logic should use the order of the fields
in the schema instead of the order in the Value::Record
Signed-off-by: Martin Tzvetanov Grigorov
> Rust: Record (de?)serialization is sensitive to order of fields in struct
> -
>
> Key: AVRO-3821
> URL: https://issues.apache.org/jira/browse/AVRO-3821
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Renar Narubin
>Assignee: Martin Tzvetanov Grigorov
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> A previous issue describes a bug where record validation failed due to a
> struct's field declaration order not matching its schema:
> https://issues.apache.org/jira/browse/AVRO-3495
>
> That issue's fix appears to be incomplete. Validation during writing works,
> however the written records cannot be read back - it appears that the fields
> are mismatched during deserialization. I've reproduced using the existing
> `avro_old_issue_47` test case with a modification to read the data back:
>
> {code:java}
> diff --git a/lang/rust/avro/tests/schema.rs b/lang/rust/avro/tests/schema.rs
> index 00be0ab01..d5b1ee904 100644
> --- a/lang/rust/avro/tests/schema.rs
> +++ b/lang/rust/avro/tests/schema.rs
> @@ -18,6 +18,7 @@
> use std::io::{Cursor, Read};
>
> use apache_avro::{
> + from_avro_datum, from_value,
> schema::{EnumSchema, FixedSchema, Name, RecordField, RecordSchema},
> to_avro_datum, to_value,
> types::{Record, Value},
> @@ -1452,7 +1453,7 @@ fn avro_old_issue_47() -> TestResult {
>
> use serde::{Deserialize, Serialize};
>
> - #[derive(Deserialize, Serialize)]
> + #[derive(Deserialize, Serialize, Debug, Clone, PartialEq)]
> pub struct MyRecord {
> b: String,
> a: i64,
> @@ -1463,7 +1464,15 @@ fn avro_old_issue_47() -> TestResult {
> a: 1,
> };
>
> - let _ = to_avro_datum(&schema, to_value(record)?)?;
> + let serialized_bytes = to_avro_datum(&schema,
> to_value(record.clone())?)?;
> +
> + let deserialized_record = from_value::(&from_avro_datum(
> + &schema,
> + &mut Cursor::new(serialized_bytes),
> + None,
> + )?)?;
> +
> + assert_eq!(record, deserialized_record);
> Ok(())
> }
>
> {code}
> This fails as written, and passes if `a` and `b` are swapped in the MyRecord
> definition.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AVRO-3821) Rust: Record (de?)serialization is sensitive to order of fields in struct
[
https://issues.apache.org/jira/browse/AVRO-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17751110#comment-17751110
]
ASF subversion and git services commented on AVRO-3821:
---
Commit 045f96c8f99457c13f291b281d06c6ee58cdef03 in avro's branch
refs/heads/branch-1.11 from Martin Grigorov
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=045f96c8f ]
AVRO-3821: [Rust] Encoding records should follow the schema (#2417)
When encoding Value::Record the logic should use the order of the fields
in the schema instead of the order in the Value::Record
Signed-off-by: Martin Tzvetanov Grigorov
(cherry picked from commit d05ca14e7de4281f5db464546dbc89537db1bcb1)
> Rust: Record (de?)serialization is sensitive to order of fields in struct
> -
>
> Key: AVRO-3821
> URL: https://issues.apache.org/jira/browse/AVRO-3821
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Renar Narubin
>Assignee: Martin Tzvetanov Grigorov
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> A previous issue describes a bug where record validation failed due to a
> struct's field declaration order not matching its schema:
> https://issues.apache.org/jira/browse/AVRO-3495
>
> That issue's fix appears to be incomplete. Validation during writing works,
> however the written records cannot be read back - it appears that the fields
> are mismatched during deserialization. I've reproduced using the existing
> `avro_old_issue_47` test case with a modification to read the data back:
>
> {code:java}
> diff --git a/lang/rust/avro/tests/schema.rs b/lang/rust/avro/tests/schema.rs
> index 00be0ab01..d5b1ee904 100644
> --- a/lang/rust/avro/tests/schema.rs
> +++ b/lang/rust/avro/tests/schema.rs
> @@ -18,6 +18,7 @@
> use std::io::{Cursor, Read};
>
> use apache_avro::{
> + from_avro_datum, from_value,
> schema::{EnumSchema, FixedSchema, Name, RecordField, RecordSchema},
> to_avro_datum, to_value,
> types::{Record, Value},
> @@ -1452,7 +1453,7 @@ fn avro_old_issue_47() -> TestResult {
>
> use serde::{Deserialize, Serialize};
>
> - #[derive(Deserialize, Serialize)]
> + #[derive(Deserialize, Serialize, Debug, Clone, PartialEq)]
> pub struct MyRecord {
> b: String,
> a: i64,
> @@ -1463,7 +1464,15 @@ fn avro_old_issue_47() -> TestResult {
> a: 1,
> };
>
> - let _ = to_avro_datum(&schema, to_value(record)?)?;
> + let serialized_bytes = to_avro_datum(&schema,
> to_value(record.clone())?)?;
> +
> + let deserialized_record = from_value::(&from_avro_datum(
> + &schema,
> + &mut Cursor::new(serialized_bytes),
> + None,
> + )?)?;
> +
> + assert_eq!(record, deserialized_record);
> Ok(())
> }
>
> {code}
> This fails as written, and passes if `a` and `b` are swapped in the MyRecord
> definition.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AVRO-3821) Rust: Record (de?)serialization is sensitive to order of fields in struct
[
https://issues.apache.org/jira/browse/AVRO-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754415#comment-17754415
]
ASF subversion and git services commented on AVRO-3821:
---
Commit d05ca14e7de4281f5db464546dbc89537db1bcb1 in avro's branch
refs/heads/dependabot/cargo/lang/rust/master/color-backtrace-0.6.0 from Martin
Grigorov
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=d05ca14e7 ]
AVRO-3821: [Rust] Encoding records should follow the schema (#2417)
When encoding Value::Record the logic should use the order of the fields
in the schema instead of the order in the Value::Record
Signed-off-by: Martin Tzvetanov Grigorov
> Rust: Record (de?)serialization is sensitive to order of fields in struct
> -
>
> Key: AVRO-3821
> URL: https://issues.apache.org/jira/browse/AVRO-3821
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Renar Narubin
>Assignee: Martin Tzvetanov Grigorov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> A previous issue describes a bug where record validation failed due to a
> struct's field declaration order not matching its schema:
> https://issues.apache.org/jira/browse/AVRO-3495
>
> That issue's fix appears to be incomplete. Validation during writing works,
> however the written records cannot be read back - it appears that the fields
> are mismatched during deserialization. I've reproduced using the existing
> `avro_old_issue_47` test case with a modification to read the data back:
>
> {code:java}
> diff --git a/lang/rust/avro/tests/schema.rs b/lang/rust/avro/tests/schema.rs
> index 00be0ab01..d5b1ee904 100644
> --- a/lang/rust/avro/tests/schema.rs
> +++ b/lang/rust/avro/tests/schema.rs
> @@ -18,6 +18,7 @@
> use std::io::{Cursor, Read};
>
> use apache_avro::{
> + from_avro_datum, from_value,
> schema::{EnumSchema, FixedSchema, Name, RecordField, RecordSchema},
> to_avro_datum, to_value,
> types::{Record, Value},
> @@ -1452,7 +1453,7 @@ fn avro_old_issue_47() -> TestResult {
>
> use serde::{Deserialize, Serialize};
>
> - #[derive(Deserialize, Serialize)]
> + #[derive(Deserialize, Serialize, Debug, Clone, PartialEq)]
> pub struct MyRecord {
> b: String,
> a: i64,
> @@ -1463,7 +1464,15 @@ fn avro_old_issue_47() -> TestResult {
> a: 1,
> };
>
> - let _ = to_avro_datum(&schema, to_value(record)?)?;
> + let serialized_bytes = to_avro_datum(&schema,
> to_value(record.clone())?)?;
> +
> + let deserialized_record = from_value::(&from_avro_datum(
> + &schema,
> + &mut Cursor::new(serialized_bytes),
> + None,
> + )?)?;
> +
> + assert_eq!(record, deserialized_record);
> Ok(())
> }
>
> {code}
> This fails as written, and passes if `a` and `b` are swapped in the MyRecord
> definition.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
