[jira] [Updated] (AVRO-3625) [Rust] UnionSchema.is_nullable() should return true if any of the variants is Schema::Null
[
https://issues.apache.org/jira/browse/AVRO-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3625:
-
Labels: pull-request-available (was: )
> [Rust] UnionSchema.is_nullable() should return true if any of the variants is
> Schema::Null
> --
>
> Key: AVRO-3625
> URL: https://issues.apache.org/jira/browse/AVRO-3625
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Affects Versions: 1.11.1
>Reporter: Martin Tzvetanov Grigorov
>Assignee: Martin Tzvetanov Grigorov
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently UnionSchema::is_nullable() returns `true` if the first variant is
> `"null"`.
> The specification
> [says|https://avro.apache.org/docs/1.11.1/specification/#unions]: "Thus, for
> unions containing “null”, the “null” is usually listed first, since the
> default value of such unions is typically null."
> That is, the `null` is *usually* at the first position{*},{*} but this is not
> mandatory!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-2831) Resolver cannot find reader/writer schema difference of fields in Union
[
https://issues.apache.org/jira/browse/AVRO-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-2831:
-
Labels: pull-request-available (was: )
> Resolver cannot find reader/writer schema difference of fields in Union
> ---
>
> Key: AVRO-2831
> URL: https://issues.apache.org/jira/browse/AVRO-2831
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0, 1.9.1, 1.9.2
>Reporter: Jinge Dai
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When avro java library deserializes and tries to resolve fields in a Union,
> it will use the writer schema if the reader field type is same as writer's,
> but it ignores the properties and logical type.
> {code:java}
> private static boolean unionEquiv(Schema write, Schema read, Map Boolean> seen) {
> final Schema.Type wt = write.getType();
> if (wt != read.getType()) {
> return false;
> }
> {code}
> However, I have hit two issues:
> # https://issues.apache.org/jira/browse/AVRO-2702 If producer is not java,
> the string field the strings of the inner record will be deserialized to
> org.apache.avro.util.Utf8 and then get error
> {code:java}
> Caused by: java.lang.ClassCastException: class org.apache.avro.util.Utf8
> cannot be cast to class java.lang.String{code}
>
> 2. I am using a built-in-house C# lib on the producer side, and it builds
> Date field schema as INT type without logicalTyple, this is bug on my side.
> However, AVRO 1.8.2 is able to handle it by using reader schema, whereas
> 1.9.2 gets error
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Parameters cannot be null!
> Parameter values:[-129, "int", null,
> org.apache.avro.data.TimeConversions$DateConversion@d3449b1]
> {code}
> The reason is, although the consumer side generates the class properly and
> sets the field to be converted to logical type, but the convertToLogicalType
> failed because the writer schema is used and the logical type is null
> {code:java}
> public static Object convertToLogicalType(Object datum, Schema schema,
> LogicalType type, Conversion conversion) {
> if (datum == null) {
> return null;
> }
> if (schema == null || type == null || conversion == null) {
> throw new IllegalArgumentException("Parameters cannot be null! Parameter
> values:"
> + Arrays.deepToString(new Object[] { datum, schema, type, conversion
> }));
> }
> {code}
>
> I am wondering if we should check type and properties in "unionEquiv" so that
> we can use the reader schema if there is difference. This complies the AVRO
> spec
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3628) [Java] JUnit 4.x tests are not executed
[
https://issues.apache.org/jira/browse/AVRO-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3628:
-
Labels: pull-request-available (was: )
> [Java] JUnit 4.x tests are not executed
> ---
>
> Key: AVRO-3628
> URL: https://issues.apache.org/jira/browse/AVRO-3628
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.11.1
>Reporter: Martin Tzvetanov Grigorov
>Assignee: Martin Tzvetanov Grigorov
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> AVRO-3579 introduced JUnit 5 and migrated some of the tests.
> This confused maven-surefire-plugin and now it does not run the JUnit 4.x
> tests, e.g.:
> {code:java}
> [INFO] --- maven-surefire-plugin:3.0.0-M7:test (default-test) @
> avro-maven-plugin ---
> [INFO] Tests will run in random order. To reproduce ordering use flag
> -Dsurefire.runOrder.random.seed=71852499748477
> [INFO] Using auto detected provider
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
> [INFO] --- maven-checkstyle-plugin:3.1.2:check (checkstyle-check) @
> avro-maven-plugin ---
> [INFO] Starting audit...
> {code}
>
> We should use JUnit 5.x Vintage engine instead of JUnit 4.x
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-1938) Python support for generating canonical forms of schema
[ https://issues.apache.org/jira/browse/AVRO-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-1938: - Labels: pull-request-available (was: ) > Python support for generating canonical forms of schema > --- > > Key: AVRO-1938 > URL: https://issues.apache.org/jira/browse/AVRO-1938 > Project: Apache Avro > Issue Type: Improvement > Components: python >Reporter: Erik Forsberg >Assignee: Michael A. Smith >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The python implementation(s) lack support for generating canonical forms of > schema. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3532) Align naming rules on code
[
https://issues.apache.org/jira/browse/AVRO-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3532:
-
Labels: pull-request-available (was: )
> Align naming rules on code
> --
>
> Key: AVRO-3532
> URL: https://issues.apache.org/jira/browse/AVRO-3532
> Project: Apache Avro
> Issue Type: Wish
>Reporter: Christophe Le Saec
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Description of [naming rule on
> documentation|https://avro.apache.org/docs/current/spec.html#names] is
> {noformat}
> - start with [A-Za-z_]
> - subsequently contain only [A-Za-z0-9_]
> {noformat}
> But [java
> code|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/Schema.java#L1578]
> use Character.isLetter method
> {code:java}
> char first = name.charAt(0);
> if (!(Character.isLetter(first) || first == '_'))
> throw new SchemaParseException("Illegal initial character: " + name);
> for (int i = 1; i < length; i++) {
> char c = name.charAt(i);
> if (!(Character.isLetterOrDigit(c) || c == '_'))
> throw new SchemaParseException("Illegal character in: " + name);
> }
> return name;
> {code}
> This method accept accent éùàçË ... and also chinese character (我) ...
> So, the aim of this ticket is to see if we can update the documentation, if
> other implementations (rust, C# ...) are also compatible with ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3611) org.apache.avro.util.RandomData generates invalid test data
[
https://issues.apache.org/jira/browse/AVRO-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3611:
-
Labels: features pull-request-available (was: features)
> org.apache.avro.util.RandomData generates invalid test data
> ---
>
> Key: AVRO-3611
> URL: https://issues.apache.org/jira/browse/AVRO-3611
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.11.1
>Reporter: Simon Klakegg
>Priority: Minor
> Labels: features, pull-request-available
> Fix For: 1.11.2
>
> Attachments: image-2022-08-18-19-05-37-323.png
>
> Original Estimate: 48h
> Time Spent: 10m
> Remaining Estimate: 47h 50m
>
> When RandomData.java generates data it does not check for Logical Types,
> which are described here: [Specification | Apache
> Avro|https://avro.apache.org/docs/1.11.1/specification/_print/]
> *For instance the following generate method would return this for INT fields:*
> {code:java}
> case INT: return random.nextInt(); {code}
>
> {*}However, an int field could be of logical type
> date:{*}!image-2022-08-18-19-05-37-323.png|width=1052,height=266!
>
> Which in many cases could create an int that is out of range for logicalType
> Date, and thus break when creating records in for instance kafka.
> My suggestion is to generated data that is valid for logicalTypes, here is an
> example I made for int and long:
> {code:java}
> case INT:
> switch (logicalTypeName) {
> case "date":
> // Random number of days between Unix Epoch start day (0) and end day
> (24855)
> int maxDaysInEpoch = (int)
> Duration.ofSeconds(Integer.MAX_VALUE).toDays();
> return ThreadLocalRandom.current().nextInt(0, maxDaysInEpoch);
> case "time-millis":
> // Random number of milliseconds between midnight 00:00:00.000 (0)
> and 23:59:59:999 (8639)
> int maxMillisecondsInDay = (int) Duration.ofDays(1).toMillis() - 1;
> return random.nextInt(0, maxMillisecondsInDay);
> default: return random.nextInt();
> }
> case LONG:
> switch (logicalTypeName) {
> case "time-micros":
> // Random number of microseconds between midnight 00:00:00.00 (0)
> and 23:59:59:99 (863)
> long maxMicrosecondsInDay = (Duration.ofDays(1).toNanos() - 1) / 1000;
> return random.nextLong(0, maxMicrosecondsInDay);
> case "timestamp-millis":
> // Random milliseconds between Unix Epoch (0) start and end
> (2147483647000)
> long maxMillisecondsInEpoch =
> TimeUnit.SECONDS.toMillis(Integer.MAX_VALUE);
> return ThreadLocalRandom.current().nextLong(0, maxMillisecondsInEpoch);
> case "timestamp-micros":
> // Random microseconds between Unix Epoch (0) start and end
> (214748364700)
> long maxMicrosecondsInEpoch =
> TimeUnit.SECONDS.toMicros(Integer.MAX_VALUE);
> return ThreadLocalRandom.current().nextLong(0, maxMicrosecondsInEpoch);
> case "local-timestamp-millis":
> // Random number of milliseconds between Unix Epoch start (0) and 100
> years from now (now() + 100)
> ZonedDateTime hundredYearsFromNow = ZonedDateTime.now().plusYears(100);
> long hundredYearsEpochMillis = ChronoUnit.MILLIS.between(Instant.EPOCH,
> hundredYearsFromNow);
> return random.nextLong(0, hundredYearsEpochMillis);
> case "local-timestamp-micros":
> // Random number of microseconds between Unix Epoch start (0) and 100
> years from now (now() + 100)
> long hundredYearsEpochMicros = ChronoUnit.MICROS.between(Instant.EPOCH,
> hundredYearsFromNow);
> return random.nextLong(0, hundredYearsEpochMicros);
> default: return random.nextLong();
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3624) Fix Avro website checks on whimsy
[ https://issues.apache.org/jira/browse/AVRO-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3624: - Labels: pull-request-available website (was: website) > Fix Avro website checks on whimsy > - > > Key: AVRO-3624 > URL: https://issues.apache.org/jira/browse/AVRO-3624 > Project: Apache Avro > Issue Type: Improvement > Components: community >Reporter: Ryan Skraba >Priority: Major > Labels: pull-request-available, website > Time Spent: 10m > Remaining Estimate: 0h > > With the new website, some of the checks on the website are no longer green: > [https://whimsy.apache.org/site/] > We should ensure all of the required links are present. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3630) [Rust] Make it possible to extend pre-existing Avro bytes
[
https://issues.apache.org/jira/browse/AVRO-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3630:
-
Labels: pull-request-available (was: )
> [Rust] Make it possible to extend pre-existing Avro bytes
> -
>
> Key: AVRO-3630
> URL: https://issues.apache.org/jira/browse/AVRO-3630
> Project: Apache Avro
> Issue Type: Improvement
> Components: rust
>Affects Versions: 1.11.1
>Reporter: Martin Tzvetanov Grigorov
>Assignee: Martin Tzvetanov Grigorov
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> From ASF Slack
> #[avro|[https://the-asf.slack.com/archives/CLUD54M1S/p1663751850569319]:]
> ===
> I'm currenlty solving it by reading the original file and appending it to a
> new Vec
> {code:java}
> let bytes: Vec = object_store
> .get(&old_manifest_location)
> .await?
> .bytes()
> .await?
> .into();
> let mut writer = apache_avro::Writer::new(&schema, Vec::new());
> let reader = apache_avro::Reader::new(&*bytes)?;
> writer.extend(reader.filter_map(Result::ok))?;
> writer.append_ser(entry)?;
> let new_bytes = writer.into_inner()?;
> {code}
> I was wondering if I could do something like:
> {code:java}
> let bytes: Vec = object_store
> .get(&old_manifest_location)
> .await?
> .bytes()
> .await?
> .into();
> let mut writer = apache_avro::Writer::new(&schema, bytes);
> writer.append_ser(entry)?;
> let new_bytes = writer.into_inner()?;
> {code}
> ==
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3631) Fix serialization of structs containing Fixed fields
[
https://issues.apache.org/jira/browse/AVRO-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3631:
-
Labels: pull-request-available (was: )
> Fix serialization of structs containing Fixed fields
>
>
> Key: AVRO-3631
> URL: https://issues.apache.org/jira/browse/AVRO-3631
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Rik Heijdens
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Consider the following minimal Avro Schema:
> {noformat}
> {
> "type": "record",
> "name": "TestStructFixedField",
> "fields": [
> {
> "name": "field",
> "type": {
> "name": "field",
> "type": "fixed",
> "size": 6
> }
> }
> ]
> }
> {noformat}
> In Rust, I might represent this schema with the following struct:
> {noformat}
> #[derive(Debug, Serialize, Deserialize)]
> struct TestStructFixedField {
> field: [u8; 6]
> }
> {noformat}
> I would then expect to be able to use `apache_avro::to_avro_datum()` to
> convert an instance of `TestStructFixedField` into an `Vec` using an
> instance of `Schema` initialized from the schema listed above.
> However, this fails because the `Value` produced by `apache_avro::to_value()`
> represents `field` as an `Value::Array` rather than a
> `Value::Fixed<6, Vec` which does not pass schema validation.
> I believe that there are two options to fix this:
> 1. Allow Value::Array> to pass validation if the array has
> the expected length, and none of the contents of the array are out-of-range
> for u8. If we go down this route, the implementation of `to_avro_datum()`
> will have to take care of converting Value::Int to u8 when converting into
> bytes.
> 2. Update `apache_avro::to_value()` such that fixed length arrays are
> converted into `Value::Fixed>` rather than `Value::Array`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3632) Union defaults are not handled as per the specification
[ https://issues.apache.org/jira/browse/AVRO-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3632: - Labels: pull-request-available (was: ) > Union defaults are not handled as per the specification > --- > > Key: AVRO-3632 > URL: https://issues.apache.org/jira/browse/AVRO-3632 > Project: Apache Avro > Issue Type: Bug > Components: rust >Affects Versions: 1.11.1 >Reporter: Santiago Fraire Willemoes >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > According to the [union > spec|https://avro.apache.org/docs/1.11.1/specification/#unions]. > ??Note that when a default value is specified for a record field whose type > is a union, the type of the default value must match the first element of the > union. Thus, for unions containing “null”, the “null” is usually listed > first, since the default value of such unions is typically null?? > I wonder if a PR would be accepted for this. I managed to add the check for > the first element in the parser. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3633) Additional attributes for 'avro_derive' crate
[
https://issues.apache.org/jira/browse/AVRO-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3633:
-
Labels: easyfix features pull-request-available (was: easyfix features)
> Additional attributes for 'avro_derive' crate
> -
>
> Key: AVRO-3633
> URL: https://issues.apache.org/jira/browse/AVRO-3633
> Project: Apache Avro
> Issue Type: Improvement
> Components: rust
>Affects Versions: 1.11.1
>Reporter: Jimmy Pramosi
>Priority: Minor
> Labels: easyfix, features, pull-request-available
> Original Estimate: 1h
> Time Spent: 10m
> Remaining Estimate: 50m
>
> Hello,
> At the moment it is not possible to skip or rename a specific field within a
> struct derived with 'AvroSchema'. For e.g.
> {code:java}
> use std::time::Instant;
> use apache_avro::AvroSchema;
> use serde::Deserialize;
> use serde::Serialize;
> #[derive(Clone, Debug, Deserialize, Serialize, AvroSchema)]
> pub struct FileDescriptor {
> #[avro(rename = "pflags")] // <--- NOT POSSIBLE
> #[serde(rename = "pflags")]
> pub flags: u64,
> #[avro(skip)] // <--- NOT POSSIBLE
> #[serde(skip)]
> access_time: Instant,
> }{code}
> I would like to suggest to implement additional attributes 'skip' and
> 'rename' to the 'avro_derive' crate. These attributes are also common in
> other procedural macro crates like 'serde'.
> Exemplary proposal:
> [https://www.diffchecker.com/iju3tSuX]
>
> Best Regards,
> Jimmy
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3634) Implement schema for missing types
[
https://issues.apache.org/jira/browse/AVRO-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3634:
-
Labels: easyfix pull-request-available (was: easyfix)
> Implement schema for missing types
> --
>
> Key: AVRO-3634
> URL: https://issues.apache.org/jira/browse/AVRO-3634
> Project: Apache Avro
> Issue Type: Improvement
> Components: rust
>Affects Versions: 1.11.1
>Reporter: Jimmy Pramosi
>Priority: Trivial
> Labels: easyfix, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hello,
> I would like to suggest to implement a schema for the type 'bool' and maybe
> 'u64'.
> The change is pretty trivial and involves adding
> {code:java}
> impl_schema!(bool, Schema::Boolean);
> impl_schema!(u64, Schema::Long);{code}
> to
> [https://github.com/apache/avro/blob/63c58f3b29d20ba94a6c68d9e4c09816d448bf34/lang/rust/avro/src/schema.rs#L1830]
>
> Best Regards,
> Jimmy
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3638) Support mapping between avro namespace and java packagename
[ https://issues.apache.org/jira/browse/AVRO-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3638: - Labels: pull-request-available (was: ) > Support mapping between avro namespace and java packagename > --- > > Key: AVRO-3638 > URL: https://issues.apache.org/jira/browse/AVRO-3638 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Simon >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When Java code is generated from an avro schema, the created classes are > given a "package" which is the namespace of the schema. However some users > prefer to have the java classes placed under some specific base package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3642) GenericSingleObjectReader::read_value fails on non-exhaustive read
[
https://issues.apache.org/jira/browse/AVRO-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3642:
-
Labels: pull-request-available (was: )
> GenericSingleObjectReader::read_value fails on non-exhaustive read
> --
>
> Key: AVRO-3642
> URL: https://issues.apache.org/jira/browse/AVRO-3642
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: David Peklak
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is a bug in the Rust implementation of
> [GenericSingleObjectReader::read_value|https://docs.rs/apache-avro/0.14.0/apache_avro/struct.GenericSingleObjectReader.html#method.read_value],
> which is exposed when a call to reader.read reads less than 10 bytes. This
> is a valid scenario, as according to the documentation of
> [Read::read|https://doc.rust-lang.org/std/io/trait.Read.html#tymethod.read],
> "It is not an error if the returned value {{n}} is smaller than the buffer
> size, even when the reader is not at the end of the stream yet." This is also
> a real-life scenario: in my use-case, I am chaining three implementations of
> the Read trait to messages that can be deserialized (one for the two magic
> bytes, one for the schema fingerprint, and one for the actual message
> content). Each call to Read::read only returns the bytes of one chain link,
> so the first call to Read::read only returns 2 bytes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3644) [JAVA] Support java.util.Optional in reflect package
[
https://issues.apache.org/jira/browse/AVRO-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3644:
-
Labels: pull-request-available (was: )
> [JAVA] Support java.util.Optional in reflect package
>
>
> Key: AVRO-3644
> URL: https://issues.apache.org/jira/browse/AVRO-3644
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Daniel Heinrich
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> java.util.Optional can currently not be used at all when using reflection
> based schemas.
> _org.apache.avro.reflect.ReflectData#getSchema_ will currently produce the
> following type for fields using Optional:
> {"type":"record","name":"Optional","namespace":"java.util","fields":[]}
> The generated schema is completly unusable (it does not contain any data). I
> would like to propose to map Optional fields as if they were not using
> Optional and would be annotated with @Nullable. So resulting in the following
> type:
> ["null","something"]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3645) Fix deserialization of enum with unit () type
[
https://issues.apache.org/jira/browse/AVRO-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3645:
-
Labels: pull-request-available rust (was: rust)
> Fix deserialization of enum with unit () type
> -
>
> Key: AVRO-3645
> URL: https://issues.apache.org/jira/browse/AVRO-3645
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Lucas Javaudin
>Priority: Major
> Labels: pull-request-available, rust
> Original Estimate: 2h
> Time Spent: 10m
> Remaining Estimate: 1h 50m
>
> Consider the following test:
> {code:java}
> #[derive(Debug, Deserialize, Serialize, PartialEq, Eq)]
> struct TestNullExternalEnum {
> a: NullExternalEnum,
> }
> #[derive(Debug, Deserialize, Serialize, PartialEq, Eq)]
> enum NullExternalEnum {
> Val1(()),
> Val2(u64),
> }
> #[test]
> fn test_from_value_null_enum() {
> let expected = TestNullExternalEnum {
> a: NullExternalEnum::Val1(()),
> };
> let test = Value::Record(vec![(
> "a".to_owned(),
> Value::Record(vec![
> ("type".to_owned(), Value::String("Val1".to_owned())),
> ("value".to_owned(), Value::Union(0, Box::new(Value::Null))),
> ]),
> )]);
> let final_value: TestNullExternalEnum = from_value(&test).unwrap();
> assert_eq!(
> final_value, expected,
> "Error deserializing null external enum"
> );
> }{code}
> On version `0.15.0`, it fails with the following error message:
> {code:java}
> called `Result::unwrap()` on an `Err` value: DeserializeValue("not a null")
> {code}
>
> If my understanding is correct, it fails because rust is expecting a
> `Value::Null` but it got a `Value::Union(0, Box::new(Value::Null))`.
> A fix that I found is to replace the `deserialize_unit`function in `de.rs`
> from
> {code:java}
> fn deserialize_unit(self, visitor: V) -> Result
> where
> V: Visitor<'de>,
> {
> match *self.input {
> Value::Null => visitor.visit_unit(),
> _ => Err(de::Error::custom("not a null")),
> }
> } {code}
> to
> {code:java}
> fn deserialize_unit(self, visitor: V) -> Result
> where
> V: Visitor<'de>,
> {
> match *self.input {
> Value::Null => visitor.visit_unit(),
> Value::Union(_i, ref x) => match **x {
> Value::Null => visitor.visit_unit(),
> _ => Err(de::Error::custom("not a null")),
> },
> _ => Err(de::Error::custom("not a null")),
> }
> } {code}
> i.e., we allow it to accept `Value::Union`, in the same way that the
> `deserialize_string` function accepts such values.
>
> If you ask yourself what is the point of having an enum with unit type, it is
> because apache-avro is not able to handle the following enum (as far as I
> know):
> {code:java}
> enum NullExternalEnum {
> Val1,
> Val2(u64),
> } {code}
> I am sorry if there is any problem with this issue, it's the first time I
> open an issue here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3649) [JAVA] reorder union types to match default value
[
https://issues.apache.org/jira/browse/AVRO-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3649:
-
Labels: pull-request-available (was: )
> [JAVA] reorder union types to match default value
> -
>
> Key: AVRO-3649
> URL: https://issues.apache.org/jira/browse/AVRO-3649
> Project: Apache Avro
> Issue Type: Improvement
>Reporter: Daniel Heinrich
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The Avro specification requires that a default value matches the first schema
> of a union type.
>
> Using @Nullable on a field will return a union type of the form ["null",
> ""].
> @Nullable also sets the default value to "null".
>
> It is not possible to override this default value with
> {color:#bbb529}{color:#172b4d}@AvroDefault because of the order of the
> schemas in the generated union.{color}
> {color}
>
> It would be nice that union types are automatically reordered to match the
> type of the provided default value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3650) Fix C++ Build on Manjaro
[
https://issues.apache.org/jira/browse/AVRO-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3650:
-
Labels: pull-request-available (was: )
> Fix C++ Build on Manjaro
>
>
> Key: AVRO-3650
> URL: https://issues.apache.org/jira/browse/AVRO-3650
> Project: Apache Avro
> Issue Type: Bug
>Reporter: John McClean
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Manjaro has a recent version of boost, which results in this.
> {quote}[ 19%] Building CXX object
> CMakeFiles/avrocpp.dir/impl/parsing/JsonCodec.cc.o
> In file included from /usr/include/boost/math/tools/real_cast.hpp:9,
> from
> /usr/include/boost/math/special_functions/fpclassify.hpp:17,
> from
> /home/john/third-party/avro/lang/c++/impl/parsing/JsonCodec.cc:20:
> /usr/include/boost/math/tools/config.hpp:23:6: error: #warning "The minimum
> language standard to use Boost.Math will be C++14 starting in July 2023
> (Boost 1.82 release)" [-Werror=cpp]
> 23 | # warning "The minimum language standard to use Boost.Math will be
> C++14 starting in July 2023 (Boost 1.82 release)"
> | ^~~
> cc1plus: all warnings being treated as errors
> make[2]: *** [CMakeFiles/avrocpp.dir/build.make:328:
> CMakeFiles/avrocpp.dir/impl/parsing/JsonCodec.cc.o] Error 1
> make[1]: *** [CMakeFiles/Makefile2:129: CMakeFiles/avrocpp.dir/all] Error 2
> make: *** [Makefile:166: all] Error 2
> {quote}
> In addition, once C++14 is used, the list of cpp keywords needs to be updated
> to include 'constinit'.
> {quote}[ 98%] Building CXX object
> CMakeFiles/AvrogencppTestReservedWords.dir/test/AvrogencppTestReservedWords.cc.o
> In file included from
> /home/john/third-party/avro/lang/c++/test/AvrogencppTestReservedWords.cc:18:
> /home/john/third-party/avro/lang/c++/build/cpp_reserved_words.hh:57:17:
> error: identifier ‘constinit’ is a keyword in C++20 [-Werror=c++20-compat]
> 57 | std::string constinit;
> | ^
> cc1plus: all warnings being treated as errors
> make[2]: *** [CMakeFiles/AvrogencppTestReservedWords.dir/build.make:76:
> CMakeFiles/AvrogencppTestReservedWords.dir/test/AvrogencppTestReservedWords.cc.o]
> Error 1
> make[1]: *** [CMakeFiles/Makefile2:951:
> CMakeFiles/AvrogencppTestReservedWords.dir/all] Error 2
> make: *** [Makefile:166: all] Error 2
> {quote}
> I'll put in a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3646) Serialization / deserialization for enum with mixed variants (with and without data)
[
https://issues.apache.org/jira/browse/AVRO-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3646:
-
Labels: pull-request-available rust (was: rust)
> Serialization / deserialization for enum with mixed variants (with and
> without data)
>
>
> Key: AVRO-3646
> URL: https://issues.apache.org/jira/browse/AVRO-3646
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Lucas Javaudin
>Priority: Major
> Labels: pull-request-available, rust
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Consider the following example of what I call an "enum with mixed variants":
> {code:java}
> enum MixedExternalEnum {
> Val1,
> Val2(f32),
> } {code}
> For the two variants, this enum can be properly serialized to and
> deserialized from a {{types::Value}} (using {{ser::to_value}} and
> {{{}de::from_value{}}}). The problem is that the way they are represented as
> a \{{types::Value}} cannot be represented using an Avro Schema.
> The serialized version of {{MixedExternalEnum::Val1}} is
> {code:java}
> Value::Record(vec![("a".to_owned(), Value::Enum(0, "Val1".to_owned()))])
> {code}
> while the serialized version of {{MixedExternalEnum::Val2(0.0)}} is
> {code:java}
> Value::Record(vec![(
> "a".to_owned(),
> Value::Record(vec![
> ("type".to_owned(), Value::Enum(1, "Val2".to_owned())),
> (
> "value".to_owned(),
> Value::Union(1, Box::new(Value::Float(0.0))),
> ),
> ]),
> )])
> {code}
> As far as I know, there is no Avro Schema compatible with both possible
> values.
> One solution I can think of is to change the serialization of
> {{MixedExternalEnum::Val1}} to
> {code:java}
> Value::Record(vec![(
> "a".to_owned(),
> Value::Record(vec![
> ("type".to_owned(), Value::Enum(0, "Val1".to_owned())),
> (
> "value".to_owned(),
> Value::Union(0, Box::new(Value::Null)),
> ),
> ]),
> )])
> {code}
>
> A workaround is to replace {{Val1}} by {{Val1(())}} in the enum definition
> (see AVRO-3645) but this can have undesirable effects for other parts of the
> code.
>
> Then, it's another story for adjacently tagged, internally tagged and
> untagged enum...
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3658) Bump jackson to address CVE-2020-36518
[
https://issues.apache.org/jira/browse/AVRO-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3658:
-
Labels: pull-request-available (was: )
> Bump jackson to address CVE-2020-36518
> --
>
> Key: AVRO-3658
> URL: https://issues.apache.org/jira/browse/AVRO-3658
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.11.1
>Reporter: Pavel Moskotin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Current version of Jackson dependency for AVRO/Java
> {code:xml}
> 2.12.7.20221012
> {code}
> brings CVE-2020-36518.
> This is covered by next versions, for example - in
> {code:xml}
> 2.13.4
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3661) [Rust] Fix new clippy errors introduced with Rust 1.65
[ https://issues.apache.org/jira/browse/AVRO-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3661: - Labels: pull-request-available (was: ) > [Rust] Fix new clippy errors introduced with Rust 1.65 > -- > > Key: AVRO-3661 > URL: https://issues.apache.org/jira/browse/AVRO-3661 > Project: Apache Avro > Issue Type: Task > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Rust 1.65.0 has been released and it comes with some new clippy checks, and > few of them lead to CI failures. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3659) Typo in python example
[ https://issues.apache.org/jira/browse/AVRO-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3659: - Labels: pull-request-available (was: ) > Typo in python example > -- > > Key: AVRO-3659 > URL: https://issues.apache.org/jira/browse/AVRO-3659 > Project: Apache Avro > Issue Type: Bug > Components: doc >Affects Versions: 1.11.1 >Reporter: Maksim Syomochkin >Priority: Trivial > Labels: pull-request-available > Attachments: typo.patch > > Time Spent: 10m > Remaining Estimate: 0h > > I found a small typo in python example (see attachment). Probably this is too > small a change to make a pull request -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3663) rust crate apache_avro_derive creates invalid schemas for raw identifiers
[
https://issues.apache.org/jira/browse/AVRO-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3663:
-
Labels: pull-request-available (was: )
> rust crate apache_avro_derive creates invalid schemas for raw identifiers
> -
>
> Key: AVRO-3663
> URL: https://issues.apache.org/jira/browse/AVRO-3663
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Neidhart Orlich
>Priority: Major
> Labels: pull-request-available
> Original Estimate: 0.5h
> Time Spent: 10m
> Remaining Estimate: 20m
>
> When using a raw identifier as field name, the `r#` prefix is used as field
> name for the resulting schema which results in an invalid AVRO schema.
> Example:
>
>
> {code:java}
> struct TestRawIdent { r#type: bool }{code}
>
>
> I implemented a fix and I am currently in the process of creating a PR
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3653) [build] Move off Travis CI
[ https://issues.apache.org/jira/browse/AVRO-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3653: - Labels: pull-request-available (was: ) > [build] Move off Travis CI > -- > > Key: AVRO-3653 > URL: https://issues.apache.org/jira/browse/AVRO-3653 > Project: Apache Avro > Issue Type: Task >Reporter: Ryan Skraba >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Infra is asking us to migrate away from Travis-CI definitively: > [https://cwiki.apache.org/confluence/display/INFRA/Travis+Migrations] > We had removed the Travis build, but restored some parts for ARM64 (AVRO-3242) > https://github.com/apache/avro/blob/master/.travis.yml -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3664) Add Rust Avro Value conversion derive implementations
[
https://issues.apache.org/jira/browse/AVRO-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3664:
-
Labels: pull-request-available (was: )
> Add Rust Avro Value conversion derive implementations
> -
>
> Key: AVRO-3664
> URL: https://issues.apache.org/jira/browse/AVRO-3664
> Project: Apache Avro
> Issue Type: New Feature
> Components: rust
>Reporter: Aaron Spiegel
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Given the current constraints of the _AvroSchema_ derive macro, it is
> possible to also derive the following implementations:
> {code:java}
> impl From for X {...}
> impl from for apache_avro::types::Value {...}{code}
> With these traits implemented, serialization in Avro becomes considerably
> simpler, for schemas that are already generated through a derive macro.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3669) Missing py.typed file
[
https://issues.apache.org/jira/browse/AVRO-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3669:
-
Labels: pull-request-available (was: )
> Missing py.typed file
> -
>
> Key: AVRO-3669
> URL: https://issues.apache.org/jira/browse/AVRO-3669
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Kaisen Chen
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The python avro package has type annotations but does not advertise itself as
> such because there is no py.typed marker file (see
> [PEP561|https://peps.python.org/pep-0561/#packaging-type-information]).
> Symptom:
> If python packages import avro and run mypy, error messages like this come up:
> {code:java}
> error: Skipping analyzing "avro": module is installed, but missing library
> stubs or py.typed marker {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3666) New schema parser for all supported schema formats
[ https://issues.apache.org/jira/browse/AVRO-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3666: - Labels: pull-request-available (was: ) > New schema parser for all supported schema formats > -- > > Key: AVRO-3666 > URL: https://issues.apache.org/jira/browse/AVRO-3666 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Oscar Westra van Holthe - Kind >Assignee: Oscar Westra van Holthe - Kind >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > As the new IDL syntax also supports schema files, it's time to refactor > schema parsing in Java to support all schema formats. > The goals: > * Parse JSON schemata with the same results > * Parse IDL schemata in the same way > * Use a pluggable API that supports other schema formats as well > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3670) Add .NET 7 support
[ https://issues.apache.org/jira/browse/AVRO-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3670: - Labels: pull-request-available (was: ) > Add .NET 7 support > -- > > Key: AVRO-3670 > URL: https://issues.apache.org/jira/browse/AVRO-3670 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Reporter: Zoltan Csizmadia >Priority: Minor > Labels: pull-request-available > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > .NET 7.0 SDK is released > [https://dotnet.microsoft.com/en-us/download/dotnet/7.0] > Add support for building and testing using NET 7 SDK as well. Published > class libraries will be still target netstandard2.0 and netstandard2.1, > however published tools and executables (e.g. AvroGen) will traget . NET 7 as > well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3657) Computation of initial buffer size in OutputBuffer makes no sense
[
https://issues.apache.org/jira/browse/AVRO-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3657:
-
Labels: pull-request-available (was: )
> Computation of initial buffer size in OutputBuffer makes no sense
> -
>
> Key: AVRO-3657
> URL: https://issues.apache.org/jira/browse/AVRO-3657
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Reporter: Tagir Valeev
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As you can see, OutputBuffer calls the superclass constructor in the
> following way:
> [https://github.com/apache/avro/blame/1119b6eb5b92730b27e9798793bc67f192591c15/lang/java/trevni/core/src/main/java/org/apache/trevni/OutputBuffer.java#L32]
>
> public OutputBuffer()
> { super((BLOCK_SIZE + BLOCK_SIZE) >> 2); }
>
> Before 3d51ba5e965561 change it was
>
> public OutputBuffer()
> { super(BLOCK_SIZE + BLOCK_SIZE >> 2); }
>
> Which looked like the author did not know that >> has lower precedence than +
> and forgot to put parentheses. It looks like, the 3d51ba5e965561 change
> blindly applied static analyzer suggestions and added parentheses to preserve
> the current semantics. However, `(BLOCK_SIZE + BLOCK_SIZE) >> 2` is simply
> `BLOCK_SIZE / 2`.
>
> My suggestion is the following:
> * If everybody is fine with current behavior, let's update this to
> `super(BLOCK_SIZE / 2);` to avoid confusion.
> * If we would like to preserve the original author's intention to allocate
> 1.25x more, let's place the parentheses in another way: `super(BLOCK_SIZE +
> (BLOCK_SIZE >> 2));`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3672) Add CI testing for Python 3.11
[ https://issues.apache.org/jira/browse/AVRO-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3672: - Labels: pull-request-available (was: ) > Add CI testing for Python 3.11 > -- > > Key: AVRO-3672 > URL: https://issues.apache.org/jira/browse/AVRO-3672 > Project: Apache Avro > Issue Type: Task > Components: python >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Python 3.11 has been released few weeks ago: > [https://www.python.org/downloads/release/python-3110/] > It would be good to test with it at Github Actions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3667) [Python] Python 3.10 CI test fails since a while
[ https://issues.apache.org/jira/browse/AVRO-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3667: - Labels: pull-request-available (was: ) > [Python] Python 3.10 CI test fails since a while > > > Key: AVRO-3667 > URL: https://issues.apache.org/jira/browse/AVRO-3667 > Project: Apache Avro > Issue Type: Bug > Components: python >Reporter: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The last 4 builds are failing at > [https://github.com/apache/avro/actions/workflows/test-lang-py.yml > |https://github.com/apache/avro/actions/workflows/test-lang-py.yml] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3662) [Ruby] Ruby 2.6 CI workflow fails since a while
[
https://issues.apache.org/jira/browse/AVRO-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3662:
-
Labels: pull-request-available (was: )
> [Ruby] Ruby 2.6 CI workflow fails since a while
> ---
>
> Key: AVRO-3662
> URL: https://issues.apache.org/jira/browse/AVRO-3662
> Project: Apache Avro
> Issue Type: Bug
> Components: ruby
>Reporter: Martin Tzvetanov Grigorov
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [https://github.com/apache/avro/actions/workflows/test-lang-ruby.yml]
>
> {code:java}
> ===
> Failure: test_bytes_decimal_object_allocations_decode(TestLogicalTypes):
> <1> was expected to be
> >=
> <2>.
> /home/runner/work/avro/avro/lang/ruby/test/test_logical_types.rb:234:in
> `block in test_bytes_decimal_object_allocations_decode'
> 231:
> 232: assert_equal 5, report.total_allocated
> 233: # Ruby 2.7 does not retain anything. Ruby 2.6 retains 1
> => 234: assert_operator 1, :>=, report.total_retained
> 235: end
> 236: end
> 237:
>
> /home/runner/work/avro/avro/lang/ruby/test/test_logical_types.rb:227:in
> `each'
>
> /home/runner/work/avro/avro/lang/ruby/test/test_logical_types.rb:227:in
> `test_bytes_decimal_object_allocations_decode' {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3673) Revive MSBuild AvroGenTask
[ https://issues.apache.org/jira/browse/AVRO-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3673: - Labels: pull-request-available (was: ) > Revive MSBuild AvroGenTask > -- > > Key: AVRO-3673 > URL: https://issues.apache.org/jira/browse/AVRO-3673 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Reporter: Zoltan Csizmadia >Priority: Minor > Labels: pull-request-available > Original Estimate: 48h > Time Spent: 10m > Remaining Estimate: 47h 50m > > Currently the avrogen tool must be calledfrom a csproj file to generate cs > files from schema or protocol files. This requires the Avro.Tools package to > be installed. > > This PR adds support for a native MSBuild task without explicitly lancing the > avrogen binary. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3674) Value::Record containing enums fail to validate when using namespaces in Schema
[
https://issues.apache.org/jira/browse/AVRO-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3674:
-
Labels: pull-request-available (was: )
> Value::Record containing enums fail to validate when using namespaces in
> Schema
> ---
>
> Key: AVRO-3674
> URL: https://issues.apache.org/jira/browse/AVRO-3674
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Rik Heijdens
>Assignee: Martin Tzvetanov Grigorov
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Consider the following schema:
> {noformat}
> {
> "type": "record",
> "name": "NamespacedMessage",
> "namespace": "com.domain",
> "fields": [
> {
> "type": "record",
> "name": "field_a",
> "fields": [
> {
> "name": "enum_a",
> "type": {
> "type": "enum",
> "name": "EnumType",
> "symbols": [
> "SYMBOL_1",
> "SYMBOL_2"
> ],
> "default": "SYMBOL_1"
> }
> },
> {
> "name": "enum_b",
> "type": "EnumType"
> }
> ]
> }
> ]
> }{noformat}
> I might represent this in Rust using the following structs:
> {noformat}
> #[derive(Serialize)]
> enum EnumType {
> #[serde(rename = "SYMBOL_1")]
> Symbol1,
> #[serde(rename = "SYMBOL_2")]
> Symbol2,
> }
> #[derive(Serialize)]
> struct FieldA {
> enum_a: EnumType,
> enum_b: EnumType,
> }
> #[derive(Serialize)]
> struct NamespacedMessage {
> field_a: FieldA,
> }
> let msg = NamespacedMessage {
> field_a: FieldA {
> enum_a: EnumType::Symbol2,
> enum_b: EnumType::Symbol1,
> },
> };
> {noformat}
> and then serialize this into a `Value` using the following logic:
> {noformat}
> let mut ser = Serializer::default();
> let test_value: Value = msg.serialize(&mut ser).unwrap();
> {noformat}
> After serializing into `test_value` I would expect that
> `test_value.validate(&schema)` yields True. However this is not the case.
> I can work around it by removing the `namespace` definition from my schema
> which allows the validation to proceed.
> I believe the cause of schema validation failure is [this lookup
> failing](https://github.com/apache/avro/blob/release-1.11.1-rc1/lang/rust/avro/src/types.rs#L370)
> when schemas are utilized as the `Value` will not have a namespace
> associated with it.
> I believe this could be fixed by altering `validate_internal` to accept an
> optional namespace that is derived from the provided schema to `validate`.
> However, I'm not sure if this is an appropriate fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3676) [Java] toString produces incorrect JSON for UUID
[ https://issues.apache.org/jira/browse/AVRO-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3676: - Labels: pull-request-available (was: ) > [Java] toString produces incorrect JSON for UUID > > > Key: AVRO-3676 > URL: https://issues.apache.org/jira/browse/AVRO-3676 > Project: Apache Avro > Issue Type: Bug >Reporter: Ryan Skraba >Priority: Major > Labels: pull-request-available > Fix For: 1.11.2 > > Time Spent: 10m > Remaining Estimate: 0h > > Like AVRO-3124, but for UUID classes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3678) [Rust] Support write float value to field defined as double
[ https://issues.apache.org/jira/browse/AVRO-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3678: - Labels: pull-request-available (was: ) > [Rust] Support write float value to field defined as double > --- > > Key: AVRO-3678 > URL: https://issues.apache.org/jira/browse/AVRO-3678 > Project: Apache Avro > Issue Type: New Feature > Components: rust >Reporter: Shaeq Ahmed >Assignee: Shaeq Ahmed >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.2 > > Time Spent: 10m > Remaining Estimate: 0h > > [https://github.com/apache/avro/blob/b8937526d32928b3b29040e862f7689d7230bddb/lang/rust/avro/src/types.rs#L405] > Currently a Avro value of the type float (32) cannot be resolved against a > schema where the type is defined as double (64). We already support type > widening from int -> long (dest. schema) as this is completely lossless, so I > think we should also support the same for float -> double for consistency & > convenience as this is a common use case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3679) [Rust] Enable 'perf' feature of regex dependency
[ https://issues.apache.org/jira/browse/AVRO-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3679: - Labels: pull-request-available (was: ) > [Rust] Enable 'perf' feature of regex dependency > > > Key: AVRO-3679 > URL: https://issues.apache.org/jira/browse/AVRO-3679 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Affects Versions: 1.11.1 >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The 'regex' crate has a 'perf' feature that improves the speed of the regular > expressions evaluation. > It would be good to make use of it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3680) [Python] Allow disabling naming validation for interoperability
[ https://issues.apache.org/jira/browse/AVRO-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3680: - Labels: pull-request-available (was: ) > [Python] Allow disabling naming validation for interoperability > --- > > Key: AVRO-3680 > URL: https://issues.apache.org/jira/browse/AVRO-3680 > Project: Apache Avro > Issue Type: New Feature > Components: python >Reporter: Ryan Skraba >Priority: Major > Labels: pull-request-available > Fix For: 1.11.2 > > Time Spent: 10m > Remaining Estimate: 0h > > The Java SDK accepts some Avro names that are not valid according to the > specification, or compatible with other SDKs. There is an [ongoing > discussion|https://lists.apache.org/thread/39v98os6wdpyr6w31xdkz0yzol51fsrr] > about the "right" thing to do. > Currently, schemas that meet the specification rules are guaranteed to be > interoperable. > When parsing schemas that might not be interoperable (but still might work > under certain circumstances), the Python SDK should provide a way to disable > name validation, like it provides a way to disable symbol [name > validation|https://github.com/apache/avro/blob/34845fd49af9cd00e98f5930c6ac6190da5ea6a3/lang/py/avro/schema.py#L1194]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3682) [Build] Remove forrest from Avro build
[
https://issues.apache.org/jira/browse/AVRO-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3682:
-
Labels: pull-request-available (was: )
> [Build] Remove forrest from Avro build
> --
>
> Key: AVRO-3682
> URL: https://issues.apache.org/jira/browse/AVRO-3682
> Project: Apache Avro
> Issue Type: Task
>Reporter: Ryan Skraba
>Assignee: Ryan Skraba
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We are no longer using forrest in the Avro build, and it can be removed from
> the project.
> This should include the deprecated files in {{docs-deprecated}} and the
> {{{}DockerfileLocal{}}}.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3683) Rust Writer, Reader can't use Schemas with dependencies in other Schemas. i.e. The output of Schema::parse_list
[
https://issues.apache.org/jira/browse/AVRO-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3683:
-
Labels: pull-request-available (was: )
> Rust Writer, Reader can't use Schemas with dependencies in other Schemas.
> i.e. The output of Schema::parse_list
> ---
>
> Key: AVRO-3683
> URL: https://issues.apache.org/jira/browse/AVRO-3683
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Affects Versions: 1.11.1
>Reporter: Mark Farnan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Avro permits Schema definitions to reference Types defined in other Schema
> files.
> Currently these can be parsed with Schema:parse_list, but the output of this
> can't be used in Writer, Reader, (or to_avro_datum, from_avro_datum)
> * These functions only take a single 'schema' and can't resolve the
> References
> * Schema:parse_list puts in 'refs' for the external dependencies, but
> doesn't seem to populate the UnionSchema's, so they can be resolved.
>
> +Example of dependant schemas:+
> letschema_TypeA=r#"
> {
> "type": "record",
> "namespace": "base.name.space",
> "name": "MultiSchemaTestTypeA",
> "fields":
> [
> { "name": "b", "type": ["null", "some.name.space.MultiSchemaTestTypeB"] },
> {
> "name": "c",
> "type": \{ "type": "map", "values": "some.name.space.MultiSchemaTestTypeB" },
> "default": {}
> }
> ]
> }
> "#;
> letschema_TypeB=r#"
> {
> "type": "record",
> "namespace": "some.name.space",
> "name": "MultiSchemaTestTypeB",
> "fields":
> [
> { "name": "d", "type": "string" },
> { "name": "e", "type": "int" }
> ]
> }
> "#;
>
> Note: In process of creating a PR with a failing test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3640) Ignores schema files when source folder is linked
[ https://issues.apache.org/jira/browse/AVRO-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3640: - Labels: pull-request-available (was: ) > Ignores schema files when source folder is linked > - > > Key: AVRO-3640 > URL: https://issues.apache.org/jira/browse/AVRO-3640 > Project: Apache Avro > Issue Type: Bug >Affects Versions: 1.11.1 > Environment: - java 17 > - maven 3.8.6 > - OS: ubuntu > >Reporter: Sławomir Siudek >Assignee: Rens Groothuijsen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > avro-maven-plugin > when source directory is created as linked, plugin skips code generation > silently > > > Full reproduction with README > https://github.com/siudeks/tech-spikes/tree/master/bugrepro_avro_code_generation -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3681) [Python] GitHub actions failing with python 3.6
[
https://issues.apache.org/jira/browse/AVRO-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3681:
-
Labels: pull-request-available (was: )
> [Python] GitHub actions failing with python 3.6
> ---
>
> Key: AVRO-3681
> URL: https://issues.apache.org/jira/browse/AVRO-3681
> Project: Apache Avro
> Issue Type: Task
>Reporter: Ryan Skraba
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The GitHub actions build is failing on Python 3.6, apparently because it is
> not available on ubuntu-latest anymore.
> From the logs in [this
> job|https://github.com/apache/avro/actions/runs/3603280401/jobs/6071285008]:
> {code}
> 2022-12-02T16:13:26.6605941Z Requested labels: ubuntu-latest
> 2022-12-02T16:13:26.6606006Z Job defined at:
> apache/avro/.github/workflows/test-lang-py.yml@refs/pull/1995/merge
>
> 2022-12-02T16:13:38.7515296Z ##[group]Run actions/setup-python@v4
> 2022-12-02T16:13:38.7515540Z with:
> 2022-12-02T16:13:38.7515728Z python-version: 3.6
> 2022-12-02T16:13:38.7515931Z check-latest: false
> 2022-12-02T16:13:38.7516303Z token: ***
> 2022-12-02T16:13:38.7516513Z update-environment: true
> 2022-12-02T16:13:38.7516716Z ##[endgroup]
> 2022-12-02T16:13:38.9235198Z Version 3.6 was not found in the local cache
> 2022-12-02T16:13:39.2572679Z ##[error]Version 3.6 with arch x64 not found
> The list of all available versions can be found here:
> https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json
> 2022-12-02T16:13:39.2689115Z Post job cleanup.
>
> {code}
> We can see that there is no 3.6 with a platform version of 22.04 in the
> referenced version manifest:
> https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json
> This issue is discussed here:
> https://github.com/actions/setup-python/issues/544
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3688) Schema resolution panics when a custom record field is included multiple times
[
https://issues.apache.org/jira/browse/AVRO-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3688:
-
Labels: pull-request-available (was: )
> Schema resolution panics when a custom record field is included multiple times
> --
>
> Key: AVRO-3688
> URL: https://issues.apache.org/jira/browse/AVRO-3688
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Rik Heijdens
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Consider the following Avro schema:
> {noformat}
> {
> "type": "record",
> "name": "Message",
> "fields": [
> {
> "name": "field_a",
> "type": [
> "null",
> {
> "name": "Inner",
> "type": "record",
> "fields": [
> {
> "name": "inner_a",
> "type": "string"
> }
> ]
> }
> ],
> "default": null
> },
> {
> "name": "field_b",
> "type": [
> "null",
> "Inner"
> ],
> "default": null
> }
> ]
> }{noformat}
> This can be represented in Rust through the following structs:
> {noformat}
> #[derive(Serialize, Deserialize)]
> struct Inner {
> inner_a: String
> }
> #[derive(Serialize, Deserialize)]
> struct Message {
> field_a: Option,
> field_b: Option
> }
> {noformat}
> If I instantiate an instance of `message`, set `field_a` and then serialize
> it using the following code:
> {noformat}
> let schema = Schema::parse_str(&schema_str).unwrap();
> let msg = Message {
> field_a: Some(Inner {
> inner_a: "foo".to_string()
> }),
> field_b: None
> };
> let mut ser = Serializer::default();
> let test_value: Value = msg.serialize(&mut ser).unwrap();
> assert!(test_value.validate(&schema), "test_value should validate");
> assert!(
> test_value.resolve(&schema).is_ok(),
> "test_value should resolve"
> );
> {noformat}
> Then my assertions pass. However if I set field_b to `Inner`, then my call to
> `test_value.resolve(&schema)` panics:
> {noformat}
> let schema = Schema::parse_str(&schema_str).unwrap();
> let msg = Message {
> field_a: Some(Inner {
> inner_a: "foo".to_string()
> }),
> field_b: Some(Inner {
> inner_a: "bar".to_string()
> })
> };
> let mut ser = Serializer::default();
> let test_value: Value = msg.serialize(&mut ser).unwrap();
> assert!(test_value.validate(&schema), "test_value should validate");
> assert!(
> test_value.resolve(&schema).is_ok(),
> "test_value should resolve"
> );
> {noformat}
> This seems to be a bug in the schema resolution logic that's causing an
> unhandled panic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3689) Detected a flaky test 'testAnnotationMultiAvroMeta'
[ https://issues.apache.org/jira/browse/AVRO-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3689: - Labels: pull-request-available (was: ) > Detected a flaky test 'testAnnotationMultiAvroMeta' > --- > > Key: AVRO-3689 > URL: https://issues.apache.org/jira/browse/AVRO-3689 > Project: Apache Avro > Issue Type: Test > Environment: > >Reporter: Zhengxu Jin >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Test `testAnnotationMultiAvroMeta' under 'TestReflect.java' is a flaky test. > To check its flakiness, you can run maven with nondex using the following > command: > ``` > mvn -pl lang/java/avro edu.illinois:nondex-maven-plugin:2.1.1:nondex > -Dtest=org.apache.avro.reflect.TestReflect#testAnnotationMultiAvroMeta > ``` > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3690) Detected a flaky test ‘testMultipleFieldAliases’
[ https://issues.apache.org/jira/browse/AVRO-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3690: - Labels: pull-request-available (was: ) > Detected a flaky test ‘testMultipleFieldAliases’ > > > Key: AVRO-3690 > URL: https://issues.apache.org/jira/browse/AVRO-3690 > Project: Apache Avro > Issue Type: Test >Reporter: Zhengxu Jin >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Test `testMultipleFieldAliases' under 'TestReflect.java' is a flaky test. To > check its flakiness, you can run maven with nondex using the following > command: > ``` > mvn -pl lang/java/avro edu.illinois:nondex-maven-plugin:2.1.1:nondex > -Dtest=org.apache.avro.reflect.TestReflect#testMultipleFieldAliases > ``` > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3691) Add Nondex:2.1.1 to Maven build section
[ https://issues.apache.org/jira/browse/AVRO-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3691: - Labels: pull-request-available (was: ) > Add Nondex:2.1.1 to Maven build section > --- > > Key: AVRO-3691 > URL: https://issues.apache.org/jira/browse/AVRO-3691 > Project: Apache Avro > Issue Type: Wish > Environment: NonDex is a tool for detecting and debugging wrong > assumptions on under-determined Java APIs. An example of such an assumption > is when code assumes the order of iterating through the entries in a > java.util.HashMap is in a specific, deterministic order, but the > specification for java.util.HashMap is under-determined and states that this > iteration order is not guaranteed to be in any particular order. Such > assumptions can hurt portability for an application when they are moved to > other environments with a different Java runtime. NonDex explores different > behaviors of under-determined APIs and reports test failures under different > explored behaviors; NonDex only explores behaviors that are allowed by the > specification and any test failure indicates an assumption on an > under-determined Java API. NonDex helps expose such brittle assumptions to > the developers early, so they can fix the assumption before it becomes a > problem far in the future and more difficult to fix. > > >Reporter: Zhengxu Jin >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3685) .NET/C#: ClassCash can not load two types with the compex type properties that has same name but different type
[ https://issues.apache.org/jira/browse/AVRO-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3685: - Labels: pull-request-available (was: ) > .NET/C#: ClassCash can not load two types with the compex type properties > that has same name but different type > --- > > Key: AVRO-3685 > URL: https://issues.apache.org/jira/browse/AVRO-3685 > Project: Apache Avro > Issue Type: Bug >Reporter: Khrystyna Popadyuk >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When load two types that have complex type properties with: > * same name > * different type > Only first one is loaded. As a result only one type can be serialized or > deserialized. > > It happens because of this check > [https://github.com/KhrystynaPopadyuk/avro/blob/dee749bde457ba8760e0e5e1e2c1d1844610f2ee/lang/csharp/src/apache/main/Reflect/ClassCache.cs#L223] > That code check load or not type base on property name. > Unit test that illustrate bug: -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3692) Serde flatten is not supported when deserializing
[
https://issues.apache.org/jira/browse/AVRO-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3692:
-
Labels: pull-request-available (was: )
> Serde flatten is not supported when deserializing
> -
>
> Key: AVRO-3692
> URL: https://issues.apache.org/jira/browse/AVRO-3692
> Project: Apache Avro
> Issue Type: Improvement
> Components: rust
>Reporter: Ten
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following:
> {code:java}
> #[test]
> fn test_from_value_struct_flatten() {
> #[derive(Deserialize, PartialEq, Debug)]
> struct S1 {
> f1: String,
> #[serde(flatten)]
> inner: S2,
> }
> #[derive(Deserialize, PartialEq, Debug)]
> struct S2 {
> f2: String,
> }
> let expected = S1 {
> f1: "Hello".to_owned(),
> inner: S2 {
> f2: "World".to_owned(),
> },
> }; let test = Value::Record(vec![
> ("f1".to_owned(), "Hello".into()),
> ("f2".to_owned(), "World".into()),
> ]);
> let final_value: S1 = from_value(&test).unwrap();
> assert_eq!(final_value, expected);
> } {code}
> fails to deserialize.
> This is due to serde hinting for map deserialization when structs contain
> flatten. It is super easy to fix, I'm opening a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3693) avrogencpp Invalid type for union exception does not identify which union
[
https://issues.apache.org/jira/browse/AVRO-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3693:
-
Labels: pull-request-available (was: )
> avrogencpp Invalid type for union exception does not identify which union
> -
>
> Key: AVRO-3693
> URL: https://issues.apache.org/jira/browse/AVRO-3693
> Project: Apache Avro
> Issue Type: Improvement
> Components: c++
>Affects Versions: 1.11.0
>Reporter: Andrew Peter Marlow
>Priority: Minor
> Labels: pull-request-available
> Original Estimate: 24h
> Time Spent: 10m
> Remaining Estimate: 23h 50m
>
> This is a small enhancement request for the avrogencpp code generator, to
> help identify a deserialisation issue better. When a structure sent by avro
> contains unions and a union part of a message is de-serialized without
> checking first that it is present, invalid type for union is thrown. However
> it does not identify which union. It could include the structName in the
> exception, then it would be possible to tie down the exception to the precise
> union that got the error.
> If the file below, Empdetails.avsc is given to avrogencpp, it will show a
> get_int function for a union where the exception text does not identify the
> union.
> {code:java}
> {
> "type" : "record",
> "namespace" : "tutorialspoint",
> "name" : "Empdetails",
> "fields" :
> [
> { "name" : "experience", "type": ["int", "null"] }, { "name" : "age",
> "type": "int" }
> ]
> } {code}
> In this particular case there is only one union so there is no problem but
> when there are multiple unions they all get the same exception text so it is
> hard to tell from where the throw came.
> I have the change waiting in the wings for a PR. The change includes the
> structName in the exception. With this change the avrogencpp generate code
> looks as follows:
> {code:java}
> inline
> int32_t Empdetails_avsc_Union__0__::get_int() const {
> if (idx_ != 0) {
> throw avro::Exception("Invalid type for union
> Empdetails_avsc_Union__0__");
> }
> return std::any_cast(value_);
> } {code}
> With this done for every union in a structure it will be clear which throw
> has executed when the exception is reported.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3695) [Rust] Can't validate schema with value when Nullable union has multiple types.
[
https://issues.apache.org/jira/browse/AVRO-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3695:
-
Labels: pull-request-available rust (was: rust)
> [Rust] Can't validate schema with value when Nullable union has multiple
> types.
> ---
>
> Key: AVRO-3695
> URL: https://issues.apache.org/jira/browse/AVRO-3695
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Mark Farnan
>Priority: Blocker
> Labels: pull-request-available, rust
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If a union has more than one 'type' (other than null), to_avro_datum is
> unable to validate the schema with the value type, and throws a validation
> error.
>
> Schema:
>
> {code:java}
> { "type": "record", "namespace": "datatypes", "name": "nullunion", "fields":
> [ {"name": "item", "type": [ "null", "long", "double" ] } ] } {code}
>
> Struct: (This was generated by rsgen-avro)
> {code:java}
> /// Auto-generated type for unnamed Avro union variants.
> #[derive(Debug, PartialEq, Clone, serde::Deserialize, serde::Serialize)]
> pub enum UnionLongDouble {Long(i64),Double(f64),}
> #[derive(Debug, PartialEq, Clone, serde::Deserialize, serde::Serialize)]
> pub struct NullUnion {pub item: Option,}{code}
>
> {code:java}
> #[test]
> fn test_multivalue_union_tovalue() {
> let nuschema = r#"
> {
> "type": "record",
> "namespace": "datatypes",
> "name": "nullunion",
> "fields":
> [
> {"name": "item",
> "type": [
> "null",
> "long",
> "double"
> ]
> }
> ]
> }
> "#;
> let nullunion = NullUnion {
> item: Some(UnionLongDouble::Long(34)),
> }; let schema = Schema::parse_str(nuschema).unwrap();
> let nu_value = to_value(nullunion).unwrap();
> let nu_encoded = to_avro_datum(&schema, nu_value).unwrap();
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3696) [Python] Replace tox-wheel with upstream tox 4
[ https://issues.apache.org/jira/browse/AVRO-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3696: - Labels: pull-request-available (was: ) > [Python] Replace tox-wheel with upstream tox 4 > -- > > Key: AVRO-3696 > URL: https://issues.apache.org/jira/browse/AVRO-3696 > Project: Apache Avro > Issue Type: Test > Components: python >Reporter: Tommi Vainikainen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Usage of third party tox-wheel package started failing since tox released > version 4 with native wheel. Any usage of tox-wheel should be replaced with > standard tox package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3697) Test against Ruby 3.2
[ https://issues.apache.org/jira/browse/AVRO-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3697: - Labels: pull-request-available (was: ) > Test against Ruby 3.2 > - > > Key: AVRO-3697 > URL: https://issues.apache.org/jira/browse/AVRO-3697 > Project: Apache Avro > Issue Type: Test > Components: ruby >Affects Versions: 1.12.0, 1.11.2 >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Ruby 3.2 was released on 12/25/2022 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3278) Drop support for Ruby 2.6
[ https://issues.apache.org/jira/browse/AVRO-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3278: - Labels: pull-request-available (was: ) > Drop support for Ruby 2.6 > - > > Key: AVRO-3278 > URL: https://issues.apache.org/jira/browse/AVRO-3278 > Project: Apache Avro > Issue Type: Test > Components: ruby >Affects Versions: 1.12.0 >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Ruby 2.6 will reach EOL on March 31, 2022. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3700) Publish SBOM artifacts
[ https://issues.apache.org/jira/browse/AVRO-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3700: - Labels: pull-request-available (was: ) > Publish SBOM artifacts > -- > > Key: AVRO-3700 > URL: https://issues.apache.org/jira/browse/AVRO-3700 > Project: Apache Avro > Issue Type: Improvement > Components: build >Affects Versions: 1.12.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3701) Add github action to validate maven 4 build compatibility
[ https://issues.apache.org/jira/browse/AVRO-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3701: - Labels: pull-request-available (was: ) > Add github action to validate maven 4 build compatibility > - > > Key: AVRO-3701 > URL: https://issues.apache.org/jira/browse/AVRO-3701 > Project: Apache Avro > Issue Type: Test > Components: build >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We can validate in advance compatility with Maven 4 for a future migration. > We can also validate with the new [maven build cache > extension|https://maven.apache.org/extensions/maven-build-cache-extension/] > that brings big improvements into the build via smart caching of build > artifacts and results. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3698) [Java] SpecificData.getClassName must replace reserved words
[
https://issues.apache.org/jira/browse/AVRO-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3698:
-
Labels: newbie patch pull-request-available (was: newbie patch)
> [Java] SpecificData.getClassName must replace reserved words
>
>
> Key: AVRO-3698
> URL: https://issues.apache.org/jira/browse/AVRO-3698
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.11.1
>Reporter: Seung-Hyun Cheong
>Priority: Major
> Labels: newbie, patch, pull-request-available
> Fix For: 1.11.2
>
> Attachments: AVRO-3698.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [Java] SpecificData.getClassName must replace reserved words
>
> When makes Java Class, SpecificCompiler adds "$" after reserved words like
> public, new, etc...
> [https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L1129]
> [https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L643]
>
> For example, if the namespace of a schema is {*}db.public.table{*}, then the
> package name of the generated Java Class, made by SpecificCompiler, is
> {*}db.public$.table{*}. (Because *public* is a reserved word in java)
>
> But when deserializes an Avro Record to a SpecificRecord,
> SpecificData.getClassName returns {*}db.pulic.table.Name{*}.
>
> So, it can't find a class by name from SpecificData.getClassName.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-2918) Schema polymorphism
[ https://issues.apache.org/jira/browse/AVRO-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-2918: - Labels: features pull-request-available (was: features) > Schema polymorphism > --- > > Key: AVRO-2918 > URL: https://issues.apache.org/jira/browse/AVRO-2918 > Project: Apache Avro > Issue Type: New Feature > Components: logical types, misc, spec >Reporter: Jonathan Rapoport >Priority: Critical > Labels: features, pull-request-available > Original Estimate: 96h > Time Spent: 10m > Remaining Estimate: 95h 50m > > Include the option to use named types as base types for a new schema. Allow > for MRO generation. Field inheritance. > The benefits of this approach include: > * Defining a schema as validation for a certain wire, and so allowing the > receiver to be certain of the structure of the data (this works today). > However, defining an extension of this schema, or certain schemas which can > be normalized to the original schema, but contain additional information, > will not allow it to be sent over the same wire. > * Backwards compatibility through inheritance - you never break the old > schema, thus allowing a long integration period, with no need to recode all > processes familiar with the schema. The new schema will simply inherit the > old one, and only add information. > * Allow for full data control through polymorphism, and the ability to > replace structures within any supported language. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3704) Naming rules : multiple choice
[ https://issues.apache.org/jira/browse/AVRO-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3704: - Labels: pull-request-available (was: ) > Naming rules : multiple choice > -- > > Key: AVRO-3704 > URL: https://issues.apache.org/jira/browse/AVRO-3704 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Christophe Le Saec >Assignee: Christophe Le Saec >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Long discussion about [naming > rules|https://lists.apache.org/thread/39v98os6wdpyr6w31xdkz0yzol51fsrr] and > the fact that currently, name checking is not the same depends of module > (rust follow current doc, java is more flexible and accept accent), and also > you can now deactivate control, i propose a change to allow to choose "name > controler" between some proposed one (no control, current controller, strict > controller that follow doc) to client custom one. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3706) AVDL nested imports cannot be resolved if path contains spaces
[
https://issues.apache.org/jira/browse/AVRO-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3706:
-
Labels: pull-request-available (was: )
> AVDL nested imports cannot be resolved if path contains spaces
> --
>
> Key: AVRO-3706
> URL: https://issues.apache.org/jira/browse/AVRO-3706
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.11.0, 1.11.1
>Reporter: fanguad
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is an issue when running the IDL tool if the IDL files have nested
> import statements, and the path includes a space (or probably any character
> that must be URL encoded). After the first level of import, the files can no
> longer be found. Consider the following three files in a {{avro bug}}
> directory:
> {{root.avdl}}
> {code:java}
> protocol Root {
> import idl "level1.avdl";
> } {code}
> {{level1.avdl}}
> {code:java}
> protocol Level1 {
> import idl "level2.avdl";
> } {code}
> {{level2.avdl}}
> {code:java}
> protocol Level2 {
> } {code}
> executing {{java -jar avro-tools-1.10.2.jar idl "avro bug/root.avdl"}}
> results in the expected
>
> {code:java}
> {
> "protocol" : "Root",
> "namespace" : null,
> "types" : [ ],
> "messages" : { }
> } {code}
> whereas {{java -jar avro-tools-1.11.1.jar idl "avro bug/root.avdl"}} causes
> an exception (the bug is also present in 1.11.0 with a slightly different
> stack trace)
>
> {code:java}
> Exception in thread "main" org.apache.avro.compiler.idl.ParseException: Error
> importing level2.avdl: java.io.FileNotFoundException: level2.avdl, at line 2,
> column 26
> at org.apache.avro.compiler.idl.Idl.error(Idl.java:88)
> at org.apache.avro.compiler.idl.Idl.ImportIdl(Idl.java:537)
> at org.apache.avro.compiler.idl.Idl.ProtocolBody(Idl.java:411)
> at org.apache.avro.compiler.idl.Idl.ProtocolDeclaration(Idl.java:286)
> at org.apache.avro.compiler.idl.Idl.CompilationUnit(Idl.java:167)
> at org.apache.avro.compiler.idl.Idl.ImportIdl(Idl.java:535)
> at org.apache.avro.compiler.idl.Idl.ProtocolBody(Idl.java:411)
> at org.apache.avro.compiler.idl.Idl.ProtocolDeclaration(Idl.java:286)
> at org.apache.avro.compiler.idl.Idl.CompilationUnit(Idl.java:167)
> at org.apache.avro.tool.IdlTool.run(IdlTool.java:61)
> at org.apache.avro.tool.Main.run(Main.java:67)
> at org.apache.avro.tool.Main.main(Main.java:56)
> {code}
>
> The same issue occurs if the files are located in "avro" but the current
> folder has spaces (ie, the absolute path to the avdl files contains spaces).
>
> The bug appears to have been introduced in this commit
> [https://github.com/apache/avro/commit/3fe5e306eaa43bdc06cef291321a8c04058a6be9#diff-376865a18691674de38817b4eea7e64c2c3848094a4730030c67f57f54f1028a]
> in {{idl.jj}} lines 130 and 169. The second time that line 130 is reached,
> it has an absolute path that is already URL-encoded. In 1.10.2, the code
> {{new File(input.getPath()).getParentFile()}} returns a String like
> {{[file:///home/user/avro%20bug/]}} whereas in 1.11.0 and later {{new
> File(input.getPath()).getParentFile().toURI()}} returns a String like
> {{[file:///home/user/avro%2520bug]}} - note that URL encoding has happened
> twice resulting in %2520 and there is no trailing slash. When resolved on
> line 169, this results in {{[file:///home/user/level2.avdl]}} instead of the
> correct {{{}[file:///home/user/avro%20bug/level2.avdl]{}}}.
>
> This bug can be easily worked around by changing the location of the avdl
> files to one without spaces or other characters that must be URL encoded.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3708) [Rust] Fix clippy warnings introduced with Rust 1.67.0
[ https://issues.apache.org/jira/browse/AVRO-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3708: - Labels: pull-request-available (was: ) > [Rust] Fix clippy warnings introduced with Rust 1.67.0 > -- > > Key: AVRO-3708 > URL: https://issues.apache.org/jira/browse/AVRO-3708 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Rust 1.67.0 has been released and it introduces some new clippy (lint) > warnings. > To fix them we need to update "edition" from 2018 to 2021 in Cargo.toml. > In addition we can make use of the new "rust-version" property that makes it > clear that the minimum supported Rust version is 1.60.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3705) avrogencpp needs an option to generate code using std instead of boost
[ https://issues.apache.org/jira/browse/AVRO-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3705: - Labels: pull-request-available (was: ) > avrogencpp needs an option to generate code using std instead of boost > -- > > Key: AVRO-3705 > URL: https://issues.apache.org/jira/browse/AVRO-3705 > Project: Apache Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.11.0 > Environment: RHEL8. The gcc installed from the Red Hat repo is > version 8.5.0 which has an option to use c++17 since that dialect is not the > default. The avro-cpp package in the Red Hat repo must have been built with > that since it emits code that uses boost. >Reporter: Andrew Peter Marlow >Assignee: Andrew Peter Marlow >Priority: Minor > Labels: pull-request-available > Fix For: 1.12.0, 1.11.2 > > Time Spent: 10m > Remaining Estimate: 0h > > avrogencpp uses either std or boost in the generated code. It does this > depending on how it was built, using #if __cplusplus >= 201703L. If the > compiler supports std::any then it used, otherwise boost is used. > The assumption is that if the compiler used to build avrogencpp supports > C\+\+17 by default then emitted code can use C\+\+17 otherwise the emitted > code should use boost. This assumption turns out to have a weakness - if a > project uses a compiler where C+17 is given as an option because it is not > the default then the code avrogencpp generates will use boost instead of std. > This problem could be solved if avrogencpp supported a command line option to > use c\+\+17. The option could be used by those environments where avrogencpp > was built with a version of gcc that does not support c17 by default but the > desire is to use C+17. > This is a serious need because there is a significant difference between > boost::any and std::any. The standard any has a small buffer optimisation but > boost does not. This means that code generated by avrogencpp that uses > boost::any will use heap memory for every case where the generated code uses > the any type. This could be avoided if the c++17 option was available. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3712) C++ Build Failure on Manjaro
[
https://issues.apache.org/jira/browse/AVRO-3712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3712:
-
Labels: pull-request-available (was: )
> C++ Build Failure on Manjaro
> -
>
> Key: AVRO-3712
> URL: https://issues.apache.org/jira/browse/AVRO-3712
> Project: Apache Avro
> Issue Type: Bug
> Environment: Manjaro
>Reporter: John McClean
>Assignee: John McClean
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {quote}
> In member function ‘void avro::ReaderImpl::readValue(double&)
> [with ValidatorType = avro::NullValidator]’,
> inlined from ‘double avro::Parser::readDouble() [with Reader =
> avro::ReaderImpl]’ at
> /home/john/third-party/avro/lang/c++/api/Parser.hh:75:26,
> inlined from ‘void TestSchema::readArray(Parser&) [with Parser =
> avro::Parser >]’ at
> /home/john/third-party/avro/lang/c++/test/unittest.cc:324:33,
> inlined from ‘void TestSchema::readData(Parser&) [with Parser =
> avro::Parser >]’ at
> /home/john/third-party/avro/lang/c++/test/unittest.cc:366:18,
> inlined from ‘void TestSchema::readRawData()’ at
> /home/john/third-party/avro/lang/c++/test/unittest.cc:404:17:
> /home/john/third-party/avro/lang/c++/api/Reader.hh:89:17: error:
> ‘v.avro::ReaderImpl::readValue(double&):: union>::d’ may be used uninitialized [-Werror=maybe-uninitialized]
> 89 | val = v.d;
> | ~~^
> {quote}
> I'll put in a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3709) [Rust] Add aliases to RecordField
[
https://issues.apache.org/jira/browse/AVRO-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3709:
-
Labels: pull-request-available rust (was: rust)
> [Rust] Add aliases to RecordField
> -
>
> Key: AVRO-3709
> URL: https://issues.apache.org/jira/browse/AVRO-3709
> Project: Apache Avro
> Issue Type: Improvement
> Components: rust
>Affects Versions: 1.11.1
>Reporter: Santiago Fraire Willemoes
>Assignee: Martin Tzvetanov Grigorov
>Priority: Minor
> Labels: pull-request-available, rust
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hello everyone, I'm working on an [avdl parser for
> rust|https://github.com/woile/avdl-rs] and I'm having an issue with the lack
> of aliases on the `RecordField`.
> For an Avdl like this:
> {code:java}
> protocol MyProtocol {
> record Hello {
> string @aliases(["nickname"]) name;
> }
> }
> {code}
> The schema is rendered as:
> {code:java}
> {
> "type" : "record",
> "name" : "Hello",
> "fields" : [ {
> "name" : "name",
> "type" : "string",
> "aliases" : [ "nickname" ]
> } ]
> }
> {code}
> But when I generate the schema on rust using the `RecordField`, it doesn't
> have aliases.
> See
> [https://github.com/apache/avro/blob/master/lang/rust/avro/src/schema.rs#L559-L580]
> And even if I use `custom_attributes` with aliases, when generating the
> schema using `schema.canonical_form()`, they don't appear.
> The `RecordField` has `custom_attributes` for unknown fields but aliases are
> known. See documentation here:
> [https://avro.apache.org/docs/1.11.1/idl-language/#other-language-features]
> The interesting thing is that the Specification doesn't mention aliases in a
> RecordField, but they are used widely everywhere.
> I would be willing to submit a PR, not sure if there's another reason for the
> lack of aliases that I'm missing.
> I would also like to ask for advice, is the Schema supposed to be used to
> generate a json schema? or is it solely for deserializing a json schema?
> Should I create a fork of it? I find it strange that it implements serde's
> serialize but it doesn't fully serialize all the attributes.
> Thanks a lot.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3713) Regression AVRO-1760 : Thread scalability problem with the use of SynchronizedMap
[ https://issues.apache.org/jira/browse/AVRO-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3713: - Labels: pull-request-available (was: ) > Regression AVRO-1760 : Thread scalability problem with the use of > SynchronizedMap > - > > Key: AVRO-3713 > URL: https://issues.apache.org/jira/browse/AVRO-3713 > Project: Apache Avro > Issue Type: Bug >Reporter: Niels Basjes >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This is a regression of AVRO-1760 which is causing production performance > problems. > In AVRO-1760 the issue was fixed that there is a scalability problem with > AVRO being used in multithreaded situations. > The solution that was implemented for this ticket uses a different Map > implementation (part of Guava) that does not have these locking issues: > [https://github.com/apache/avro/commit/363360229e7ea5f816d87f9d5747dd4494d3b862#diff-422f44890f5e0ab1f508a7c75d0d18fc7831a1c3ca13e0c97780617570c50d5cL970-L973] > Investigating the performance of our internal application has made us > conclude the original problem was reintroduced with this commit: > [https://github.com/apache/avro/commit/c78a3d5ea4e7e511ba9f6182157956db3e65e1a0#diff-422f44890f5e0ab1f508a7c75d0d18fc7831a1c3ca13e0c97780617570c50d5cL986-L988] > which seems to be related to this jira AVRO-2265 "Remove Guava as a > dependency". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3711) Add documentation about uuid in IDL
[
https://issues.apache.org/jira/browse/AVRO-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3711:
-
Labels: pull-request-available (was: )
> Add documentation about uuid in IDL
> ---
>
> Key: AVRO-3711
> URL: https://issues.apache.org/jira/browse/AVRO-3711
> Project: Apache Avro
> Issue Type: Improvement
> Components: doc
>Reporter: Santiago Fraire Willemoes
>Assignee: Santiago Fraire Willemoes
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It's possible to parse an uuid in IDL using a logical type, but it is not
> documented in the IDL section:
> [https://avro.apache.org/docs/1.11.1/idl-language/#logical-types]
> The following is valid:
> {code:java}
> record SyntaxFields {
>uuid pk = "pepe";
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3715) The build consistently fails.
[ https://issues.apache.org/jira/browse/AVRO-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3715: - Labels: pull-request-available (was: ) > The build consistently fails. > - > > Key: AVRO-3715 > URL: https://issues.apache.org/jira/browse/AVRO-3715 > Project: Apache Avro > Issue Type: Bug >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > At least 3 reasons: > - Oracle Java 18 no longer exists. > - CycloneDX does not generate reproducible BOM > > https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/doc/BUILDSPEC.md#cyclonedx-does-not-generate-reproducible-bom > - During ./build.sh dist this dependency does not exist > org.apache.maven.plugin-tools:maven-plugin-tools-javadoc:3.7.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3684) appending to existing file with outputstream not working
[ https://issues.apache.org/jira/browse/AVRO-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3684: - Labels: pull-request-available (was: ) > appending to existing file with outputstream not working > > > Key: AVRO-3684 > URL: https://issues.apache.org/jira/browse/AVRO-3684 > Project: Apache Avro > Issue Type: Bug >Reporter: Mikko Kortelainen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > appending to existing file with outputstream not working > > i created a test case at [https://github.com/apache/avro/pull/2000/files] for > the reference. If this is an incorrect way then please redirect to > documentation or if no such thing exist, please link to existing working > example or test case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3718) Flaky NettyServer test
[
https://issues.apache.org/jira/browse/AVRO-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3718:
-
Labels: pull-request-available (was: )
> Flaky NettyServer test
> --
>
> Key: AVRO-3718
> URL: https://issues.apache.org/jira/browse/AVRO-3718
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Niels Basjes
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> On a regular basis the TestNettyServer.connectionsCount fails in the Github
> Actions CI.
> When you rerun the same test is usually passes.
> {code:java}
> [INFO] --- maven-surefire-plugin:3.0.0-M8:test (default-test) @
> avro-ipc-netty ---
> 3498[INFO] Tests will run in random order. To reproduce ordering use flag
> -Dsurefire.runOrder.random.seed=371358370820
> 3499[INFO] Using auto detected provider
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
> 3500[INFO]
> 3501[INFO] ---
> 3502[INFO] T E S T S
> 3503[INFO] ---
> 3504[INFO] Running org.apache.avro.ipc.netty.TestNettyServer
> 3505Error: Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 4.626 s <<< FAILURE! - in org.apache.avro.ipc.netty.TestNettyServer
> 3506Error: org.apache.avro.ipc.netty.TestNettyServer.connectionsCount Time
> elapsed: 0.01 s <<< FAILURE!
> 3507org.opentest4j.AssertionFailedError: expected: <2> but was: <3>
> 3508 at
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> 3509 at
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> 3510 at
> org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
> 3511{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-2404) Security sweep of build scripts and urls
[ https://issues.apache.org/jira/browse/AVRO-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-2404: - Labels: pull-request-available (was: ) > Security sweep of build scripts and urls > > > Key: AVRO-2404 > URL: https://issues.apache.org/jira/browse/AVRO-2404 > Project: Apache Avro > Issue Type: Task > Components: build >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Major > Labels: pull-request-available > Fix For: 1.9.2 > > Time Spent: 10m > Remaining Estimate: 0h > > In some places we still use insecure URLs. > We should fix that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3722) Eagerly Initialize Instance Variables in Ruby Implementation
[ https://issues.apache.org/jira/browse/AVRO-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3722: - Labels: pull-request-available (was: ) > Eagerly Initialize Instance Variables in Ruby Implementation > > > Key: AVRO-3722 > URL: https://issues.apache.org/jira/browse/AVRO-3722 > Project: Apache Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.11.1 >Reporter: Aaron Patterson >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Ruby 3.2 introduced an inline cache scheme based on the "shape" of an object. > The object's shape depends on what instance variables were set on the object > and in what order. Lazily initializing an instance variable can cause poor > inline cache performance because the order in which the IV is set may not be > predictable. > The Ruby version of Avro has an IV that is lazily initialized, and I would > like to eagerly initialize it so that we can improve cache performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3723) [Rust] Make schema::ResolvedSchema and schema::Names public
[ https://issues.apache.org/jira/browse/AVRO-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3723: - Labels: pull-request-available (was: ) > [Rust] Make schema::ResolvedSchema and schema::Names public > --- > > Key: AVRO-3723 > URL: https://issues.apache.org/jira/browse/AVRO-3723 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > schema::ResolvedSchema and schema::Names are often needed by third party > traits so they should be made public. > > https://github.com/apache/avro/pull/2131 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-2943) Map comparison between Utf8 and String keys fails
[
https://issues.apache.org/jira/browse/AVRO-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-2943:
-
Labels: pull-request-available (was: )
> Map comparison between Utf8 and String keys fails
> -
>
> Key: AVRO-2943
> URL: https://issues.apache.org/jira/browse/AVRO-2943
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.10.0
> Environment: Mac OS Catalina 10.15.6
>
> openjdk version "1.8.0_265"
> OpenJDK Runtime Environment Corretto-8.265.01.1 (build 1.8.0_265-b01)
> OpenJDK 64-Bit Server VM Corretto-8.265.01.1 (build 25.265-b01, mixed mode)
>Reporter: Frank Grimes
>Priority: Major
> Labels: pull-request-available
> Attachments: AVRO-2943-frankgrimes97.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following test I locally added to org.apache.avro.generic.TestGenericData
> on master demonstrates the problem:
> {code:java}
> @Test
> public void testMapKeyEquals() {
> Schema mapSchema = new Schema.Parser().parse("{\"type\": \"map\",
> \"values\": \"string\"}");
> Field myMapField = new Field("my_map", Schema.createMap(mapSchema), null,
> null);
> Schema schema = Schema.createRecord("my_record", "doc", "mytest", false);
> schema.setFields(Arrays.asList(myMapField));
> GenericRecord r0 = new GenericData.Record(schema);
> GenericRecord r1 = new GenericData.Record(schema);
> HashMap pair1 = new HashMap<>();
> pair1.put("keyOne", "valueOne");
> r0.put("my_map", pair1);
> HashMap pair2 = new HashMap<>();
> pair2.put(new Utf8("keyOne"), "valueOne");
> r1.put("my_map", pair2);
> assertEquals(r0, r1);
> assertEquals(r1, r0);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3721) [Java] Add cache to org.apache.avro.JsonProperties.getObjectProps
[ https://issues.apache.org/jira/browse/AVRO-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3721: - Labels: pull-request-available (was: ) > [Java] Add cache to org.apache.avro.JsonProperties.getObjectProps > - > > Key: AVRO-3721 > URL: https://issues.apache.org/jira/browse/AVRO-3721 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.11.1 >Reporter: Moshe Elisha >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently every time you invoke JsonProperties.getObjectProps on same > instance, the code loops over the props and creates a new LinkedHashMap and a > new UnmodifiableMap. > > In our use case, we store a Protocol (and the JsonProperties) and access the > same instances multiple times. > > Storing a simple cache "private Map objectProps;" inside > JsonProperties will slightly improve performance and memory usage. > objectProps can be lazy initialized in JsonProperties.getObjectProps and > reset when "JsonProperties.addProp(String, JsonNode)" is invoked. > > Thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3641) Add support for Nullable and NotNull annotations for SpecificRecords
[
https://issues.apache.org/jira/browse/AVRO-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3641:
-
Labels: pull-request-available (was: )
> Add support for Nullable and NotNull annotations for SpecificRecords
>
>
> Key: AVRO-3641
> URL: https://issues.apache.org/jira/browse/AVRO-3641
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Reporter: Peter Knuts
>Priority: Major
> Labels: pull-request-available
> Attachments: AVRO-3641.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Update the Java SpecificCompiler to add support for jetbrains {{@Nullable}}
> and {{@NotNull}} annotations to getters, setters, builders and all-arg
> constructor. This will provide compile time safety for nullability when used.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-1603) maven avro plugin to also generate avsc files.
[ https://issues.apache.org/jira/browse/AVRO-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-1603: - Labels: pull-request-available (was: ) > maven avro plugin to also generate avsc files. > -- > > Key: AVRO-1603 > URL: https://issues.apache.org/jira/browse/AVRO-1603 > Project: Apache Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.8.0 >Reporter: Zoltan Farkas >Priority: Minor > Labels: pull-request-available > > It would be nice to be able to generate also all avsc schema files during > compilation. > This schema files than could be packages, versioned, distributed with maven... -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3264) Improve the Avro landing page
[ https://issues.apache.org/jira/browse/AVRO-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3264: - Labels: pull-request-available (was: ) > Improve the Avro landing page > - > > Key: AVRO-3264 > URL: https://issues.apache.org/jira/browse/AVRO-3264 > Project: Apache Avro > Issue Type: Improvement > Components: website >Reporter: Ryan Skraba >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This [mailing list > thread|https://lists.apache.org/thread/ktjh7kyd92hf0jsmpp16c3k121dc8grg] has > some good suggestions for improving the main landing page. > An improved message can also be put into packages and artifact pages like: > [https://pypi.org/project/avro/] > [https://metacpan.org/release/RSKRABA/Avro-1.11.0/source/README] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3725) fix documentation of functions and variables
[ https://issues.apache.org/jira/browse/AVRO-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3725: - Labels: pull-request-available (was: ) > fix documentation of functions and variables > > > Key: AVRO-3725 > URL: https://issues.apache.org/jira/browse/AVRO-3725 > Project: Apache Avro > Issue Type: New Feature > Components: rust >Affects Versions: 1.11.1 >Reporter: Igor Izvekov >Assignee: Igor Izvekov >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.2 > > Time Spent: 10m > Remaining Estimate: 0h > > In "avro-rs" documentation there is no description of the data type "Ref". > These changes translate "//" to "///" where the object's documentation is > present. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3724) C# JsonEncoder can't handle nested array of records
[
https://issues.apache.org/jira/browse/AVRO-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3724:
-
Labels: pull-request-available (was: )
> C# JsonEncoder can't handle nested array of records
> ---
>
> Key: AVRO-3724
> URL: https://issues.apache.org/jira/browse/AVRO-3724
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Affects Versions: 1.11.2
>Reporter: Andrew Benedek
>Priority: Major
> Labels: pull-request-available
> Attachments: PAYMENT.cs, Program.cs, Transaction.cs, sample2.avsc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The exception is:
> Avro.AvroTypeException
> HResult=0x80131500
> Message=Incorrect number of items written. 1 more required.
> Source=Avro
> StackTrace:
> at Avro.IO.ParsingEncoder.Pop() in
> C:\code\avro\lang\csharp\src\apache\main\IO\ParsingEncoder.cs:line 135
> at Avro.IO.JsonEncoder.WriteArrayEnd() in
> C:\code\avro\lang\csharp\src\apache\main\IO\JsonEncoder.cs:line 259
> at Avro.Generic.PreresolvingDatumWriter`1.WriteArray(WriteItem itemWriter,
> Object array, Encoder encoder) in
> C:\code\avro\lang\csharp\src\apache\main\Generic\PreresolvingDatumWriter.cs:line
> 235
> at
> Avro.Generic.PreresolvingDatumWriter`1.<>c_{_}DisplayClass20_0.b{_}_0(Object
> d, Encoder e) in
> C:\code\avro\lang\csharp\src\apache\main\Generic\PreresolvingDatumWriter.cs:line
> 225
> at Avro.Specific.SpecificDatumWriter`1.WriteRecordFields(Object recordObj,
> RecordFieldWriter[] writers, Encoder encoder) in
> C:\code\avro\lang\csharp\src\apache\main\Specific\SpecificDatumWriter.cs:line
> 46
> at
> Avro.Generic.PreresolvingDatumWriter`1.<>c_{_}DisplayClass14_0.b{_}_0(Object
> v, Encoder e) in
> C:\code\avro\lang\csharp\src\apache\main\Generic\PreresolvingDatumWriter.cs:line
> 147
> at Avro.Generic.PreresolvingDatumWriter`1.Write(T datum, Encoder encoder)
> in
> C:\code\avro\lang\csharp\src\apache\main\Generic\PreresolvingDatumWriter.cs:line
> 52
> at Program.$(String[] args) in
> C:\code\ips-sales-replication\producer\AvroConsoleApp\Program.cs:line 210
>
> I have cloned the GitHub repo and built the C# solution (Avro.dll) since
> 1.11.2 with the JsonEncoder has not yet been released as a Nuget package.
> A schema with a nested record works fine.
> A schema with a nested array of record throws the above exception.
> The BinaryEncoder however is able to produce the binary Avro with the same
> schema.
> I have attached the schema.
> My code is a simple C# .NET console application.
> Program.cs is also attached along with the schema data model that was
> generated using CodeGen.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3727) Add RollForward to C# avrogen tool
[ https://issues.apache.org/jira/browse/AVRO-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3727: - Labels: pull-request-available (was: ) > Add RollForward to C# avrogen tool > -- > > Key: AVRO-3727 > URL: https://issues.apache.org/jira/browse/AVRO-3727 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Reporter: Zoltan Csizmadia >Priority: Minor > Labels: pull-request-available > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > When the avrogen C# tool is used with an SDk installed which is not supported > (built for) yet, avrogen fails with anerror that the required NET SDk is > missing. RollForward instructs the tool to look for the next major SDK > version and use that, if the required SDK is not installed, > > SInce NET 8.0 preview is out, as of now, avrogen cannot run in NET 8 without > this option. > > https://learn.microsoft.com/en-us/dotnet/core/project-sdk/msbuild-props#rollforward -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3729) Improve the Avro page
[ https://issues.apache.org/jira/browse/AVRO-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3729: - Labels: pull-request-available (was: ) > Improve the Avro page > - > > Key: AVRO-3729 > URL: https://issues.apache.org/jira/browse/AVRO-3729 > Project: Apache Avro > Issue Type: Improvement > Components: website >Reporter: Nishchal D V >Assignee: Nishchal D V >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The documentation pages are outdated and need to be updated. We also need to > add a dropdown for the project and blog sections on the landing page and > improve the overall user experience of the landing page. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3735) [Rust] Validation fails for
[
https://issues.apache.org/jira/browse/AVRO-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3735:
-
Labels: pull-request-available rust (was: rust)
> [Rust] Validation fails for
>
>
> Key: AVRO-3735
> URL: https://issues.apache.org/jira/browse/AVRO-3735
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Matthew Cargille
>Assignee: Martin Tzvetanov Grigorov
>Priority: Major
> Labels: pull-request-available, rust
> Attachments: avro_example.zip
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Schema validation fails when referencing a previously defined type which has
> inherited a namespace.
>
> Schema:
>
> {code:java}
> {
> "type": "record",
> "name": "Foo",
> "namespace": "name.space",
> "fields":
> [
> {
> "name": "barInit",
> "type":
> {
> "type": "enum",
> "name": "Bar",
> "symbols":
> [
> "bar0",
> "bar1"
> ]
> }
> },
> {
> "name": "barUse",
> "type": "Bar"
> }
> ]
> } {code}
>
>
> Rust Objects (from rsgen)
>
> {code:java}
> #[derive(Debug, PartialEq, Eq, Hash, PartialOrd, Ord, Clone,
> serde::Deserialize, serde::Serialize,)]
> pub enum Bar {
> #[serde(rename = "bar0")]
> Bar0,
> #[serde(rename = "bar1")]
> Bar1,
> }
> #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize, serde::Serialize)]
> pub struct Foo {
> #[serde(rename = "barInit")]
> pub bar_init: Bar,
> #[serde(rename = "barUse")]
> pub bar_use: Bar,
> }
> {code}
>
>
> Test
>
> {code:java}
> #[test]
> fn test_serialize() {
> testing_logger::setup();
> let schema = Schema::parse_str(&get_raw_example_schema()).unwrap();
> let foo = Foo {
> bar_init: Bar::Bar0,
> bar_use: Bar::Bar1,
> };
> let avro_value = to_value(foo).unwrap();
> println!(
> "value is valid for schema: {}",
> avro_value.validate(&schema)
> );
> testing_logger::validate(|logs| {
> for log in logs {
> println!("{}", log.body)
> }
> });
> let mut writer = Writer::new(&schema, Vec::new());
> // schema validation happens here
> writer.append(avro_value).unwrap();
> } {code}
>
>
> Test & Log Output
>
> {code:java}
> value is valid for schema: false
> Invalid value: Record([("barInit", Enum(0, "bar0")), ("barUse", Enum(1,
> "bar1"))]) for schema: Record { name: Name { name: "Foo", namespace:
> Some("name.space") }, aliases: None, doc: None, fields: [RecordField { name:
> "barInit", doc: None, default: None, schema: Enum { name: Name { name: "Bar",
> namespace: None }, aliases: None, doc: None, symbols: ["bar0", "bar1"] },
> order: Ascending, position: 0 }, RecordField { name: "barUse", doc: None,
> default: None, schema: Ref { name: Name { name: "Bar", namespace: None } },
> order: Ascending, position: 1 }], lookup: {"barInit": 0, "barUse": 1} }.
> Reason: Unresolved schema reference: 'Bar'. Parsed names: [Name { name:
> "Bar", namespace: Some("name.space") }, Name { name: "Foo", namespace:
> Some("name.space") }]
> thread 'test::test_serialize' panicked at 'called `Result::unwrap()` on an
> `Err` value: ValidationWithReason("Unresolved schema reference: 'Bar'. Parsed
> names: [Name { name: \"Foo\", namespace: Some(\"name.space\") }, Name { name:
> \"Bar\", namespace: Some(\"name.space\") }]")', src/main.rs:33:35 {code}
>
> full test repo is attached
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3262) Include Avro record class name in toString
[
https://issues.apache.org/jira/browse/AVRO-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3262:
-
Labels: pull-request-available (was: )
> Include Avro record class name in toString
> --
>
> Key: AVRO-3262
> URL: https://issues.apache.org/jira/browse/AVRO-3262
> Project: Apache Avro
> Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Dan Newton
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Avro objects output JSON when they are `toString`'d.
> This makes it difficult to understand what Avro objects are being logged when
> passing the whole object into the logging string.
> It would be convenient if it contained the class name of the Avro object, so
> that when used in strings or logging its easier to read.
> Note, this could be bypassed manually in user code by writing something like
> `"logging ${avroObject::class.java.name} $avroObject"` (I'd write it in Java
> but my head hurts right now).
> From what I looked at, possibly changing `SpecificRecordBase.toString` to the
> following would resolve the issue:
>
> {code:java}
> @Override
> public String toString() {
> return this.class + " " + getSpecificData().toString(this);
> }{code}
>
> Possibly the solution might be more elegant than that or in a slightly
> different place.
> The solution above assumes that the `toString` isn't being used for anything
> within the Avro code. If other code depends on the fact it outputs JSON then
> what I've suggested won't be suitable.
> In that situation, we could add a new method that contains the class name
> which can be called from user code and is still an improvement.
> On the flip side, the JSON only output could be moved and called from other
> Avro code, leaving the class name `toString` as the default. This would be
> preferred as Avro objects contained within other objects will be output with
> their class names when turned into strings, which does not happen with the
> previous option.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3686) [website] Use .asf.yaml for deploying the webside and documentation.
[ https://issues.apache.org/jira/browse/AVRO-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3686: - Labels: pull-request-available (was: ) > [website] Use .asf.yaml for deploying the webside and documentation. > > > Key: AVRO-3686 > URL: https://issues.apache.org/jira/browse/AVRO-3686 > Project: Apache Avro > Issue Type: Improvement > Components: doc, website >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > I'd like to propose changing the way the site/documentation is deployed at > [https://avro.apache.org|https://avro.apache.org/]. > Currently it comes from Subversion: > [http://svn-master.apache.org/repos/asf/avro/site/publish] > I suggest to make use of .asf.yaml > ([https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-WebsitedeploymentserviceforGitrepositories]) > instead. > > With the new approach it will be possible to: > 1) deploy the docs from master to > [https://avro.staged.apache.org|https://avro.staged.apache.org/] > 2) deploy latest release, e.g. 1.11.1, to > [https://avro.apache.org|https://avro.apache.org/] > 3) deploy older versions, e.g. 1.10.0, to [https://avro.apache.org/1.10.0/] > > > We will either redirect to the old versions of the docs from Subversion or > create Git branches/tags and copy the content. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3736) [Build][Ruby] The release build fails "bundle not found"
[
https://issues.apache.org/jira/browse/AVRO-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3736:
-
Labels: pull-request-available (was: )
> [Build][Ruby] The release build fails "bundle not found"
>
>
> Key: AVRO-3736
> URL: https://issues.apache.org/jira/browse/AVRO-3736
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Ryan Skraba
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After some reorganisation in the Dockerfile and build scripts from AVRO-2666
> and [this
> commit|https://github.com/apache/avro/commit/cf7bfe7710febdc84e70fcc8ad21d859dbf1be1f],
> it looks like the ubertool docker can no longer be used to build the Ruby
> Avro SDK:
> {code}
> # In the main or branch-1.11 directory
> ./build.sh docker
> # Inside the container
> ./build.sh clean
>
> + cd lang/ruby
> + ./build.sh clean
> ./build.sh: line 27: bundle: command not found
> {code}
> This is a bit subtle, because if the `lang/ruby/.gem/` directory exists from
> a previous build, it might still work. A clean build will fail.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3741) Note about the version requirement of Rust in BUILD.md
[ https://issues.apache.org/jira/browse/AVRO-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3741: - Labels: pull-request-available (was: ) > Note about the version requirement of Rust in BUILD.md > -- > > Key: AVRO-3741 > URL: https://issues.apache.org/jira/browse/AVRO-3741 > Project: Apache Avro > Issue Type: Improvement > Components: doc >Affects Versions: 1.12.0 >Reporter: Kousuke Saruta >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The requirement version of Rust is not noted in BUILD.md. > It's better to note like other languages. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3742) Bump maven-plugin-plugin from 3.8.1 to 3.8.2
[ https://issues.apache.org/jira/browse/AVRO-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3742: - Labels: pull-request-available (was: ) > Bump maven-plugin-plugin from 3.8.1 to 3.8.2 > > > Key: AVRO-3742 > URL: https://issues.apache.org/jira/browse/AVRO-3742 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.11.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3744) Bump maven-checkstyle-plugin from 3.2.1 to 3.2.2
[ https://issues.apache.org/jira/browse/AVRO-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3744: - Labels: pull-request-available (was: ) > Bump maven-checkstyle-plugin from 3.2.1 to 3.2.2 > > > Key: AVRO-3744 > URL: https://issues.apache.org/jira/browse/AVRO-3744 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.11.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3745) Bump zstd-jni from 1.5.4-2 to 1.5.5-2
[ https://issues.apache.org/jira/browse/AVRO-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3745: - Labels: pull-request-available (was: ) > Bump zstd-jni from 1.5.4-2 to 1.5.5-2 > - > > Key: AVRO-3745 > URL: https://issues.apache.org/jira/browse/AVRO-3745 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.11.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3746) Bump grpc.version from 1.54.0 to 1.54.1
[ https://issues.apache.org/jira/browse/AVRO-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3746: - Labels: pull-request-available (was: ) > Bump grpc.version from 1.54.0 to 1.54.1 > --- > > Key: AVRO-3746 > URL: https://issues.apache.org/jira/browse/AVRO-3746 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.11.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3743) Bump cyclonedx-maven-plugin from 2.7.6 to 2.7.7
[ https://issues.apache.org/jira/browse/AVRO-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3743: - Labels: pull-request-available (was: ) > Bump cyclonedx-maven-plugin from 2.7.6 to 2.7.7 > --- > > Key: AVRO-3743 > URL: https://issues.apache.org/jira/browse/AVRO-3743 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.11.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3747) Set `is_human_readable` to `false` for serde serialization
[ https://issues.apache.org/jira/browse/AVRO-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3747: - Labels: pull-request-available (was: ) > Set `is_human_readable` to `false` for serde serialization > -- > > Key: AVRO-3747 > URL: https://issues.apache.org/jira/browse/AVRO-3747 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: Cetra Free >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Serde's `Serializer` trait has a type hint for dictating when a format will > be "human readable" or not. This allows implementers of the `Serialize` > trait to define whether they want a compact representation of data. > I need this change so that I can ensure timestamps are either rfc 3389 string > or millisecond integers depending on type. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3748) issue with DataFileSeekableInput.SeekableInputStream.skip
[
https://issues.apache.org/jira/browse/AVRO-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3748:
-
Labels: pull-request-available (was: )
> issue with DataFileSeekableInput.SeekableInputStream.skip
> -
>
> Key: AVRO-3748
> URL: https://issues.apache.org/jira/browse/AVRO-3748
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.11.1
>Reporter: Steven Aerts
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We found a longstanding bug in the implementation of
> {{DataFileSeekableInput.SeekableInputStream.skip.}}
> This skip function is not hit that often. It can for example be hit when the
> FastReader is enabled and it tries to skip a significant amount of data.
> The implmentation of this function is however fault and can result in data
> corruption or
> {{{}java.io.EOFException{}}}, as instead of skipping the number of bytes, it
> will seek to a wrong place in the file.
>
> We have a pull request ready to fix and test this issue.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3737) [C] memcheck_test_avro_commons_schema is failing
[
https://issues.apache.org/jira/browse/AVRO-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3737:
-
Labels: pull-request-available (was: )
> [C] memcheck_test_avro_commons_schema is failing
>
>
> Key: AVRO-3737
> URL: https://issues.apache.org/jira/browse/AVRO-3737
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.11.1, 1.12.0
>Reporter: Ryan Skraba
>Assignee: Christophe Le Saec
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Even with the fix in AVRO-3454, one of the memcheck test is failing in the
> release docker:
> *{{cd lang/c && ./build.sh test}}*
> {code}
> ... (snip) ...
> 98% tests passed, 1 tests failed out of 48
> Total Test time (real) = 20.64 sec
> The following tests FAILED:
> 7 - memcheck_test_avro_commons_schema (Failed)
> Errors while running CTest
> make: *** [Makefile:85: test] Error 8
> {code}
> WIth a bit more detail in the logs:
> *{{tail -n 25 ../../build/c/tests/memcheck_test_avro_commons_schema.log}}*
> {code}
> ==1209==by 0x10F97C: create_writer (test_avro_commons_schema.c:70)
> ==1209==by 0x10F97C: read_data (test_avro_commons_schema.c:92)
> ==1209==by 0x10F97C: run_tests (test_avro_commons_schema.c:113)
> ==1209==by 0x10F97C: main (test_avro_commons_schema.c:139)
> ==1209==
> ==1209== 172,133 (131,232 direct, 40,901 indirect) bytes in 2 blocks are
> definitely lost in loss record 80 of 80
> ==1209==at 0x483B723: malloc (in
> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==1209==by 0x483E017: realloc (in
> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==1209==by 0x11032F: avro_file_writer_create_with_codec_fp
> (datafile.c:211)
> ==1209==by 0x11032F: avro_file_writer_create_with_codec_fp
> (datafile.c:196)
> ==1209==by 0x1107A5: avro_file_writer_create (datafile.c:179)
> ==1209==by 0x10F97C: create_writer (test_avro_commons_schema.c:70)
> ==1209==by 0x10F97C: read_data (test_avro_commons_schema.c:92)
> ==1209==by 0x10F97C: run_tests (test_avro_commons_schema.c:113)
> ==1209==by 0x10F97C: main (test_avro_commons_schema.c:139)
> ==1209==
> ==1209== LEAK SUMMARY:
> ==1209==definitely lost: 131,392 bytes in 4 blocks
> ==1209==indirectly lost: 57,315 bytes in 146 blocks
> ==1209== possibly lost: 0 bytes in 0 blocks
> ==1209==still reachable: 1,888 bytes in 4 blocks
> ==1209== suppressed: 0 bytes in 0 blocks
> ==1209==
> ==1209== For lists of detected and suppressed errors, rerun with: -s
> ==1209== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3756) Support writing types back to the user in memory without writing files to disk
[ https://issues.apache.org/jira/browse/AVRO-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3756: - Labels: pull-request-available (was: ) > Support writing types back to the user in memory without writing files to disk > -- > > Key: AVRO-3756 > URL: https://issues.apache.org/jira/browse/AVRO-3756 > Project: Apache Avro > Issue Type: Bug > Components: csharp >Reporter: Alexander Rosenfeld >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > As per the title, Avro C# should support writing types directly back to the > user, instead of [forcing the user to write to > disk|https://github.com/apache/avro/blob/master/lang/csharp/src/apache/main/CodeGen/CodeGen.cs#L1144]. > This is critical for use in scenarios where the code generation is done in a > CI job, where disk permissions might not be flexible. > > I already have code for this and intend to submit a patch along with this > ticket. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3755) [Rust] Deserialization fails for reader schema with namespace
[
https://issues.apache.org/jira/browse/AVRO-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3755:
-
Labels: pull-request-available (was: )
> [Rust] Deserialization fails for reader schema with namespace
> --
>
> Key: AVRO-3755
> URL: https://issues.apache.org/jira/browse/AVRO-3755
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Matthew Cargille
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Deserializing an avro value fails when using a reader schema with a namespace
> in avro
> 0.14.0. Probably related to AVRO-3735. May be working in 0.15.0 but it's an
> issue that's impacting our code and we'd like to double-check that a test
> case will cover it.
>
> Structs (from rs-gen)
> {code:java}
> #[derive(
> Debug, PartialEq, Eq, Hash, PartialOrd, Ord, Clone, serde::Deserialize,
> serde::Serialize,
> )]
> pub enum Bar {
> #[serde(rename = "bar0")]
> Bar0,
> #[serde(rename = "bar1")]
> Bar1,
> #[serde(rename = "bar2")]
> Bar2,
> }
> #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize, serde::Serialize)]
> pub struct Foo {
> #[serde(rename = "barInit")]
> pub bar_init: Bar,
> #[serde(rename = "barUse")]
> pub bar_use: Bar,
> } {code}
>
> Writer schema (serializes successfully)
> {code:java}
> fn get_raw_example_schema() -> String {
> r#"{
> "type": "record",
> "name": "Foo",
> "fields":
> [
> {
> "name": "barInit",
> "type":
> {
> "type": "enum",
> "name": "Bar",
> "symbols":
> [
> "bar0",
> "bar1"
> ]
> }
> },
> {
> "name": "barUse",
> "type": "Bar"
> }
> ]
> }"#
> .to_string()
> } {code}
> Reader Schema
> {code:java}
> fn get_raw_example_schema_v2() -> String {
> r#"{
> "type": "record",
> "name": "Foo",
> "namespace": "name.space",
> "fields":
> [
> {
> "name": "barInit",
> "type":
> {
> "type": "enum",
> "name": "Bar",
> "symbols":
> [
> "bar0",
> "bar1",
> "bar2"
> ]
> }
> },
> {
> "name": "barUse",
> "type": "Bar"
> }
> ]
> }"#
> .to_string()
> } {code}
> Test code:
> {code:java}
> #[test]
> fn test_deserialize() {
> testing_logger::setup();
> let schema = Schema::parse_str(&get_raw_example_schema()).unwrap();
> let foo = Foo {
> bar_init: Bar::Bar0,
> bar_use: Bar::Bar1,
> }; let avro_value = to_value(foo).unwrap();
> println!(
> "value is valid for schema: {}",
> avro_value.validate(&schema)
> ); let datum = to_avro_datum(&schema, avro_value).unwrap();
> let mut x = &datum[..];
> let read_schema =
> Schema::parse_str(&get_raw_example_schema_v2()).unwrap();
> let deser_value = from_avro_datum(&schema, &mut x,
> Some(&read_schema)).unwrap();
> testing_logger::validate(|logs| {
> for log in logs {
> println!("{}", log.body)
> }
> });
> } {code}
>
> error
> {code:java}
> panicked at 'called `Result::unwrap()` on an `Err` value:
> SchemaResolutionError(Name { name: "Bar", namespace: None }) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3757) [rust] Update syn to 2.x
[ https://issues.apache.org/jira/browse/AVRO-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3757: - Labels: pull-request-available (was: ) > [rust] Update syn to 2.x > > > Key: AVRO-3757 > URL: https://issues.apache.org/jira/browse/AVRO-3757 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Darling 0.20.x has been released - > https://github.com/TedDriggs/darling/pull/226. > Now we can update the syn dependency from 1.x to 2.x > (https://github.com/apache/avro/pull/2158) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3758) [Rust] Use AtomicXyz types instead of static mutable ones
[
https://issues.apache.org/jira/browse/AVRO-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3758:
-
Labels: pull-request-available (was: )
> [Rust] Use AtomicXyz types instead of static mutable ones
> -
>
> Key: AVRO-3758
> URL: https://issues.apache.org/jira/browse/AVRO-3758
> Project: Apache Avro
> Issue Type: Improvement
> Components: rust
>Reporter: Martin Tzvetanov Grigorov
>Assignee: Martin Tzvetanov Grigorov
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> apache_avro::util uses "static mut" for MAX_ALLOCATION_BYTES and
> SERDE_HUMAN_READABLE. Because of this it has to use "unsafe {}" blocks to
> update their values.
>
> It would be cleaner to use AtomicBool/AtomicUsize instead and get rid of the
> unsafe blocks.
>
> Suggested at https://github.com/apache/avro/pull/2202#issuecomment-1542887124
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3764) [Rust] Add schemata-based resolve method
[ https://issues.apache.org/jira/browse/AVRO-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3764: - Labels: pull-request-available (was: ) > [Rust] Add schemata-based resolve method > > > Key: AVRO-3764 > URL: https://issues.apache.org/jira/browse/AVRO-3764 > Project: Apache Avro > Issue Type: New Feature > Components: rust >Reporter: Fedor Telnov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There is existing validate method on apache_avro::Value that does validation > based on schemata. It would be good if we add resolve method based on > schemata. > I provided a draft - maybe it is not intended to work this way. Correct me if > it is the case. > Draft pull request: https://github.com/apache/avro/pull/2262 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3766) [Rust] Print fliendlier errors when test cases fail
[ https://issues.apache.org/jira/browse/AVRO-3766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3766: - Labels: pull-request-available (was: ) > [Rust] Print fliendlier errors when test cases fail > --- > > Key: AVRO-3766 > URL: https://issues.apache.org/jira/browse/AVRO-3766 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > [https://bluxte.net/musings/2023/01/08/improving_failure_messages_rust_tests/] > > The idea is to use #[track_caller] attribute on a method that panics > immediately. > This way the "useless" frames from the unwinding are not printed. > An extra benefit is the removal of calls to Result::unwrap() in the tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3770) SpecificDefaultReader and PreresolvingDatumReader don't handle aliases properly
[
https://issues.apache.org/jira/browse/AVRO-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3770:
-
Labels: pull-request-available (was: )
> SpecificDefaultReader and PreresolvingDatumReader don't handle aliases
> properly
> ---
>
> Key: AVRO-3770
> URL: https://issues.apache.org/jira/browse/AVRO-3770
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Affects Versions: 1.11.1
>Reporter: Morgan Parry
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Both of these classes will read from an aliased value correctly but then will
> go on to try and insert a default value reader, because they're also still
> looking for the new name in the writer schema.
> For example, with a field like
> {code:json}
> {
> "name": "i",
> "type": "int",
> "aliases": "j"
> }
> {code}
> and some old serialised data that was still writing the field {{j}} instead,
> the readers will read that {{j}} value into {{i}} (as expected) but then also
> add some fallback handling for the 'missing' {{i}} field.
> This is due to the following loop in {{PreresolvingDatumReader}}, for example:
> {code:c#}
> foreach (Field rf in readerSchema)
> {
> if (writerSchema.Contains(rf.Name)) continue;
> {code}
> That condition isn't sufficient in the case of aliasing.
> I believe it's also the case that {{SpecificDefaultReader}} doesn't handle
> aliases at all, since it uses {{TryGetField()}} rather than
> {{TryGetFieldAlias()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
