[jira] [Updated] (AVRO-3625) [Rust] UnionSchema.is_nullable() should return true if any of the variants is Schema::Null
[ https://issues.apache.org/jira/browse/AVRO-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3625: - Labels: pull-request-available (was: ) > [Rust] UnionSchema.is_nullable() should return true if any of the variants is > Schema::Null > -- > > Key: AVRO-3625 > URL: https://issues.apache.org/jira/browse/AVRO-3625 > Project: Apache Avro > Issue Type: Bug > Components: rust >Affects Versions: 1.11.1 >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently UnionSchema::is_nullable() returns `true` if the first variant is > `"null"`. > The specification > [says|https://avro.apache.org/docs/1.11.1/specification/#unions]: "Thus, for > unions containing “null”, the “null” is usually listed first, since the > default value of such unions is typically null." > That is, the `null` is *usually* at the first position{*},{*} but this is not > mandatory! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-2831) Resolver cannot find reader/writer schema difference of fields in Union
[ https://issues.apache.org/jira/browse/AVRO-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-2831: - Labels: pull-request-available (was: ) > Resolver cannot find reader/writer schema difference of fields in Union > --- > > Key: AVRO-2831 > URL: https://issues.apache.org/jira/browse/AVRO-2831 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.9.0, 1.9.1, 1.9.2 >Reporter: Jinge Dai >Priority: Blocker > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When avro java library deserializes and tries to resolve fields in a Union, > it will use the writer schema if the reader field type is same as writer's, > but it ignores the properties and logical type. > {code:java} > private static boolean unionEquiv(Schema write, Schema read, Map Boolean> seen) { > final Schema.Type wt = write.getType(); > if (wt != read.getType()) { > return false; > } > {code} > However, I have hit two issues: > # https://issues.apache.org/jira/browse/AVRO-2702 If producer is not java, > the string field the strings of the inner record will be deserialized to > org.apache.avro.util.Utf8 and then get error > {code:java} > Caused by: java.lang.ClassCastException: class org.apache.avro.util.Utf8 > cannot be cast to class java.lang.String{code} > > 2. I am using a built-in-house C# lib on the producer side, and it builds > Date field schema as INT type without logicalTyple, this is bug on my side. > However, AVRO 1.8.2 is able to handle it by using reader schema, whereas > 1.9.2 gets error > {code:java} > Caused by: java.lang.IllegalArgumentException: Parameters cannot be null! > Parameter values:[-129, "int", null, > org.apache.avro.data.TimeConversions$DateConversion@d3449b1] > {code} > The reason is, although the consumer side generates the class properly and > sets the field to be converted to logical type, but the convertToLogicalType > failed because the writer schema is used and the logical type is null > {code:java} > public static Object convertToLogicalType(Object datum, Schema schema, > LogicalType type, Conversion conversion) { > if (datum == null) { > return null; > } > if (schema == null || type == null || conversion == null) { > throw new IllegalArgumentException("Parameters cannot be null! Parameter > values:" > + Arrays.deepToString(new Object[] { datum, schema, type, conversion > })); > } > {code} > > I am wondering if we should check type and properties in "unionEquiv" so that > we can use the reader schema if there is difference. This complies the AVRO > spec > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3628) [Java] JUnit 4.x tests are not executed
[ https://issues.apache.org/jira/browse/AVRO-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3628: - Labels: pull-request-available (was: ) > [Java] JUnit 4.x tests are not executed > --- > > Key: AVRO-3628 > URL: https://issues.apache.org/jira/browse/AVRO-3628 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.11.1 >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > AVRO-3579 introduced JUnit 5 and migrated some of the tests. > This confused maven-surefire-plugin and now it does not run the JUnit 4.x > tests, e.g.: > {code:java} > [INFO] --- maven-surefire-plugin:3.0.0-M7:test (default-test) @ > avro-maven-plugin --- > [INFO] Tests will run in random order. To reproduce ordering use flag > -Dsurefire.runOrder.random.seed=71852499748477 > [INFO] Using auto detected provider > org.apache.maven.surefire.junitplatform.JUnitPlatformProvider > [INFO] > [INFO] --- > [INFO] T E S T S > [INFO] --- > [INFO] > [INFO] Results: > [INFO] > [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 > [INFO] > [INFO] > [INFO] --- maven-checkstyle-plugin:3.1.2:check (checkstyle-check) @ > avro-maven-plugin --- > [INFO] Starting audit... > {code} > > We should use JUnit 5.x Vintage engine instead of JUnit 4.x -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-1938) Python support for generating canonical forms of schema
[ https://issues.apache.org/jira/browse/AVRO-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-1938: - Labels: pull-request-available (was: ) > Python support for generating canonical forms of schema > --- > > Key: AVRO-1938 > URL: https://issues.apache.org/jira/browse/AVRO-1938 > Project: Apache Avro > Issue Type: Improvement > Components: python >Reporter: Erik Forsberg >Assignee: Michael A. Smith >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The python implementation(s) lack support for generating canonical forms of > schema. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3532) Align naming rules on code
[ https://issues.apache.org/jira/browse/AVRO-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3532: - Labels: pull-request-available (was: ) > Align naming rules on code > -- > > Key: AVRO-3532 > URL: https://issues.apache.org/jira/browse/AVRO-3532 > Project: Apache Avro > Issue Type: Wish >Reporter: Christophe Le Saec >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Description of [naming rule on > documentation|https://avro.apache.org/docs/current/spec.html#names] is > {noformat} > - start with [A-Za-z_] > - subsequently contain only [A-Za-z0-9_] > {noformat} > But [java > code|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/Schema.java#L1578] > use Character.isLetter method > {code:java} > char first = name.charAt(0); > if (!(Character.isLetter(first) || first == '_')) > throw new SchemaParseException("Illegal initial character: " + name); > for (int i = 1; i < length; i++) { > char c = name.charAt(i); > if (!(Character.isLetterOrDigit(c) || c == '_')) > throw new SchemaParseException("Illegal character in: " + name); > } > return name; > {code} > This method accept accent éùàçË ... and also chinese character (我) ... > So, the aim of this ticket is to see if we can update the documentation, if > other implementations (rust, C# ...) are also compatible with ? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3611) org.apache.avro.util.RandomData generates invalid test data
[ https://issues.apache.org/jira/browse/AVRO-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3611: - Labels: features pull-request-available (was: features) > org.apache.avro.util.RandomData generates invalid test data > --- > > Key: AVRO-3611 > URL: https://issues.apache.org/jira/browse/AVRO-3611 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.11.1 >Reporter: Simon Klakegg >Priority: Minor > Labels: features, pull-request-available > Fix For: 1.11.2 > > Attachments: image-2022-08-18-19-05-37-323.png > > Original Estimate: 48h > Time Spent: 10m > Remaining Estimate: 47h 50m > > When RandomData.java generates data it does not check for Logical Types, > which are described here: [Specification | Apache > Avro|https://avro.apache.org/docs/1.11.1/specification/_print/] > *For instance the following generate method would return this for INT fields:* > {code:java} > case INT: return random.nextInt(); {code} > > {*}However, an int field could be of logical type > date:{*}!image-2022-08-18-19-05-37-323.png|width=1052,height=266! > > Which in many cases could create an int that is out of range for logicalType > Date, and thus break when creating records in for instance kafka. > My suggestion is to generated data that is valid for logicalTypes, here is an > example I made for int and long: > {code:java} > case INT: > switch (logicalTypeName) { > case "date": > // Random number of days between Unix Epoch start day (0) and end day > (24855) > int maxDaysInEpoch = (int) > Duration.ofSeconds(Integer.MAX_VALUE).toDays(); > return ThreadLocalRandom.current().nextInt(0, maxDaysInEpoch); > case "time-millis": > // Random number of milliseconds between midnight 00:00:00.000 (0) > and 23:59:59:999 (8639) > int maxMillisecondsInDay = (int) Duration.ofDays(1).toMillis() - 1; > return random.nextInt(0, maxMillisecondsInDay); > default: return random.nextInt(); > } > case LONG: > switch (logicalTypeName) { > case "time-micros": > // Random number of microseconds between midnight 00:00:00.00 (0) > and 23:59:59:99 (863) > long maxMicrosecondsInDay = (Duration.ofDays(1).toNanos() - 1) / 1000; > return random.nextLong(0, maxMicrosecondsInDay); > case "timestamp-millis": > // Random milliseconds between Unix Epoch (0) start and end > (2147483647000) > long maxMillisecondsInEpoch = > TimeUnit.SECONDS.toMillis(Integer.MAX_VALUE); > return ThreadLocalRandom.current().nextLong(0, maxMillisecondsInEpoch); > case "timestamp-micros": > // Random microseconds between Unix Epoch (0) start and end > (214748364700) > long maxMicrosecondsInEpoch = > TimeUnit.SECONDS.toMicros(Integer.MAX_VALUE); > return ThreadLocalRandom.current().nextLong(0, maxMicrosecondsInEpoch); > case "local-timestamp-millis": > // Random number of milliseconds between Unix Epoch start (0) and 100 > years from now (now() + 100) > ZonedDateTime hundredYearsFromNow = ZonedDateTime.now().plusYears(100); > long hundredYearsEpochMillis = ChronoUnit.MILLIS.between(Instant.EPOCH, > hundredYearsFromNow); > return random.nextLong(0, hundredYearsEpochMillis); > case "local-timestamp-micros": > // Random number of microseconds between Unix Epoch start (0) and 100 > years from now (now() + 100) > long hundredYearsEpochMicros = ChronoUnit.MICROS.between(Instant.EPOCH, > hundredYearsFromNow); > return random.nextLong(0, hundredYearsEpochMicros); > default: return random.nextLong(); > } {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3624) Fix Avro website checks on whimsy
[ https://issues.apache.org/jira/browse/AVRO-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3624: - Labels: pull-request-available website (was: website) > Fix Avro website checks on whimsy > - > > Key: AVRO-3624 > URL: https://issues.apache.org/jira/browse/AVRO-3624 > Project: Apache Avro > Issue Type: Improvement > Components: community >Reporter: Ryan Skraba >Priority: Major > Labels: pull-request-available, website > Time Spent: 10m > Remaining Estimate: 0h > > With the new website, some of the checks on the website are no longer green: > [https://whimsy.apache.org/site/] > We should ensure all of the required links are present. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3630) [Rust] Make it possible to extend pre-existing Avro bytes
[ https://issues.apache.org/jira/browse/AVRO-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3630: - Labels: pull-request-available (was: ) > [Rust] Make it possible to extend pre-existing Avro bytes > - > > Key: AVRO-3630 > URL: https://issues.apache.org/jira/browse/AVRO-3630 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Affects Versions: 1.11.1 >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > From ASF Slack > #[avro|[https://the-asf.slack.com/archives/CLUD54M1S/p1663751850569319]:] > === > I'm currenlty solving it by reading the original file and appending it to a > new Vec > {code:java} > let bytes: Vec = object_store > .get(&old_manifest_location) > .await? > .bytes() > .await? > .into(); > let mut writer = apache_avro::Writer::new(&schema, Vec::new()); > let reader = apache_avro::Reader::new(&*bytes)?; > writer.extend(reader.filter_map(Result::ok))?; > writer.append_ser(entry)?; > let new_bytes = writer.into_inner()?; > {code} > I was wondering if I could do something like: > {code:java} > let bytes: Vec = object_store > .get(&old_manifest_location) > .await? > .bytes() > .await? > .into(); > let mut writer = apache_avro::Writer::new(&schema, bytes); > writer.append_ser(entry)?; > let new_bytes = writer.into_inner()?; > {code} > == -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3631) Fix serialization of structs containing Fixed fields
[ https://issues.apache.org/jira/browse/AVRO-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3631: - Labels: pull-request-available (was: ) > Fix serialization of structs containing Fixed fields > > > Key: AVRO-3631 > URL: https://issues.apache.org/jira/browse/AVRO-3631 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: Rik Heijdens >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Consider the following minimal Avro Schema: > {noformat} > { > "type": "record", > "name": "TestStructFixedField", > "fields": [ > { > "name": "field", > "type": { > "name": "field", > "type": "fixed", > "size": 6 > } > } > ] > } > {noformat} > In Rust, I might represent this schema with the following struct: > {noformat} > #[derive(Debug, Serialize, Deserialize)] > struct TestStructFixedField { > field: [u8; 6] > } > {noformat} > I would then expect to be able to use `apache_avro::to_avro_datum()` to > convert an instance of `TestStructFixedField` into an `Vec` using an > instance of `Schema` initialized from the schema listed above. > However, this fails because the `Value` produced by `apache_avro::to_value()` > represents `field` as an `Value::Array` rather than a > `Value::Fixed<6, Vec` which does not pass schema validation. > I believe that there are two options to fix this: > 1. Allow Value::Array> to pass validation if the array has > the expected length, and none of the contents of the array are out-of-range > for u8. If we go down this route, the implementation of `to_avro_datum()` > will have to take care of converting Value::Int to u8 when converting into > bytes. > 2. Update `apache_avro::to_value()` such that fixed length arrays are > converted into `Value::Fixed>` rather than `Value::Array`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3632) Union defaults are not handled as per the specification
[ https://issues.apache.org/jira/browse/AVRO-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3632: - Labels: pull-request-available (was: ) > Union defaults are not handled as per the specification > --- > > Key: AVRO-3632 > URL: https://issues.apache.org/jira/browse/AVRO-3632 > Project: Apache Avro > Issue Type: Bug > Components: rust >Affects Versions: 1.11.1 >Reporter: Santiago Fraire Willemoes >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > According to the [union > spec|https://avro.apache.org/docs/1.11.1/specification/#unions]. > ??Note that when a default value is specified for a record field whose type > is a union, the type of the default value must match the first element of the > union. Thus, for unions containing “null”, the “null” is usually listed > first, since the default value of such unions is typically null?? > I wonder if a PR would be accepted for this. I managed to add the check for > the first element in the parser. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3633) Additional attributes for 'avro_derive' crate
[ https://issues.apache.org/jira/browse/AVRO-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3633: - Labels: easyfix features pull-request-available (was: easyfix features) > Additional attributes for 'avro_derive' crate > - > > Key: AVRO-3633 > URL: https://issues.apache.org/jira/browse/AVRO-3633 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Affects Versions: 1.11.1 >Reporter: Jimmy Pramosi >Priority: Minor > Labels: easyfix, features, pull-request-available > Original Estimate: 1h > Time Spent: 10m > Remaining Estimate: 50m > > Hello, > At the moment it is not possible to skip or rename a specific field within a > struct derived with 'AvroSchema'. For e.g. > {code:java} > use std::time::Instant; > use apache_avro::AvroSchema; > use serde::Deserialize; > use serde::Serialize; > #[derive(Clone, Debug, Deserialize, Serialize, AvroSchema)] > pub struct FileDescriptor { > #[avro(rename = "pflags")] // <--- NOT POSSIBLE > #[serde(rename = "pflags")] > pub flags: u64, > #[avro(skip)] // <--- NOT POSSIBLE > #[serde(skip)] > access_time: Instant, > }{code} > I would like to suggest to implement additional attributes 'skip' and > 'rename' to the 'avro_derive' crate. These attributes are also common in > other procedural macro crates like 'serde'. > Exemplary proposal: > [https://www.diffchecker.com/iju3tSuX] > > Best Regards, > Jimmy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3634) Implement schema for missing types
[ https://issues.apache.org/jira/browse/AVRO-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3634: - Labels: easyfix pull-request-available (was: easyfix) > Implement schema for missing types > -- > > Key: AVRO-3634 > URL: https://issues.apache.org/jira/browse/AVRO-3634 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Affects Versions: 1.11.1 >Reporter: Jimmy Pramosi >Priority: Trivial > Labels: easyfix, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Hello, > I would like to suggest to implement a schema for the type 'bool' and maybe > 'u64'. > The change is pretty trivial and involves adding > {code:java} > impl_schema!(bool, Schema::Boolean); > impl_schema!(u64, Schema::Long);{code} > to > [https://github.com/apache/avro/blob/63c58f3b29d20ba94a6c68d9e4c09816d448bf34/lang/rust/avro/src/schema.rs#L1830] > > Best Regards, > Jimmy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3638) Support mapping between avro namespace and java packagename
[ https://issues.apache.org/jira/browse/AVRO-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3638: - Labels: pull-request-available (was: ) > Support mapping between avro namespace and java packagename > --- > > Key: AVRO-3638 > URL: https://issues.apache.org/jira/browse/AVRO-3638 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Simon >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When Java code is generated from an avro schema, the created classes are > given a "package" which is the namespace of the schema. However some users > prefer to have the java classes placed under some specific base package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3642) GenericSingleObjectReader::read_value fails on non-exhaustive read
[ https://issues.apache.org/jira/browse/AVRO-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3642: - Labels: pull-request-available (was: ) > GenericSingleObjectReader::read_value fails on non-exhaustive read > -- > > Key: AVRO-3642 > URL: https://issues.apache.org/jira/browse/AVRO-3642 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: David Peklak >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There is a bug in the Rust implementation of > [GenericSingleObjectReader::read_value|https://docs.rs/apache-avro/0.14.0/apache_avro/struct.GenericSingleObjectReader.html#method.read_value], > which is exposed when a call to reader.read reads less than 10 bytes. This > is a valid scenario, as according to the documentation of > [Read::read|https://doc.rust-lang.org/std/io/trait.Read.html#tymethod.read], > "It is not an error if the returned value {{n}} is smaller than the buffer > size, even when the reader is not at the end of the stream yet." This is also > a real-life scenario: in my use-case, I am chaining three implementations of > the Read trait to messages that can be deserialized (one for the two magic > bytes, one for the schema fingerprint, and one for the actual message > content). Each call to Read::read only returns the bytes of one chain link, > so the first call to Read::read only returns 2 bytes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3644) [JAVA] Support java.util.Optional in reflect package
[ https://issues.apache.org/jira/browse/AVRO-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3644: - Labels: pull-request-available (was: ) > [JAVA] Support java.util.Optional in reflect package > > > Key: AVRO-3644 > URL: https://issues.apache.org/jira/browse/AVRO-3644 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Daniel Heinrich >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > java.util.Optional can currently not be used at all when using reflection > based schemas. > _org.apache.avro.reflect.ReflectData#getSchema_ will currently produce the > following type for fields using Optional: > {"type":"record","name":"Optional","namespace":"java.util","fields":[]} > The generated schema is completly unusable (it does not contain any data). I > would like to propose to map Optional fields as if they were not using > Optional and would be annotated with @Nullable. So resulting in the following > type: > ["null","something"] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3645) Fix deserialization of enum with unit () type
[ https://issues.apache.org/jira/browse/AVRO-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3645: - Labels: pull-request-available rust (was: rust) > Fix deserialization of enum with unit () type > - > > Key: AVRO-3645 > URL: https://issues.apache.org/jira/browse/AVRO-3645 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: Lucas Javaudin >Priority: Major > Labels: pull-request-available, rust > Original Estimate: 2h > Time Spent: 10m > Remaining Estimate: 1h 50m > > Consider the following test: > {code:java} > #[derive(Debug, Deserialize, Serialize, PartialEq, Eq)] > struct TestNullExternalEnum { > a: NullExternalEnum, > } > #[derive(Debug, Deserialize, Serialize, PartialEq, Eq)] > enum NullExternalEnum { > Val1(()), > Val2(u64), > } > #[test] > fn test_from_value_null_enum() { > let expected = TestNullExternalEnum { > a: NullExternalEnum::Val1(()), > }; > let test = Value::Record(vec![( > "a".to_owned(), > Value::Record(vec![ > ("type".to_owned(), Value::String("Val1".to_owned())), > ("value".to_owned(), Value::Union(0, Box::new(Value::Null))), > ]), > )]); > let final_value: TestNullExternalEnum = from_value(&test).unwrap(); > assert_eq!( > final_value, expected, > "Error deserializing null external enum" > ); > }{code} > On version `0.15.0`, it fails with the following error message: > {code:java} > called `Result::unwrap()` on an `Err` value: DeserializeValue("not a null") > {code} > > If my understanding is correct, it fails because rust is expecting a > `Value::Null` but it got a `Value::Union(0, Box::new(Value::Null))`. > A fix that I found is to replace the `deserialize_unit`function in `de.rs` > from > {code:java} > fn deserialize_unit(self, visitor: V) -> Result > where > V: Visitor<'de>, > { > match *self.input { > Value::Null => visitor.visit_unit(), > _ => Err(de::Error::custom("not a null")), > } > } {code} > to > {code:java} > fn deserialize_unit(self, visitor: V) -> Result > where > V: Visitor<'de>, > { > match *self.input { > Value::Null => visitor.visit_unit(), > Value::Union(_i, ref x) => match **x { > Value::Null => visitor.visit_unit(), > _ => Err(de::Error::custom("not a null")), > }, > _ => Err(de::Error::custom("not a null")), > } > } {code} > i.e., we allow it to accept `Value::Union`, in the same way that the > `deserialize_string` function accepts such values. > > If you ask yourself what is the point of having an enum with unit type, it is > because apache-avro is not able to handle the following enum (as far as I > know): > {code:java} > enum NullExternalEnum { > Val1, > Val2(u64), > } {code} > I am sorry if there is any problem with this issue, it's the first time I > open an issue here. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3649) [JAVA] reorder union types to match default value
[ https://issues.apache.org/jira/browse/AVRO-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3649: - Labels: pull-request-available (was: ) > [JAVA] reorder union types to match default value > - > > Key: AVRO-3649 > URL: https://issues.apache.org/jira/browse/AVRO-3649 > Project: Apache Avro > Issue Type: Improvement >Reporter: Daniel Heinrich >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The Avro specification requires that a default value matches the first schema > of a union type. > > Using @Nullable on a field will return a union type of the form ["null", > ""]. > @Nullable also sets the default value to "null". > > It is not possible to override this default value with > {color:#bbb529}{color:#172b4d}@AvroDefault because of the order of the > schemas in the generated union.{color} > {color} > > It would be nice that union types are automatically reordered to match the > type of the provided default value. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3650) Fix C++ Build on Manjaro
[ https://issues.apache.org/jira/browse/AVRO-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3650: - Labels: pull-request-available (was: ) > Fix C++ Build on Manjaro > > > Key: AVRO-3650 > URL: https://issues.apache.org/jira/browse/AVRO-3650 > Project: Apache Avro > Issue Type: Bug >Reporter: John McClean >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Manjaro has a recent version of boost, which results in this. > {quote}[ 19%] Building CXX object > CMakeFiles/avrocpp.dir/impl/parsing/JsonCodec.cc.o > In file included from /usr/include/boost/math/tools/real_cast.hpp:9, > from > /usr/include/boost/math/special_functions/fpclassify.hpp:17, > from > /home/john/third-party/avro/lang/c++/impl/parsing/JsonCodec.cc:20: > /usr/include/boost/math/tools/config.hpp:23:6: error: #warning "The minimum > language standard to use Boost.Math will be C++14 starting in July 2023 > (Boost 1.82 release)" [-Werror=cpp] > 23 | # warning "The minimum language standard to use Boost.Math will be > C++14 starting in July 2023 (Boost 1.82 release)" > | ^~~ > cc1plus: all warnings being treated as errors > make[2]: *** [CMakeFiles/avrocpp.dir/build.make:328: > CMakeFiles/avrocpp.dir/impl/parsing/JsonCodec.cc.o] Error 1 > make[1]: *** [CMakeFiles/Makefile2:129: CMakeFiles/avrocpp.dir/all] Error 2 > make: *** [Makefile:166: all] Error 2 > {quote} > In addition, once C++14 is used, the list of cpp keywords needs to be updated > to include 'constinit'. > {quote}[ 98%] Building CXX object > CMakeFiles/AvrogencppTestReservedWords.dir/test/AvrogencppTestReservedWords.cc.o > In file included from > /home/john/third-party/avro/lang/c++/test/AvrogencppTestReservedWords.cc:18: > /home/john/third-party/avro/lang/c++/build/cpp_reserved_words.hh:57:17: > error: identifier ‘constinit’ is a keyword in C++20 [-Werror=c++20-compat] > 57 | std::string constinit; > | ^ > cc1plus: all warnings being treated as errors > make[2]: *** [CMakeFiles/AvrogencppTestReservedWords.dir/build.make:76: > CMakeFiles/AvrogencppTestReservedWords.dir/test/AvrogencppTestReservedWords.cc.o] > Error 1 > make[1]: *** [CMakeFiles/Makefile2:951: > CMakeFiles/AvrogencppTestReservedWords.dir/all] Error 2 > make: *** [Makefile:166: all] Error 2 > {quote} > I'll put in a PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3646) Serialization / deserialization for enum with mixed variants (with and without data)
[ https://issues.apache.org/jira/browse/AVRO-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3646: - Labels: pull-request-available rust (was: rust) > Serialization / deserialization for enum with mixed variants (with and > without data) > > > Key: AVRO-3646 > URL: https://issues.apache.org/jira/browse/AVRO-3646 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: Lucas Javaudin >Priority: Major > Labels: pull-request-available, rust > Time Spent: 10m > Remaining Estimate: 0h > > Consider the following example of what I call an "enum with mixed variants": > {code:java} > enum MixedExternalEnum { > Val1, > Val2(f32), > } {code} > For the two variants, this enum can be properly serialized to and > deserialized from a {{types::Value}} (using {{ser::to_value}} and > {{{}de::from_value{}}}). The problem is that the way they are represented as > a \{{types::Value}} cannot be represented using an Avro Schema. > The serialized version of {{MixedExternalEnum::Val1}} is > {code:java} > Value::Record(vec![("a".to_owned(), Value::Enum(0, "Val1".to_owned()))]) > {code} > while the serialized version of {{MixedExternalEnum::Val2(0.0)}} is > {code:java} > Value::Record(vec![( > "a".to_owned(), > Value::Record(vec![ > ("type".to_owned(), Value::Enum(1, "Val2".to_owned())), > ( > "value".to_owned(), > Value::Union(1, Box::new(Value::Float(0.0))), > ), > ]), > )]) > {code} > As far as I know, there is no Avro Schema compatible with both possible > values. > One solution I can think of is to change the serialization of > {{MixedExternalEnum::Val1}} to > {code:java} > Value::Record(vec![( > "a".to_owned(), > Value::Record(vec![ > ("type".to_owned(), Value::Enum(0, "Val1".to_owned())), > ( > "value".to_owned(), > Value::Union(0, Box::new(Value::Null)), > ), > ]), > )]) > {code} > > A workaround is to replace {{Val1}} by {{Val1(())}} in the enum definition > (see AVRO-3645) but this can have undesirable effects for other parts of the > code. > > Then, it's another story for adjacently tagged, internally tagged and > untagged enum... -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3658) Bump jackson to address CVE-2020-36518
[ https://issues.apache.org/jira/browse/AVRO-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3658: - Labels: pull-request-available (was: ) > Bump jackson to address CVE-2020-36518 > -- > > Key: AVRO-3658 > URL: https://issues.apache.org/jira/browse/AVRO-3658 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.11.1 >Reporter: Pavel Moskotin >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Current version of Jackson dependency for AVRO/Java > {code:xml} > 2.12.7.20221012 > {code} > brings CVE-2020-36518. > This is covered by next versions, for example - in > {code:xml} > 2.13.4 > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3661) [Rust] Fix new clippy errors introduced with Rust 1.65
[ https://issues.apache.org/jira/browse/AVRO-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3661: - Labels: pull-request-available (was: ) > [Rust] Fix new clippy errors introduced with Rust 1.65 > -- > > Key: AVRO-3661 > URL: https://issues.apache.org/jira/browse/AVRO-3661 > Project: Apache Avro > Issue Type: Task > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Rust 1.65.0 has been released and it comes with some new clippy checks, and > few of them lead to CI failures. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3659) Typo in python example
[ https://issues.apache.org/jira/browse/AVRO-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3659: - Labels: pull-request-available (was: ) > Typo in python example > -- > > Key: AVRO-3659 > URL: https://issues.apache.org/jira/browse/AVRO-3659 > Project: Apache Avro > Issue Type: Bug > Components: doc >Affects Versions: 1.11.1 >Reporter: Maksim Syomochkin >Priority: Trivial > Labels: pull-request-available > Attachments: typo.patch > > Time Spent: 10m > Remaining Estimate: 0h > > I found a small typo in python example (see attachment). Probably this is too > small a change to make a pull request -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3663) rust crate apache_avro_derive creates invalid schemas for raw identifiers
[ https://issues.apache.org/jira/browse/AVRO-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3663: - Labels: pull-request-available (was: ) > rust crate apache_avro_derive creates invalid schemas for raw identifiers > - > > Key: AVRO-3663 > URL: https://issues.apache.org/jira/browse/AVRO-3663 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: Neidhart Orlich >Priority: Major > Labels: pull-request-available > Original Estimate: 0.5h > Time Spent: 10m > Remaining Estimate: 20m > > When using a raw identifier as field name, the `r#` prefix is used as field > name for the resulting schema which results in an invalid AVRO schema. > Example: > > > {code:java} > struct TestRawIdent { r#type: bool }{code} > > > I implemented a fix and I am currently in the process of creating a PR -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3653) [build] Move off Travis CI
[ https://issues.apache.org/jira/browse/AVRO-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3653: - Labels: pull-request-available (was: ) > [build] Move off Travis CI > -- > > Key: AVRO-3653 > URL: https://issues.apache.org/jira/browse/AVRO-3653 > Project: Apache Avro > Issue Type: Task >Reporter: Ryan Skraba >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Infra is asking us to migrate away from Travis-CI definitively: > [https://cwiki.apache.org/confluence/display/INFRA/Travis+Migrations] > We had removed the Travis build, but restored some parts for ARM64 (AVRO-3242) > https://github.com/apache/avro/blob/master/.travis.yml -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3664) Add Rust Avro Value conversion derive implementations
[ https://issues.apache.org/jira/browse/AVRO-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3664: - Labels: pull-request-available (was: ) > Add Rust Avro Value conversion derive implementations > - > > Key: AVRO-3664 > URL: https://issues.apache.org/jira/browse/AVRO-3664 > Project: Apache Avro > Issue Type: New Feature > Components: rust >Reporter: Aaron Spiegel >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Given the current constraints of the _AvroSchema_ derive macro, it is > possible to also derive the following implementations: > {code:java} > impl From for X {...} > impl from for apache_avro::types::Value {...}{code} > With these traits implemented, serialization in Avro becomes considerably > simpler, for schemas that are already generated through a derive macro. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3669) Missing py.typed file
[ https://issues.apache.org/jira/browse/AVRO-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3669: - Labels: pull-request-available (was: ) > Missing py.typed file > - > > Key: AVRO-3669 > URL: https://issues.apache.org/jira/browse/AVRO-3669 > Project: Apache Avro > Issue Type: Bug >Reporter: Kaisen Chen >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The python avro package has type annotations but does not advertise itself as > such because there is no py.typed marker file (see > [PEP561|https://peps.python.org/pep-0561/#packaging-type-information]). > Symptom: > If python packages import avro and run mypy, error messages like this come up: > {code:java} > error: Skipping analyzing "avro": module is installed, but missing library > stubs or py.typed marker {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3666) New schema parser for all supported schema formats
[ https://issues.apache.org/jira/browse/AVRO-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3666: - Labels: pull-request-available (was: ) > New schema parser for all supported schema formats > -- > > Key: AVRO-3666 > URL: https://issues.apache.org/jira/browse/AVRO-3666 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Oscar Westra van Holthe - Kind >Assignee: Oscar Westra van Holthe - Kind >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > As the new IDL syntax also supports schema files, it's time to refactor > schema parsing in Java to support all schema formats. > The goals: > * Parse JSON schemata with the same results > * Parse IDL schemata in the same way > * Use a pluggable API that supports other schema formats as well > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3670) Add .NET 7 support
[ https://issues.apache.org/jira/browse/AVRO-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3670: - Labels: pull-request-available (was: ) > Add .NET 7 support > -- > > Key: AVRO-3670 > URL: https://issues.apache.org/jira/browse/AVRO-3670 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Reporter: Zoltan Csizmadia >Priority: Minor > Labels: pull-request-available > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > .NET 7.0 SDK is released > [https://dotnet.microsoft.com/en-us/download/dotnet/7.0] > Add support for building and testing using NET 7 SDK as well. Published > class libraries will be still target netstandard2.0 and netstandard2.1, > however published tools and executables (e.g. AvroGen) will traget . NET 7 as > well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3657) Computation of initial buffer size in OutputBuffer makes no sense
[ https://issues.apache.org/jira/browse/AVRO-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3657: - Labels: pull-request-available (was: ) > Computation of initial buffer size in OutputBuffer makes no sense > - > > Key: AVRO-3657 > URL: https://issues.apache.org/jira/browse/AVRO-3657 > Project: Apache Avro > Issue Type: Bug > Components: java >Reporter: Tagir Valeev >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > As you can see, OutputBuffer calls the superclass constructor in the > following way: > [https://github.com/apache/avro/blame/1119b6eb5b92730b27e9798793bc67f192591c15/lang/java/trevni/core/src/main/java/org/apache/trevni/OutputBuffer.java#L32] > > public OutputBuffer() > { super((BLOCK_SIZE + BLOCK_SIZE) >> 2); } > > Before 3d51ba5e965561 change it was > > public OutputBuffer() > { super(BLOCK_SIZE + BLOCK_SIZE >> 2); } > > Which looked like the author did not know that >> has lower precedence than + > and forgot to put parentheses. It looks like, the 3d51ba5e965561 change > blindly applied static analyzer suggestions and added parentheses to preserve > the current semantics. However, `(BLOCK_SIZE + BLOCK_SIZE) >> 2` is simply > `BLOCK_SIZE / 2`. > > My suggestion is the following: > * If everybody is fine with current behavior, let's update this to > `super(BLOCK_SIZE / 2);` to avoid confusion. > * If we would like to preserve the original author's intention to allocate > 1.25x more, let's place the parentheses in another way: `super(BLOCK_SIZE + > (BLOCK_SIZE >> 2));` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3672) Add CI testing for Python 3.11
[ https://issues.apache.org/jira/browse/AVRO-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3672: - Labels: pull-request-available (was: ) > Add CI testing for Python 3.11 > -- > > Key: AVRO-3672 > URL: https://issues.apache.org/jira/browse/AVRO-3672 > Project: Apache Avro > Issue Type: Task > Components: python >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Python 3.11 has been released few weeks ago: > [https://www.python.org/downloads/release/python-3110/] > It would be good to test with it at Github Actions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3667) [Python] Python 3.10 CI test fails since a while
[ https://issues.apache.org/jira/browse/AVRO-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3667: - Labels: pull-request-available (was: ) > [Python] Python 3.10 CI test fails since a while > > > Key: AVRO-3667 > URL: https://issues.apache.org/jira/browse/AVRO-3667 > Project: Apache Avro > Issue Type: Bug > Components: python >Reporter: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The last 4 builds are failing at > [https://github.com/apache/avro/actions/workflows/test-lang-py.yml > |https://github.com/apache/avro/actions/workflows/test-lang-py.yml] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3662) [Ruby] Ruby 2.6 CI workflow fails since a while
[ https://issues.apache.org/jira/browse/AVRO-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3662: - Labels: pull-request-available (was: ) > [Ruby] Ruby 2.6 CI workflow fails since a while > --- > > Key: AVRO-3662 > URL: https://issues.apache.org/jira/browse/AVRO-3662 > Project: Apache Avro > Issue Type: Bug > Components: ruby >Reporter: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > [https://github.com/apache/avro/actions/workflows/test-lang-ruby.yml] > > {code:java} > === > Failure: test_bytes_decimal_object_allocations_decode(TestLogicalTypes): > <1> was expected to be > >= > <2>. > /home/runner/work/avro/avro/lang/ruby/test/test_logical_types.rb:234:in > `block in test_bytes_decimal_object_allocations_decode' > 231: > 232: assert_equal 5, report.total_allocated > 233: # Ruby 2.7 does not retain anything. Ruby 2.6 retains 1 > => 234: assert_operator 1, :>=, report.total_retained > 235: end > 236: end > 237: > > /home/runner/work/avro/avro/lang/ruby/test/test_logical_types.rb:227:in > `each' > > /home/runner/work/avro/avro/lang/ruby/test/test_logical_types.rb:227:in > `test_bytes_decimal_object_allocations_decode' {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3673) Revive MSBuild AvroGenTask
[ https://issues.apache.org/jira/browse/AVRO-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3673: - Labels: pull-request-available (was: ) > Revive MSBuild AvroGenTask > -- > > Key: AVRO-3673 > URL: https://issues.apache.org/jira/browse/AVRO-3673 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Reporter: Zoltan Csizmadia >Priority: Minor > Labels: pull-request-available > Original Estimate: 48h > Time Spent: 10m > Remaining Estimate: 47h 50m > > Currently the avrogen tool must be calledfrom a csproj file to generate cs > files from schema or protocol files. This requires the Avro.Tools package to > be installed. > > This PR adds support for a native MSBuild task without explicitly lancing the > avrogen binary. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3674) Value::Record containing enums fail to validate when using namespaces in Schema
[ https://issues.apache.org/jira/browse/AVRO-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3674: - Labels: pull-request-available (was: ) > Value::Record containing enums fail to validate when using namespaces in > Schema > --- > > Key: AVRO-3674 > URL: https://issues.apache.org/jira/browse/AVRO-3674 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: Rik Heijdens >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Consider the following schema: > {noformat} > { > "type": "record", > "name": "NamespacedMessage", > "namespace": "com.domain", > "fields": [ > { > "type": "record", > "name": "field_a", > "fields": [ > { > "name": "enum_a", > "type": { > "type": "enum", > "name": "EnumType", > "symbols": [ > "SYMBOL_1", > "SYMBOL_2" > ], > "default": "SYMBOL_1" > } > }, > { > "name": "enum_b", > "type": "EnumType" > } > ] > } > ] > }{noformat} > I might represent this in Rust using the following structs: > {noformat} > #[derive(Serialize)] > enum EnumType { > #[serde(rename = "SYMBOL_1")] > Symbol1, > #[serde(rename = "SYMBOL_2")] > Symbol2, > } > #[derive(Serialize)] > struct FieldA { > enum_a: EnumType, > enum_b: EnumType, > } > #[derive(Serialize)] > struct NamespacedMessage { > field_a: FieldA, > } > let msg = NamespacedMessage { > field_a: FieldA { > enum_a: EnumType::Symbol2, > enum_b: EnumType::Symbol1, > }, > }; > {noformat} > and then serialize this into a `Value` using the following logic: > {noformat} > let mut ser = Serializer::default(); > let test_value: Value = msg.serialize(&mut ser).unwrap(); > {noformat} > After serializing into `test_value` I would expect that > `test_value.validate(&schema)` yields True. However this is not the case. > I can work around it by removing the `namespace` definition from my schema > which allows the validation to proceed. > I believe the cause of schema validation failure is [this lookup > failing](https://github.com/apache/avro/blob/release-1.11.1-rc1/lang/rust/avro/src/types.rs#L370) > when schemas are utilized as the `Value` will not have a namespace > associated with it. > I believe this could be fixed by altering `validate_internal` to accept an > optional namespace that is derived from the provided schema to `validate`. > However, I'm not sure if this is an appropriate fix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3676) [Java] toString produces incorrect JSON for UUID
[ https://issues.apache.org/jira/browse/AVRO-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3676: - Labels: pull-request-available (was: ) > [Java] toString produces incorrect JSON for UUID > > > Key: AVRO-3676 > URL: https://issues.apache.org/jira/browse/AVRO-3676 > Project: Apache Avro > Issue Type: Bug >Reporter: Ryan Skraba >Priority: Major > Labels: pull-request-available > Fix For: 1.11.2 > > Time Spent: 10m > Remaining Estimate: 0h > > Like AVRO-3124, but for UUID classes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3678) [Rust] Support write float value to field defined as double
[ https://issues.apache.org/jira/browse/AVRO-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3678: - Labels: pull-request-available (was: ) > [Rust] Support write float value to field defined as double > --- > > Key: AVRO-3678 > URL: https://issues.apache.org/jira/browse/AVRO-3678 > Project: Apache Avro > Issue Type: New Feature > Components: rust >Reporter: Shaeq Ahmed >Assignee: Shaeq Ahmed >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.2 > > Time Spent: 10m > Remaining Estimate: 0h > > [https://github.com/apache/avro/blob/b8937526d32928b3b29040e862f7689d7230bddb/lang/rust/avro/src/types.rs#L405] > Currently a Avro value of the type float (32) cannot be resolved against a > schema where the type is defined as double (64). We already support type > widening from int -> long (dest. schema) as this is completely lossless, so I > think we should also support the same for float -> double for consistency & > convenience as this is a common use case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3679) [Rust] Enable 'perf' feature of regex dependency
[ https://issues.apache.org/jira/browse/AVRO-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3679: - Labels: pull-request-available (was: ) > [Rust] Enable 'perf' feature of regex dependency > > > Key: AVRO-3679 > URL: https://issues.apache.org/jira/browse/AVRO-3679 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Affects Versions: 1.11.1 >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The 'regex' crate has a 'perf' feature that improves the speed of the regular > expressions evaluation. > It would be good to make use of it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3680) [Python] Allow disabling naming validation for interoperability
[ https://issues.apache.org/jira/browse/AVRO-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3680: - Labels: pull-request-available (was: ) > [Python] Allow disabling naming validation for interoperability > --- > > Key: AVRO-3680 > URL: https://issues.apache.org/jira/browse/AVRO-3680 > Project: Apache Avro > Issue Type: New Feature > Components: python >Reporter: Ryan Skraba >Priority: Major > Labels: pull-request-available > Fix For: 1.11.2 > > Time Spent: 10m > Remaining Estimate: 0h > > The Java SDK accepts some Avro names that are not valid according to the > specification, or compatible with other SDKs. There is an [ongoing > discussion|https://lists.apache.org/thread/39v98os6wdpyr6w31xdkz0yzol51fsrr] > about the "right" thing to do. > Currently, schemas that meet the specification rules are guaranteed to be > interoperable. > When parsing schemas that might not be interoperable (but still might work > under certain circumstances), the Python SDK should provide a way to disable > name validation, like it provides a way to disable symbol [name > validation|https://github.com/apache/avro/blob/34845fd49af9cd00e98f5930c6ac6190da5ea6a3/lang/py/avro/schema.py#L1194]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3682) [Build] Remove forrest from Avro build
[ https://issues.apache.org/jira/browse/AVRO-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3682: - Labels: pull-request-available (was: ) > [Build] Remove forrest from Avro build > -- > > Key: AVRO-3682 > URL: https://issues.apache.org/jira/browse/AVRO-3682 > Project: Apache Avro > Issue Type: Task >Reporter: Ryan Skraba >Assignee: Ryan Skraba >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We are no longer using forrest in the Avro build, and it can be removed from > the project. > This should include the deprecated files in {{docs-deprecated}} and the > {{{}DockerfileLocal{}}}. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3683) Rust Writer, Reader can't use Schemas with dependencies in other Schemas. i.e. The output of Schema::parse_list
[ https://issues.apache.org/jira/browse/AVRO-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3683: - Labels: pull-request-available (was: ) > Rust Writer, Reader can't use Schemas with dependencies in other Schemas. > i.e. The output of Schema::parse_list > --- > > Key: AVRO-3683 > URL: https://issues.apache.org/jira/browse/AVRO-3683 > Project: Apache Avro > Issue Type: Bug > Components: rust >Affects Versions: 1.11.1 >Reporter: Mark Farnan >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Avro permits Schema definitions to reference Types defined in other Schema > files. > Currently these can be parsed with Schema:parse_list, but the output of this > can't be used in Writer, Reader, (or to_avro_datum, from_avro_datum) > * These functions only take a single 'schema' and can't resolve the > References > * Schema:parse_list puts in 'refs' for the external dependencies, but > doesn't seem to populate the UnionSchema's, so they can be resolved. > > +Example of dependant schemas:+ > letschema_TypeA=r#" > { > "type": "record", > "namespace": "base.name.space", > "name": "MultiSchemaTestTypeA", > "fields": > [ > { "name": "b", "type": ["null", "some.name.space.MultiSchemaTestTypeB"] }, > { > "name": "c", > "type": \{ "type": "map", "values": "some.name.space.MultiSchemaTestTypeB" }, > "default": {} > } > ] > } > "#; > letschema_TypeB=r#" > { > "type": "record", > "namespace": "some.name.space", > "name": "MultiSchemaTestTypeB", > "fields": > [ > { "name": "d", "type": "string" }, > { "name": "e", "type": "int" } > ] > } > "#; > > Note: In process of creating a PR with a failing test. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3640) Ignores schema files when source folder is linked
[ https://issues.apache.org/jira/browse/AVRO-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3640: - Labels: pull-request-available (was: ) > Ignores schema files when source folder is linked > - > > Key: AVRO-3640 > URL: https://issues.apache.org/jira/browse/AVRO-3640 > Project: Apache Avro > Issue Type: Bug >Affects Versions: 1.11.1 > Environment: - java 17 > - maven 3.8.6 > - OS: ubuntu > >Reporter: Sławomir Siudek >Assignee: Rens Groothuijsen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > avro-maven-plugin > when source directory is created as linked, plugin skips code generation > silently > > > Full reproduction with README > https://github.com/siudeks/tech-spikes/tree/master/bugrepro_avro_code_generation -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3681) [Python] GitHub actions failing with python 3.6
[ https://issues.apache.org/jira/browse/AVRO-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3681: - Labels: pull-request-available (was: ) > [Python] GitHub actions failing with python 3.6 > --- > > Key: AVRO-3681 > URL: https://issues.apache.org/jira/browse/AVRO-3681 > Project: Apache Avro > Issue Type: Task >Reporter: Ryan Skraba >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The GitHub actions build is failing on Python 3.6, apparently because it is > not available on ubuntu-latest anymore. > From the logs in [this > job|https://github.com/apache/avro/actions/runs/3603280401/jobs/6071285008]: > {code} > 2022-12-02T16:13:26.6605941Z Requested labels: ubuntu-latest > 2022-12-02T16:13:26.6606006Z Job defined at: > apache/avro/.github/workflows/test-lang-py.yml@refs/pull/1995/merge > > 2022-12-02T16:13:38.7515296Z ##[group]Run actions/setup-python@v4 > 2022-12-02T16:13:38.7515540Z with: > 2022-12-02T16:13:38.7515728Z python-version: 3.6 > 2022-12-02T16:13:38.7515931Z check-latest: false > 2022-12-02T16:13:38.7516303Z token: *** > 2022-12-02T16:13:38.7516513Z update-environment: true > 2022-12-02T16:13:38.7516716Z ##[endgroup] > 2022-12-02T16:13:38.9235198Z Version 3.6 was not found in the local cache > 2022-12-02T16:13:39.2572679Z ##[error]Version 3.6 with arch x64 not found > The list of all available versions can be found here: > https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json > 2022-12-02T16:13:39.2689115Z Post job cleanup. > > {code} > We can see that there is no 3.6 with a platform version of 22.04 in the > referenced version manifest: > https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json > This issue is discussed here: > https://github.com/actions/setup-python/issues/544 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3688) Schema resolution panics when a custom record field is included multiple times
[ https://issues.apache.org/jira/browse/AVRO-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3688: - Labels: pull-request-available (was: ) > Schema resolution panics when a custom record field is included multiple times > -- > > Key: AVRO-3688 > URL: https://issues.apache.org/jira/browse/AVRO-3688 > Project: Apache Avro > Issue Type: Bug >Reporter: Rik Heijdens >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Consider the following Avro schema: > {noformat} > { > "type": "record", > "name": "Message", > "fields": [ > { > "name": "field_a", > "type": [ > "null", > { > "name": "Inner", > "type": "record", > "fields": [ > { > "name": "inner_a", > "type": "string" > } > ] > } > ], > "default": null > }, > { > "name": "field_b", > "type": [ > "null", > "Inner" > ], > "default": null > } > ] > }{noformat} > This can be represented in Rust through the following structs: > {noformat} > #[derive(Serialize, Deserialize)] > struct Inner { > inner_a: String > } > #[derive(Serialize, Deserialize)] > struct Message { > field_a: Option, > field_b: Option > } > {noformat} > If I instantiate an instance of `message`, set `field_a` and then serialize > it using the following code: > {noformat} > let schema = Schema::parse_str(&schema_str).unwrap(); > let msg = Message { > field_a: Some(Inner { > inner_a: "foo".to_string() > }), > field_b: None > }; > let mut ser = Serializer::default(); > let test_value: Value = msg.serialize(&mut ser).unwrap(); > assert!(test_value.validate(&schema), "test_value should validate"); > assert!( > test_value.resolve(&schema).is_ok(), > "test_value should resolve" > ); > {noformat} > Then my assertions pass. However if I set field_b to `Inner`, then my call to > `test_value.resolve(&schema)` panics: > {noformat} > let schema = Schema::parse_str(&schema_str).unwrap(); > let msg = Message { > field_a: Some(Inner { > inner_a: "foo".to_string() > }), > field_b: Some(Inner { > inner_a: "bar".to_string() > }) > }; > let mut ser = Serializer::default(); > let test_value: Value = msg.serialize(&mut ser).unwrap(); > assert!(test_value.validate(&schema), "test_value should validate"); > assert!( > test_value.resolve(&schema).is_ok(), > "test_value should resolve" > ); > {noformat} > This seems to be a bug in the schema resolution logic that's causing an > unhandled panic. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3689) Detected a flaky test 'testAnnotationMultiAvroMeta'
[ https://issues.apache.org/jira/browse/AVRO-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3689: - Labels: pull-request-available (was: ) > Detected a flaky test 'testAnnotationMultiAvroMeta' > --- > > Key: AVRO-3689 > URL: https://issues.apache.org/jira/browse/AVRO-3689 > Project: Apache Avro > Issue Type: Test > Environment: > >Reporter: Zhengxu Jin >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Test `testAnnotationMultiAvroMeta' under 'TestReflect.java' is a flaky test. > To check its flakiness, you can run maven with nondex using the following > command: > ``` > mvn -pl lang/java/avro edu.illinois:nondex-maven-plugin:2.1.1:nondex > -Dtest=org.apache.avro.reflect.TestReflect#testAnnotationMultiAvroMeta > ``` > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3690) Detected a flaky test ‘testMultipleFieldAliases’
[ https://issues.apache.org/jira/browse/AVRO-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3690: - Labels: pull-request-available (was: ) > Detected a flaky test ‘testMultipleFieldAliases’ > > > Key: AVRO-3690 > URL: https://issues.apache.org/jira/browse/AVRO-3690 > Project: Apache Avro > Issue Type: Test >Reporter: Zhengxu Jin >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Test `testMultipleFieldAliases' under 'TestReflect.java' is a flaky test. To > check its flakiness, you can run maven with nondex using the following > command: > ``` > mvn -pl lang/java/avro edu.illinois:nondex-maven-plugin:2.1.1:nondex > -Dtest=org.apache.avro.reflect.TestReflect#testMultipleFieldAliases > ``` > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3691) Add Nondex:2.1.1 to Maven build section
[ https://issues.apache.org/jira/browse/AVRO-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3691: - Labels: pull-request-available (was: ) > Add Nondex:2.1.1 to Maven build section > --- > > Key: AVRO-3691 > URL: https://issues.apache.org/jira/browse/AVRO-3691 > Project: Apache Avro > Issue Type: Wish > Environment: NonDex is a tool for detecting and debugging wrong > assumptions on under-determined Java APIs. An example of such an assumption > is when code assumes the order of iterating through the entries in a > java.util.HashMap is in a specific, deterministic order, but the > specification for java.util.HashMap is under-determined and states that this > iteration order is not guaranteed to be in any particular order. Such > assumptions can hurt portability for an application when they are moved to > other environments with a different Java runtime. NonDex explores different > behaviors of under-determined APIs and reports test failures under different > explored behaviors; NonDex only explores behaviors that are allowed by the > specification and any test failure indicates an assumption on an > under-determined Java API. NonDex helps expose such brittle assumptions to > the developers early, so they can fix the assumption before it becomes a > problem far in the future and more difficult to fix. > > >Reporter: Zhengxu Jin >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3685) .NET/C#: ClassCash can not load two types with the compex type properties that has same name but different type
[ https://issues.apache.org/jira/browse/AVRO-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3685: - Labels: pull-request-available (was: ) > .NET/C#: ClassCash can not load two types with the compex type properties > that has same name but different type > --- > > Key: AVRO-3685 > URL: https://issues.apache.org/jira/browse/AVRO-3685 > Project: Apache Avro > Issue Type: Bug >Reporter: Khrystyna Popadyuk >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When load two types that have complex type properties with: > * same name > * different type > Only first one is loaded. As a result only one type can be serialized or > deserialized. > > It happens because of this check > [https://github.com/KhrystynaPopadyuk/avro/blob/dee749bde457ba8760e0e5e1e2c1d1844610f2ee/lang/csharp/src/apache/main/Reflect/ClassCache.cs#L223] > That code check load or not type base on property name. > Unit test that illustrate bug: -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3692) Serde flatten is not supported when deserializing
[ https://issues.apache.org/jira/browse/AVRO-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3692: - Labels: pull-request-available (was: ) > Serde flatten is not supported when deserializing > - > > Key: AVRO-3692 > URL: https://issues.apache.org/jira/browse/AVRO-3692 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Reporter: Ten >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The following: > {code:java} > #[test] > fn test_from_value_struct_flatten() { > #[derive(Deserialize, PartialEq, Debug)] > struct S1 { > f1: String, > #[serde(flatten)] > inner: S2, > } > #[derive(Deserialize, PartialEq, Debug)] > struct S2 { > f2: String, > } > let expected = S1 { > f1: "Hello".to_owned(), > inner: S2 { > f2: "World".to_owned(), > }, > }; let test = Value::Record(vec![ > ("f1".to_owned(), "Hello".into()), > ("f2".to_owned(), "World".into()), > ]); > let final_value: S1 = from_value(&test).unwrap(); > assert_eq!(final_value, expected); > } {code} > fails to deserialize. > This is due to serde hinting for map deserialization when structs contain > flatten. It is super easy to fix, I'm opening a PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3693) avrogencpp Invalid type for union exception does not identify which union
[ https://issues.apache.org/jira/browse/AVRO-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3693: - Labels: pull-request-available (was: ) > avrogencpp Invalid type for union exception does not identify which union > - > > Key: AVRO-3693 > URL: https://issues.apache.org/jira/browse/AVRO-3693 > Project: Apache Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.11.0 >Reporter: Andrew Peter Marlow >Priority: Minor > Labels: pull-request-available > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > This is a small enhancement request for the avrogencpp code generator, to > help identify a deserialisation issue better. When a structure sent by avro > contains unions and a union part of a message is de-serialized without > checking first that it is present, invalid type for union is thrown. However > it does not identify which union. It could include the structName in the > exception, then it would be possible to tie down the exception to the precise > union that got the error. > If the file below, Empdetails.avsc is given to avrogencpp, it will show a > get_int function for a union where the exception text does not identify the > union. > {code:java} > { > "type" : "record", > "namespace" : "tutorialspoint", > "name" : "Empdetails", > "fields" : > [ > { "name" : "experience", "type": ["int", "null"] }, { "name" : "age", > "type": "int" } > ] > } {code} > In this particular case there is only one union so there is no problem but > when there are multiple unions they all get the same exception text so it is > hard to tell from where the throw came. > I have the change waiting in the wings for a PR. The change includes the > structName in the exception. With this change the avrogencpp generate code > looks as follows: > {code:java} > inline > int32_t Empdetails_avsc_Union__0__::get_int() const { > if (idx_ != 0) { > throw avro::Exception("Invalid type for union > Empdetails_avsc_Union__0__"); > } > return std::any_cast(value_); > } {code} > With this done for every union in a structure it will be clear which throw > has executed when the exception is reported. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3695) [Rust] Can't validate schema with value when Nullable union has multiple types.
[ https://issues.apache.org/jira/browse/AVRO-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3695: - Labels: pull-request-available rust (was: rust) > [Rust] Can't validate schema with value when Nullable union has multiple > types. > --- > > Key: AVRO-3695 > URL: https://issues.apache.org/jira/browse/AVRO-3695 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: Mark Farnan >Priority: Blocker > Labels: pull-request-available, rust > Time Spent: 10m > Remaining Estimate: 0h > > If a union has more than one 'type' (other than null), to_avro_datum is > unable to validate the schema with the value type, and throws a validation > error. > > Schema: > > {code:java} > { "type": "record", "namespace": "datatypes", "name": "nullunion", "fields": > [ {"name": "item", "type": [ "null", "long", "double" ] } ] } {code} > > Struct: (This was generated by rsgen-avro) > {code:java} > /// Auto-generated type for unnamed Avro union variants. > #[derive(Debug, PartialEq, Clone, serde::Deserialize, serde::Serialize)] > pub enum UnionLongDouble {Long(i64),Double(f64),} > #[derive(Debug, PartialEq, Clone, serde::Deserialize, serde::Serialize)] > pub struct NullUnion {pub item: Option,}{code} > > {code:java} > #[test] > fn test_multivalue_union_tovalue() { > let nuschema = r#" > { > "type": "record", > "namespace": "datatypes", > "name": "nullunion", > "fields": > [ > {"name": "item", > "type": [ > "null", > "long", > "double" > ] > } > ] > } > "#; > let nullunion = NullUnion { > item: Some(UnionLongDouble::Long(34)), > }; let schema = Schema::parse_str(nuschema).unwrap(); > let nu_value = to_value(nullunion).unwrap(); > let nu_encoded = to_avro_datum(&schema, nu_value).unwrap(); > } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3696) [Python] Replace tox-wheel with upstream tox 4
[ https://issues.apache.org/jira/browse/AVRO-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3696: - Labels: pull-request-available (was: ) > [Python] Replace tox-wheel with upstream tox 4 > -- > > Key: AVRO-3696 > URL: https://issues.apache.org/jira/browse/AVRO-3696 > Project: Apache Avro > Issue Type: Test > Components: python >Reporter: Tommi Vainikainen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Usage of third party tox-wheel package started failing since tox released > version 4 with native wheel. Any usage of tox-wheel should be replaced with > standard tox package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3697) Test against Ruby 3.2
[ https://issues.apache.org/jira/browse/AVRO-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3697: - Labels: pull-request-available (was: ) > Test against Ruby 3.2 > - > > Key: AVRO-3697 > URL: https://issues.apache.org/jira/browse/AVRO-3697 > Project: Apache Avro > Issue Type: Test > Components: ruby >Affects Versions: 1.12.0, 1.11.2 >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Ruby 3.2 was released on 12/25/2022 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3278) Drop support for Ruby 2.6
[ https://issues.apache.org/jira/browse/AVRO-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3278: - Labels: pull-request-available (was: ) > Drop support for Ruby 2.6 > - > > Key: AVRO-3278 > URL: https://issues.apache.org/jira/browse/AVRO-3278 > Project: Apache Avro > Issue Type: Test > Components: ruby >Affects Versions: 1.12.0 >Reporter: Tim Perkins >Assignee: Tim Perkins >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Ruby 2.6 will reach EOL on March 31, 2022. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3700) Publish SBOM artifacts
[ https://issues.apache.org/jira/browse/AVRO-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3700: - Labels: pull-request-available (was: ) > Publish SBOM artifacts > -- > > Key: AVRO-3700 > URL: https://issues.apache.org/jira/browse/AVRO-3700 > Project: Apache Avro > Issue Type: Improvement > Components: build >Affects Versions: 1.12.0 >Reporter: Dongjoon Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3701) Add github action to validate maven 4 build compatibility
[ https://issues.apache.org/jira/browse/AVRO-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3701: - Labels: pull-request-available (was: ) > Add github action to validate maven 4 build compatibility > - > > Key: AVRO-3701 > URL: https://issues.apache.org/jira/browse/AVRO-3701 > Project: Apache Avro > Issue Type: Test > Components: build >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We can validate in advance compatility with Maven 4 for a future migration. > We can also validate with the new [maven build cache > extension|https://maven.apache.org/extensions/maven-build-cache-extension/] > that brings big improvements into the build via smart caching of build > artifacts and results. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3698) [Java] SpecificData.getClassName must replace reserved words
[ https://issues.apache.org/jira/browse/AVRO-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3698: - Labels: newbie patch pull-request-available (was: newbie patch) > [Java] SpecificData.getClassName must replace reserved words > > > Key: AVRO-3698 > URL: https://issues.apache.org/jira/browse/AVRO-3698 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.11.1 >Reporter: Seung-Hyun Cheong >Priority: Major > Labels: newbie, patch, pull-request-available > Fix For: 1.11.2 > > Attachments: AVRO-3698.patch > > Time Spent: 10m > Remaining Estimate: 0h > > [Java] SpecificData.getClassName must replace reserved words > > When makes Java Class, SpecificCompiler adds "$" after reserved words like > public, new, etc... > [https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L1129] > [https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L643] > > For example, if the namespace of a schema is {*}db.public.table{*}, then the > package name of the generated Java Class, made by SpecificCompiler, is > {*}db.public$.table{*}. (Because *public* is a reserved word in java) > > But when deserializes an Avro Record to a SpecificRecord, > SpecificData.getClassName returns {*}db.pulic.table.Name{*}. > > So, it can't find a class by name from SpecificData.getClassName. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-2918) Schema polymorphism
[ https://issues.apache.org/jira/browse/AVRO-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-2918: - Labels: features pull-request-available (was: features) > Schema polymorphism > --- > > Key: AVRO-2918 > URL: https://issues.apache.org/jira/browse/AVRO-2918 > Project: Apache Avro > Issue Type: New Feature > Components: logical types, misc, spec >Reporter: Jonathan Rapoport >Priority: Critical > Labels: features, pull-request-available > Original Estimate: 96h > Time Spent: 10m > Remaining Estimate: 95h 50m > > Include the option to use named types as base types for a new schema. Allow > for MRO generation. Field inheritance. > The benefits of this approach include: > * Defining a schema as validation for a certain wire, and so allowing the > receiver to be certain of the structure of the data (this works today). > However, defining an extension of this schema, or certain schemas which can > be normalized to the original schema, but contain additional information, > will not allow it to be sent over the same wire. > * Backwards compatibility through inheritance - you never break the old > schema, thus allowing a long integration period, with no need to recode all > processes familiar with the schema. The new schema will simply inherit the > old one, and only add information. > * Allow for full data control through polymorphism, and the ability to > replace structures within any supported language. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3704) Naming rules : multiple choice
[ https://issues.apache.org/jira/browse/AVRO-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3704: - Labels: pull-request-available (was: ) > Naming rules : multiple choice > -- > > Key: AVRO-3704 > URL: https://issues.apache.org/jira/browse/AVRO-3704 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Christophe Le Saec >Assignee: Christophe Le Saec >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Long discussion about [naming > rules|https://lists.apache.org/thread/39v98os6wdpyr6w31xdkz0yzol51fsrr] and > the fact that currently, name checking is not the same depends of module > (rust follow current doc, java is more flexible and accept accent), and also > you can now deactivate control, i propose a change to allow to choose "name > controler" between some proposed one (no control, current controller, strict > controller that follow doc) to client custom one. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3706) AVDL nested imports cannot be resolved if path contains spaces
[ https://issues.apache.org/jira/browse/AVRO-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3706: - Labels: pull-request-available (was: ) > AVDL nested imports cannot be resolved if path contains spaces > -- > > Key: AVRO-3706 > URL: https://issues.apache.org/jira/browse/AVRO-3706 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.11.0, 1.11.1 >Reporter: fanguad >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There is an issue when running the IDL tool if the IDL files have nested > import statements, and the path includes a space (or probably any character > that must be URL encoded). After the first level of import, the files can no > longer be found. Consider the following three files in a {{avro bug}} > directory: > {{root.avdl}} > {code:java} > protocol Root { > import idl "level1.avdl"; > } {code} > {{level1.avdl}} > {code:java} > protocol Level1 { > import idl "level2.avdl"; > } {code} > {{level2.avdl}} > {code:java} > protocol Level2 { > } {code} > executing {{java -jar avro-tools-1.10.2.jar idl "avro bug/root.avdl"}} > results in the expected > > {code:java} > { > "protocol" : "Root", > "namespace" : null, > "types" : [ ], > "messages" : { } > } {code} > whereas {{java -jar avro-tools-1.11.1.jar idl "avro bug/root.avdl"}} causes > an exception (the bug is also present in 1.11.0 with a slightly different > stack trace) > > {code:java} > Exception in thread "main" org.apache.avro.compiler.idl.ParseException: Error > importing level2.avdl: java.io.FileNotFoundException: level2.avdl, at line 2, > column 26 > at org.apache.avro.compiler.idl.Idl.error(Idl.java:88) > at org.apache.avro.compiler.idl.Idl.ImportIdl(Idl.java:537) > at org.apache.avro.compiler.idl.Idl.ProtocolBody(Idl.java:411) > at org.apache.avro.compiler.idl.Idl.ProtocolDeclaration(Idl.java:286) > at org.apache.avro.compiler.idl.Idl.CompilationUnit(Idl.java:167) > at org.apache.avro.compiler.idl.Idl.ImportIdl(Idl.java:535) > at org.apache.avro.compiler.idl.Idl.ProtocolBody(Idl.java:411) > at org.apache.avro.compiler.idl.Idl.ProtocolDeclaration(Idl.java:286) > at org.apache.avro.compiler.idl.Idl.CompilationUnit(Idl.java:167) > at org.apache.avro.tool.IdlTool.run(IdlTool.java:61) > at org.apache.avro.tool.Main.run(Main.java:67) > at org.apache.avro.tool.Main.main(Main.java:56) > {code} > > The same issue occurs if the files are located in "avro" but the current > folder has spaces (ie, the absolute path to the avdl files contains spaces). > > The bug appears to have been introduced in this commit > [https://github.com/apache/avro/commit/3fe5e306eaa43bdc06cef291321a8c04058a6be9#diff-376865a18691674de38817b4eea7e64c2c3848094a4730030c67f57f54f1028a] > in {{idl.jj}} lines 130 and 169. The second time that line 130 is reached, > it has an absolute path that is already URL-encoded. In 1.10.2, the code > {{new File(input.getPath()).getParentFile()}} returns a String like > {{[file:///home/user/avro%20bug/]}} whereas in 1.11.0 and later {{new > File(input.getPath()).getParentFile().toURI()}} returns a String like > {{[file:///home/user/avro%2520bug]}} - note that URL encoding has happened > twice resulting in %2520 and there is no trailing slash. When resolved on > line 169, this results in {{[file:///home/user/level2.avdl]}} instead of the > correct {{{}[file:///home/user/avro%20bug/level2.avdl]{}}}. > > This bug can be easily worked around by changing the location of the avdl > files to one without spaces or other characters that must be URL encoded. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3708) [Rust] Fix clippy warnings introduced with Rust 1.67.0
[ https://issues.apache.org/jira/browse/AVRO-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3708: - Labels: pull-request-available (was: ) > [Rust] Fix clippy warnings introduced with Rust 1.67.0 > -- > > Key: AVRO-3708 > URL: https://issues.apache.org/jira/browse/AVRO-3708 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Rust 1.67.0 has been released and it introduces some new clippy (lint) > warnings. > To fix them we need to update "edition" from 2018 to 2021 in Cargo.toml. > In addition we can make use of the new "rust-version" property that makes it > clear that the minimum supported Rust version is 1.60.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3705) avrogencpp needs an option to generate code using std instead of boost
[ https://issues.apache.org/jira/browse/AVRO-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3705: - Labels: pull-request-available (was: ) > avrogencpp needs an option to generate code using std instead of boost > -- > > Key: AVRO-3705 > URL: https://issues.apache.org/jira/browse/AVRO-3705 > Project: Apache Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.11.0 > Environment: RHEL8. The gcc installed from the Red Hat repo is > version 8.5.0 which has an option to use c++17 since that dialect is not the > default. The avro-cpp package in the Red Hat repo must have been built with > that since it emits code that uses boost. >Reporter: Andrew Peter Marlow >Assignee: Andrew Peter Marlow >Priority: Minor > Labels: pull-request-available > Fix For: 1.12.0, 1.11.2 > > Time Spent: 10m > Remaining Estimate: 0h > > avrogencpp uses either std or boost in the generated code. It does this > depending on how it was built, using #if __cplusplus >= 201703L. If the > compiler supports std::any then it used, otherwise boost is used. > The assumption is that if the compiler used to build avrogencpp supports > C\+\+17 by default then emitted code can use C\+\+17 otherwise the emitted > code should use boost. This assumption turns out to have a weakness - if a > project uses a compiler where C+17 is given as an option because it is not > the default then the code avrogencpp generates will use boost instead of std. > This problem could be solved if avrogencpp supported a command line option to > use c\+\+17. The option could be used by those environments where avrogencpp > was built with a version of gcc that does not support c17 by default but the > desire is to use C+17. > This is a serious need because there is a significant difference between > boost::any and std::any. The standard any has a small buffer optimisation but > boost does not. This means that code generated by avrogencpp that uses > boost::any will use heap memory for every case where the generated code uses > the any type. This could be avoided if the c++17 option was available. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3712) C++ Build Failure on Manjaro
[ https://issues.apache.org/jira/browse/AVRO-3712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3712: - Labels: pull-request-available (was: ) > C++ Build Failure on Manjaro > - > > Key: AVRO-3712 > URL: https://issues.apache.org/jira/browse/AVRO-3712 > Project: Apache Avro > Issue Type: Bug > Environment: Manjaro >Reporter: John McClean >Assignee: John McClean >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > {quote} > In member function ‘void avro::ReaderImpl::readValue(double&) > [with ValidatorType = avro::NullValidator]’, > inlined from ‘double avro::Parser::readDouble() [with Reader = > avro::ReaderImpl]’ at > /home/john/third-party/avro/lang/c++/api/Parser.hh:75:26, > inlined from ‘void TestSchema::readArray(Parser&) [with Parser = > avro::Parser >]’ at > /home/john/third-party/avro/lang/c++/test/unittest.cc:324:33, > inlined from ‘void TestSchema::readData(Parser&) [with Parser = > avro::Parser >]’ at > /home/john/third-party/avro/lang/c++/test/unittest.cc:366:18, > inlined from ‘void TestSchema::readRawData()’ at > /home/john/third-party/avro/lang/c++/test/unittest.cc:404:17: > /home/john/third-party/avro/lang/c++/api/Reader.hh:89:17: error: > ‘v.avro::ReaderImpl::readValue(double&):: union>::d’ may be used uninitialized [-Werror=maybe-uninitialized] > 89 | val = v.d; > | ~~^ > {quote} > I'll put in a PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3709) [Rust] Add aliases to RecordField
[ https://issues.apache.org/jira/browse/AVRO-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3709: - Labels: pull-request-available rust (was: rust) > [Rust] Add aliases to RecordField > - > > Key: AVRO-3709 > URL: https://issues.apache.org/jira/browse/AVRO-3709 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Affects Versions: 1.11.1 >Reporter: Santiago Fraire Willemoes >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available, rust > Time Spent: 10m > Remaining Estimate: 0h > > Hello everyone, I'm working on an [avdl parser for > rust|https://github.com/woile/avdl-rs] and I'm having an issue with the lack > of aliases on the `RecordField`. > For an Avdl like this: > {code:java} > protocol MyProtocol { > record Hello { > string @aliases(["nickname"]) name; > } > } > {code} > The schema is rendered as: > {code:java} > { > "type" : "record", > "name" : "Hello", > "fields" : [ { > "name" : "name", > "type" : "string", > "aliases" : [ "nickname" ] > } ] > } > {code} > But when I generate the schema on rust using the `RecordField`, it doesn't > have aliases. > See > [https://github.com/apache/avro/blob/master/lang/rust/avro/src/schema.rs#L559-L580] > And even if I use `custom_attributes` with aliases, when generating the > schema using `schema.canonical_form()`, they don't appear. > The `RecordField` has `custom_attributes` for unknown fields but aliases are > known. See documentation here: > [https://avro.apache.org/docs/1.11.1/idl-language/#other-language-features] > The interesting thing is that the Specification doesn't mention aliases in a > RecordField, but they are used widely everywhere. > I would be willing to submit a PR, not sure if there's another reason for the > lack of aliases that I'm missing. > I would also like to ask for advice, is the Schema supposed to be used to > generate a json schema? or is it solely for deserializing a json schema? > Should I create a fork of it? I find it strange that it implements serde's > serialize but it doesn't fully serialize all the attributes. > Thanks a lot. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3713) Regression AVRO-1760 : Thread scalability problem with the use of SynchronizedMap
[ https://issues.apache.org/jira/browse/AVRO-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3713: - Labels: pull-request-available (was: ) > Regression AVRO-1760 : Thread scalability problem with the use of > SynchronizedMap > - > > Key: AVRO-3713 > URL: https://issues.apache.org/jira/browse/AVRO-3713 > Project: Apache Avro > Issue Type: Bug >Reporter: Niels Basjes >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This is a regression of AVRO-1760 which is causing production performance > problems. > In AVRO-1760 the issue was fixed that there is a scalability problem with > AVRO being used in multithreaded situations. > The solution that was implemented for this ticket uses a different Map > implementation (part of Guava) that does not have these locking issues: > [https://github.com/apache/avro/commit/363360229e7ea5f816d87f9d5747dd4494d3b862#diff-422f44890f5e0ab1f508a7c75d0d18fc7831a1c3ca13e0c97780617570c50d5cL970-L973] > Investigating the performance of our internal application has made us > conclude the original problem was reintroduced with this commit: > [https://github.com/apache/avro/commit/c78a3d5ea4e7e511ba9f6182157956db3e65e1a0#diff-422f44890f5e0ab1f508a7c75d0d18fc7831a1c3ca13e0c97780617570c50d5cL986-L988] > which seems to be related to this jira AVRO-2265 "Remove Guava as a > dependency". -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3711) Add documentation about uuid in IDL
[ https://issues.apache.org/jira/browse/AVRO-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3711: - Labels: pull-request-available (was: ) > Add documentation about uuid in IDL > --- > > Key: AVRO-3711 > URL: https://issues.apache.org/jira/browse/AVRO-3711 > Project: Apache Avro > Issue Type: Improvement > Components: doc >Reporter: Santiago Fraire Willemoes >Assignee: Santiago Fraire Willemoes >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > It's possible to parse an uuid in IDL using a logical type, but it is not > documented in the IDL section: > [https://avro.apache.org/docs/1.11.1/idl-language/#logical-types] > The following is valid: > {code:java} > record SyntaxFields { >uuid pk = "pepe"; > }{code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3715) The build consistently fails.
[ https://issues.apache.org/jira/browse/AVRO-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3715: - Labels: pull-request-available (was: ) > The build consistently fails. > - > > Key: AVRO-3715 > URL: https://issues.apache.org/jira/browse/AVRO-3715 > Project: Apache Avro > Issue Type: Bug >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > At least 3 reasons: > - Oracle Java 18 no longer exists. > - CycloneDX does not generate reproducible BOM > > https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/doc/BUILDSPEC.md#cyclonedx-does-not-generate-reproducible-bom > - During ./build.sh dist this dependency does not exist > org.apache.maven.plugin-tools:maven-plugin-tools-javadoc:3.7.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3684) appending to existing file with outputstream not working
[ https://issues.apache.org/jira/browse/AVRO-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3684: - Labels: pull-request-available (was: ) > appending to existing file with outputstream not working > > > Key: AVRO-3684 > URL: https://issues.apache.org/jira/browse/AVRO-3684 > Project: Apache Avro > Issue Type: Bug >Reporter: Mikko Kortelainen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > appending to existing file with outputstream not working > > i created a test case at [https://github.com/apache/avro/pull/2000/files] for > the reference. If this is an incorrect way then please redirect to > documentation or if no such thing exist, please link to existing working > example or test case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3718) Flaky NettyServer test
[ https://issues.apache.org/jira/browse/AVRO-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3718: - Labels: pull-request-available (was: ) > Flaky NettyServer test > -- > > Key: AVRO-3718 > URL: https://issues.apache.org/jira/browse/AVRO-3718 > Project: Apache Avro > Issue Type: Bug >Reporter: Niels Basjes >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > On a regular basis the TestNettyServer.connectionsCount fails in the Github > Actions CI. > When you rerun the same test is usually passes. > {code:java} > [INFO] --- maven-surefire-plugin:3.0.0-M8:test (default-test) @ > avro-ipc-netty --- > 3498[INFO] Tests will run in random order. To reproduce ordering use flag > -Dsurefire.runOrder.random.seed=371358370820 > 3499[INFO] Using auto detected provider > org.apache.maven.surefire.junitplatform.JUnitPlatformProvider > 3500[INFO] > 3501[INFO] --- > 3502[INFO] T E S T S > 3503[INFO] --- > 3504[INFO] Running org.apache.avro.ipc.netty.TestNettyServer > 3505Error: Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: > 4.626 s <<< FAILURE! - in org.apache.avro.ipc.netty.TestNettyServer > 3506Error: org.apache.avro.ipc.netty.TestNettyServer.connectionsCount Time > elapsed: 0.01 s <<< FAILURE! > 3507org.opentest4j.AssertionFailedError: expected: <2> but was: <3> > 3508 at > org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) > 3509 at > org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) > 3510 at > org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197) > 3511{code} > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-2404) Security sweep of build scripts and urls
[ https://issues.apache.org/jira/browse/AVRO-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-2404: - Labels: pull-request-available (was: ) > Security sweep of build scripts and urls > > > Key: AVRO-2404 > URL: https://issues.apache.org/jira/browse/AVRO-2404 > Project: Apache Avro > Issue Type: Task > Components: build >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Major > Labels: pull-request-available > Fix For: 1.9.2 > > Time Spent: 10m > Remaining Estimate: 0h > > In some places we still use insecure URLs. > We should fix that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3722) Eagerly Initialize Instance Variables in Ruby Implementation
[ https://issues.apache.org/jira/browse/AVRO-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3722: - Labels: pull-request-available (was: ) > Eagerly Initialize Instance Variables in Ruby Implementation > > > Key: AVRO-3722 > URL: https://issues.apache.org/jira/browse/AVRO-3722 > Project: Apache Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.11.1 >Reporter: Aaron Patterson >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Ruby 3.2 introduced an inline cache scheme based on the "shape" of an object. > The object's shape depends on what instance variables were set on the object > and in what order. Lazily initializing an instance variable can cause poor > inline cache performance because the order in which the IV is set may not be > predictable. > The Ruby version of Avro has an IV that is lazily initialized, and I would > like to eagerly initialize it so that we can improve cache performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3723) [Rust] Make schema::ResolvedSchema and schema::Names public
[ https://issues.apache.org/jira/browse/AVRO-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3723: - Labels: pull-request-available (was: ) > [Rust] Make schema::ResolvedSchema and schema::Names public > --- > > Key: AVRO-3723 > URL: https://issues.apache.org/jira/browse/AVRO-3723 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > schema::ResolvedSchema and schema::Names are often needed by third party > traits so they should be made public. > > https://github.com/apache/avro/pull/2131 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-2943) Map comparison between Utf8 and String keys fails
[ https://issues.apache.org/jira/browse/AVRO-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-2943: - Labels: pull-request-available (was: ) > Map comparison between Utf8 and String keys fails > - > > Key: AVRO-2943 > URL: https://issues.apache.org/jira/browse/AVRO-2943 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.10.0 > Environment: Mac OS Catalina 10.15.6 > > openjdk version "1.8.0_265" > OpenJDK Runtime Environment Corretto-8.265.01.1 (build 1.8.0_265-b01) > OpenJDK 64-Bit Server VM Corretto-8.265.01.1 (build 25.265-b01, mixed mode) >Reporter: Frank Grimes >Priority: Major > Labels: pull-request-available > Attachments: AVRO-2943-frankgrimes97.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The following test I locally added to org.apache.avro.generic.TestGenericData > on master demonstrates the problem: > {code:java} > @Test > public void testMapKeyEquals() { > Schema mapSchema = new Schema.Parser().parse("{\"type\": \"map\", > \"values\": \"string\"}"); > Field myMapField = new Field("my_map", Schema.createMap(mapSchema), null, > null); > Schema schema = Schema.createRecord("my_record", "doc", "mytest", false); > schema.setFields(Arrays.asList(myMapField)); > GenericRecord r0 = new GenericData.Record(schema); > GenericRecord r1 = new GenericData.Record(schema); > HashMap pair1 = new HashMap<>(); > pair1.put("keyOne", "valueOne"); > r0.put("my_map", pair1); > HashMap pair2 = new HashMap<>(); > pair2.put(new Utf8("keyOne"), "valueOne"); > r1.put("my_map", pair2); > assertEquals(r0, r1); > assertEquals(r1, r0); > } > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3721) [Java] Add cache to org.apache.avro.JsonProperties.getObjectProps
[ https://issues.apache.org/jira/browse/AVRO-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3721: - Labels: pull-request-available (was: ) > [Java] Add cache to org.apache.avro.JsonProperties.getObjectProps > - > > Key: AVRO-3721 > URL: https://issues.apache.org/jira/browse/AVRO-3721 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.11.1 >Reporter: Moshe Elisha >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently every time you invoke JsonProperties.getObjectProps on same > instance, the code loops over the props and creates a new LinkedHashMap and a > new UnmodifiableMap. > > In our use case, we store a Protocol (and the JsonProperties) and access the > same instances multiple times. > > Storing a simple cache "private Map objectProps;" inside > JsonProperties will slightly improve performance and memory usage. > objectProps can be lazy initialized in JsonProperties.getObjectProps and > reset when "JsonProperties.addProp(String, JsonNode)" is invoked. > > Thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3641) Add support for Nullable and NotNull annotations for SpecificRecords
[ https://issues.apache.org/jira/browse/AVRO-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3641: - Labels: pull-request-available (was: ) > Add support for Nullable and NotNull annotations for SpecificRecords > > > Key: AVRO-3641 > URL: https://issues.apache.org/jira/browse/AVRO-3641 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Peter Knuts >Priority: Major > Labels: pull-request-available > Attachments: AVRO-3641.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Update the Java SpecificCompiler to add support for jetbrains {{@Nullable}} > and {{@NotNull}} annotations to getters, setters, builders and all-arg > constructor. This will provide compile time safety for nullability when used. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-1603) maven avro plugin to also generate avsc files.
[ https://issues.apache.org/jira/browse/AVRO-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-1603: - Labels: pull-request-available (was: ) > maven avro plugin to also generate avsc files. > -- > > Key: AVRO-1603 > URL: https://issues.apache.org/jira/browse/AVRO-1603 > Project: Apache Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.8.0 >Reporter: Zoltan Farkas >Priority: Minor > Labels: pull-request-available > > It would be nice to be able to generate also all avsc schema files during > compilation. > This schema files than could be packages, versioned, distributed with maven... -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3264) Improve the Avro landing page
[ https://issues.apache.org/jira/browse/AVRO-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3264: - Labels: pull-request-available (was: ) > Improve the Avro landing page > - > > Key: AVRO-3264 > URL: https://issues.apache.org/jira/browse/AVRO-3264 > Project: Apache Avro > Issue Type: Improvement > Components: website >Reporter: Ryan Skraba >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This [mailing list > thread|https://lists.apache.org/thread/ktjh7kyd92hf0jsmpp16c3k121dc8grg] has > some good suggestions for improving the main landing page. > An improved message can also be put into packages and artifact pages like: > [https://pypi.org/project/avro/] > [https://metacpan.org/release/RSKRABA/Avro-1.11.0/source/README] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3725) fix documentation of functions and variables
[ https://issues.apache.org/jira/browse/AVRO-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3725: - Labels: pull-request-available (was: ) > fix documentation of functions and variables > > > Key: AVRO-3725 > URL: https://issues.apache.org/jira/browse/AVRO-3725 > Project: Apache Avro > Issue Type: New Feature > Components: rust >Affects Versions: 1.11.1 >Reporter: Igor Izvekov >Assignee: Igor Izvekov >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.2 > > Time Spent: 10m > Remaining Estimate: 0h > > In "avro-rs" documentation there is no description of the data type "Ref". > These changes translate "//" to "///" where the object's documentation is > present. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3724) C# JsonEncoder can't handle nested array of records
[ https://issues.apache.org/jira/browse/AVRO-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3724: - Labels: pull-request-available (was: ) > C# JsonEncoder can't handle nested array of records > --- > > Key: AVRO-3724 > URL: https://issues.apache.org/jira/browse/AVRO-3724 > Project: Apache Avro > Issue Type: Bug > Components: csharp >Affects Versions: 1.11.2 >Reporter: Andrew Benedek >Priority: Major > Labels: pull-request-available > Attachments: PAYMENT.cs, Program.cs, Transaction.cs, sample2.avsc > > Time Spent: 10m > Remaining Estimate: 0h > > The exception is: > Avro.AvroTypeException > HResult=0x80131500 > Message=Incorrect number of items written. 1 more required. > Source=Avro > StackTrace: > at Avro.IO.ParsingEncoder.Pop() in > C:\code\avro\lang\csharp\src\apache\main\IO\ParsingEncoder.cs:line 135 > at Avro.IO.JsonEncoder.WriteArrayEnd() in > C:\code\avro\lang\csharp\src\apache\main\IO\JsonEncoder.cs:line 259 > at Avro.Generic.PreresolvingDatumWriter`1.WriteArray(WriteItem itemWriter, > Object array, Encoder encoder) in > C:\code\avro\lang\csharp\src\apache\main\Generic\PreresolvingDatumWriter.cs:line > 235 > at > Avro.Generic.PreresolvingDatumWriter`1.<>c_{_}DisplayClass20_0.b{_}_0(Object > d, Encoder e) in > C:\code\avro\lang\csharp\src\apache\main\Generic\PreresolvingDatumWriter.cs:line > 225 > at Avro.Specific.SpecificDatumWriter`1.WriteRecordFields(Object recordObj, > RecordFieldWriter[] writers, Encoder encoder) in > C:\code\avro\lang\csharp\src\apache\main\Specific\SpecificDatumWriter.cs:line > 46 > at > Avro.Generic.PreresolvingDatumWriter`1.<>c_{_}DisplayClass14_0.b{_}_0(Object > v, Encoder e) in > C:\code\avro\lang\csharp\src\apache\main\Generic\PreresolvingDatumWriter.cs:line > 147 > at Avro.Generic.PreresolvingDatumWriter`1.Write(T datum, Encoder encoder) > in > C:\code\avro\lang\csharp\src\apache\main\Generic\PreresolvingDatumWriter.cs:line > 52 > at Program.$(String[] args) in > C:\code\ips-sales-replication\producer\AvroConsoleApp\Program.cs:line 210 > > I have cloned the GitHub repo and built the C# solution (Avro.dll) since > 1.11.2 with the JsonEncoder has not yet been released as a Nuget package. > A schema with a nested record works fine. > A schema with a nested array of record throws the above exception. > The BinaryEncoder however is able to produce the binary Avro with the same > schema. > I have attached the schema. > My code is a simple C# .NET console application. > Program.cs is also attached along with the schema data model that was > generated using CodeGen. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3727) Add RollForward to C# avrogen tool
[ https://issues.apache.org/jira/browse/AVRO-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3727: - Labels: pull-request-available (was: ) > Add RollForward to C# avrogen tool > -- > > Key: AVRO-3727 > URL: https://issues.apache.org/jira/browse/AVRO-3727 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Reporter: Zoltan Csizmadia >Priority: Minor > Labels: pull-request-available > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > When the avrogen C# tool is used with an SDk installed which is not supported > (built for) yet, avrogen fails with anerror that the required NET SDk is > missing. RollForward instructs the tool to look for the next major SDK > version and use that, if the required SDK is not installed, > > SInce NET 8.0 preview is out, as of now, avrogen cannot run in NET 8 without > this option. > > https://learn.microsoft.com/en-us/dotnet/core/project-sdk/msbuild-props#rollforward -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3729) Improve the Avro page
[ https://issues.apache.org/jira/browse/AVRO-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3729: - Labels: pull-request-available (was: ) > Improve the Avro page > - > > Key: AVRO-3729 > URL: https://issues.apache.org/jira/browse/AVRO-3729 > Project: Apache Avro > Issue Type: Improvement > Components: website >Reporter: Nishchal D V >Assignee: Nishchal D V >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The documentation pages are outdated and need to be updated. We also need to > add a dropdown for the project and blog sections on the landing page and > improve the overall user experience of the landing page. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3735) [Rust] Validation fails for
[ https://issues.apache.org/jira/browse/AVRO-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3735: - Labels: pull-request-available rust (was: rust) > [Rust] Validation fails for > > > Key: AVRO-3735 > URL: https://issues.apache.org/jira/browse/AVRO-3735 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: Matthew Cargille >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available, rust > Attachments: avro_example.zip > > Time Spent: 10m > Remaining Estimate: 0h > > Schema validation fails when referencing a previously defined type which has > inherited a namespace. > > Schema: > > {code:java} > { > "type": "record", > "name": "Foo", > "namespace": "name.space", > "fields": > [ > { > "name": "barInit", > "type": > { > "type": "enum", > "name": "Bar", > "symbols": > [ > "bar0", > "bar1" > ] > } > }, > { > "name": "barUse", > "type": "Bar" > } > ] > } {code} > > > Rust Objects (from rsgen) > > {code:java} > #[derive(Debug, PartialEq, Eq, Hash, PartialOrd, Ord, Clone, > serde::Deserialize, serde::Serialize,)] > pub enum Bar { > #[serde(rename = "bar0")] > Bar0, > #[serde(rename = "bar1")] > Bar1, > } > #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize, serde::Serialize)] > pub struct Foo { > #[serde(rename = "barInit")] > pub bar_init: Bar, > #[serde(rename = "barUse")] > pub bar_use: Bar, > } > {code} > > > Test > > {code:java} > #[test] > fn test_serialize() { > testing_logger::setup(); > let schema = Schema::parse_str(&get_raw_example_schema()).unwrap(); > let foo = Foo { > bar_init: Bar::Bar0, > bar_use: Bar::Bar1, > }; > let avro_value = to_value(foo).unwrap(); > println!( > "value is valid for schema: {}", > avro_value.validate(&schema) > ); > testing_logger::validate(|logs| { > for log in logs { > println!("{}", log.body) > } > }); > let mut writer = Writer::new(&schema, Vec::new()); > // schema validation happens here > writer.append(avro_value).unwrap(); > } {code} > > > Test & Log Output > > {code:java} > value is valid for schema: false > Invalid value: Record([("barInit", Enum(0, "bar0")), ("barUse", Enum(1, > "bar1"))]) for schema: Record { name: Name { name: "Foo", namespace: > Some("name.space") }, aliases: None, doc: None, fields: [RecordField { name: > "barInit", doc: None, default: None, schema: Enum { name: Name { name: "Bar", > namespace: None }, aliases: None, doc: None, symbols: ["bar0", "bar1"] }, > order: Ascending, position: 0 }, RecordField { name: "barUse", doc: None, > default: None, schema: Ref { name: Name { name: "Bar", namespace: None } }, > order: Ascending, position: 1 }], lookup: {"barInit": 0, "barUse": 1} }. > Reason: Unresolved schema reference: 'Bar'. Parsed names: [Name { name: > "Bar", namespace: Some("name.space") }, Name { name: "Foo", namespace: > Some("name.space") }] > thread 'test::test_serialize' panicked at 'called `Result::unwrap()` on an > `Err` value: ValidationWithReason("Unresolved schema reference: 'Bar'. Parsed > names: [Name { name: \"Foo\", namespace: Some(\"name.space\") }, Name { name: > \"Bar\", namespace: Some(\"name.space\") }]")', src/main.rs:33:35 {code} > > full test repo is attached > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3262) Include Avro record class name in toString
[ https://issues.apache.org/jira/browse/AVRO-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3262: - Labels: pull-request-available (was: ) > Include Avro record class name in toString > -- > > Key: AVRO-3262 > URL: https://issues.apache.org/jira/browse/AVRO-3262 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.11.0 >Reporter: Dan Newton >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Avro objects output JSON when they are `toString`'d. > This makes it difficult to understand what Avro objects are being logged when > passing the whole object into the logging string. > It would be convenient if it contained the class name of the Avro object, so > that when used in strings or logging its easier to read. > Note, this could be bypassed manually in user code by writing something like > `"logging ${avroObject::class.java.name} $avroObject"` (I'd write it in Java > but my head hurts right now). > From what I looked at, possibly changing `SpecificRecordBase.toString` to the > following would resolve the issue: > > {code:java} > @Override > public String toString() { > return this.class + " " + getSpecificData().toString(this); > }{code} > > Possibly the solution might be more elegant than that or in a slightly > different place. > The solution above assumes that the `toString` isn't being used for anything > within the Avro code. If other code depends on the fact it outputs JSON then > what I've suggested won't be suitable. > In that situation, we could add a new method that contains the class name > which can be called from user code and is still an improvement. > On the flip side, the JSON only output could be moved and called from other > Avro code, leaving the class name `toString` as the default. This would be > preferred as Avro objects contained within other objects will be output with > their class names when turned into strings, which does not happen with the > previous option. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3686) [website] Use .asf.yaml for deploying the webside and documentation.
[ https://issues.apache.org/jira/browse/AVRO-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3686: - Labels: pull-request-available (was: ) > [website] Use .asf.yaml for deploying the webside and documentation. > > > Key: AVRO-3686 > URL: https://issues.apache.org/jira/browse/AVRO-3686 > Project: Apache Avro > Issue Type: Improvement > Components: doc, website >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > I'd like to propose changing the way the site/documentation is deployed at > [https://avro.apache.org|https://avro.apache.org/]. > Currently it comes from Subversion: > [http://svn-master.apache.org/repos/asf/avro/site/publish] > I suggest to make use of .asf.yaml > ([https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-WebsitedeploymentserviceforGitrepositories]) > instead. > > With the new approach it will be possible to: > 1) deploy the docs from master to > [https://avro.staged.apache.org|https://avro.staged.apache.org/] > 2) deploy latest release, e.g. 1.11.1, to > [https://avro.apache.org|https://avro.apache.org/] > 3) deploy older versions, e.g. 1.10.0, to [https://avro.apache.org/1.10.0/] > > > We will either redirect to the old versions of the docs from Subversion or > create Git branches/tags and copy the content. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3736) [Build][Ruby] The release build fails "bundle not found"
[ https://issues.apache.org/jira/browse/AVRO-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3736: - Labels: pull-request-available (was: ) > [Build][Ruby] The release build fails "bundle not found" > > > Key: AVRO-3736 > URL: https://issues.apache.org/jira/browse/AVRO-3736 > Project: Apache Avro > Issue Type: Bug >Reporter: Ryan Skraba >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > After some reorganisation in the Dockerfile and build scripts from AVRO-2666 > and [this > commit|https://github.com/apache/avro/commit/cf7bfe7710febdc84e70fcc8ad21d859dbf1be1f], > it looks like the ubertool docker can no longer be used to build the Ruby > Avro SDK: > {code} > # In the main or branch-1.11 directory > ./build.sh docker > # Inside the container > ./build.sh clean > > + cd lang/ruby > + ./build.sh clean > ./build.sh: line 27: bundle: command not found > {code} > This is a bit subtle, because if the `lang/ruby/.gem/` directory exists from > a previous build, it might still work. A clean build will fail. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3741) Note about the version requirement of Rust in BUILD.md
[ https://issues.apache.org/jira/browse/AVRO-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3741: - Labels: pull-request-available (was: ) > Note about the version requirement of Rust in BUILD.md > -- > > Key: AVRO-3741 > URL: https://issues.apache.org/jira/browse/AVRO-3741 > Project: Apache Avro > Issue Type: Improvement > Components: doc >Affects Versions: 1.12.0 >Reporter: Kousuke Saruta >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The requirement version of Rust is not noted in BUILD.md. > It's better to note like other languages. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3742) Bump maven-plugin-plugin from 3.8.1 to 3.8.2
[ https://issues.apache.org/jira/browse/AVRO-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3742: - Labels: pull-request-available (was: ) > Bump maven-plugin-plugin from 3.8.1 to 3.8.2 > > > Key: AVRO-3742 > URL: https://issues.apache.org/jira/browse/AVRO-3742 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.11.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3744) Bump maven-checkstyle-plugin from 3.2.1 to 3.2.2
[ https://issues.apache.org/jira/browse/AVRO-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3744: - Labels: pull-request-available (was: ) > Bump maven-checkstyle-plugin from 3.2.1 to 3.2.2 > > > Key: AVRO-3744 > URL: https://issues.apache.org/jira/browse/AVRO-3744 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.11.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3745) Bump zstd-jni from 1.5.4-2 to 1.5.5-2
[ https://issues.apache.org/jira/browse/AVRO-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3745: - Labels: pull-request-available (was: ) > Bump zstd-jni from 1.5.4-2 to 1.5.5-2 > - > > Key: AVRO-3745 > URL: https://issues.apache.org/jira/browse/AVRO-3745 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.11.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3746) Bump grpc.version from 1.54.0 to 1.54.1
[ https://issues.apache.org/jira/browse/AVRO-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3746: - Labels: pull-request-available (was: ) > Bump grpc.version from 1.54.0 to 1.54.1 > --- > > Key: AVRO-3746 > URL: https://issues.apache.org/jira/browse/AVRO-3746 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.11.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3743) Bump cyclonedx-maven-plugin from 2.7.6 to 2.7.7
[ https://issues.apache.org/jira/browse/AVRO-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3743: - Labels: pull-request-available (was: ) > Bump cyclonedx-maven-plugin from 2.7.6 to 2.7.7 > --- > > Key: AVRO-3743 > URL: https://issues.apache.org/jira/browse/AVRO-3743 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.11.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3747) Set `is_human_readable` to `false` for serde serialization
[ https://issues.apache.org/jira/browse/AVRO-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3747: - Labels: pull-request-available (was: ) > Set `is_human_readable` to `false` for serde serialization > -- > > Key: AVRO-3747 > URL: https://issues.apache.org/jira/browse/AVRO-3747 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: Cetra Free >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Serde's `Serializer` trait has a type hint for dictating when a format will > be "human readable" or not. This allows implementers of the `Serialize` > trait to define whether they want a compact representation of data. > I need this change so that I can ensure timestamps are either rfc 3389 string > or millisecond integers depending on type. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3748) issue with DataFileSeekableInput.SeekableInputStream.skip
[ https://issues.apache.org/jira/browse/AVRO-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3748: - Labels: pull-request-available (was: ) > issue with DataFileSeekableInput.SeekableInputStream.skip > - > > Key: AVRO-3748 > URL: https://issues.apache.org/jira/browse/AVRO-3748 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.11.1 >Reporter: Steven Aerts >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We found a longstanding bug in the implementation of > {{DataFileSeekableInput.SeekableInputStream.skip.}} > This skip function is not hit that often. It can for example be hit when the > FastReader is enabled and it tries to skip a significant amount of data. > The implmentation of this function is however fault and can result in data > corruption or > {{{}java.io.EOFException{}}}, as instead of skipping the number of bytes, it > will seek to a wrong place in the file. > > We have a pull request ready to fix and test this issue. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3737) [C] memcheck_test_avro_commons_schema is failing
[ https://issues.apache.org/jira/browse/AVRO-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3737: - Labels: pull-request-available (was: ) > [C] memcheck_test_avro_commons_schema is failing > > > Key: AVRO-3737 > URL: https://issues.apache.org/jira/browse/AVRO-3737 > Project: Apache Avro > Issue Type: Bug >Affects Versions: 1.11.1, 1.12.0 >Reporter: Ryan Skraba >Assignee: Christophe Le Saec >Priority: Critical > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Even with the fix in AVRO-3454, one of the memcheck test is failing in the > release docker: > *{{cd lang/c && ./build.sh test}}* > {code} > ... (snip) ... > 98% tests passed, 1 tests failed out of 48 > Total Test time (real) = 20.64 sec > The following tests FAILED: > 7 - memcheck_test_avro_commons_schema (Failed) > Errors while running CTest > make: *** [Makefile:85: test] Error 8 > {code} > WIth a bit more detail in the logs: > *{{tail -n 25 ../../build/c/tests/memcheck_test_avro_commons_schema.log}}* > {code} > ==1209==by 0x10F97C: create_writer (test_avro_commons_schema.c:70) > ==1209==by 0x10F97C: read_data (test_avro_commons_schema.c:92) > ==1209==by 0x10F97C: run_tests (test_avro_commons_schema.c:113) > ==1209==by 0x10F97C: main (test_avro_commons_schema.c:139) > ==1209== > ==1209== 172,133 (131,232 direct, 40,901 indirect) bytes in 2 blocks are > definitely lost in loss record 80 of 80 > ==1209==at 0x483B723: malloc (in > /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) > ==1209==by 0x483E017: realloc (in > /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) > ==1209==by 0x11032F: avro_file_writer_create_with_codec_fp > (datafile.c:211) > ==1209==by 0x11032F: avro_file_writer_create_with_codec_fp > (datafile.c:196) > ==1209==by 0x1107A5: avro_file_writer_create (datafile.c:179) > ==1209==by 0x10F97C: create_writer (test_avro_commons_schema.c:70) > ==1209==by 0x10F97C: read_data (test_avro_commons_schema.c:92) > ==1209==by 0x10F97C: run_tests (test_avro_commons_schema.c:113) > ==1209==by 0x10F97C: main (test_avro_commons_schema.c:139) > ==1209== > ==1209== LEAK SUMMARY: > ==1209==definitely lost: 131,392 bytes in 4 blocks > ==1209==indirectly lost: 57,315 bytes in 146 blocks > ==1209== possibly lost: 0 bytes in 0 blocks > ==1209==still reachable: 1,888 bytes in 4 blocks > ==1209== suppressed: 0 bytes in 0 blocks > ==1209== > ==1209== For lists of detected and suppressed errors, rerun with: -s > ==1209== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0) > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3756) Support writing types back to the user in memory without writing files to disk
[ https://issues.apache.org/jira/browse/AVRO-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3756: - Labels: pull-request-available (was: ) > Support writing types back to the user in memory without writing files to disk > -- > > Key: AVRO-3756 > URL: https://issues.apache.org/jira/browse/AVRO-3756 > Project: Apache Avro > Issue Type: Bug > Components: csharp >Reporter: Alexander Rosenfeld >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > As per the title, Avro C# should support writing types directly back to the > user, instead of [forcing the user to write to > disk|https://github.com/apache/avro/blob/master/lang/csharp/src/apache/main/CodeGen/CodeGen.cs#L1144]. > This is critical for use in scenarios where the code generation is done in a > CI job, where disk permissions might not be flexible. > > I already have code for this and intend to submit a patch along with this > ticket. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3755) [Rust] Deserialization fails for reader schema with namespace
[ https://issues.apache.org/jira/browse/AVRO-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3755: - Labels: pull-request-available (was: ) > [Rust] Deserialization fails for reader schema with namespace > -- > > Key: AVRO-3755 > URL: https://issues.apache.org/jira/browse/AVRO-3755 > Project: Apache Avro > Issue Type: Bug > Components: rust >Reporter: Matthew Cargille >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Deserializing an avro value fails when using a reader schema with a namespace > in avro > 0.14.0. Probably related to AVRO-3735. May be working in 0.15.0 but it's an > issue that's impacting our code and we'd like to double-check that a test > case will cover it. > > Structs (from rs-gen) > {code:java} > #[derive( > Debug, PartialEq, Eq, Hash, PartialOrd, Ord, Clone, serde::Deserialize, > serde::Serialize, > )] > pub enum Bar { > #[serde(rename = "bar0")] > Bar0, > #[serde(rename = "bar1")] > Bar1, > #[serde(rename = "bar2")] > Bar2, > } > #[derive(Debug, PartialEq, Eq, Clone, serde::Deserialize, serde::Serialize)] > pub struct Foo { > #[serde(rename = "barInit")] > pub bar_init: Bar, > #[serde(rename = "barUse")] > pub bar_use: Bar, > } {code} > > Writer schema (serializes successfully) > {code:java} > fn get_raw_example_schema() -> String { > r#"{ > "type": "record", > "name": "Foo", > "fields": > [ > { > "name": "barInit", > "type": > { > "type": "enum", > "name": "Bar", > "symbols": > [ > "bar0", > "bar1" > ] > } > }, > { > "name": "barUse", > "type": "Bar" > } > ] > }"# > .to_string() > } {code} > Reader Schema > {code:java} > fn get_raw_example_schema_v2() -> String { > r#"{ > "type": "record", > "name": "Foo", > "namespace": "name.space", > "fields": > [ > { > "name": "barInit", > "type": > { > "type": "enum", > "name": "Bar", > "symbols": > [ > "bar0", > "bar1", > "bar2" > ] > } > }, > { > "name": "barUse", > "type": "Bar" > } > ] > }"# > .to_string() > } {code} > Test code: > {code:java} > #[test] > fn test_deserialize() { > testing_logger::setup(); > let schema = Schema::parse_str(&get_raw_example_schema()).unwrap(); > let foo = Foo { > bar_init: Bar::Bar0, > bar_use: Bar::Bar1, > }; let avro_value = to_value(foo).unwrap(); > println!( > "value is valid for schema: {}", > avro_value.validate(&schema) > ); let datum = to_avro_datum(&schema, avro_value).unwrap(); > let mut x = &datum[..]; > let read_schema = > Schema::parse_str(&get_raw_example_schema_v2()).unwrap(); > let deser_value = from_avro_datum(&schema, &mut x, > Some(&read_schema)).unwrap(); > testing_logger::validate(|logs| { > for log in logs { > println!("{}", log.body) > } > }); > } {code} > > error > {code:java} > panicked at 'called `Result::unwrap()` on an `Err` value: > SchemaResolutionError(Name { name: "Bar", namespace: None }) {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3757) [rust] Update syn to 2.x
[ https://issues.apache.org/jira/browse/AVRO-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3757: - Labels: pull-request-available (was: ) > [rust] Update syn to 2.x > > > Key: AVRO-3757 > URL: https://issues.apache.org/jira/browse/AVRO-3757 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Darling 0.20.x has been released - > https://github.com/TedDriggs/darling/pull/226. > Now we can update the syn dependency from 1.x to 2.x > (https://github.com/apache/avro/pull/2158) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3758) [Rust] Use AtomicXyz types instead of static mutable ones
[ https://issues.apache.org/jira/browse/AVRO-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3758: - Labels: pull-request-available (was: ) > [Rust] Use AtomicXyz types instead of static mutable ones > - > > Key: AVRO-3758 > URL: https://issues.apache.org/jira/browse/AVRO-3758 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > apache_avro::util uses "static mut" for MAX_ALLOCATION_BYTES and > SERDE_HUMAN_READABLE. Because of this it has to use "unsafe {}" blocks to > update their values. > > It would be cleaner to use AtomicBool/AtomicUsize instead and get rid of the > unsafe blocks. > > Suggested at https://github.com/apache/avro/pull/2202#issuecomment-1542887124 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3764) [Rust] Add schemata-based resolve method
[ https://issues.apache.org/jira/browse/AVRO-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3764: - Labels: pull-request-available (was: ) > [Rust] Add schemata-based resolve method > > > Key: AVRO-3764 > URL: https://issues.apache.org/jira/browse/AVRO-3764 > Project: Apache Avro > Issue Type: New Feature > Components: rust >Reporter: Fedor Telnov >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There is existing validate method on apache_avro::Value that does validation > based on schemata. It would be good if we add resolve method based on > schemata. > I provided a draft - maybe it is not intended to work this way. Correct me if > it is the case. > Draft pull request: https://github.com/apache/avro/pull/2262 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3766) [Rust] Print fliendlier errors when test cases fail
[ https://issues.apache.org/jira/browse/AVRO-3766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3766: - Labels: pull-request-available (was: ) > [Rust] Print fliendlier errors when test cases fail > --- > > Key: AVRO-3766 > URL: https://issues.apache.org/jira/browse/AVRO-3766 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > [https://bluxte.net/musings/2023/01/08/improving_failure_messages_rust_tests/] > > The idea is to use #[track_caller] attribute on a method that panics > immediately. > This way the "useless" frames from the unwinding are not printed. > An extra benefit is the removal of calls to Result::unwrap() in the tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3770) SpecificDefaultReader and PreresolvingDatumReader don't handle aliases properly
[ https://issues.apache.org/jira/browse/AVRO-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AVRO-3770: - Labels: pull-request-available (was: ) > SpecificDefaultReader and PreresolvingDatumReader don't handle aliases > properly > --- > > Key: AVRO-3770 > URL: https://issues.apache.org/jira/browse/AVRO-3770 > Project: Apache Avro > Issue Type: Bug > Components: csharp >Affects Versions: 1.11.1 >Reporter: Morgan Parry >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Both of these classes will read from an aliased value correctly but then will > go on to try and insert a default value reader, because they're also still > looking for the new name in the writer schema. > For example, with a field like > {code:json} > { > "name": "i", > "type": "int", > "aliases": "j" > } > {code} > and some old serialised data that was still writing the field {{j}} instead, > the readers will read that {{j}} value into {{i}} (as expected) but then also > add some fallback handling for the 'missing' {{i}} field. > This is due to the following loop in {{PreresolvingDatumReader}}, for example: > {code:c#} > foreach (Field rf in readerSchema) > { > if (writerSchema.Contains(rf.Name)) continue; > {code} > That condition isn't sufficient in the case of aliasing. > I believe it's also the case that {{SpecificDefaultReader}} doesn't handle > aliases at all, since it uses {{TryGetField()}} rather than > {{TryGetFieldAlias()}}. -- This message was sent by Atlassian Jira (v8.20.10#820010)