[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547253#comment-16547253 ] ASF GitHub Bot commented on AVRO-2199: -- tjwp commented on a change in pull request #320: AVRO-2199: Validate that field defaults have the correct type URL: https://github.com/apache/avro/pull/320#discussion_r203221105 ## File path: lang/ruby/test/test_default_validation.rb ## @@ -0,0 +1,127 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +require 'test_help' + +class TestDefaultValidation < Test::Unit::TestCase Review comment: I separate these tests out for the avro-patches gem, but it may make sense to move them into `test_schema.rb` here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547252#comment-16547252 ] ASF GitHub Bot commented on AVRO-2199: -- tjwp commented on a change in pull request #320: AVRO-2199: Validate that field defaults have the correct type URL: https://github.com/apache/avro/pull/320#discussion_r203220999 ## File path: lang/ruby/lib/avro/schema.rb ## @@ -373,6 +374,18 @@ def to_avro(names=Set.new) avro['doc'] = doc if doc end end + + def validate_default! Review comment: Should this method be made private? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1927) If a default value is set, Avro allows null values in non-nullable fields.
[
https://issues.apache.org/jira/browse/AVRO-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547243#comment-16547243
]
Ryan Blue commented on AVRO-1927:
-
Sounds like the problem here is actually that there is no validation in the
avro-specific builders or in object creation. Avro is doing the right thing and
not allowing you to serialize invalid data.
I think it makes sense to add null checks to the generated builders. Feel free
to fix this and open a PR!
> If a default value is set, Avro allows null values in non-nullable fields.
> --
>
> Key: AVRO-1927
> URL: https://issues.apache.org/jira/browse/AVRO-1927
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
>Reporter: Andreas Maier
>Priority: Major
>
> With an avro schema like
> {code}
> {
> "name": "myfield",
> "type": "string",
> "default": ""
> }
> {code}
> the following code should throw an exception
> {code}
> MyObject myObject = MyObject.newBuilder().setMyfield(null).build();
> {code}
> But instead the value of myfield is set to null, which causes an exception
> later when serializing myObject, because null is not a valid value for
> myfield.
> I believe in this case setMyfield(null) should throw an exception,
> independent of the value of default.
> See also
> https://stackoverflow.com/questions/38509279/generated-avro-builder-set-null-doesnt-overwrite-with-default
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2200) Option to fail when extra fields are in the payload
[ https://issues.apache.org/jira/browse/AVRO-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546810#comment-16546810 ] ASF GitHub Bot commented on AVRO-2200: -- dorner opened a new pull request #321: AVRO-2200: Option to fail when extra fields are in the payload URL: https://github.com/apache/avro/pull/321 See https://issues.apache.org/jira/browse/AVRO-2200 Currently, when passing a datum into the schema validator, the datum will pass even if the datum has more fields than the schema does. This can be dangerous because the client code is assuming that the schema has these values, but they are actually being silently swallowed when they are encoded. This PR provides an option to make the validator fail on any extra fields. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Option to fail when extra fields are in the payload > --- > > Key: AVRO-2200 > URL: https://issues.apache.org/jira/browse/AVRO-2200 > Project: Avro > Issue Type: Improvement > Components: ruby >Reporter: Daniel Orner >Priority: Minor > > Currently, when passing a datum into the schema validator, the datum will > pass even if the datum has more fields than the schema does. This can be > dangerous because the client code is assuming that the schema has these > values, but they are actually being silently swallowed when they are encoded. > I'd like to open a PR where we can provide an option to make the validator > fail on any extra fields. See https://github.com/salsify/avro-patches/pull/15 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546795#comment-16546795 ] ASF GitHub Bot commented on AVRO-2199: -- dorner opened a new pull request #320: AVRO-2199: Validate that field defaults have the correct type URL: https://github.com/apache/avro/pull/320 See https://issues.apache.org/jira/browse/AVRO-2199 Currently, on the master branch, when a schema is parsed, it is possible to define a field with a type and a default of a totally different type. E.g. if the field has type "string", the default can be set to "null". This PR fixes this by running the default through the SchemaValidator whenever a new Field is created. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2198) BigDecimal (logical type=Decimal / type = bytes) GenericRecord to SpecificRecord Conversion Issue
[
https://issues.apache.org/jira/browse/AVRO-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ABourg updated AVRO-2198:
-
Duplicate of *https://issues.apache.org/jira/browse/AVRO-2087*
> BigDecimal (logical type=Decimal / type = bytes) GenericRecord to
> SpecificRecord Conversion Issue
> -
>
> Key: AVRO-2198
> URL: https://issues.apache.org/jira/browse/AVRO-2198
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
> Environment: !image-2018-07-12-13-02-20-961.png!
>Reporter: ABourg
>Priority: Major
> Fix For: 1.8.2
>
>
> There seems to be an issue with the conversion process from a byte array to a
> BigDecimal when converting to a SpecificRecord from a GenericRecord object.
> Below is a simple avro definition with "amount" defined as logical type
> *decimal* and type *bytes*. The avroData specific class has been generated
> with enablebigdecimal = true.
> See below example.
> An amount value of *2000.11* is converted to BigDecimal value of
> *606738534879530359915932.65*
> {code:java}
> String schema =
> "{\"type\":\"record\",\"name\":\"avroTrans\",\"namespace\":\"com.demo.KafkaStream\",\"fields\":[{\"name\":\"amount\",\"type\":{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":5,\"scale\":2}}]}";
> String json = "{\"amount\": \"2000.11\"}";
> Schema avroSchema = new Schema.Parser().parse(schema);
> GenericRecord obj = Utils.jsonToAvro(json, avroSchema);
> System.out.println("GenericRecord Object Value ->" + obj);
> GenericDatumWriter writer = new
> GenericDatumWriter(avroTrans.getClassSchema());
> ByteArrayOutputStream out = new ByteArrayOutputStream();
> Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
> writer.write(obj, encoder);
> encoder.flush();
> byte[] avroData = out.toByteArray();
> out.close();
> SpecificDatumReader reader2 = new
> SpecificDatumReader(avroTrans.class);
> Decoder decoder2 = DecoderFactory.get().binaryDecoder(avroData, null);
> avroTrans customRecord = reader2.read(null, decoder2);
> System.out.println("SpecificRecord Object Value -> " + customRecord);
> {code}
> *Output:*
> GenericRecord Object Value ->\{"amount": {"bytes": "2000.11"}}
> SpecificRecord Object Value -> \{"amount": 606738534879530359915932.65}
> Within *org.apache.avro.Conversion* there is a fromBytes conversion method
> which takes a bytebuffer as input (see below).
>
> {code:java}
> @Override
> public BigDecimal fromBytes(ByteBuffer value, Schema schema, LogicalType
> type) {
> int scale = ((LogicalTypes.Decimal) type).getScale();
> // always copy the bytes out because BigInteger has no offset/length
> ctor
> byte[] bytes = new byte[value.remaining()];
> value.get(bytes);
> return new BigDecimal(new BigInteger(bytes), scale);
> }
> {code}
> The BigInteger constructor (bytes) used version is to "*_Translates a byte
> array containing the two's-complement binary representation of a BigInteger
> into a BigInteger_*."
> Could use of the BigInteger(Bytes) causing the incorrect conversion to huge
> number?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
