Avro logical type

2018-09-04 Thread Indhumathi M
Hello all,
I am working on avro logical type decimal, but was not sure what data
has to be provided.
As mentioned in the Avro document,
A decimal logical type annotates Avro bytes or fixed types. The byte
array must contain the two's-complement representation of the unscaled
integer value in big-endian byte order. The scale is fixed, and is
specified using an attribute.

But in code, in Conversions class, I can see that, from a decimal
value, it is forming a ByteArray.

@Override
public BigDecimal fromBytes(ByteBuffer value, Schema schema, LogicalType type) {
  int scale = ((LogicalTypes.Decimal) type).getScale();
  // always copy the bytes out because BigInteger has no offset/length ctor
  byte[] bytes = value.get(new byte[value.remaining()]).array();
  return new BigDecimal(new BigInteger(bytes), scale);
}

@Override
public ByteBuffer toBytes(BigDecimal value, Schema schema, LogicalType type) {
  int scale = ((LogicalTypes.Decimal) type).getScale();
  if (scale != value.scale()) {
throw new AvroTypeException("Cannot encode decimal with scale " +
value.scale() + " as scale " + scale);
  }
  return ByteBuffer.wrap(value.unscaledValue().toByteArray());
}

but, I find that, nowhere two's-complement representation of the
unscaled integer
value in big-endian byte order is stored in byte array.

Can anyone help me with this?

Thanks,

Indhumathi M


Avro Schema with Large Number of fields AVRO-1642

2018-09-04 Thread chinchu chinchu
Hi All,
We have an avro schema that has around 2k fields .When we do not use a
nested structure(nested record types)  we run into  jvm specification
issues related to method size and number of parameters .My preference is to
keep the avro schema completely flattened without any nested types .How
ever the avro maven plugin generates classes that violates the jvm spec.Is
there a way to over come this scenario so truely flattened avro schemas can
be used.

Thanks,
Chinchu


Can we have Avro-1852 in master or next release?

2018-09-04 Thread kant kodali
Hi All,

Can we have https://issues.apache.org/jira/browse/AVRO-1852 in master or as
part of any upcoming release? It turns out this is extremely useful and
necessary when used with Spark or Hadoop.

Thanks,
Kant