Hi Nandor,
It's not the numeric portion that is the problem for me, but the $date
field name. Mongo apparently requires the structure I provided in the
example, and whenever I use $date as the field name the Java Avro API
throws an exception about an invalid character in the field definition.
The logical type thing is good to know for future reference.
I admit that this is likely a really uncommon edge case for Avro. The work
around I found for defining a schema that is at least compatible with the
Mongo Extended JSON requirements was to do this (one field example):
{
"namespace": "test",
"name": "PutTestRecord",
"type": "record",
"fields": [{
"name": "timestampField",
"type": {
"type": "map",
"values": "long"
}
}]
}
It doesn't give you the full validation that would be ideal if we could
define a field with the name "$date," but it's an 80% solution that works
with NiFi and other tools that have to generate Extended JSON for Mongo.
Thanks,
Mike
On Wed, Oct 25, 2017 at 4:48 AM, Nandor Kollar <nkol...@cloudera.com> wrote:
> Hi Mike,
>
> This JSON doesn't seems like a valid Avro schema
> <https://avro.apache.org/docs/1.8.1/spec.html#schemas>. If you'd like to
> use timestamps in your schema, you should use Timestamp logical types,
> <https://avro.apache.org/docs/1.8.1/spec.html#Timestamp+%28millisecond+precision%29>
> which annotate Avro longs. In this case the schema of this field should
> look like this:
>
> {
> "name":"timestamp",
> "type":"long",
> "logicalType":"timestamp-millis"
> }
>
> If you'd like to create Avro files with this schema, there's on Avro wiki
> you can find a brief tutorial
> <https://avro.apache.org/docs/1.8.1/gettingstartedjava.html#Compiling+the+schema>
> how to create and write Avro files with this schema in Java.
>
> Regards,
> Nandor
>
> On Tue, Oct 24, 2017 at 8:18 PM, Mike Thomsen <mikerthom...@gmail.com>
> wrote:
>
>> I am trying to build an avro schema for a NiFi flow that is going to
>> insert data into Mongo, and Mongo extended JSON requires the use of $
>> characters in cases like this (to represent a date):
>>
>> {
>> "timestamp": {
>> "$date": TIMESTAMP_LONG_HERE
>> }
>> }
>>
>> I tried building a schema with that, and it failed saying there was an
>> invalid character in the schema. just wanted to check and see if there was
>> a work around for this or if I'll have to choose another option.
>>
>> Thanks,
>>
>> Mike
>>
>
>
