Hi, thank you for this hint. I’ve done some checks and indeed we’re using schema inference in the sql processor which loads the data into NiFi. But the schema inference derives the correct data type. It uses long and the same pipeline processes values that are larger that the upper value for integer.
This is the schema that is inferred for the failing record. I tried it with
plain json and avro connected to the schema registry. The schema registry also
provides long as data type. Unfortunately it fails again.
avro.schemaæ{"type":"record","name":"nifiRecord","namespace":"org.apache.nifi","fields":[{"name":"_hdl_loadstamp","type":["string","null"]},{"name":"Id","type":["string","null"]},{"name":"NumericId","type":"long"},….
Did I miss something else?
From: Pierre Villard <[email protected]>
Date: Thursday, 11. July 2024 at 12:09
To: [email protected] <[email protected]>
Subject: Re: PublishKafkaRecord_2_6 produces negative partition number
Achtung! Die Absender-Adresse ist möglicherweise gefälscht oder wurde
nachträglich manipuliert. Bitte überprüfen Sie die Plausibilität der Email und
lassen bei enthaltenen Anhängen und Links besondere Vorsicht walten.
Wenden Sie sich im Zweifelsfall an das CIT unter [email protected] oder 06122 536 350.
(Stichwort: DKIM Test Fehlgeschlagen)
----------------------------------------------------------------------------------------------------------------
Hi,
Upper value for int type is 2147483647 and your field value is 2147483648 and I
believe that's the reason why you get this error. You may want to check the
schema you're using for your data in the configuration of the record reader. If
you're using schema inference, that may be the reason.
HTH,
Pierre
Le jeu. 11 juil. 2024 à 16:37, Richter, Robert
<[email protected]<mailto:[email protected]>> a écrit :
Hello NiFi Community,
we’ve encountered an error in one of our pipelines. Exactly one record cannot
be processed by the PublishKafkaRecord_2_6 processor. The Processor throws the
following exception.
java.lang.IllegalArgumentException: Invalid partition: -8. Partition number
should always be non-negative or null.
The processor is configured as follows and we’re on NiFi version 1.23.2. The
topic we’re publishing on has 20 partitions:
* partitioner class: record path partitioner
* partition: /NumericId
* Message Key Field: NumericId
The processor fails only for records with value 2147483648 in the message key
field. All other records can be processed without failure. (millions per day)
We tried different reader and writer. Avro with and without schema registry and
plain json. The result is the same.
Here’s an example record.
[ {
"_loadstamp" : "20240711075814350",
"Id" : "04948E4E-94B1-84HD-94CA-AB84020F729",
"NumericId" : 2147483648,
"TimeStamp" : "2024-06-12 15:30:58.341 +02:00",
"SequenceNumber" : 123456789,
"IsTrendUploaded" : false,
"CompoundValueId" : null,
"Value" : 1.0,
"Score" : 65535,
"ScoreColor" : 0,
"CompositeId" : null
} ]
Kind regard
Robert
Das SVA Mail-System ist mit einem Mailverschluesselungs-Gateway ausgestattet.
Wenn Sie moechten, dass an Sie gerichtete E-Mails verschluesselt werden, senden
Sie einfach eine S/MIME-signierte E-Mail oder Ihren PGP Public Key an
[email protected]<mailto:[email protected]>.
The SVA mail system is equipped with an email encryption gateway. If you want
email sent to you to be encrypted please send a S/MIME signed email or your PGP
public key to [email protected]<mailto:[email protected]>.
smime.p7s
Description: S/MIME cryptographic signature
