[ https://issues.apache.org/jira/browse/NIFI-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355610#comment-16355610 ]
ASF subversion and git services commented on NIFI-4844: ------------------------------------------------------- Commit 2b062e211f36a0833c273e1a552d42723fdf54fe in nifi's branch refs/heads/master from [~ijokarumawak] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=2b062e2 ] NIFI-4844: Adjust BigDecimal scale to the target Avro schema - Applied the same scale adjustment not only to BigDecimal inputs, but also to Double values. Signed-off-by: Matthew Burgess <mattyb...@apache.org> This closes #2450 > AvroRecordSetWriter should be able to convert a double having less scale than > intended target Avro schema instead of throwing an AvroTypeException > -------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: NIFI-4844 > URL: https://issues.apache.org/jira/browse/NIFI-4844 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Affects Versions: 1.3.0 > Reporter: Koji Kawamura > Assignee: Koji Kawamura > Priority: Major > > Current AvroTypeUtil conversion logic can throw AvroTypeException when it > maps double values into Avro decimal logical type fields if the double value > has less scale than the one defined at the target Avro decimal field schema. > For example, with following schema: > {code} > { > "type": "record", > "name": "logicalDecimalTest", > "fields": [ > {"name": "id", "type": "int"}, > {"name": "name", "type": "string"}, > { > "name": "price", > "type": { > "type": "bytes", > "logicalType": "decimal", > "precision": 18, > "scale": 8 > }}]} > {code} > And following CSV records: > {code} > id|name|price > 1|one|1.23 > 2|two|2.34 > {code} > Would produce this Exception: > {code} > 2018-02-06 09:57:27,461 ERROR [Timer-Driven Process Thread-7] > o.a.n.processors.standard.ConvertRecord > ConvertRecord[id=6897bc30-0161-1000-a8e7-9ce0ce8eb9ae] Failed to process > StandardFlowFileRecord[uuid=a97366a0-79bb-42ff-9023-c5d62ecfdbc5,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1517878123416-2, container=default, > section=2], offset=5, length=48],offset=0,name=220105646548465,size=48]; will > route to failure: org.apache.avro.AvroTypeException: Cannot encode decimal > with scale 17 as scale 8 > org.apache.avro.AvroTypeException: Cannot encode decimal with scale 17 as > scale 8 > at > org.apache.avro.Conversions$DecimalConversion.toBytes(Conversions.java:86) > at > org.apache.nifi.avro.AvroTypeUtil.convertToAvroObject(AvroTypeUtil.java:546) > at > org.apache.nifi.avro.AvroTypeUtil.createAvroRecord(AvroTypeUtil.java:457) > at > org.apache.nifi.avro.WriteAvroResultWithExternalSchema.writeRecord(WriteAvroResultWithExternalSchema.java:76) > at > org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59) > at > org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:122) > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2827) > at > org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:109) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122) > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} > The same issue is reported in the Avro project, > [AVRO-1864|https://issues.apache.org/jira/browse/AVRO-1864]. The recommended > approach is to adjust the scale at NiFi side. Actually, for BigDecimal input > values, NiFi already does this, but not with double values. AvroTypeUtil > should do the same scale adjustment for double values, too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)