Good question -- no, this must be an artifact of running InferAvroSchema before the QueryRecord processor. InferAvroSchema infers the schema and places it in a flow file attribute, which the CSVReader uses in its configuration.
I just debugged nifi and found that it thinks one of my fields is null. There is indeed a record with an empty string in the csv for this field, but the same thing happens every day when this flow runs, so I suspect something must have changed in the implementation of either CSVReader or InferAvroSchema that made the parsing stricter. I'm going to try to figure out how to allow blank values in the CSV. On Mon, Oct 2, 2017 at 4:29 PM, Mark Payne <[email protected]> wrote: > Joe, > > Thanks. I was going to suggest we spawn off a new thread if this continued > with more back-and-forth. > > I did notice that in the JIRA you indicated that > you are using a CSVReader service to read the data. In the message, it > indicates that the FlowFIle's > filename is > "billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro" > -- which would imply > to me that it is Avro (perhaps converted from CSV to avro upstream?) Is it > possible that you just selected > the wrong reader? > > Thanks > -Mark > > > On Oct 2, 2017, at 12:24 PM, Joe Gresock <[email protected]> wrote: > > > > I'm changing the subject name so as not to muddy the [VOTE] thread. > > > > I finally found the error in the logs. This was also not happening prior > > to the upgrade. I thinkI'll debug the server to determine the root > cause: > > > > 2017-10-02 16:09:47,004 ERROR [Timer-Driven Process Thread-81] > > o.a.nifi.processors.standard.QueryRecord > > QueryRecord[id=21570e18-015c-1000-0000-000058832883] Unable to query > > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim= > StandardContentClaim > > [resourceClaim=StandardResourceClaim[id=1506960462126-919, > > container=default, section=919], offset=55116, > > length=616757],offset=0,name=billing-detailed-line-items- > with-resources-and-tags-2017-10.csv.avro,size=616757] > > due to org.apache.nifi.processor.exception.ProcessException: Failed to > read > > next record in stream for > > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim= > StandardContentClaim > > [resourceClaim=StandardResourceClaim[id=1506960462126-919, > > container=default, section=919], offset=55116, > > length=616757],offset=0,name=billing-detailed-line-items- > with-resources-and-tags-2017-10.csv.avro,size=616757]: > > {} > > org.apache.nifi.processor.exception.ProcessException: Failed to read > next > > record in stream for > > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim= > StandardContentClaim > > [resourceClaim=StandardResourceClaim[id=1506960462126-919, > > container=default, section=919], offset=55116, > > length=616757],offset=0,name=billing-detailed-line-items- > with-resources-and-tags-2017-10.csv.avro,size=616757] > > at > > org.apache.nifi.queryrecord.FlowFileEnumerator.moveNext( > FlowFileEnumerator.java:65) > > at Baz$1$1.moveNext(Unknown Source) > > at > > org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.< > init>(Linq4j.java:664) > > at > > org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98) > > at > > org.apache.calcite.linq4j.AbstractEnumerable.iterator( > AbstractEnumerable.java:33) > > at > > org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:89) > > at > > org.apache.calcite.avatica.AvaticaResultSet.execute( > AvaticaResultSet.java:196) > > at > > org.apache.calcite.jdbc.CalciteResultSet.execute( > CalciteResultSet.java:67) > > at > > org.apache.calcite.jdbc.CalciteResultSet.execute( > CalciteResultSet.java:44) > > at > > org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal( > AvaticaConnection.java:513) > > at > > org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery( > AvaticaPreparedStatement.java:132) > > at > > org.apache.nifi.processors.standard.QueryRecord. > queryWithCache(QueryRecord.java:445) > > at > > org.apache.nifi.processors.standard.QueryRecord. > onTrigger(QueryRecord.java:276) > > at > > org.apache.nifi.processor.AbstractProcessor.onTrigger( > AbstractProcessor.java:27) > > at > > org.apache.nifi.controller.StandardProcessorNode.onTrigger( > StandardProcessorNode.java:1119) > > at > > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call( > ContinuallyRunProcessorTask.java:147) > > at > > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call( > ContinuallyRunProcessorTask.java:47) > > at > > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run( > TimerDrivenSchedulingAgent.java:128) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.runAndReset( > FutureTask.java:308) > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:748) > > > > > > On Mon, Oct 2, 2017 at 4:03 PM, Mark Payne <[email protected]> wrote: > > > >> Hey Joe, > >> > >> I've seen this error crop up a time or two before. The issue is a bug in > >> how we are handling a failure > >> condition. Haven't yet figured out the best way to tackle it, but the > >> thing to note is that this error occurs > >> when there is some other sort of failure -- typically, a failure to > obtain > >> the schema for the Record Reader > >> or Record Writer. Can you check your logs for any other errors that may > be > >> occurring? > >> > >> Thanks > >> -Mark > >> > >> > >>> On Oct 2, 2017, at 11:32 AM, Joe Gresock <[email protected]> wrote: > >>> > >>> I have to change my non-binding +1 vote to a 0, unless there is a > >>> workaround for the bug [1] I just submitted, regarding the QueryRecord > >>> processor being broken. > >>> > >>> > >>> [1] https://issues.apache.org/jira/browse/NIFI-4452 > >>> > >>> > >>> On Mon, Oct 2, 2017 at 3:23 AM, Kevin Doran <[email protected]> > >> wrote: > >>> > >>>> +1, non-binding > >>>> > >>>> - Verified hashes > >>>> - Full build with contrib-check passes > >>>> - Was able to run nifi server and send flow files to remote input port > >>>> over s2s protocol. Web UI and API working as expected. > >>>> > >>>> Platform: > >>>> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; > >>>> 2017-04-03T15:39:06-04:00) > >>>> Java version: 1.8.0_131, vendor: Oracle Corporation > >>>> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: "mac" > >>>> > >>>> Looks good! Nice work everyone! > >>>> > >>>> Kevin > >>>> > >>>> On 9/28/17, 15:54, "Jeff" <[email protected]> wrote: > >>>> > >>>> Hello, > >>>> > >>>> I am pleased to be calling this vote for the source release of > Apache > >>>> NiFi > >>>> nifi-1.4.0. > >>>> > >>>> The source zip, including signatures, digests, etc. can be found at: > >>>> https://repository.apache.org/content/repositories/ > >> orgapachenifi-1111 > >>>> > >>>> The Git tag is nifi-1.4.0-RC2 > >>>> The Git commit ID is e6508ba7d3da5bba54abd6233a7a8f9dd4c32151 > >>>> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h= > >>>> e6508ba7d3da5bba54abd6233a7a8f9dd4c32151 > >>>> > >>>> Checksums of nifi-1.4.0-source-release.zip: > >>>> MD5: 41e4083e602883a3e180032f32913414 > >>>> SHA1: 26770625138126f45bed4989adb0a6b65a767aa2 > >>>> > >>>> Release artifacts are signed with the following key: > >>>> https://people.apache.org/keys/committer/jstorck.asc > >>>> > >>>> KEYS file available here: > >>>> https://dist.apache.org/repos/dist/release/nifi/KEYS > >>>> > >>>> 199 issues were closed/resolved for this release: > >>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa? > >>>> projectId=12316020&version=12340589 > >>>> > >>>> Release note highlights can be found here: > >>>> https://cwiki.apache.org/confluence/display/NIFI/ > >>>> Release+Notes#ReleaseNotes-Version-1.4.0 > >>>> > >>>> The vote will be open for 72 hours. > >>>> Please download the release candidate and evaluate the necessary > >> items > >>>> including checking hashes, signatures, build > >>>> from source, and test. The please vote: > >>>> > >>>> [ ] +1 Release this package as nifi-1.4.0 > >>>> [ ] +0 no opinion > >>>> [ ] -1 Do not release this package because... > >>>> > >>>> > >>>> > >>>> > >>> > >>> > >>> -- > >>> I know what it is to be in need, and I know what it is to have > plenty. I > >>> have learned the secret of being content in any and every situation, > >>> whether well fed or hungry, whether living in plenty or in want. I can > >> do > >>> all this through him who gives me strength. *-Philippians 4:12-13* > >> > >> > > > > > > -- > > I know what it is to be in need, and I know what it is to have plenty. I > > have learned the secret of being content in any and every situation, > > whether well fed or hungry, whether living in plenty or in want. I can > do > > all this through him who gives me strength. *-Philippians 4:12-13* > > -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength. *-Philippians 4:12-13*
