I was able to configure the reader to infer the schema from the first line
of the CSV, and the writer to use the same schema, and now the processor
works. I think these are new properties in 1.4.0, and they certainly help
simplify the flow! I was able to remove the InferAvroSchema processor as a
result.
Joe
On Mon, Oct 2, 2017 at 4:44 PM, Joe Gresock <jgres...@gmail.com> wrote:
> Good question -- no, this must be an artifact of running InferAvroSchema
> before the QueryRecord processor. InferAvroSchema infers the schema and
> places it in a flow file attribute, which the CSVReader uses in its
> configuration.
>
> I just debugged nifi and found that it thinks one of my fields is null.
> There is indeed a record with an empty string in the csv for this field,
> but the same thing happens every day when this flow runs, so I suspect
> something must have changed in the implementation of either CSVReader or
> InferAvroSchema that made the parsing stricter. I'm going to try to figure
> out how to allow blank values in the CSV.
>
> On Mon, Oct 2, 2017 at 4:29 PM, Mark Payne <marka...@hotmail.com> wrote:
>
>> Joe,
>>
>> Thanks. I was going to suggest we spawn off a new thread if this
>> continued with more back-and-forth.
>>
>> I did notice that in the JIRA you indicated that
>> you are using a CSVReader service to read the data. In the message, it
>> indicates that the FlowFIle's
>> filename is
>> "billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro"
>> -- which would imply
>> to me that it is Avro (perhaps converted from CSV to avro upstream?) Is
>> it possible that you just selected
>> the wrong reader?
>>
>> Thanks
>> -Mark
>>
>> > On Oct 2, 2017, at 12:24 PM, Joe Gresock <jgres...@gmail.com> wrote:
>> >
>> > I'm changing the subject name so as not to muddy the [VOTE] thread.
>> >
>> > I finally found the error in the logs. This was also not happening
>> prior
>> > to the upgrade. I thinkI'll debug the server to determine the root
>> cause:
>> >
>> > 2017-10-02 16:09:47,004 ERROR [Timer-Driven Process Thread-81]
>> > o.a.nifi.processors.standard.QueryRecord
>> > QueryRecord[id=21570e18-015c-1000-0000-000058832883] Unable to query
>> > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f7
>> 4911,claim=StandardContentClaim
>> > [resourceClaim=StandardResourceClaim[id=1506960462126-919,
>> > container=default, section=919], offset=55116,
>> > length=616757],offset=0,name=billing-detailed-line-items-wit
>> h-resources-and-tags-2017-10.csv.avro,size=616757]
>> > due to org.apache.nifi.processor.exception.ProcessException: Failed to
>> read
>> > next record in stream for
>> > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f7
>> 4911,claim=StandardContentClaim
>> > [resourceClaim=StandardResourceClaim[id=1506960462126-919,
>> > container=default, section=919], offset=55116,
>> > length=616757],offset=0,name=billing-detailed-line-items-wit
>> h-resources-and-tags-2017-10.csv.avro,size=616757]:
>> > {}
>> > org.apache.nifi.processor.exception.ProcessException: Failed to read
>> next
>> > record in stream for
>> > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f7
>> 4911,claim=StandardContentClaim
>> > [resourceClaim=StandardResourceClaim[id=1506960462126-919,
>> > container=default, section=919], offset=55116,
>> > length=616757],offset=0,name=billing-detailed-line-items-wit
>> h-resources-and-tags-2017-10.csv.avro,size=616757]
>> > at
>> > org.apache.nifi.queryrecord.FlowFileEnumerator.moveNext(Flow
>> FileEnumerator.java:65)
>> > at Baz$1$1.moveNext(Unknown Source)
>> > at
>> > org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(
>> Linq4j.java:664)
>> > at
>> > org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
>> > at
>> > org.apache.calcite.linq4j.AbstractEnumerable.iterator(Abstra
>> ctEnumerable.java:33)
>> > at
>> > org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:89)
>> > at
>> > org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaR
>> esultSet.java:196)
>> > at
>> > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResu
>> ltSet.java:67)
>> > at
>> > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResu
>> ltSet.java:44)
>> > at
>> > org.apache.calcite.avatica.AvaticaConnection.executeQueryInt
>> ernal(AvaticaConnection.java:513)
>> > at
>> > org.apache.calcite.avatica.AvaticaPreparedStatement.executeQ
>> uery(AvaticaPreparedStatement.java:132)
>> > at
>> > org.apache.nifi.processors.standard.QueryRecord.queryWithCac
>> he(QueryRecord.java:445)
>> > at
>> > org.apache.nifi.processors.standard.QueryRecord.onTrigger(
>> QueryRecord.java:276)
>> > at
>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
>> ctProcessor.java:27)
>> > at
>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>> tandardProcessorNode.java:1119)
>> > at
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:147)
>> > at
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:47)
>> > at
>> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>> gent$1.run(TimerDrivenSchedulingAgent.java:128)
>> > at
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:
>> 308)
>> > at
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> > at
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:294)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:748)
>> >
>> >
>> > On Mon, Oct 2, 2017 at 4:03 PM, Mark Payne <marka...@hotmail.com>
>> wrote:
>> >
>> >> Hey Joe,
>> >>
>> >> I've seen this error crop up a time or two before. The issue is a bug
>> in
>> >> how we are handling a failure
>> >> condition. Haven't yet figured out the best way to tackle it, but the
>> >> thing to note is that this error occurs
>> >> when there is some other sort of failure -- typically, a failure to
>> obtain
>> >> the schema for the Record Reader
>> >> or Record Writer. Can you check your logs for any other errors that
>> may be
>> >> occurring?
>> >>
>> >> Thanks
>> >> -Mark
>> >>
>> >>
>> >>> On Oct 2, 2017, at 11:32 AM, Joe Gresock <jgres...@gmail.com> wrote:
>> >>>
>> >>> I have to change my non-binding +1 vote to a 0, unless there is a
>> >>> workaround for the bug [1] I just submitted, regarding the QueryRecord
>> >>> processor being broken.
>> >>>
>> >>>
>> >>> [1] https://issues.apache.org/jira/browse/NIFI-4452
>> >>>
>> >>>
>> >>> On Mon, Oct 2, 2017 at 3:23 AM, Kevin Doran <kdoran.apa...@gmail.com>
>> >> wrote:
>> >>>
>> >>>> +1, non-binding
>> >>>>
>> >>>> - Verified hashes
>> >>>> - Full build with contrib-check passes
>> >>>> - Was able to run nifi server and send flow files to remote input
>> port
>> >>>> over s2s protocol. Web UI and API working as expected.
>> >>>>
>> >>>> Platform:
>> >>>> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426;
>> >>>> 2017-04-03T15:39:06-04:00)
>> >>>> Java version: 1.8.0_131, vendor: Oracle Corporation
>> >>>> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family:
>> "mac"
>> >>>>
>> >>>> Looks good! Nice work everyone!
>> >>>>
>> >>>> Kevin
>> >>>>
>> >>>> On 9/28/17, 15:54, "Jeff" <jsto...@apache.org> wrote:
>> >>>>
>> >>>> Hello,
>> >>>>
>> >>>> I am pleased to be calling this vote for the source release of
>> Apache
>> >>>> NiFi
>> >>>> nifi-1.4.0.
>> >>>>
>> >>>> The source zip, including signatures, digests, etc. can be found
>> at:
>> >>>> https://repository.apache.org/content/repositories/
>> >> orgapachenifi-1111
>> >>>>
>> >>>> The Git tag is nifi-1.4.0-RC2
>> >>>> The Git commit ID is e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
>> >>>> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
>> >>>> e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
>> >>>>
>> >>>> Checksums of nifi-1.4.0-source-release.zip:
>> >>>> MD5: 41e4083e602883a3e180032f32913414
>> >>>> SHA1: 26770625138126f45bed4989adb0a6b65a767aa2
>> >>>>
>> >>>> Release artifacts are signed with the following key:
>> >>>> https://people.apache.org/keys/committer/jstorck.asc
>> >>>>
>> >>>> KEYS file available here:
>> >>>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>> >>>>
>> >>>> 199 issues were closed/resolved for this release:
>> >>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> >>>> projectId=12316020&version=12340589
>> >>>>
>> >>>> Release note highlights can be found here:
>> >>>> https://cwiki.apache.org/confluence/display/NIFI/
>> >>>> Release+Notes#ReleaseNotes-Version-1.4.0
>> >>>>
>> >>>> The vote will be open for 72 hours.
>> >>>> Please download the release candidate and evaluate the necessary
>> >> items
>> >>>> including checking hashes, signatures, build
>> >>>> from source, and test. The please vote:
>> >>>>
>> >>>> [ ] +1 Release this package as nifi-1.4.0
>> >>>> [ ] +0 no opinion
>> >>>> [ ] -1 Do not release this package because...
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> I know what it is to be in need, and I know what it is to have
>> plenty. I
>> >>> have learned the secret of being content in any and every situation,
>> >>> whether well fed or hungry, whether living in plenty or in want. I
>> can
>> >> do
>> >>> all this through him who gives me strength. *-Philippians 4:12-13*
>> >>
>> >>
>> >
>> >
>> > --
>> > I know what it is to be in need, and I know what it is to have plenty.
>> I
>> > have learned the secret of being content in any and every situation,
>> > whether well fed or hungry, whether living in plenty or in want. I can
>> do
>> > all this through him who gives me strength. *-Philippians 4:12-13*
>>
>>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty. I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want. I can
> do all this through him who gives me strength. *-Philippians 4:12-13*
>
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
