adding original attribute values to InvokeHttp response?
How can I add original attribute values into InvokeHttp response context (Json)? Thank you,
Re: CSV Record Reader - No Quote and No Escape
This example was for a different issue. The examples for CSV Reader are on the JIRA. Sorry. Thanks Shawn From: Shawn Weeks Reply-To: "users@nifi.apache.org" Date: Thursday, December 26, 2019 at 12:28 PM To: "users@nifi.apache.org" Subject: Re: CSV Record Reader - No Quote and No Escape Attached is an example. Do not try this on a NiFi cluster you care about as you might have to delete your flow.xml.gz file to kill it. I increased my xmx and xms to 4gb for this test. Thanks Shawn Weeks From: Pierre Villard Reply-To: "users@nifi.apache.org" Date: Thursday, December 26, 2019 at 9:33 AM To: "users@nifi.apache.org" Subject: Re: CSV Record Reader - No Quote and No Escape Hey Shawn, Could you provide an example of what data you have and how you would like to have it processed? It'll be easier to add a unit test and make a code change if necessary. Thanks, Pierre Le ven. 20 déc. 2019 à 18:51, Shawn Weeks mailto:swe...@weeksconsulting.us>> a écrit : I’m trying to process some tab delimited text that may contain embedded quotes and slashes. How do I disable quote and escape in the CSVReader so that they aren’t used. Setting them to empty text doesn’t work and I’m getting the following error because there are quotes and slashes in the data despite setting the format to tab delimited. Failed to write due to org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter: java.io.IOException: org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter
Re: Advanced QueryRecord Brings NiFi Down
Realized I attached my example to the wrong chain, sorry. Don’t run this on a cluster you care about as you’ll probably have to drop the flow.xml.gz to stop it. Thanks Shawn From: Mike Thomsen Reply-To: "users@nifi.apache.org" Date: Tuesday, December 24, 2019 at 7:42 PM To: "users@nifi.apache.org" Subject: Re: Advanced QueryRecord Brings NiFi Down If you could share more details like query, schema, etc. that would be a big help toward setting up for a Jira ticket to investigate. On Mon, Dec 23, 2019 at 4:11 PM Shawn Weeks mailto:swe...@weeksconsulting.us>> wrote: I was playing around with QueryRecord this afternoon and I succeeded in bring a decent sized(16 Core 64gb AWS m5) NiFi instance down. The input file is an 8gb CSV file and I was using the row_number() analytic function. Memory usage went to 51 GB and CPU went to 100% on all cores despite only having QueryRecord running in a single thread. Has anyone else run into this with QueryRecord? I’m assuming it’s Calcite trying to run the query multi-threaded but even if it put the whole thing in memory it shouldn’t have gone through that much. Documentation on Calcite query execution is pretty much nonexistent so I’m not sure where to even begin debugging. Thanks Shawn Weeks Query_Record_Bug.xml Description: Query_Record_Bug.xml
Re: CSV Record Reader - No Quote and No Escape
Attached is an example. Do not try this on a NiFi cluster you care about as you might have to delete your flow.xml.gz file to kill it. I increased my xmx and xms to 4gb for this test. Thanks Shawn Weeks From: Pierre Villard Reply-To: "users@nifi.apache.org" Date: Thursday, December 26, 2019 at 9:33 AM To: "users@nifi.apache.org" Subject: Re: CSV Record Reader - No Quote and No Escape Hey Shawn, Could you provide an example of what data you have and how you would like to have it processed? It'll be easier to add a unit test and make a code change if necessary. Thanks, Pierre Le ven. 20 déc. 2019 à 18:51, Shawn Weeks mailto:swe...@weeksconsulting.us>> a écrit : I’m trying to process some tab delimited text that may contain embedded quotes and slashes. How do I disable quote and escape in the CSVReader so that they aren’t used. Setting them to empty text doesn’t work and I’m getting the following error because there are quotes and slashes in the data despite setting the format to tab delimited. Failed to write due to org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter: java.io.IOException: org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter Query_Record_Bug.xml Description: Query_Record_Bug.xml
Re: CSV Record Reader - No Quote and No Escape
I’ve created NIFI-6966 and NIFI-6967 with some examples. Thanks Shawn Weeks From: Pierre Villard Reply-To: "users@nifi.apache.org" Date: Thursday, December 26, 2019 at 9:33 AM To: "users@nifi.apache.org" Subject: Re: CSV Record Reader - No Quote and No Escape Hey Shawn, Could you provide an example of what data you have and how you would like to have it processed? It'll be easier to add a unit test and make a code change if necessary. Thanks, Pierre Le ven. 20 déc. 2019 à 18:51, Shawn Weeks mailto:swe...@weeksconsulting.us>> a écrit : I’m trying to process some tab delimited text that may contain embedded quotes and slashes. How do I disable quote and escape in the CSVReader so that they aren’t used. Setting them to empty text doesn’t work and I’m getting the following error because there are quotes and slashes in the data despite setting the format to tab delimited. Failed to write due to org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter: java.io.IOException: org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter
Re: CSV Record Reader - No Quote and No Escape
Hey Shawn, Could you provide an example of what data you have and how you would like to have it processed? It'll be easier to add a unit test and make a code change if necessary. Thanks, Pierre Le ven. 20 déc. 2019 à 18:51, Shawn Weeks a écrit : > I’m trying to process some tab delimited text that may contain embedded > quotes and slashes. How do I disable quote and escape in the CSVReader so > that they aren’t used. Setting them to empty text doesn’t work and I’m > getting the following error because there are quotes and slashes in the > data despite setting the format to tab delimited. > > > > Failed to write due to > org.apache.nifi.serialization.MalformedRecordException: Error while getting > next record. Root cause: java.io.IOException: (line 1050916) invalid char > between encapsulated token and delimiter: java.io.IOException: > org.apache.nifi.serialization.MalformedRecordException: Error while getting > next record. Root cause: java.io.IOException: (line 1050916) invalid char > between encapsulated token and delimiter >
Re: CaptureChangeMySQL Error.
Thank you very much Shawn. I followed your advice and it was settled. Have a nice day. :) Thanks DC Gong 2019년 12월 25일 (수) 오후 11:24, Shawn Weeks 님이 작성: > That means you haven’t setup state management usually. Are you running a > cluster or a single node? If it’s a cluster you’ll need to set your > zookeeper string in the state management config file. > > Thanks > Shawn > > Sent from my iPhone > > On Dec 24, 2019, at 7:05 AM, DC Gong wrote: > > > Sorry, i was missing it in email posting. > > The exception is below. > > - > > 2019-12-24 11:10:31,830 ERROR [Timer-Driven Process Thread-6] > o.a.n.c.m.processors.CaptureChangeMySQL > CaptureChangeMySQL[id=f0e81363-d438-157a--d1676ca4] > CaptureChangeMySQL[id=f0e81363-d438-157a--d1676ca4] failed to > process session due to java.lang.IllegalArgumentException: A HostProvider > may not be empty!; Processor Administratively Yielded for 1 sec: > java.lang.IllegalArgumentException: A HostProvider may not be empty! > java.lang.IllegalArgumentException: A HostProvider may not be empty! > at > org.apache.zookeeper.client.StaticHostProvider.init(StaticHostProvider.java:136) > at > org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:87) > at > org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1312) > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:951) > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:688) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getState(ZooKeeperStateProvider.java:403) > at > org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.getState(StandardStateManagerProvider.java:305) > at > org.apache.nifi.controller.state.StandardStateManager.getState(StandardStateManager.java:63) > at > org.apache.nifi.controller.lifecycle.TaskTerminationAwareStateManager.getState(TaskTerminationAwareStateManager.java:52) > at > org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.setup(CaptureChangeMySQL.java:435) > at > org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:544) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > > > Thanks. > > > > > 2019년 12월 24일 (화) 오후 9:59, Shawn Weeks 님이 작성: > >> What the exception your seeing? >> >> >> >> Thanks >> >> Shawn >> >> >> >> *From: *DC Gong >> *Reply-To: *"users@nifi.apache.org" >> *Date: *Tuesday, December 24, 2019 at 1:39 AM >> *To: *"users@nifi.apache.org" >> *Subject: *CaptureChangeMySQL Error. >> >> >> >> Hello, >> >> >> >> I want to use a CaptureChangeMySQL processor. >> >> >> >> The properties settings are set, but the error below occurs. >> I think it's a ZOOKEEPER error, but I don't know what the problem is. >> >> Can you tell me what settings to look at? >> >> >> >> My setting is below. >> >> >> >> MySQL Hosts : localhost:3306 >> >> MySQL Driver Class Name : com.mysql.jdbc.Driver >> >> MySQL Driver Location(s) : /home/test/mysql-connector-java-8.0.18.jar >> >> Username : test >> >> Password: test >> >> Server ID : 1 >> >> Database/Schema Name Pattern : testdb >> >> Table Name Pattern : test_table >> >> OTHERS : default. >> >> >> >> Please reply. >> >> >> >> Best Regards ggong. >> >> >> >
