That probably makes sense. You may not even need back-pressure at all between ListenSyslog -> Control Rate, this way the data will just queue up there. I guess it depends how fast your syslog data is coming in and how long you think spark will be down.
NiFi will swap out flow files to disk when the queue gets very large and should be able to hold a significant number of flow files in a queue (millions). The swapping is controlled by the following properties in nifi.properties: nifi.queue.swap.threshold=20000 nifi.swap.in.period=5 sec nifi.swap.in.threads=1 nifi.swap.out.period=5 sec nifi.swap.out.threads=4 On Wed, Jul 6, 2016 at 3:55 PM, pradeepbill <pradeep.b...@gmail.com> wrote: > Thanks Bryan, then I guess in my case for the time being, I am going to use > it like syslog processor-> control rate processor->data output port->spark > application , and apply back pressure like really huge between syslog > processor -> control rate processor and very small back pressure between > control rate processor -> data output port , that way when the spark > application comes back, a fixed data flows and wont overwhelm the spark > application, decent idea ? > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/conditional-Listen-Syslog-tp12652p12662.html > Sent from the Apache NiFi Developer List mailing list archive at > Nabble.com. >
