Re: Architecting the MS SQL CDC Processor

2017-10-16 Thread Matt Burgess
Peter, This is great to hear, I'm sure the community is looking forward to such a solution! I worked on the first offering of the CaptureChangeMySQL processor, so here are some notes, comments, and (hopefully!) answers to your questions: * If you support a RecordSetWriter controller service as

Architecting the MS SQL CDC Processor

2017-10-16 Thread Peter Wicks (pwicks)
I've been working on a new processor that does Change Data Capture with Microsoft SQL Server. I followed Microsoft's documentation on how CDC works, and I've got some code that gets me the changes and is testing well. Right now, I don't actually have a processor, but a number of scripts that

RE: [EXT] Re: JAVA_HOME trouble in nifi.sh

2017-10-16 Thread Peter Wicks (pwicks)
Aldrin, My branch isn't pure, in fact it has a few build only related changes... I'll play around with it. I'm working on moving my builds to a Unix box with Jenkins to reduce the CPU strain on my development box. But my builds were failing due to the NPM/Node Maven plugin. So I upgraded the

RE: [EXT] Re: Please refresh my memory on NAR dependencies

2017-10-16 Thread Peter Wicks (pwicks)
I gave this a shot and it worked well for me. https://github.com/apache/nifi/pull/2194 -Original Message- From: Koji Kawamura [mailto:ijokaruma...@gmail.com] Sent: Monday, October 16, 2017 12:03 PM To: dev Subject: Re: [EXT] Re: Please refresh my memory on NAR

RE: [EXT] Re: Funnel Queue Slowness

2017-10-16 Thread Peter Wicks (pwicks)
Pierre, I agree with you all around. It would be nice if it was a little smarter. --Peter -Original Message- From: Pierre Villard [mailto:pierre.villard...@gmail.com] Sent: Monday, October 16, 2017 4:00 PM To: dev Subject: Re: [EXT] Re: Funnel Queue Slowness

Re: [EXT] Re: Funnel Queue Slowness

2017-10-16 Thread Pierre Villard
Peter, This behaviour is by design and it's the case for processors as well. Back pressure is only checked by the component each time it is scheduled to see whether the component can run or not. If yes, the component will run as configured and will process as many flow files as it is supposed to