[Discuss] DeleteSFTP

[Andre]

Folks,

I was recently asked by a user if we have a DeleteSFTP processor to handle
file deletion AFTER some logic has been performed.

I noticed we don't have a processor covering that, and while ExecuteScript
could be used to make this happen I wondered if it we should support that?

The logic is simple:

ListSFTP -> FetchSFTP  - (if success) -> DoSomething - (if success) ->
DeleteSFTP


I think this pattern would fit those who are just too cautious to trust the
"Delete file" "Completion strategy" available on FetchSFTP.  The reason for
caution is the assumption that although NiFi queues are persistent, things
can still go wrong between the file being sourced and the "DoSomething"
success obtained.

What do you think?

Initial Admin Identity can't query provenance?

[Peter Wicks (pwicks)]

I just setup a new NiFi instance and setup myself as the Initial Admin 
Identity, I'm currently the only user on the box and I've made no security 
changes. I noticed today that I couldn't query provenance.

I had to go in and create a new policy and add myself to it. This seems odd for 
an initial admin identity, but maybe it's by design?

Thanks,
  Peter

VolatileProvenanceRepository for MiNiFi

[Sumanth Chinthagunta]

Hi,

In NiFi ,  I can set following properties in 'nifi.property'.

nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository
nifi.provenance.repository.buffer.size=1000


What is the equivalent process for MiNiFi configuration ?

-- 
Thanks,
Sumanth

ConvertAvroToOrc enconding problem

[rodrigomar...@id.uff.br]

Hello guys, I`m trying to convert a avro file encoded with utf-8 to orc, but 
when I use this processor the encoding seems to change and the text fields on 
the orc file turns wrong. Can someone help me with this question?

Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

[Andy Christianson]

PR is in [1].

Regards,

Andy I.C.

[1]: https://github.com/apache/nifi-minifi-cpp/pull/154

From: Andy Christianson 
Sent: Wednesday, October 25, 2017 10:47 AM
To: dev@nifi.apache.org
Subject: Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

Sounds good. I have a local commit which I'll move to a branch and submit a PR 
for, after adding a relevant JIRA.

Regards,

Andy I.C.

From: Marc 
Sent: Wednesday, October 25, 2017 8:08 AM
To: dev@nifi.apache.org
Subject: Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

Andy,
  I agree with your suggestion. There's no reason why we couldn't make this
an option in the future after its removal to help clean up DEBUG.

  Thanks,
  Marc

On Tue, Oct 24, 2017 at 9:48 AM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> MiNiFi - C++ Devs:
>
>
> On an earlier commit, I added the following to the root CMakeLists.txt:
>
>
> # Enable asan in DEBUG for compatibility with civet
> set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fsanitize=address")
>
>
> This was, as stated in the comment, to address a compatibility issue when
> compiling in DEBUG where civet would fail to link.
>
>
> On more recent dev toolchains (devtoolset-6 SCL CentOS/GCC 6.3), this
> results in extremely verbose and unhelpful output during test runs. While I
> am not opposed to using these tools to detect legitimate memory leaks, as
> it stands currently the setting is more of a detriment to the development
> process. I think that we should either only turn this on when we
> deliberately intend to analyze memory usage, or possibly create additional
> unit tests to detect memory leaks.
>
>
> In the meantime, I think we should remove the statement from our
> CMakeLists.txt and disable it in the CMakeLists.txt for civet as well.
>
>
> I would like to hear additional feedback from the development community to
> see if we have consensus on this change.
>
>
> Regards,
>
>
> Andy I.C.
>

Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

[Andy Christianson]

Sounds good. I have a local commit which I'll move to a branch and submit a PR 
for, after adding a relevant JIRA.

Regards,

Andy I.C.

From: Marc 
Sent: Wednesday, October 25, 2017 8:08 AM
To: dev@nifi.apache.org
Subject: Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

Andy,
  I agree with your suggestion. There's no reason why we couldn't make this
an option in the future after its removal to help clean up DEBUG.

  Thanks,
  Marc

On Tue, Oct 24, 2017 at 9:48 AM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> MiNiFi - C++ Devs:
>
>
> On an earlier commit, I added the following to the root CMakeLists.txt:
>
>
> # Enable asan in DEBUG for compatibility with civet
> set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fsanitize=address")
>
>
> This was, as stated in the comment, to address a compatibility issue when
> compiling in DEBUG where civet would fail to link.
>
>
> On more recent dev toolchains (devtoolset-6 SCL CentOS/GCC 6.3), this
> results in extremely verbose and unhelpful output during test runs. While I
> am not opposed to using these tools to detect legitimate memory leaks, as
> it stands currently the setting is more of a detriment to the development
> process. I think that we should either only turn this on when we
> deliberately intend to analyze memory usage, or possibly create additional
> unit tests to detect memory leaks.
>
>
> In the meantime, I think we should remove the statement from our
> CMakeLists.txt and disable it in the CMakeLists.txt for civet as well.
>
>
> I would like to hear additional feedback from the development community to
> see if we have consensus on this change.
>
>
> Regards,
>
>
> Andy I.C.
>

Re: minifi-cpp - Consensus regarding enablement of fsanitize=address

[Marc]

Andy,
  I agree with your suggestion. There's no reason why we couldn't make this
an option in the future after its removal to help clean up DEBUG.

  Thanks,
  Marc

On Tue, Oct 24, 2017 at 9:48 AM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> MiNiFi - C++ Devs:
>
>
> On an earlier commit, I added the following to the root CMakeLists.txt:
>
>
> # Enable asan in DEBUG for compatibility with civet
> set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fsanitize=address")
>
>
> This was, as stated in the comment, to address a compatibility issue when
> compiling in DEBUG where civet would fail to link.
>
>
> On more recent dev toolchains (devtoolset-6 SCL CentOS/GCC 6.3), this
> results in extremely verbose and unhelpful output during test runs. While I
> am not opposed to using these tools to detect legitimate memory leaks, as
> it stands currently the setting is more of a detriment to the development
> process. I think that we should either only turn this on when we
> deliberately intend to analyze memory usage, or possibly create additional
> unit tests to detect memory leaks.
>
>
> In the meantime, I think we should remove the statement from our
> CMakeLists.txt and disable it in the CMakeLists.txt for civet as well.
>
>
> I would like to hear additional feedback from the development community to
> see if we have consensus on this change.
>
>
> Regards,
>
>
> Andy I.C.
>

Re: Proper way to finish a review

[Joe Witt]

Mike

Thanks for jumping in to the review game.  We definitely need the help!

I'd say leave your comments/+1 for things that you've reviewed and
validated and you think are ready to merge.  Then anyone with commit
privileges can take that and get it pushed in.  On your reviews please
try to let it be known to what level you're confident.  Ideally a +1
would mean you've done a full clean build with contrib check, any new
tests that come with it are passing, that the code looks good, and
that you've had a chance to run it and verify the behavior yourself.
Some things are harder to review than others for sure but every bit
helps.

Thanks

On Wed, Oct 25, 2017 at 7:43 AM, Mike Thomsen  wrote:
> When I am satisfied with a review, should I choose approve or the option to
> just leave comment and ping some folks on the PMC to come see if they want
> to sign off for merging?
>
> Thanks,
>
> Mike