[jira] [Commented] (NIFI-4434) ListHDFS applies File Filter also to subdirectory names in recursive search
[ https://issues.apache.org/jira/browse/NIFI-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572466#comment-16572466 ] ASF GitHub Bot commented on NIFI-4434: -- Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/2937 @ottobackwards @zenfenan I added additional details to give some examples on how to use the filters. Please let me know if you think more detail is needed. I appreciate the review! > ListHDFS applies File Filter also to subdirectory names in recursive search > --- > > Key: NIFI-4434 > URL: https://issues.apache.org/jira/browse/NIFI-4434 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Holger Frydrych >Assignee: Jeff Storck >Priority: Major > > The File Filter regex configured in the ListHDFS processor is applied not > just to files found, but also to subdirectories. > If you try to set up a recursive search to list e.g. all csv files in a > directory hierarchy via a regex like ".*\.csv", it will only pick up csv > files in the base directory, not in any subdirectory. This is because > subdirectories don't typically match that regex pattern. > To fix this, either subdirectories should not be matched against the file > filter, or the file filter should be applied to the full path of all files > (relative to the base directory). The GetHDFS processor offers both options > via a switch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2937: NIFI-4434 Fixed recursive listing with a custom regex filt...
Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/2937 @ottobackwards @zenfenan I added additional details to give some examples on how to use the filters. Please let me know if you think more detail is needed. I appreciate the review! ---
[jira] [Created] (NIFIREG-190) Support for Event Whitelisting in the Registry Event Hooks
Jeremy Dyer created NIFIREG-190: --- Summary: Support for Event Whitelisting in the Registry Event Hooks Key: NIFIREG-190 URL: https://issues.apache.org/jira/browse/NIFIREG-190 Project: NiFi Registry Issue Type: New Feature Affects Versions: 0.2.0 Reporter: Jeremy Dyer Assignee: Jeremy Dyer Fix For: 0.3.0 Today when an event hook is configured it will be invoked for all of the NiFi Registry events. While a user can parse the arguments in the script and manually write scripts which ignore certain events it makes more sense to provide this event whitelisting the the registry itself. I propose adding a new property called something like "Event Whitelist" to the current configuration logic. If this property is not present things should continue to operate just as they do now, AKA the script is sent all of the events, if the property is specified it should contain a comma delimited list of events that the hook provider should be triggered for. This will be extremely useful when providers that do not provider any sort of filtering logic like the script hook provider come along. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4517) Allow SQL results to be output as records in any supported format
[ https://issues.apache.org/jira/browse/NIFI-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572329#comment-16572329 ] Peter Wicks commented on NIFI-4517: --- Sounds good. They don't even need to be in the same release, it's just that a user would not necessarily see the same results if they used no RecordWriter vs using the Avro Record Writer. > Allow SQL results to be output as records in any supported format > - > > Key: NIFI-4517 > URL: https://issues.apache.org/jira/browse/NIFI-4517 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Peter Wicks >Priority: Major > > ExecuteSQL and QueryDatabaseTable currently only outputs Avro, and the schema > is only available as embedded within the flow file, not as an attribute such > as record-aware processors can handle. > ExecuteSQL and QueryDatabaseTable processors should be updated with a > RecordSetWriter implementation. This will allow output using any writer > format (Avro, JSON, CSV, Free-form text, etc.), as well as all the other > features therein (such as writing the schema to an attribute, and will avoid > the need for a ConvertAvroToXYZ or ConvertRecord processor downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (NIFI-4246) OAuth 2 Authorization support - Client Credentials Grant
[
https://issues.apache.org/jira/browse/NIFI-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572049#comment-16572049
]
Romain Guay edited comment on NIFI-4246 at 8/7/18 8:55 PM:
---
Hi all and thanks for this useful work.
I am new to this conversation and relatively new to Nifi, but I have of course
the need to OAuth authentication, and I also have a similar need with a custom
token-based authentication which I think is not so different. I wonder if we
could take this opportunity to open our options by defining a slightly more
generic interface for this authentication service and allow multiple
implementations for the same InvokeHttp processor.
Here is what I would suggest:
{noformat}
public interface AuthenticationProviderClientService extends ControllerService {
boolean isTokenExpired();
boolean isTokenExpired(int statusCode, String message);
Map getAuthenticationHeaders();
boolean authenticate();
}{noformat}
The isTokenExpired() method is used as currently when preparing the request.
The isTokenExpired(String response) method would be used in cases where the
token does not know its expiration, but we can determine it from the statusCode
and message from the HTTP response. If it is expired, authenticate() is called
and the flow file sent to retry. So each implementation would use either one or
the other, returning false by default.
I also think having getAuthenticationHeaders() instead of getAccessToken()
makes it more generic.
Please let me know if this all makes sense.
was (Author: rg...@modulusdata.com):
Hi all and thanks for this useful work.
I am new to this conversation and relatively new to Nifi, but I have of course
the need to OAuth authentication, and I also have a similar need with a custom
token-based authentication which I think is not so different. I wonder if we
could take this opportunity to open our options by defining a slightly more
genetic interface for this authentication service and allow multiple
implementations for the same InvokeHttp processor.
Here is what I would suggest:
{noformat}
public interface AuthenticationProviderClientService extends ControllerService {
boolean isTokenExpired();
boolean isTokenExpired(int statusCode, String message);
Map getAuthenticationHeaders();
boolean authenticate();
}{noformat}
The isTokenExpired() method is used as currently when preparing the request.
The isTokenExpired(String response) method would be used in cases where the
token does not know its expiration, but we can determine it from the statusCode
and message from the HTTP response. If it is expired, authenticate() is called
and the flow file sent to retry. So each implementation would use either one or
the other, returning false by default.
I also think having getAuthenticationHeaders() instead of getAccessToken()
makes it more generic.
Please let me know if this all makes sense.
> OAuth 2 Authorization support - Client Credentials Grant
>
>
> Key: NIFI-4246
> URL: https://issues.apache.org/jira/browse/NIFI-4246
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Jeremy Dyer
>Assignee: Jeremy Dyer
>Priority: Major
>
> If your interacting with REST endpoints on the web chances are you are going
> to run into an OAuth2 secured webservice. The IETF (Internet Engineering Task
> Force) defines 4 methods in which OAuth2 authorization can occur. This JIRA
> is focused solely on the Client Credentials Grant method defined at
> https://tools.ietf.org/html/rfc6749#section-4.4
> This implementation should provide a ControllerService in which the enduser
> can configure the credentials for obtaining the authorization grant (access
> token) from the resource owner. In turn a new property will be added to the
> InvokeHTTP processor (if it doesn't already exist from one of the other JIRA
> efforts similar to this one) where the processor can reference this
> controller service to obtain the access token and insert the appropriate HTTP
> header (Authorization: Bearer{access_token}) so that the InvokeHTTP processor
> can interact with the OAuth protected resources without having to worry about
> setting up the credentials for each InvokeHTTP processor saving time and
> complexity.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5496) Remove finalizer from StandardProcessSession
[ https://issues.apache.org/jira/browse/NIFI-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-5496: - Attachment: gc-without-finalizer.png > Remove finalizer from StandardProcessSession > > > Key: NIFI-5496 > URL: https://issues.apache.org/jira/browse/NIFI-5496 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Attachments: gc-with-finalizer.png, gc-without-finalizer.png > > > The NiFi API and documentation have always been pretty clear that Processors > are responsible for properly committing/rolling back their Process Sessions. > However, what happens when a Processor fails to do so? > The AbstractProcessor (from which 99+% of all Processors extend) will > automatically call ProcessSession.commit() if the subclass's onTrigger() > method returns successfully and call ProcessSession.rollback() if any > Throwable is thrown from the subclass's onTrigger(). > However, those Processors that extend from AbstractSessionFactoryProcessor do > not have this protection. As a fail-safe, we have always had a finalizer in > StandardProcessSession. The finalizer simply calls rollback(). This ensures > that any FlowFiles that were not properly handled (transferred, removed, > etc.) are accounted for and re-queued/discarded. > Java 9 has now made finalizers deprecated, though. Additionally, through > doing some research, I have seen that the finalizer ends up being quite > expensive in the effects that it causes on the system's Garbage Collection. > Specifically, GC ends up reclaiming less and less and less heap until a point > at which a lot more is reclaimed. This results in a JVM that is sometimes > "choppy" in its responsiveness, which can lead to instabilities. (Will attach > screenshots showing this behavior). > Given this, and given that only a handful of Processors need to make use of > the AbstractSessionFactoryProcessor, we should remove the Process Session's > finalizer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5496) Remove finalizer from StandardProcessSession
[ https://issues.apache.org/jira/browse/NIFI-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-5496: - Attachment: gc-with-finalizer.png > Remove finalizer from StandardProcessSession > > > Key: NIFI-5496 > URL: https://issues.apache.org/jira/browse/NIFI-5496 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Attachments: gc-with-finalizer.png, gc-without-finalizer.png > > > The NiFi API and documentation have always been pretty clear that Processors > are responsible for properly committing/rolling back their Process Sessions. > However, what happens when a Processor fails to do so? > The AbstractProcessor (from which 99+% of all Processors extend) will > automatically call ProcessSession.commit() if the subclass's onTrigger() > method returns successfully and call ProcessSession.rollback() if any > Throwable is thrown from the subclass's onTrigger(). > However, those Processors that extend from AbstractSessionFactoryProcessor do > not have this protection. As a fail-safe, we have always had a finalizer in > StandardProcessSession. The finalizer simply calls rollback(). This ensures > that any FlowFiles that were not properly handled (transferred, removed, > etc.) are accounted for and re-queued/discarded. > Java 9 has now made finalizers deprecated, though. Additionally, through > doing some research, I have seen that the finalizer ends up being quite > expensive in the effects that it causes on the system's Garbage Collection. > Specifically, GC ends up reclaiming less and less and less heap until a point > at which a lot more is reclaimed. This results in a JVM that is sometimes > "choppy" in its responsiveness, which can lead to instabilities. (Will attach > screenshots showing this behavior). > Given this, and given that only a handful of Processors need to make use of > the AbstractSessionFactoryProcessor, we should remove the Process Session's > finalizer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5496) Remove finalizer from StandardProcessSession
Mark Payne created NIFI-5496: Summary: Remove finalizer from StandardProcessSession Key: NIFI-5496 URL: https://issues.apache.org/jira/browse/NIFI-5496 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne The NiFi API and documentation have always been pretty clear that Processors are responsible for properly committing/rolling back their Process Sessions. However, what happens when a Processor fails to do so? The AbstractProcessor (from which 99+% of all Processors extend) will automatically call ProcessSession.commit() if the subclass's onTrigger() method returns successfully and call ProcessSession.rollback() if any Throwable is thrown from the subclass's onTrigger(). However, those Processors that extend from AbstractSessionFactoryProcessor do not have this protection. As a fail-safe, we have always had a finalizer in StandardProcessSession. The finalizer simply calls rollback(). This ensures that any FlowFiles that were not properly handled (transferred, removed, etc.) are accounted for and re-queued/discarded. Java 9 has now made finalizers deprecated, though. Additionally, through doing some research, I have seen that the finalizer ends up being quite expensive in the effects that it causes on the system's Garbage Collection. Specifically, GC ends up reclaiming less and less and less heap until a point at which a lot more is reclaimed. This results in a JVM that is sometimes "choppy" in its responsiveness, which can lead to instabilities. (Will attach screenshots showing this behavior). Given this, and given that only a handful of Processors need to make use of the AbstractSessionFactoryProcessor, we should remove the Process Session's finalizer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5476) Enable TLS Toolkit (standalone) to sign certificates with external CA certificate
[
https://issues.apache.org/jira/browse/NIFI-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy LoPresto updated NIFI-5476:
Resolution: Fixed
Fix Version/s: 1.8.0
Status: Resolved (was: Patch Available)
> Enable TLS Toolkit (standalone) to sign certificates with external CA
> certificate
> -
>
> Key: NIFI-5476
> URL: https://issues.apache.org/jira/browse/NIFI-5476
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Security, Tools and Build
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, pem, pkcs1, pkcs8, pki, security, tls,
> tls-toolkit
> Fix For: 1.8.0
>
>
> The TLS Toolkit can sign certificates using a public certificate and private
> key generated and signed elsewhere by injecting them into the
> {{nifi-cert.pem}} and {{nifi-key.key}} files as long as they are in the
> proper format and self-signed. The toolkit should be enhanced to handle PKCS
> #8 formatted private keys (in addition to the PKCS #1 formatted keys it
> handles now) and to allow for non self-signed certificates.
> To verify this, use certificates generated by
> [TinyCert|https://tinycert.org].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5476) Enable TLS Toolkit (standalone) to sign certificates with external CA certificate
[
https://issues.apache.org/jira/browse/NIFI-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572178#comment-16572178
]
ASF GitHub Bot commented on NIFI-5476:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2935
> Enable TLS Toolkit (standalone) to sign certificates with external CA
> certificate
> -
>
> Key: NIFI-5476
> URL: https://issues.apache.org/jira/browse/NIFI-5476
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Security, Tools and Build
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, pem, pkcs1, pkcs8, pki, security, tls,
> tls-toolkit
>
> The TLS Toolkit can sign certificates using a public certificate and private
> key generated and signed elsewhere by injecting them into the
> {{nifi-cert.pem}} and {{nifi-key.key}} files as long as they are in the
> proper format and self-signed. The toolkit should be enhanced to handle PKCS
> #8 formatted private keys (in addition to the PKCS #1 formatted keys it
> handles now) and to allow for non self-signed certificates.
> To verify this, use certificates generated by
> [TinyCert|https://tinycert.org].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2935: NIFI-5476 Allow TLS toolkit to use externally-signe...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2935 ---
[jira] [Commented] (NIFI-5473) Add documentation for using intermediate CA with TLS toolkit
[ https://issues.apache.org/jira/browse/NIFI-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572176#comment-16572176 ] ASF subversion and git services commented on NIFI-5473: --- Commit 57baae9ae28f63877e66cbc178cdd127a82d4841 in nifi's branch refs/heads/master from [~alopresto] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=57baae9 ] NIFI-5476 Added logic to check CA certificate signature against additional certificates. Moved utility code to TlsHelper. Added unit tests. Added command-line parsing for additional CA certificate path. Added documentation on using the TLS Toolkit to generate and sign certificates using an externally-signed CA. Updated toolkit external CA documentation to be inline with additional context from NIFI-5473. Cleaned up toolkit documentation. Improved error message by changing to absolute path. Added Javadoc to and removed unthrown exception declarations from TlsHelper#verifyCertificateSignature(). Cleaned up unit tests with utility method. Fixed checkstyle error. Support conversion of a PKCS#8 formatted private key automatically to avoid forcing the user to do that. Also add some log messages for debugging when the parser fails to parse the appropriate object Incorporated Peter's contribution for PKCS #8 to PKCS #1 conversion. Added documentation and refactored methods. Refactored unit test. Added RAT exclusion for test resource. This closes #2935. Co-authored-by: pepov Signed-off-by: Matt Gilman > Add documentation for using intermediate CA with TLS toolkit > > > Key: NIFI-5473 > URL: https://issues.apache.org/jira/browse/NIFI-5473 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website, Security, Tools and Build >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: certificate, documentation, security, tls, tls-toolkit > Fix For: 1.8.0 > > > With some manual work, the TLS toolkit can be used with a pre-existing > certificate and private key that has been signed by an organization's > certificate authority (CA) to sign toolkit-generated certificates. The Admin > Guide should be improved to cover the necessary steps. > When the separate "Security Guide" / "Toolkit Guide" is created, this content > should be migrated there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5476) Enable TLS Toolkit (standalone) to sign certificates with external CA certificate
[
https://issues.apache.org/jira/browse/NIFI-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572175#comment-16572175
]
ASF subversion and git services commented on NIFI-5476:
---
Commit 57baae9ae28f63877e66cbc178cdd127a82d4841 in nifi's branch
refs/heads/master from [~alopresto]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=57baae9 ]
NIFI-5476 Added logic to check CA certificate signature against additional
certificates.
Moved utility code to TlsHelper.
Added unit tests.
Added command-line parsing for additional CA certificate path.
Added documentation on using the TLS Toolkit to generate and sign certificates
using an externally-signed CA.
Updated toolkit external CA documentation to be inline with additional context
from NIFI-5473.
Cleaned up toolkit documentation.
Improved error message by changing to absolute path.
Added Javadoc to and removed unthrown exception declarations from
TlsHelper#verifyCertificateSignature().
Cleaned up unit tests with utility method.
Fixed checkstyle error.
Support conversion of a PKCS#8 formatted private key automatically to avoid
forcing the user to do that. Also add some log messages for debugging when the
parser fails to parse the appropriate object
Incorporated Peter's contribution for PKCS #8 to PKCS #1 conversion.
Added documentation and refactored methods.
Refactored unit test.
Added RAT exclusion for test resource.
This closes #2935.
Co-authored-by: pepov
Signed-off-by: Matt Gilman
> Enable TLS Toolkit (standalone) to sign certificates with external CA
> certificate
> -
>
> Key: NIFI-5476
> URL: https://issues.apache.org/jira/browse/NIFI-5476
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Security, Tools and Build
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, pem, pkcs1, pkcs8, pki, security, tls,
> tls-toolkit
>
> The TLS Toolkit can sign certificates using a public certificate and private
> key generated and signed elsewhere by injecting them into the
> {{nifi-cert.pem}} and {{nifi-key.key}} files as long as they are in the
> proper format and self-signed. The toolkit should be enhanced to handle PKCS
> #8 formatted private keys (in addition to the PKCS #1 formatted keys it
> handles now) and to allow for non self-signed certificates.
> To verify this, use certificates generated by
> [TinyCert|https://tinycert.org].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5466) Improve efficiency of RingBufferEventRepository
[
https://issues.apache.org/jira/browse/NIFI-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572161#comment-16572161
]
ASF GitHub Bot commented on NIFI-5466:
--
Github user bbende commented on the issue:
https://github.com/apache/nifi/pull/2939
Will review...
> Improve efficiency of RingBufferEventRepository
> ---
>
> Key: NIFI-5466
> URL: https://issues.apache.org/jira/browse/NIFI-5466
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.8.0
>
>
> The implementation of the FlowFileEventRepository (i.e., the
> RingBufferEventRepository) is implemented in such a way that 300 buckets are
> kept for each component. Each bucket holds a set of 16+ stats (bytes read,
> bytes written, flowfiles in, flowfiles out, etc. as well as counters)
> Each time that a user refreshes stats, we calculate the 5-minute window for
> each component by iterating over all 300 buckets, then summing together the
> values of the stats. So, for every 1,000 processors, we have 1000 * 300 * 16
> = 4.8 MM calculations being performed each time that the user refreshes.
> We can improve this significantly. To do this, we can keep a 'running tally'
> for each stat. Every time that we update a bucket in the
> FlowFileEventRepository, we should similarly update the running tally. Each
> time that we 'replace' a bucket in the FlowFileEventRepository, we should
> subtract from the running tally the values in the bucket, in addition to
> adding in the new value. This would result in amortizing the cost of
> calculation over the 5-minute period and means that every time that we get
> the processor stats we could do so by simply performing a 'get' operation
> without any calculations.
> But wait, there's more! The EventSumValue class has an {{add(FlowFileEvent)}}
> method. If the EventSumValue were to simply keep a boolean value indicating
> whether or not it was empty, then when adding to an empty bucket we could
> avoid the calculations all together, and when replacing an empty bucket we
> could avoid the calculation against the running tally as well.
> Similarly, in the VolatileComponentStatusRepository, when we perform a
> 'capture' we should detect any 'empty' stats objects and use a single static
> instance for this, which would significantly reduce the amount of heap that
> is used to store metrics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2939: NIFI-5466: Keep a running total of stats for each componen...
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2939 Will review... ---
[jira] [Commented] (NIFI-5476) Enable TLS Toolkit (standalone) to sign certificates with external CA certificate
[
https://issues.apache.org/jira/browse/NIFI-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572149#comment-16572149
]
ASF GitHub Bot commented on NIFI-5476:
--
Github user mcgilman commented on the issue:
https://github.com/apache/nifi/pull/2935
Thanks for the contributions @alopresto and @pepov! I'm also a +1 on the
changes proposed here.
> Enable TLS Toolkit (standalone) to sign certificates with external CA
> certificate
> -
>
> Key: NIFI-5476
> URL: https://issues.apache.org/jira/browse/NIFI-5476
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Security, Tools and Build
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: certificate, pem, pkcs1, pkcs8, pki, security, tls,
> tls-toolkit
>
> The TLS Toolkit can sign certificates using a public certificate and private
> key generated and signed elsewhere by injecting them into the
> {{nifi-cert.pem}} and {{nifi-key.key}} files as long as they are in the
> proper format and self-signed. The toolkit should be enhanced to handle PKCS
> #8 formatted private keys (in addition to the PKCS #1 formatted keys it
> handles now) and to allow for non self-signed certificates.
> To verify this, use certificates generated by
> [TinyCert|https://tinycert.org].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2935: NIFI-5476 Allow TLS toolkit to use externally-signed CA in...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2935 Thanks for the contributions @alopresto and @pepov! I'm also a +1 on the changes proposed here. ---
[jira] [Commented] (NIFI-5448) Failed EL date parsing live-locks processors without a failure relationship
[
https://issues.apache.org/jira/browse/NIFI-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572132#comment-16572132
]
ASF GitHub Bot commented on NIFI-5448:
--
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2914
@zenfenan changed it.
> Failed EL date parsing live-locks processors without a failure relationship
> ---
>
> Key: NIFI-5448
> URL: https://issues.apache.org/jira/browse/NIFI-5448
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: David Koster
>Assignee: Mike Thomsen
>Priority: Major
>
> Processors that utilize the Expression Language need to always present a
> failure relationship.
> If a processor with only a success relationship, for example UpdateAttribute,
> utilizes the expression language to perform type coercion to a date and
> fails, the processor will be unable to dispose of the FlowFile and remain
> blocked indefinitely.
> Recreation flow:
> GenerateFlowFile -> Update Attribute #1 -> Update Attribute #2 -> Anything
> Update Attribute #1 - test = "Hello World"
> Update Attribute #2 - test = ${test:toDate('-MM-dd')}
>
> Generates an IllegalAttributeException on UpdateAttribute.
>
> The behavior should match numerical type coercion and silently skip the
> processing or offer failure relationships on processors supporting EL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2914: NIFI-5448 Added failure relationship to UpdateAttributes t...
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2914 @zenfenan changed it. ---
[jira] [Assigned] (NIFI-5494) Add Expression Language support to "Hive Configuration Resources" in PutHiveStreaming processor
[ https://issues.apache.org/jira/browse/NIFI-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaprasanna Sethuraman reassigned NIFI-5494: - Assignee: Sivaprasanna Sethuraman > Add Expression Language support to "Hive Configuration Resources" in > PutHiveStreaming processor > --- > > Key: NIFI-5494 > URL: https://issues.apache.org/jira/browse/NIFI-5494 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.0 >Reporter: Rahul Soni >Assignee: Sivaprasanna Sethuraman >Priority: Minor > > PutHiveSteaming processor's "Hive Configuration Resources" property do not > support Expression Language and it causes issues while promoting flows using > NiFi registry as changing the value of this property to point to some other > cluster render the flow out of sync. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi pull request #134: MINIFI-468 Updating Docker resources to next ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi/pull/134 ---
[GitHub] nifi-minifi pull request #135: MINIFI-469 - support OSX in travis automated ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi/pull/135 ---
[jira] [Commented] (NIFI-5489) Support Attribute Expressions with AMQP Processors
[ https://issues.apache.org/jira/browse/NIFI-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572106#comment-16572106 ] ASF GitHub Bot commented on NIFI-5489: -- Github user mosermw commented on the issue: https://github.com/apache/nifi/pull/2936 Please do add Port to your PR, if you can, then I think this will also cover NIFI-4723. It looks like PORT_VALIDATOR already supports expression language, so you are good there. > Support Attribute Expressions with AMQP Processors > -- > > Key: NIFI-5489 > URL: https://issues.apache.org/jira/browse/NIFI-5489 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.1 >Reporter: Daniel >Priority: Major > > Particularly the fields: host, virtualhost and username. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user mosermw commented on the issue: https://github.com/apache/nifi/pull/2936 Please do add Port to your PR, if you can, then I think this will also cover NIFI-4723. It looks like PORT_VALIDATOR already supports expression language, so you are good there. ---
[jira] [Assigned] (NIFI-4771) Consolidate Docker resources into nifi-container
[ https://issues.apache.org/jira/browse/NIFI-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned NIFI-4771: - Assignee: (was: Aldrin Piri) > Consolidate Docker resources into nifi-container > > > Key: NIFI-4771 > URL: https://issues.apache.org/jira/browse/NIFI-4771 > Project: Apache NiFi > Issue Type: Task > Components: Docker >Reporter: Aldrin Piri >Priority: Major > > With the creation of DockerHub repositories for all major components within > NiFi, we can consolidate efforts and resources in the nifi-container. > Currently we have existing Docker work for NiFi, NiFi Toolkit, NiFi MiNiFI > Java & C++. > These should be transitioned to the container repo and established with > Docker Hub for the associated convenience binaries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5489) Support Attribute Expressions with AMQP Processors
[ https://issues.apache.org/jira/browse/NIFI-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572103#comment-16572103 ] ASF GitHub Bot commented on NIFI-5489: -- Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 @mosermw I can, I just enabled the fields most likely to be variables for most use cases. If you'd like me to add that to this PR, I certainly can. > Support Attribute Expressions with AMQP Processors > -- > > Key: NIFI-5489 > URL: https://issues.apache.org/jira/browse/NIFI-5489 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.1 >Reporter: Daniel >Priority: Major > > Particularly the fields: host, virtualhost and username. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2914: NIFI-5448 Added failure relationship to UpdateAttributes t...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2914 @MikeThomsen The suggestion made by @mcgilman sounds good to me. If we could rename `Stop Processing` to `Penalize`, that would make more sense. Thoughts? ---
[jira] [Commented] (NIFI-5448) Failed EL date parsing live-locks processors without a failure relationship
[
https://issues.apache.org/jira/browse/NIFI-5448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572102#comment-16572102
]
ASF GitHub Bot commented on NIFI-5448:
--
Github user zenfenan commented on the issue:
https://github.com/apache/nifi/pull/2914
@MikeThomsen The suggestion made by @mcgilman sounds good to me. If we
could rename `Stop Processing` to `Penalize`, that would make more sense.
Thoughts?
> Failed EL date parsing live-locks processors without a failure relationship
> ---
>
> Key: NIFI-5448
> URL: https://issues.apache.org/jira/browse/NIFI-5448
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: David Koster
>Assignee: Mike Thomsen
>Priority: Major
>
> Processors that utilize the Expression Language need to always present a
> failure relationship.
> If a processor with only a success relationship, for example UpdateAttribute,
> utilizes the expression language to perform type coercion to a date and
> fails, the processor will be unable to dispose of the FlowFile and remain
> blocked indefinitely.
> Recreation flow:
> GenerateFlowFile -> Update Attribute #1 -> Update Attribute #2 -> Anything
> Update Attribute #1 - test = "Hello World"
> Update Attribute #2 - test = ${test:toDate('-MM-dd')}
>
> Generates an IllegalAttributeException on UpdateAttribute.
>
> The behavior should match numerical type coercion and silently skip the
> processing or offer failure relationships on processors supporting EL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 @mosermw I can, I just enabled the fields most likely to be variables for most use cases. If you'd like me to add that to this PR, I certainly can. ---
[jira] [Commented] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572098#comment-16572098
]
Otto Fowler commented on NIFI-5495:
---
[~mike.thomsen]
Part of the issue is that the SimpleDate validators we have are buried in
Serialization. I think they need to be more centrally available ( moved to
StandardValidators ).
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Assignee: Mike Thomsen
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5489) Support Attribute Expressions with AMQP Processors
[ https://issues.apache.org/jira/browse/NIFI-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572090#comment-16572090 ] Michael Moser commented on NIFI-5489: - See also NIFI-4723 and NIFI-5118 > Support Attribute Expressions with AMQP Processors > -- > > Key: NIFI-5489 > URL: https://issues.apache.org/jira/browse/NIFI-5489 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.1 >Reporter: Daniel >Priority: Major > > Particularly the fields: host, virtualhost and username. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4434) ListHDFS applies File Filter also to subdirectory names in recursive search
[ https://issues.apache.org/jira/browse/NIFI-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572091#comment-16572091 ] ASF GitHub Bot commented on NIFI-4434: -- Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/2937 @ottobackwards @zenfenan I'll break out my rusty HTML skills and try to write up some extra documentation with examples/usecases. Hopefully in an hour or two I'll have an update for the PR. Thanks for the reviews! > ListHDFS applies File Filter also to subdirectory names in recursive search > --- > > Key: NIFI-4434 > URL: https://issues.apache.org/jira/browse/NIFI-4434 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Holger Frydrych >Assignee: Jeff Storck >Priority: Major > > The File Filter regex configured in the ListHDFS processor is applied not > just to files found, but also to subdirectories. > If you try to set up a recursive search to list e.g. all csv files in a > directory hierarchy via a regex like ".*\.csv", it will only pick up csv > files in the base directory, not in any subdirectory. This is because > subdirectories don't typically match that regex pattern. > To fix this, either subdirectories should not be matched against the file > filter, or the file filter should be applied to the full path of all files > (relative to the base directory). The GetHDFS processor offers both options > via a switch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2937: NIFI-4434 Fixed recursive listing with a custom regex filt...
Github user jtstorck commented on the issue: https://github.com/apache/nifi/pull/2937 @ottobackwards @zenfenan I'll break out my rusty HTML skills and try to write up some extra documentation with examples/usecases. Hopefully in an hour or two I'll have an update for the PR. Thanks for the reviews! ---
[jira] [Commented] (NIFI-5489) Support Attribute Expressions with AMQP Processors
[ https://issues.apache.org/jira/browse/NIFI-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572086#comment-16572086 ] ASF GitHub Bot commented on NIFI-5489: -- Github user mosermw commented on the issue: https://github.com/apache/nifi/pull/2936 If you are using VARIABLE_REGISTRY to dynamically evaluate the Host, shouldn't you also allow it for Port? > Support Attribute Expressions with AMQP Processors > -- > > Key: NIFI-5489 > URL: https://issues.apache.org/jira/browse/NIFI-5489 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.1 >Reporter: Daniel >Priority: Major > > Particularly the fields: host, virtualhost and username. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user mosermw commented on the issue: https://github.com/apache/nifi/pull/2936 If you are using VARIABLE_REGISTRY to dynamically evaluate the Host, shouldn't you also allow it for Port? ---
[jira] [Commented] (NIFI-4246) OAuth 2 Authorization support - Client Credentials Grant
[
https://issues.apache.org/jira/browse/NIFI-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572049#comment-16572049
]
Romain Guay commented on NIFI-4246:
---
Hi all and thanks for this useful work.
I am new to this conversation and relatively new to Nifi, but I have of course
the need to OAuth authentication, and I also have a similar need with a custom
token-based authentication which I think is not so different. I wonder if we
could take this opportunity to open our options by defining a slightly more
genetic interface for this authentication service and allow multiple
implementations for the same InvokeHttp processor.
Here is what I would suggest:
{noformat}
public interface AuthenticationProviderClientService extends ControllerService {
boolean isTokenExpired();
boolean isTokenExpired(int statusCode, String message);
Map getAuthenticationHeaders();
boolean authenticate();
}{noformat}
The isTokenExpired() method is used as currently when preparing the request.
The isTokenExpired(String response) method would be used in cases where the
token does not know its expiration, but we can determine it from the statusCode
and message from the HTTP response. If it is expired, authenticate() is called
and the flow file sent to retry. So each implementation would use either one or
the other, returning false by default.
I also think having getAuthenticationHeaders() instead of getAccessToken()
makes it more generic.
Please let me know if this all makes sense.
> OAuth 2 Authorization support - Client Credentials Grant
>
>
> Key: NIFI-4246
> URL: https://issues.apache.org/jira/browse/NIFI-4246
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Jeremy Dyer
>Assignee: Jeremy Dyer
>Priority: Major
>
> If your interacting with REST endpoints on the web chances are you are going
> to run into an OAuth2 secured webservice. The IETF (Internet Engineering Task
> Force) defines 4 methods in which OAuth2 authorization can occur. This JIRA
> is focused solely on the Client Credentials Grant method defined at
> https://tools.ietf.org/html/rfc6749#section-4.4
> This implementation should provide a ControllerService in which the enduser
> can configure the credentials for obtaining the authorization grant (access
> token) from the resource owner. In turn a new property will be added to the
> InvokeHTTP processor (if it doesn't already exist from one of the other JIRA
> efforts similar to this one) where the processor can reference this
> controller service to obtain the access token and insert the appropriate HTTP
> header (Authorization: Bearer{access_token}) so that the InvokeHTTP processor
> can interact with the OAuth protected resources without having to worry about
> setting up the credentials for each InvokeHTTP processor saving time and
> complexity.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Thomsen reassigned NIFI-5495:
--
Assignee: Mike Thomsen
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Assignee: Mike Thomsen
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572033#comment-16572033 ] ASF GitHub Bot commented on NIFI-5479: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2933 @joewitt Thanks for the PR! When starting up in secure mode using a configuration that works with current master branch, I received some stack traces regarding the initialization of the `SSLContext`. There appears to be a runtime difference introduced here that affects the loading of providers. > 1305 Caused by: java.security.NoSuchAlgorithmException: no such algorithm: JKS for provider BC 1306 at sun.security.jca.GetInstance.getService(GetInstance.java:87) 1307 at sun.security.jca.GetInstance.getInstance(GetInstance.java:206) 1308 at java.security.Security.getImpl(Security.java:698) 1309 at java.security.KeyStore.getInstance(KeyStore.java:896) 1310 ... 21 common frames omitted > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Major > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2933: NIFI-5479 Upgraded Jetty. Moved where we unpack bundled de...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2933 @joewitt Thanks for the PR! When starting up in secure mode using a configuration that works with current master branch, I received some stack traces regarding the initialization of the `SSLContext`. There appears to be a runtime difference introduced here that affects the loading of providers. > 1305 Caused by: java.security.NoSuchAlgorithmException: no such algorithm: JKS for provider BC 1306 at sun.security.jca.GetInstance.getService(GetInstance.java:87) 1307 at sun.security.jca.GetInstance.getInstance(GetInstance.java:206) 1308 at java.security.Security.getImpl(Security.java:698) 1309 at java.security.KeyStore.getInstance(KeyStore.java:896) 1310 ... 21 common frames omitted ---
[jira] [Commented] (NIFI-5489) Support Attribute Expressions with AMQP Processors
[ https://issues.apache.org/jira/browse/NIFI-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572028#comment-16572028 ] ASF GitHub Bot commented on NIFI-5489: -- Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2936 @danieljimenez correct, PRs merged now will be included in the next release. Because our *bug fix* releases (`x.x.1`) do not include feature work, this will be in the next *minor* release, which is `1.8.0`. > Support Attribute Expressions with AMQP Processors > -- > > Key: NIFI-5489 > URL: https://issues.apache.org/jira/browse/NIFI-5489 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.1 >Reporter: Daniel >Priority: Major > > Particularly the fields: host, virtualhost and username. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2936 @danieljimenez correct, PRs merged now will be included in the next release. Because our *bug fix* releases (`x.x.1`) do not include feature work, this will be in the next *minor* release, which is `1.8.0`. ---
[jira] [Commented] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572022#comment-16572022
]
Otto Fowler commented on NIFI-5495:
---
I was just pasting the code in here when I refreshed as saw your comment. You
beat me to it!
I changed the name of the ticket to match the actual ask. Hope you don't mind.
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5495) Allow configuration of DateFormat for Mongo Processors
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Otto Fowler updated NIFI-5495:
--
Summary: Allow configuration of DateFormat for Mongo Processors (was:
GetMongo Truncates Dates)
> Allow configuration of DateFormat for Mongo Processors
> --
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (MINIFICPP-594) CapturePCAP predates PROCESSORS.md and thus was never added
[ https://issues.apache.org/jira/browse/MINIFICPP-594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-594: -- Fix Version/s: 0.6.0 > CapturePCAP predates PROCESSORS.md and thus was never added > --- > > Key: MINIFICPP-594 > URL: https://issues.apache.org/jira/browse/MINIFICPP-594 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > NetworkControllers -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-594) CapturePCAP predates PROCESSORS.md and thus was never added
[ https://issues.apache.org/jira/browse/MINIFICPP-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572008#comment-16572008 ] ASF GitHub Bot commented on MINIFICPP-594: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/390 > CapturePCAP predates PROCESSORS.md and thus was never added > --- > > Key: MINIFICPP-594 > URL: https://issues.apache.org/jira/browse/MINIFICPP-594 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > NetworkControllers -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-594) CapturePCAP predates PROCESSORS.md and thus was never added
[ https://issues.apache.org/jira/browse/MINIFICPP-594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFICPP-594. --- Resolution: Fixed > CapturePCAP predates PROCESSORS.md and thus was never added > --- > > Key: MINIFICPP-594 > URL: https://issues.apache.org/jira/browse/MINIFICPP-594 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > NetworkControllers -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4517) Allow SQL results to be output as records in any supported format
[ https://issues.apache.org/jira/browse/NIFI-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572006#comment-16572006 ] Matt Burgess commented on NIFI-4517: How about I take this one and you take NIFI-5405 and we review each other's :) I don't think the order should matter (we aren't touching the same stuff right?) as long as both get into the same release. Thoughts? > Allow SQL results to be output as records in any supported format > - > > Key: NIFI-4517 > URL: https://issues.apache.org/jira/browse/NIFI-4517 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Peter Wicks >Priority: Major > > ExecuteSQL and QueryDatabaseTable currently only outputs Avro, and the schema > is only available as embedded within the flow file, not as an attribute such > as record-aware processors can handle. > ExecuteSQL and QueryDatabaseTable processors should be updated with a > RecordSetWriter implementation. This will allow output using any writer > format (Avro, JSON, CSV, Free-form text, etc.), as well as all the other > features therein (such as writing the schema to an attribute, and will avoid > the need for a ConvertAvroToXYZ or ConvertRecord processor downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #390: MINIFICPP-594: Add CapturePacket to Proce...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/390 ---
[jira] [Commented] (NIFI-5495) GetMongo Truncates Dates
[
https://issues.apache.org/jira/browse/NIFI-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572005#comment-16572005
]
Ryan Hendrickson commented on NIFI-5495:
Looks like the date is hard-coded to a specific format:
[https://github.com/apache/nifi/blob/b1f78d58a6f5250b4f4e6873b03459a8e987407a/nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java#L319]
I wonder if that could be a configurable property.
> GetMongo Truncates Dates
>
>
> Key: NIFI-5495
> URL: https://issues.apache.org/jira/browse/NIFI-5495
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
> Environment: CentOS 7.5, Java 1.8.0 u172
>Reporter: Ryan Hendrickson
>Priority: Blocker
>
> When using the GetMongo, configured with JSON Type of "Standard JSON", it
> truncates dates with milliseconds.
>
> I've got a document in Mongo that has a date field that looks like the
> following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4517) Allow SQL results to be output as records in any supported format
[ https://issues.apache.org/jira/browse/NIFI-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571998#comment-16571998 ] Peter Wicks commented on NIFI-4517: --- Yes and no. I started, but felt that some work was needed on the Avro front, so started NIFI-5405. I think NIFI-5405 is basically done, but I needed to do some testing when I got pulled off onto other work. Thoughts on the order? > Allow SQL results to be output as records in any supported format > - > > Key: NIFI-4517 > URL: https://issues.apache.org/jira/browse/NIFI-4517 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Peter Wicks >Priority: Major > > ExecuteSQL and QueryDatabaseTable currently only outputs Avro, and the schema > is only available as embedded within the flow file, not as an attribute such > as record-aware processors can handle. > ExecuteSQL and QueryDatabaseTable processors should be updated with a > RecordSetWriter implementation. This will allow output using any writer > format (Avro, JSON, CSV, Free-form text, etc.), as well as all the other > features therein (such as writing the schema to an attribute, and will avoid > the need for a ConvertAvroToXYZ or ConvertRecord processor downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5466) Improve efficiency of RingBufferEventRepository
[
https://issues.apache.org/jira/browse/NIFI-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-5466:
-
Fix Version/s: 1.8.0
Status: Patch Available (was: Open)
> Improve efficiency of RingBufferEventRepository
> ---
>
> Key: NIFI-5466
> URL: https://issues.apache.org/jira/browse/NIFI-5466
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.8.0
>
>
> The implementation of the FlowFileEventRepository (i.e., the
> RingBufferEventRepository) is implemented in such a way that 300 buckets are
> kept for each component. Each bucket holds a set of 16+ stats (bytes read,
> bytes written, flowfiles in, flowfiles out, etc. as well as counters)
> Each time that a user refreshes stats, we calculate the 5-minute window for
> each component by iterating over all 300 buckets, then summing together the
> values of the stats. So, for every 1,000 processors, we have 1000 * 300 * 16
> = 4.8 MM calculations being performed each time that the user refreshes.
> We can improve this significantly. To do this, we can keep a 'running tally'
> for each stat. Every time that we update a bucket in the
> FlowFileEventRepository, we should similarly update the running tally. Each
> time that we 'replace' a bucket in the FlowFileEventRepository, we should
> subtract from the running tally the values in the bucket, in addition to
> adding in the new value. This would result in amortizing the cost of
> calculation over the 5-minute period and means that every time that we get
> the processor stats we could do so by simply performing a 'get' operation
> without any calculations.
> But wait, there's more! The EventSumValue class has an {{add(FlowFileEvent)}}
> method. If the EventSumValue were to simply keep a boolean value indicating
> whether or not it was empty, then when adding to an empty bucket we could
> avoid the calculations all together, and when replacing an empty bucket we
> could avoid the calculation against the running tally as well.
> Similarly, in the VolatileComponentStatusRepository, when we perform a
> 'capture' we should detect any 'empty' stats objects and use a single static
> instance for this, which would significantly reduce the amount of heap that
> is used to store metrics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5466) Improve efficiency of RingBufferEventRepository
[ https://issues.apache.org/jira/browse/NIFI-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571991#comment-16571991 ] ASF GitHub Bot commented on NIFI-5466: -- GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/2939 NIFI-5466: Keep a running total of stats for each component. Refactor… …ed FlowFileEvent and repository in order to provide more efficient storage of objects on Java heap by allowing the same 'EMPTY' object to be reused - Refactored VolatileComponentStatusRepository to avoid holding on to ProcessorStatus objects, etc, and only keep what they need - Updated VolatileComponentStatusRepository to ensure that we are efficiently storing metrics for processors, etc. that are not running Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5466 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2939.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2939 commit 278f2ae1c395219a584debf252d8491fe507ad17 Author: Mark Payne Date: 2018-08-02T18:17:36Z NIFI-5466: Keep a running total of stats for each component. Refactored FlowFileEvent and repository in order to provide more efficient storage of objects on Java heap by allowing the same 'EMPTY' object to be reused - Refactored VolatileComponentStatusRepository to avoid holding on to ProcessorStatus objects, etc, and only keep what they need - Updated VolatileComponentStatusRepository to ensure that we are efficiently storing metrics for processors, etc. that are not running > Improve efficiency of RingBufferEventRepository > --- > > Key: NIFI-5466 > URL: https://issues.apache.org/jira/browse/NIFI-5466 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.8.0 > > > The implementation of the FlowFileEventRepository (i.e., the > RingBufferEventRepository) is implemented in such a way that 300 buckets are > kept for each component. Each bucket holds a set of 16+ stats (bytes read, > bytes written, flowfiles in, flowfiles out, etc. as well as counters) > Each time that a user refreshes stats, we calculate the 5-minute window for > each component by iterating over all 300 buckets, then summing together the > values of the stats. So, for every 1,000 processors, we have 1000 * 300 * 16 > = 4.8 MM calculations being performed each time that the user refreshes. > We can improve this significantly. To do this, we can keep a 'running tally' > for each stat. Every time that we update a bucket in the > FlowFileEventRepository, we should similarly update the running tally. Each > time that we 'replace' a bucket in the FlowFileEventRepository, we should >
[GitHub] nifi pull request #2939: NIFI-5466: Keep a running total of stats for each c...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/2939 NIFI-5466: Keep a running total of stats for each component. Refactor⦠â¦ed FlowFileEvent and repository in order to provide more efficient storage of objects on Java heap by allowing the same 'EMPTY' object to be reused - Refactored VolatileComponentStatusRepository to avoid holding on to ProcessorStatus objects, etc, and only keep what they need - Updated VolatileComponentStatusRepository to ensure that we are efficiently storing metrics for processors, etc. that are not running Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5466 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2939.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2939 commit 278f2ae1c395219a584debf252d8491fe507ad17 Author: Mark Payne Date: 2018-08-02T18:17:36Z NIFI-5466: Keep a running total of stats for each component. Refactored FlowFileEvent and repository in order to provide more efficient storage of objects on Java heap by allowing the same 'EMPTY' object to be reused - Refactored VolatileComponentStatusRepository to avoid holding on to ProcessorStatus objects, etc, and only keep what they need - Updated VolatileComponentStatusRepository to ensure that we are efficiently storing metrics for processors, etc. that are not running ---
[jira] [Resolved] (MINIFICPP-573) break apart the component manifest into separated bundles
[ https://issues.apache.org/jira/browse/MINIFICPP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFICPP-573. --- Resolution: Fixed > break apart the component manifest into separated bundles > - > > Key: MINIFICPP-573 > URL: https://issues.apache.org/jira/browse/MINIFICPP-573 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-573) break apart the component manifest into separated bundles
[ https://issues.apache.org/jira/browse/MINIFICPP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571960#comment-16571960 ] ASF GitHub Bot commented on MINIFICPP-573: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/381 > break apart the component manifest into separated bundles > - > > Key: MINIFICPP-573 > URL: https://issues.apache.org/jira/browse/MINIFICPP-573 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-573) break apart the component manifest into separated bundles
[ https://issues.apache.org/jira/browse/MINIFICPP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-573: -- Fix Version/s: 0.6.0 > break apart the component manifest into separated bundles > - > > Key: MINIFICPP-573 > URL: https://issues.apache.org/jira/browse/MINIFICPP-573 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #381: MINIFICPP-573: Remove lower level build i...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/381 ---
[jira] [Created] (NIFI-5495) GetMongo Truncates Dates
Ryan Hendrickson created NIFI-5495:
--
Summary: GetMongo Truncates Dates
Key: NIFI-5495
URL: https://issues.apache.org/jira/browse/NIFI-5495
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.7.1
Environment: CentOS 7.5, Java 1.8.0 u172
Reporter: Ryan Hendrickson
When using the GetMongo, configured with JSON Type of "Standard JSON", it
truncates dates with milliseconds.
I've got a document in Mongo that has a date field that looks like the
following:
{
...
"date" : ISODate("2018-08-06T16:20:10.912Z"
...
}
When GetMongo spits it out, the date comes out as: "2018-08-06T16:20:10Z",
noticeably missing the milliseconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4517) Allow SQL results to be output as records in any supported format
[ https://issues.apache.org/jira/browse/NIFI-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571948#comment-16571948 ] Matt Burgess commented on NIFI-4517: [~patricker] Are you actively working this? If not, I'd be happy to take over, I had started working on it before I checked the Jira and saw that it was assigned to you :) > Allow SQL results to be output as records in any supported format > - > > Key: NIFI-4517 > URL: https://issues.apache.org/jira/browse/NIFI-4517 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Peter Wicks >Priority: Major > > ExecuteSQL and QueryDatabaseTable currently only outputs Avro, and the schema > is only available as embedded within the flow file, not as an attribute such > as record-aware processors can handle. > ExecuteSQL and QueryDatabaseTable processors should be updated with a > RecordSetWriter implementation. This will allow output using any writer > format (Avro, JSON, CSV, Free-form text, etc.), as well as all the other > features therein (such as writing the schema to an attribute, and will avoid > the need for a ConvertAvroToXYZ or ConvertRecord processor downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5494) Add Expression Language support to "Hive Configuration Resources" in PutHiveStreaming processor
Rahul Soni created NIFI-5494: Summary: Add Expression Language support to "Hive Configuration Resources" in PutHiveStreaming processor Key: NIFI-5494 URL: https://issues.apache.org/jira/browse/NIFI-5494 Project: Apache NiFi Issue Type: Improvement Components: Extensions Affects Versions: 1.7.0 Reporter: Rahul Soni PutHiveSteaming processor's "Hive Configuration Resources" property do not support Expression Language and it causes issues while promoting flows using NiFi registry as changing the value of this property to point to some other cluster render the flow out of sync. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5489) Support Attribute Expressions with AMQP Processors
[ https://issues.apache.org/jira/browse/NIFI-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571929#comment-16571929 ] ASF GitHub Bot commented on NIFI-5489: -- Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 Missed a spot evaluating the attribute expressions. Fixed. > Support Attribute Expressions with AMQP Processors > -- > > Key: NIFI-5489 > URL: https://issues.apache.org/jira/browse/NIFI-5489 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.1 >Reporter: Daniel >Priority: Major > > Particularly the fields: host, virtualhost and username. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 Missed a spot evaluating the attribute expressions. Fixed. ---
[jira] [Commented] (NIFI-2415) GenerateTableFetch/QueryDatabaseTable cannot store state
[
https://issues.apache.org/jira/browse/NIFI-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571821#comment-16571821
]
Daniel commented on NIFI-2415:
--
Thanks [~mattyb149], mine is similar but not the same. I'll open a new ticket
rather than kick this dead horse. Thanks for your response.
> GenerateTableFetch/QueryDatabaseTable cannot store state
>
>
> Key: NIFI-2415
> URL: https://issues.apache.org/jira/browse/NIFI-2415
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Assignee: Matt Burgess
>Priority: Critical
> Fix For: 1.0.0, 1.0.0-Beta
>
> Attachments: Screen Shot 2016-07-27 at 11.10.45 PM.png
>
>
> running NIFI in a clustered mode, 7/27 build
> Both GenerateTableFetch and QueryDatabaseTable cannot store state in a
> clustered mode (checked ListFile, can store state successfully)
> see attached screenshot, trace below:
> 2016-07-27 23:25:00,014 INFO [ProcessThread(sid:1 cport:-1):]
> o.a.z.server.PrepRequestProcessor Got user-level KeeperException when
> processing sessionid:0x2562f7875130009 type:create cxid:0xe4d
> zxid:0x73733 txntype:-1 reqpath:n/a Error
> Path:/nifi/components/2f810bb3-0156-1000--16128f8b
> Error:KeeperErrorCode = InvalidACL for
> /nifi/components/2f810bb3-0156-1000--16128f8b
> 2016-07-27 23:25:00,017 WARN [Framework Task Thread Thread-1]
> o.a.n.c.s.m.StandardStateManagerProvider Component with ID {} was removed
> from NiFi instance but failed to clear clustered state for the component
> java.io.IOException: Failed to set cluster-wide state in ZooKeeper for
> component with ID 2f810bb3-0156-1000--16128f8b
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:343)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.clear(ZooKeeperStateProvider.java:453)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.StandardStateManager.clear(StandardStateManager.java:85)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.manager.StandardStateManagerProvider.onComponentRemoved(StandardStateManagerProvider.java:276)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.groups.StandardProcessGroup$1.run(StandardProcessGroup.java:730)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_77]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_77]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_77]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> Caused by: org.apache.zookeeper.KeeperException$InvalidACLException:
> KeeperErrorCode = InvalidACL for
> /nifi/components/2f810bb3-0156-1000--16128f8b
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> ... 13 common frames omitted
--
This message was sent by Atlassian JIRA
[jira] [Commented] (NIFI-2415) GenerateTableFetch/QueryDatabaseTable cannot store state
[
https://issues.apache.org/jira/browse/NIFI-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571805#comment-16571805
]
Matt Burgess commented on NIFI-2415:
I think it was written before 1.0.0 was released, and the value or config or
whatever was corrected for the 1.0.0 release. What version of NiFi are you
using? Is the same stacktrace or something different?
> GenerateTableFetch/QueryDatabaseTable cannot store state
>
>
> Key: NIFI-2415
> URL: https://issues.apache.org/jira/browse/NIFI-2415
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Assignee: Matt Burgess
>Priority: Critical
> Fix For: 1.0.0, 1.0.0-Beta
>
> Attachments: Screen Shot 2016-07-27 at 11.10.45 PM.png
>
>
> running NIFI in a clustered mode, 7/27 build
> Both GenerateTableFetch and QueryDatabaseTable cannot store state in a
> clustered mode (checked ListFile, can store state successfully)
> see attached screenshot, trace below:
> 2016-07-27 23:25:00,014 INFO [ProcessThread(sid:1 cport:-1):]
> o.a.z.server.PrepRequestProcessor Got user-level KeeperException when
> processing sessionid:0x2562f7875130009 type:create cxid:0xe4d
> zxid:0x73733 txntype:-1 reqpath:n/a Error
> Path:/nifi/components/2f810bb3-0156-1000--16128f8b
> Error:KeeperErrorCode = InvalidACL for
> /nifi/components/2f810bb3-0156-1000--16128f8b
> 2016-07-27 23:25:00,017 WARN [Framework Task Thread Thread-1]
> o.a.n.c.s.m.StandardStateManagerProvider Component with ID {} was removed
> from NiFi instance but failed to clear clustered state for the component
> java.io.IOException: Failed to set cluster-wide state in ZooKeeper for
> component with ID 2f810bb3-0156-1000--16128f8b
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:343)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.clear(ZooKeeperStateProvider.java:453)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.StandardStateManager.clear(StandardStateManager.java:85)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.manager.StandardStateManagerProvider.onComponentRemoved(StandardStateManagerProvider.java:276)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.groups.StandardProcessGroup$1.run(StandardProcessGroup.java:730)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_77]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_77]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_77]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> Caused by: org.apache.zookeeper.KeeperException$InvalidACLException:
> KeeperErrorCode = InvalidACL for
> /nifi/components/2f810bb3-0156-1000--16128f8b
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> ...
[jira] [Commented] (MINIFICPP-573) break apart the component manifest into separated bundles
[ https://issues.apache.org/jira/browse/MINIFICPP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571803#comment-16571803 ] ASF GitHub Bot commented on MINIFICPP-573: -- Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/381 looks like we have a test failure: ``` --- Test Valid Regex --- /home/apiri/development/code/nifi-minifi-cpp/libminifi/test/unit/PropertyValidationAgentInfoTests.cpp:47 ... /home/apiri/development/code/nifi-minifi-cpp/libminifi/test/unit/PropertyValidationAgentInfoTests.cpp:52: FAILED: REQUIRE( resp.children.size() > 0 ) with expansion: 0 > 0 ``` > break apart the component manifest into separated bundles > - > > Key: MINIFICPP-573 > URL: https://issues.apache.org/jira/browse/MINIFICPP-573 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #381: MINIFICPP-573: Remove lower level build informat...
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/381 looks like we have a test failure: ``` --- Test Valid Regex --- /home/apiri/development/code/nifi-minifi-cpp/libminifi/test/unit/PropertyValidationAgentInfoTests.cpp:47 ... /home/apiri/development/code/nifi-minifi-cpp/libminifi/test/unit/PropertyValidationAgentInfoTests.cpp:52: FAILED: REQUIRE( resp.children.size() > 0 ) with expansion: 0 > 0 ``` ---
[jira] [Commented] (NIFI-2415) GenerateTableFetch/QueryDatabaseTable cannot store state
[
https://issues.apache.org/jira/browse/NIFI-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571786#comment-16571786
]
Daniel commented on NIFI-2415:
--
Still seeing this, what was the fix then if not a bug?
> GenerateTableFetch/QueryDatabaseTable cannot store state
>
>
> Key: NIFI-2415
> URL: https://issues.apache.org/jira/browse/NIFI-2415
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Assignee: Matt Burgess
>Priority: Critical
> Fix For: 1.0.0, 1.0.0-Beta
>
> Attachments: Screen Shot 2016-07-27 at 11.10.45 PM.png
>
>
> running NIFI in a clustered mode, 7/27 build
> Both GenerateTableFetch and QueryDatabaseTable cannot store state in a
> clustered mode (checked ListFile, can store state successfully)
> see attached screenshot, trace below:
> 2016-07-27 23:25:00,014 INFO [ProcessThread(sid:1 cport:-1):]
> o.a.z.server.PrepRequestProcessor Got user-level KeeperException when
> processing sessionid:0x2562f7875130009 type:create cxid:0xe4d
> zxid:0x73733 txntype:-1 reqpath:n/a Error
> Path:/nifi/components/2f810bb3-0156-1000--16128f8b
> Error:KeeperErrorCode = InvalidACL for
> /nifi/components/2f810bb3-0156-1000--16128f8b
> 2016-07-27 23:25:00,017 WARN [Framework Task Thread Thread-1]
> o.a.n.c.s.m.StandardStateManagerProvider Component with ID {} was removed
> from NiFi instance but failed to clear clustered state for the component
> java.io.IOException: Failed to set cluster-wide state in ZooKeeper for
> component with ID 2f810bb3-0156-1000--16128f8b
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:343)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.clear(ZooKeeperStateProvider.java:453)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.StandardStateManager.clear(StandardStateManager.java:85)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.manager.StandardStateManagerProvider.onComponentRemoved(StandardStateManagerProvider.java:276)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.groups.StandardProcessGroup$1.run(StandardProcessGroup.java:730)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_77]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_77]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_77]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_77]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> Caused by: org.apache.zookeeper.KeeperException$InvalidACLException:
> KeeperErrorCode = InvalidACL for
> /nifi/components/2f810bb3-0156-1000--16128f8b
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:121)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> ... 13 common frames omitted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-594) CapturePCAP predates PROCESSORS.md and thus was never added
[ https://issues.apache.org/jira/browse/MINIFICPP-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571763#comment-16571763 ] ASF GitHub Bot commented on MINIFICPP-594: -- GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/390 MINIFICPP-594: Add CapturePacket to Processors documentation MINIFICPP-590: Update boolean to not keep source PCAP -- done in testing for MINIFICPP-590 fix. Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-594 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/390.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #390 commit 622b63174ee7c24534971906815bddb2ba4426d0 Author: Marc Parisi Date: 2018-08-07T14:33:28Z MINIFICPP-594: Add CapturePacket to Processors documentation MINIFICPP-590: Update boolean to not keep source PCAP -- done in testing for MINIFICPP-590 fix. > CapturePCAP predates PROCESSORS.md and thus was never added > --- > > Key: MINIFICPP-594 > URL: https://issues.apache.org/jira/browse/MINIFICPP-594 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > > NetworkControllers -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #390: MINIFICPP-594: Add CapturePacket to Proce...
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/390 MINIFICPP-594: Add CapturePacket to Processors documentation MINIFICPP-590: Update boolean to not keep source PCAP -- done in testing for MINIFICPP-590 fix. Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-594 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/390.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #390 commit 622b63174ee7c24534971906815bddb2ba4426d0 Author: Marc Parisi Date: 2018-08-07T14:33:28Z MINIFICPP-594: Add CapturePacket to Processors documentation MINIFICPP-590: Update boolean to not keep source PCAP -- done in testing for MINIFICPP-590 fix. ---
[jira] [Commented] (NIFI-5489) Support Attribute Expressions with AMQP Processors
[ https://issues.apache.org/jira/browse/NIFI-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571747#comment-16571747 ] ASF GitHub Bot commented on NIFI-5489: -- Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 Awesome, thanks! That would mean it's targeted for 1.8.0, correct? > Support Attribute Expressions with AMQP Processors > -- > > Key: NIFI-5489 > URL: https://issues.apache.org/jira/browse/NIFI-5489 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.7.1 >Reporter: Daniel >Priority: Major > > Particularly the fields: host, virtualhost and username. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2936: NIFI-5489: Add expression language support to AMQP process...
Github user danieljimenez commented on the issue: https://github.com/apache/nifi/pull/2936 Awesome, thanks! That would mean it's targeted for 1.8.0, correct? ---
[jira] [Created] (NIFI-5493) Reverse Proxy & OIDC
Curtis Ruck created NIFI-5493: - Summary: Reverse Proxy & OIDC Key: NIFI-5493 URL: https://issues.apache.org/jira/browse/NIFI-5493 Project: Apache NiFi Issue Type: Bug Components: Security Affects Versions: 1.7.1, 1.7.0 Environment: Reverse Proxy w/ SSL Client Auth between Nifi and Reverse Proxy Reporter: Curtis Ruck In my environment we have a reverse proxy that (httpd) that does SSL Client Auth to authenticate the TLS session between reverse proxy and application servers. When trying to configure OIDC authentication, the reverse proxy's SSL Client Auth certificate is "authenticated" prior to triggering the OIDC redirect for authentication. Ideally, I believe there should be a mechanism to disable NiFi Client Auth, or enabling a priority between authentication mechanisms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5469) Edits needed for LDAP and Kerberos login identity provider sections in Admin Guide
[
https://issues.apache.org/jira/browse/NIFI-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571735#comment-16571735
]
ASF GitHub Bot commented on NIFI-5469:
--
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2925#discussion_r208248950
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -3256,9 +3361,9 @@ stream {
image:s2s-rproxy-portnumber.svg["Port number to Node mapping"]
--- End diff --
While editing a doc (administration-guide.adoc, overview.adoc, etc.), the
screenshots won't load unless the image files are in the same directory as the
doc. But, once the docs are build the screenshots will be referenced properly:
> Edits needed for LDAP and Kerberos login identity provider sections in Admin
> Guide
> --
>
> Key: NIFI-5469
> URL: https://issues.apache.org/jira/browse/NIFI-5469
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Documentation Website
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Going through the Authentication and Authorization sections of the Admin
> Guide, I noticed the following improvements could be made:
> * Removed “Kerberos Config File” property from kerberos-provider login
> identity provider (this was done because the same property exists in
> nifi.properties)
> * Corrected the "LDAP-based Users/Groups Referencing User Attribute” login
> identity provider example to refer to “member uid"
> * Added titles to login identity provider examples for improved
> readability/search
> * Changed UserGroupProvider property examples from bulleted lists to tables
> Also, text formatting for references to config files, directories, etc. need
> to be made consistent. For example, config files like _nifi.properties_,
> _authorizers.xml_ should be italicized. Directories, properties and default
> values for properties should be monospaced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2925: NIFI-5469 Additional italics and code formatting co...
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2925#discussion_r208248950
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -3256,9 +3361,9 @@ stream {
image:s2s-rproxy-portnumber.svg["Port number to Node mapping"]
--- End diff --
While editing a doc (administration-guide.adoc, overview.adoc, etc.), the
screenshots won't load unless the image files are in the same directory as the
doc. But, once the docs are build the screenshots will be referenced properly:
---
[jira] [Commented] (NIFI-5151) Patch Nifi with Upsert functions for PutDatabaseRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571712#comment-16571712 ] Peter Wicks commented on NIFI-5151: --- Most databases that support UPSERT don't use the UPSERT keyword. The Wikipedia article shows all of the various ways it's been implemented: [https://en.wikipedia.org/wiki/Merge_(SQL).] A few examples: * MySQL uses `INSERT ... ON DUPLICATE KEY UPDATE` * SQLite uses `INSERT OR REPLACE INTO` * IBM DB2 and Teradata have a `WHEN MATCHED` / `WHEN NOT MATCHED` * Firebird uses `MERGE INTO` So when I say it would be "database specific", I mean it will work with the one or two databases that chose to implement UPSERT logic using that very specific syntax, but won't work with any of the other various ways it's been implemented over the years in other databases. This is exactly what the DatabaseAdapter's were built for, database vendor specific logic. > Patch Nifi with Upsert functions for PutDatabaseRecord processor > > > Key: NIFI-5151 > URL: https://issues.apache.org/jira/browse/NIFI-5151 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.0 >Reporter: Karl Amundsson >Priority: Major > Labels: Processor > Attachments: > 0001-NIFI-5151-Adding-support-for-UPSERT-in-PutDatabaseRe.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > Since Phoenix doesn't support the SQL statement INSERT you have to use a > process like: ConvertAttributesToJSON->ConvertJSONToSQL in Insert > mode->ReplaceText to replace "INSERT" with "UPSERT" -> PutSQL (See: > [https://community.hortonworks.com/questions/40561/nifi-phoenix-processor.html)] > With this patch you can choose to use UPSERT directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MINIFICPP-589) CapturePacket processor should allow selection of interfaces
[ https://issues.apache.org/jira/browse/MINIFICPP-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned MINIFICPP-589: - Assignee: Mr TheSegfault > CapturePacket processor should allow selection of interfaces > > > Key: MINIFICPP-589 > URL: https://issues.apache.org/jira/browse/MINIFICPP-589 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Could use a regex capability to ensure that users can select a single or > named scope of interfaces. For example, eth.* or eth0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-589) CapturePacket processor should allow selection of interfaces
[ https://issues.apache.org/jira/browse/MINIFICPP-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-589: -- Fix Version/s: 0.6.0 > CapturePacket processor should allow selection of interfaces > > > Key: MINIFICPP-589 > URL: https://issues.apache.org/jira/browse/MINIFICPP-589 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Could use a regex capability to ensure that users can select a single or > named scope of interfaces. For example, eth.* or eth0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-589) CapturePacket processor should allow selection of interfaces
[ https://issues.apache.org/jira/browse/MINIFICPP-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFICPP-589. --- Resolution: Fixed > CapturePacket processor should allow selection of interfaces > > > Key: MINIFICPP-589 > URL: https://issues.apache.org/jira/browse/MINIFICPP-589 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > Could use a regex capability to ensure that users can select a single or > named scope of interfaces. For example, eth.* or eth0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-590) Resolve zero length flow files at startup of pcap processor.
[ https://issues.apache.org/jira/browse/MINIFICPP-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFICPP-590. --- Resolution: Fixed > Resolve zero length flow files at startup of pcap processor. > - > > Key: MINIFICPP-590 > URL: https://issues.apache.org/jira/browse/MINIFICPP-590 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > Attachments: Screen Shot 2018-08-03 at 1.56.42 PM.png > > > Batching feature in PCAP appears to break at some points. Investigate this. > May be a memory issue on certain versions of GCC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-590) Resolve zero length flow files at startup of pcap processor.
[ https://issues.apache.org/jira/browse/MINIFICPP-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFICPP-590: -- Fix Version/s: 0.6.0 > Resolve zero length flow files at startup of pcap processor. > - > > Key: MINIFICPP-590 > URL: https://issues.apache.org/jira/browse/MINIFICPP-590 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > Attachments: Screen Shot 2018-08-03 at 1.56.42 PM.png > > > Batching feature in PCAP appears to break at some points. Investigate this. > May be a memory issue on certain versions of GCC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-590) Resolve zero length flow files at startup of pcap processor.
[ https://issues.apache.org/jira/browse/MINIFICPP-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571692#comment-16571692 ] ASF GitHub Bot commented on MINIFICPP-590: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/388 > Resolve zero length flow files at startup of pcap processor. > - > > Key: MINIFICPP-590 > URL: https://issues.apache.org/jira/browse/MINIFICPP-590 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > Attachments: Screen Shot 2018-08-03 at 1.56.42 PM.png > > > Batching feature in PCAP appears to break at some points. Investigate this. > May be a memory issue on certain versions of GCC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #388: MINIFICPP-590: Fix zero length files at s...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/388 ---
[jira] [Commented] (NIFI-5469) Edits needed for LDAP and Kerberos login identity provider sections in Admin Guide
[
https://issues.apache.org/jira/browse/NIFI-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571660#comment-16571660
]
ASF GitHub Bot commented on NIFI-5469:
--
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2925#discussion_r208232010
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -3256,9 +3361,9 @@ stream {
image:s2s-rproxy-portnumber.svg["Port number to Node mapping"]
--- End diff --
Does the screenshots load for you? It doesn't for me. Tried across browsers
and also confirmed that the URL is different.
> Edits needed for LDAP and Kerberos login identity provider sections in Admin
> Guide
> --
>
> Key: NIFI-5469
> URL: https://issues.apache.org/jira/browse/NIFI-5469
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Documentation Website
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Going through the Authentication and Authorization sections of the Admin
> Guide, I noticed the following improvements could be made:
> * Removed “Kerberos Config File” property from kerberos-provider login
> identity provider (this was done because the same property exists in
> nifi.properties)
> * Corrected the "LDAP-based Users/Groups Referencing User Attribute” login
> identity provider example to refer to “member uid"
> * Added titles to login identity provider examples for improved
> readability/search
> * Changed UserGroupProvider property examples from bulleted lists to tables
> Also, text formatting for references to config files, directories, etc. need
> to be made consistent. For example, config files like _nifi.properties_,
> _authorizers.xml_ should be italicized. Directories, properties and default
> values for properties should be monospaced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2925: NIFI-5469 Additional italics and code formatting co...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2925#discussion_r208232010
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -3256,9 +3361,9 @@ stream {
image:s2s-rproxy-portnumber.svg["Port number to Node mapping"]
--- End diff --
Does the screenshots load for you? It doesn't for me. Tried across browsers
and also confirmed that the URL is different.
---
[jira] [Commented] (NIFI-4434) ListHDFS applies File Filter also to subdirectory names in recursive search
[ https://issues.apache.org/jira/browse/NIFI-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571641#comment-16571641 ] ASF GitHub Bot commented on NIFI-4434: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2937 @jtstorck I think that examples would help the user get the right one the first time. Not a deal breaker though. > ListHDFS applies File Filter also to subdirectory names in recursive search > --- > > Key: NIFI-4434 > URL: https://issues.apache.org/jira/browse/NIFI-4434 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Holger Frydrych >Assignee: Jeff Storck >Priority: Major > > The File Filter regex configured in the ListHDFS processor is applied not > just to files found, but also to subdirectories. > If you try to set up a recursive search to list e.g. all csv files in a > directory hierarchy via a regex like ".*\.csv", it will only pick up csv > files in the base directory, not in any subdirectory. This is because > subdirectories don't typically match that regex pattern. > To fix this, either subdirectories should not be matched against the file > filter, or the file filter should be applied to the full path of all files > (relative to the base directory). The GetHDFS processor offers both options > via a switch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2937: NIFI-4434 Fixed recursive listing with a custom regex filt...
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2937 @jtstorck I think that examples would help the user get the right one the first time. Not a deal breaker though. ---
[jira] [Commented] (NIFI-4434) ListHDFS applies File Filter also to subdirectory names in recursive search
[
https://issues.apache.org/jira/browse/NIFI-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571636#comment-16571636
]
ASF GitHub Bot commented on NIFI-4434:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2937#discussion_r208222852
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java
---
@@ -462,11 +523,15 @@ private String getPerms(final FsAction action) {
private PathFilter createPathFilter(final ProcessContext context) {
final Pattern filePattern =
Pattern.compile(context.getProperty(FILE_FILTER).getValue());
--- End diff --
fair enough
> ListHDFS applies File Filter also to subdirectory names in recursive search
> ---
>
> Key: NIFI-4434
> URL: https://issues.apache.org/jira/browse/NIFI-4434
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Holger Frydrych
>Assignee: Jeff Storck
>Priority: Major
>
> The File Filter regex configured in the ListHDFS processor is applied not
> just to files found, but also to subdirectories.
> If you try to set up a recursive search to list e.g. all csv files in a
> directory hierarchy via a regex like ".*\.csv", it will only pick up csv
> files in the base directory, not in any subdirectory. This is because
> subdirectories don't typically match that regex pattern.
> To fix this, either subdirectories should not be matched against the file
> filter, or the file filter should be applied to the full path of all files
> (relative to the base directory). The GetHDFS processor offers both options
> via a switch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2937: NIFI-4434 Fixed recursive listing with a custom reg...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2937#discussion_r208222852
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java
---
@@ -462,11 +523,15 @@ private String getPerms(final FsAction action) {
private PathFilter createPathFilter(final ProcessContext context) {
final Pattern filePattern =
Pattern.compile(context.getProperty(FILE_FILTER).getValue());
--- End diff --
fair enough
---
[jira] [Commented] (MINIFICPP-590) Resolve zero length flow files at startup of pcap processor.
[ https://issues.apache.org/jira/browse/MINIFICPP-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571550#comment-16571550 ] ASF GitHub Bot commented on MINIFICPP-590: -- Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/388 As long as we have a ticket for it that works for me. Will review the updates. > Resolve zero length flow files at startup of pcap processor. > - > > Key: MINIFICPP-590 > URL: https://issues.apache.org/jira/browse/MINIFICPP-590 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Attachments: Screen Shot 2018-08-03 at 1.56.42 PM.png > > > Batching feature in PCAP appears to break at some points. Investigate this. > May be a memory issue on certain versions of GCC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #388: MINIFICPP-590: Fix zero length files at startup
Github user apiri commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/388 As long as we have a ticket for it that works for me. Will review the updates. ---
[jira] [Commented] (NIFI-5151) Patch Nifi with Upsert functions for PutDatabaseRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571182#comment-16571182 ] Karl Amundsson commented on NIFI-5151: -- Thanks for the input! Can I ask you in what way it is database specific? To me it seems this would work on all databases that support UPSERT. > Patch Nifi with Upsert functions for PutDatabaseRecord processor > > > Key: NIFI-5151 > URL: https://issues.apache.org/jira/browse/NIFI-5151 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.0 >Reporter: Karl Amundsson >Priority: Major > Labels: Processor > Attachments: > 0001-NIFI-5151-Adding-support-for-UPSERT-in-PutDatabaseRe.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > Since Phoenix doesn't support the SQL statement INSERT you have to use a > process like: ConvertAttributesToJSON->ConvertJSONToSQL in Insert > mode->ReplaceText to replace "INSERT" with "UPSERT" -> PutSQL (See: > [https://community.hortonworks.com/questions/40561/nifi-phoenix-processor.html)] > With this patch you can choose to use UPSERT directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry issue #131: NIFIREG-186: Adding Ranger authorizer
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi-registry/pull/131 @kevdoran I've added a minimal documentation for users to use the Ranger extension. I hope it provides enough information. Thanks! ---
[jira] [Commented] (NIFIREG-186) Create Authorizer implementation that uses Apache Ranger
[ https://issues.apache.org/jira/browse/NIFIREG-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571176#comment-16571176 ] ASF GitHub Bot commented on NIFIREG-186: Github user ijokarumawak commented on the issue: https://github.com/apache/nifi-registry/pull/131 @kevdoran I've added a minimal documentation for users to use the Ranger extension. I hope it provides enough information. Thanks! > Create Authorizer implementation that uses Apache Ranger > > > Key: NIFIREG-186 > URL: https://issues.apache.org/jira/browse/NIFIREG-186 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > In addition to the standard file-based Authorizer, we should provide an > Authorizer implementation that uses Apache Ranger, so that users implement > centralized authorization against both NiFi and NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-186) Create Authorizer implementation that uses Apache Ranger
[ https://issues.apache.org/jira/browse/NIFIREG-186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571175#comment-16571175 ] ASF GitHub Bot commented on NIFIREG-186: Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi-registry/pull/131#discussion_r208113908 --- Diff: nifi-registry-extensions/nifi-registry-ranger-extension/README.md --- @@ -0,0 +1,116 @@ + +# NiFi Registry Ranger extension + +This extension provides `org.apache.nifi.registry.ranger.RangerAuthorizer` class for NiFi Registry to authorize user requests by access policies defined at [Apache Ranger](https://ranger.apache.org/). + +## Prerequisites + +* Apache Ranger 1.2.0 or later is needed. --- End diff -- This sentence may need to be updated once the improvement at Ranger project is committed. > Create Authorizer implementation that uses Apache Ranger > > > Key: NIFIREG-186 > URL: https://issues.apache.org/jira/browse/NIFIREG-186 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > In addition to the standard file-based Authorizer, we should provide an > Authorizer implementation that uses Apache Ranger, so that users implement > centralized authorization against both NiFi and NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry pull request #131: NIFIREG-186: Adding Ranger authorizer
Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi-registry/pull/131#discussion_r208113908 --- Diff: nifi-registry-extensions/nifi-registry-ranger-extension/README.md --- @@ -0,0 +1,116 @@ + +# NiFi Registry Ranger extension + +This extension provides `org.apache.nifi.registry.ranger.RangerAuthorizer` class for NiFi Registry to authorize user requests by access policies defined at [Apache Ranger](https://ranger.apache.org/). + +## Prerequisites + +* Apache Ranger 1.2.0 or later is needed. --- End diff -- This sentence may need to be updated once the improvement at Ranger project is committed. ---
