[jira] [Created] (NIFI-9484) SECURE NIFI behind Nginx Reverse proxy

[mayki (Jira)]

mayki created NIFI-9484:
---

 Summary: SECURE NIFI behind Nginx Reverse proxy
 Key: NIFI-9484
 URL: https://issues.apache.org/jira/browse/NIFI-9484
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.11.4
Reporter: mayki


Hello,

I read a lot of documentation about how configure secure nifi before reverse 
proxy.

But it seems missing something to work completly.
 * the path to /nifi, WEB UI,  processor configuration: OK
 * the path to /nifi-api, stop processor, view provenance, 
[https://:19091/nifi-api/processors/28621407-017d-1000-31a3-58539fc2e94a/run-status|https://s3639tos.mc2.renault.fr:19091/nifi-api/processors/28621407-017d-1000-31a3-58539fc2e94a/run-status]
 : KO

 

I use this config :
{code:java}
server {
      listen    19091;
      server_name    nginx_server;
      proxy_ssl_trusted_certificate /etc/nginx/certs/xxx-cert.pem;
      proxy_ssl_certificate /etc/nginx/certs/xxx-cert.pem;
      proxy_ssl_certificate_key /etc/nginx/certs/xxx-key.pem;
    location / {
        proxy_ssl_server_name on;

        proxy_pass https://nifi_server:9091;
        proxy_set_header X-ProxyScheme https;
        proxy_set_header X-ProxyHost nifi_server;
        proxy_set_header X-ProxyPort 19091;
        #proxy_set_header X-ProxyContextPath /;
        proxy_set_header X-ProxiedEntitiesChain $ssl_client_s_dn;
 {code}
Nifi properties 1: 
{code:java}
nifi.web.proxy.context.path=
nifi.web.proxy.host=nginx_server:19091 {code}
Nifi properties 2: 
{code:java}
nifi.web.proxy.context.path=/
nifi.web.proxy.host=nginx_server:19091 {code}
 

Nifi properties 3: 
{code:java}
nifi.web.proxy.context.path=//,/nifi-api
nifi.web.proxy.host=nginx_server:19091 {code}
 

I read a lot time, but I don't sure that is need to push in 
nifi.web.proxy.context.path

What is the correct configuration to setup ?

Thanks.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1226: MINIFICPP-1673: ProcessSession::write/ProcessSession::append should o…

[GitBox]

martinzink commented on a change in pull request #1226:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1226#discussion_r768512438



##
File path: libminifi/src/core/ProcessSession.cpp
##
@@ -212,6 +212,10 @@ void ProcessSession::transfer(const 
std::shared_ptr &flow, Relat
 }
 
 void ProcessSession::write(const std::shared_ptr &flow, 
OutputStreamCallback *callback) {
+  gsl_Expects(_updatedFlowFiles.contains(flow->getUUID())
+  || _addedFlowFiles.contains(flow->getUUID())
+  || std::any_of(_clonedFlowFiles.begin(), _clonedFlowFiles.end(), 
[&flow](const auto& cloned_flow_file) { return flow == cloned_flow_file; }));
+

Review comment:
   good idea, changed it in 
https://github.com/apache/nifi-minifi-cpp/pull/1226/commits/300532be1a9347d2926c139185bbf46edb619102




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1225: MINIFICPP-1688: When storing time durations we should use std::chrono…

[GitBox]

adamdebreceni commented on a change in pull request #1225:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1225#discussion_r768542906



##
File path: libminifi/src/provenance/Provenance.cpp
##
@@ -45,13 +45,13 @@ const char 
*ProvenanceEventRecord::ProvenanceEventTypeStr[REPLAY + 1] = { "CREAT
 
 
ProvenanceEventRecord::ProvenanceEventRecord(ProvenanceEventRecord::ProvenanceEventType
 event, std::string componentId, std::string componentType)
 : core::SerializableComponent(core::getClassName()),
-  _entryDate(0),
-  _lineageStartDate(0),
+  _entryDate(std::chrono::time_point()),

Review comment:
   is there any advantage using this over 
`std::chrono::system_clock::time_point`? the latter is not much shorter so I 
don't insist on changing it, just asking




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1226: MINIFICPP-1673: ProcessSession::write/ProcessSession::append should o…

[GitBox]

szaszm commented on a change in pull request #1226:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1226#discussion_r768552299



##
File path: libminifi/src/core/ProcessSession.cpp
##
@@ -212,6 +212,10 @@ void ProcessSession::transfer(const 
std::shared_ptr &flow, Relat
 }
 
 void ProcessSession::write(const std::shared_ptr &flow, 
OutputStreamCallback *callback) {
+  gsl_Expects(_updatedFlowFiles.contains(flow->getUUID())
+  || _addedFlowFiles.contains(flow->getUUID())
+  || std::any_of(_clonedFlowFiles.begin(), _clonedFlowFiles.end(), 
[&flow](const auto& cloned_flow_file) { return flow == cloned_flow_file; }));
+

Review comment:
   Consider extracting the lambda to a variable to avoid the error with 
clang. It will start working once they implement the C++20 feature "Lambdas in 
unevaluated contexts".
   This is the disabled implementation of the macro:
   ```
   # define gsl_ELIDE_( x )  static_assert( ::std::is_constructible::value, "argument of contract check must be convertible to bool" 
)
   ```
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1225: MINIFICPP-1688: When storing time durations we should use std::chrono…

[GitBox]

szaszm commented on a change in pull request #1225:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1225#discussion_r768559940



##
File path: libminifi/src/provenance/Provenance.cpp
##
@@ -45,13 +45,13 @@ const char 
*ProvenanceEventRecord::ProvenanceEventTypeStr[REPLAY + 1] = { "CREAT
 
 
ProvenanceEventRecord::ProvenanceEventRecord(ProvenanceEventRecord::ProvenanceEventType
 event, std::string componentId, std::string componentType)
 : core::SerializableComponent(core::getClassName()),
-  _entryDate(0),
-  _lineageStartDate(0),
+  _entryDate(std::chrono::time_point()),

Review comment:
   We could also just delete the line and let the default constructor be 
called automatically.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1225: MINIFICPP-1688: When storing time durations we should use std::chrono…

[GitBox]

adamdebreceni commented on a change in pull request #1225:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1225#discussion_r768570039



##
File path: libminifi/src/RemoteProcessorGroupPort.cpp
##
@@ -167,17 +167,12 @@ void RemoteProcessorGroupPort::onSchedule(const 
std::shared_ptrgetProperty(idleTimeout.getName(), idleTimeoutStr)
-|| !core::Property::getTimeMSFromString(idleTimeoutStr, 
idleTimeoutVal)) {
-  logger_->log_debug("%s attribute is invalid, so default value of %s will 
be used", idleTimeout.getName(),
- idleTimeout.getValue());
-  if (!core::Property::getTimeMSFromString(idleTimeout.getValue(), 
idleTimeoutVal)) {
-assert(false);  // Couldn't parse our default value
-  }
+if (auto idle_timeout = 
context->getProperty(idleTimeout)) {
+  idle_timeout_ = idle_timeout->getMilliseconds();
+} else {
+  logger_->log_debug("%s attribute is invalid, so default value of %s will 
be used", idleTimeout.getName(), idleTimeout.getDefaultValue());
+  idle_timeout_ = 
static_cast(idleTimeout.getDefaultValue().to_string()).getMilliseconds();

Review comment:
   this `static_cast` feels "off", although at a glance I couldn't find 
anything in the coding guidelines that would advise against it, might even 
advise for it: 
   > Use static_cast as the equivalent of a C-style cast that does value 
conversion




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1225: MINIFICPP-1688: When storing time durations we should use std::chrono…

[GitBox]

adamdebreceni commented on a change in pull request #1225:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1225#discussion_r768570039



##
File path: libminifi/src/RemoteProcessorGroupPort.cpp
##
@@ -167,17 +167,12 @@ void RemoteProcessorGroupPort::onSchedule(const 
std::shared_ptrgetProperty(idleTimeout.getName(), idleTimeoutStr)
-|| !core::Property::getTimeMSFromString(idleTimeoutStr, 
idleTimeoutVal)) {
-  logger_->log_debug("%s attribute is invalid, so default value of %s will 
be used", idleTimeout.getName(),
- idleTimeout.getValue());
-  if (!core::Property::getTimeMSFromString(idleTimeout.getValue(), 
idleTimeoutVal)) {
-assert(false);  // Couldn't parse our default value
-  }
+if (auto idle_timeout = 
context->getProperty(idleTimeout)) {
+  idle_timeout_ = idle_timeout->getMilliseconds();
+} else {
+  logger_->log_debug("%s attribute is invalid, so default value of %s will 
be used", idleTimeout.getName(), idleTimeout.getDefaultValue());
+  idle_timeout_ = 
static_cast(idleTimeout.getDefaultValue().to_string()).getMilliseconds();

Review comment:
   this `static_cast` feels "off", although at a glance I couldn't find 
anything in the coding guidelines that would advise against it, might even 
advises for it: 
   > Use static_cast as the equivalent of a C-style cast that does value 
conversion




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (NIFI-9485) BouncyCastle conflicted when compiled and running nifi-1.16.0 on windows11

[sunyd1224 (Jira)]

sunyd1224 created NIFI-9485:
---

 Summary: BouncyCastle conflicted when compiled and running 
nifi-1.16.0 on windows11 
 Key: NIFI-9485
 URL: https://issues.apache.org/jira/browse/NIFI-9485
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Tools and Build
Affects Versions: 1.16.0
 Environment: os: windows 11
java version: 1.8.0_171
maven version: 3.8.4
Reporter: sunyd1224
 Fix For: 1.16.0
 Attachments: error.png

I built nifi-1.16.0 from the source code for a long time. When finished 
successfully,I run the run-nifi.bat.  Hower,the error occurs:

!error.png!

The reason is that there is another lower version of BouncyCastle than needed 
in the java classpath on my computer and the BouncyCastle is not specified with 
the precise version in some of the pom.xml of nifi.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1226: MINIFICPP-1673: ProcessSession::write/ProcessSession::append should o…

[GitBox]

adamdebreceni commented on a change in pull request #1226:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1226#discussion_r768631049



##
File path: libminifi/src/core/ProcessSession.cpp
##
@@ -212,6 +212,10 @@ void ProcessSession::transfer(const 
std::shared_ptr &flow, Relat
 }
 
 void ProcessSession::write(const std::shared_ptr &flow, 
OutputStreamCallback *callback) {
+  gsl_ExpectsAudit(_updatedFlowFiles.contains(flow->getUUID())

Review comment:
   as we provide access to the session to custom code through the script 
extension, this feels like an invariant we should always check, turning the 
`std::map`s (and the `_clonedFlowFiles` vector) into `std::unordered_map`s 
would make the check an `O(1)` operation




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1226: MINIFICPP-1673: ProcessSession::write/ProcessSession::append should o…

[GitBox]

szaszm commented on a change in pull request #1226:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1226#discussion_r768640855



##
File path: libminifi/src/core/ProcessSession.cpp
##
@@ -212,6 +212,10 @@ void ProcessSession::transfer(const 
std::shared_ptr &flow, Relat
 }
 
 void ProcessSession::write(const std::shared_ptr &flow, 
OutputStreamCallback *callback) {
+  gsl_ExpectsAudit(_updatedFlowFiles.contains(flow->getUUID())

Review comment:
   I prefer the narrow contract, but if you decide to have a wide contract, 
then please use if + throw or something similar to implement it, not 
gsl_Expects (which is meant to declare preconditions).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1226: MINIFICPP-1673: ProcessSession::write/ProcessSession::append should o…

[GitBox]

szaszm commented on a change in pull request #1226:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1226#discussion_r768641854



##
File path: libminifi/src/core/ProcessSession.cpp
##
@@ -212,6 +212,10 @@ void ProcessSession::transfer(const 
std::shared_ptr &flow, Relat
 }
 
 void ProcessSession::write(const std::shared_ptr &flow, 
OutputStreamCallback *callback) {
+  gsl_ExpectsAudit(_updatedFlowFiles.contains(flow->getUUID())

Review comment:
   Is it feasible to keep the precondition in the C++ world and make the 
scripting wrapper have an internal check?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1226: MINIFICPP-1673: ProcessSession::write/ProcessSession::append should o…

[GitBox]

szaszm commented on a change in pull request #1226:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1226#discussion_r768640855



##
File path: libminifi/src/core/ProcessSession.cpp
##
@@ -212,6 +212,10 @@ void ProcessSession::transfer(const 
std::shared_ptr &flow, Relat
 }
 
 void ProcessSession::write(const std::shared_ptr &flow, 
OutputStreamCallback *callback) {
+  gsl_ExpectsAudit(_updatedFlowFiles.contains(flow->getUUID())

Review comment:
   I prefer the narrow contract (we can document preconditions to scripting 
users, too), but if you decide to have a wide contract, then please use if + 
throw or something similar to implement it, not gsl_Expects (which is meant to 
declare preconditions).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1226: MINIFICPP-1673: ProcessSession::write/ProcessSession::append should o…

[GitBox]

adamdebreceni commented on a change in pull request #1226:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1226#discussion_r768656719



##
File path: libminifi/src/core/ProcessSession.cpp
##
@@ -212,6 +212,10 @@ void ProcessSession::transfer(const 
std::shared_ptr &flow, Relat
 }
 
 void ProcessSession::write(const std::shared_ptr &flow, 
OutputStreamCallback *callback) {
+  gsl_ExpectsAudit(_updatedFlowFiles.contains(flow->getUUID())

Review comment:
   we wrap the session in a `PyProcessSession` which could handle the check 
if it had access to the internals of the `ProcessSession` (not probable) or had 
its own flowfile registry (would be duplicating the session's internal registry)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1226: MINIFICPP-1673: ProcessSession::write/ProcessSession::append should o…

[GitBox]

adamdebreceni commented on a change in pull request #1226:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1226#discussion_r768662563



##
File path: libminifi/src/core/ProcessSession.cpp
##
@@ -212,6 +212,10 @@ void ProcessSession::transfer(const 
std::shared_ptr &flow, Relat
 }
 
 void ProcessSession::write(const std::shared_ptr &flow, 
OutputStreamCallback *callback) {
+  gsl_ExpectsAudit(_updatedFlowFiles.contains(flow->getUUID())

Review comment:
   I prefer having the checks if possible (unless they are too costly and 
in the hot path) instead of the "read-the-docs" hands-off strategy, as the 
scripting processor is not really used, I'm fine with leaving it as-is, but IMO 
it is something we should revisit if need arises 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1226: MINIFICPP-1673: ProcessSession::write/ProcessSession::append should o…

[GitBox]

szaszm commented on a change in pull request #1226:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1226#discussion_r768666904



##
File path: libminifi/src/core/ProcessSession.cpp
##
@@ -212,6 +212,10 @@ void ProcessSession::transfer(const 
std::shared_ptr &flow, Relat
 }
 
 void ProcessSession::write(const std::shared_ptr &flow, 
OutputStreamCallback *callback) {
+  gsl_ExpectsAudit(_updatedFlowFiles.contains(flow->getUUID())

Review comment:
   I would not be against a public member function that can check if a flow 
file can be written to. I don't know a good name though. Maybe `bool 
canBeWrittenTo(const core::FlowFile&) const noexcept;`? If we did that, 
`PyProcessSession` could check that and `write`/`append` could have that as 
their precondition.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] sunyd1224 opened a new pull request #5602: NIFI-9485 specify the precise BouncyCastle version in pom.xml

[GitBox]

sunyd1224 opened a new pull request #5602:
URL: https://github.com/apache/nifi/pull/5602


   
   Thank you for submitting a contribution to Apache NiFi.
   
   Please provide a short description of the PR here:
   
    Description of PR
   
   _Enables X functionality; fixes bug NIFI-._
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [ ] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [ ] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [ ] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [ ] Have you verified that the full build is successful on JDK 8?
   - [ ] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory closed pull request #5602: NIFI-9485 specify the precise BouncyCastle version in pom.xml

[GitBox]

exceptionfactory closed pull request #5602:
URL: https://github.com/apache/nifi/pull/5602


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory commented on pull request #5602: NIFI-9485 specify the precise BouncyCastle version in pom.xml

[GitBox]

exceptionfactory commented on pull request #5602:
URL: https://github.com/apache/nifi/pull/5602#issuecomment-993584003


   Thank you for your contribution @sunyd1224!
   
   According to the Jira issue linked to this pull request, the problem appears 
to be a runtime environment problem, not something in the NiFi build 
configuration.  The root Maven configuration includes the Bouncy Castle 
version, so all other project modules inherit the same version:
   
   
https://github.com/apache/nifi/blob/1321b25f6670fb96e3e75076addb6fbe7e691c84/pom.xml#L263
   
   For this reason, the changes in this pull request are not necessary. Further 
discussion can take place on the Jira issue, but I am closing this current pull 
request.  If we do find an issue that needs to be addressed with the Bouncy 
Castle version, opening a new pull request is certainly an option.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1225: MINIFICPP-1688: When storing time durations we should use std::chrono…

[GitBox]

martinzink commented on a change in pull request #1225:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1225#discussion_r768706698



##
File path: extensions/sftp/processors/ListSFTP.cpp
##
@@ -359,21 +360,21 @@ bool ListSFTP::filterFile(const std::string& parent_path, 
const std::string& fil
 
   /* Age */
   time_t now = time(nullptr);
-  uint64_t file_age = (now - attrs.mtime) * 1000;
+  std::chrono::milliseconds file_age = std::chrono::seconds(now - attrs.mtime);

Review comment:
   this wouldnt work because the rhs would be a `time_point` 
and not a `duration`, and afaik the attrs.mtime is measured from unix epoch and 
since the epoch of the file_clock is not necessarily the same as the unix 
epoch, the `time_point::time_since_epoch` could return something 
completely different from what we want.
   
   we could create a `time_point` from the attrs.mtime but for that 
we would need a platform specific helper function (because the file_clock 
interface is quite different on msvc, libc++ and on stdlibc++) it could be done 
with something similar to this (only in reverse):
   
https://github.com/rustammendel/nifi-minifi-cpp/pull/1/files#diff-796b46848475896d2982dcb681e0f547bac6aec3c98da80e8c03139562092604R89-R97
   
   or we could stick to using system_clock (which has the unix epoch)
   `auto file_age = std::chrono::system_clock::now() - 
std::chrono::system_clock::from_time_t(attrs.mtime);`
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] sunyd1224 commented on pull request #5602: NIFI-9485 specify the precise BouncyCastle version in pom.xml

[GitBox]

sunyd1224 commented on pull request #5602:
URL: https://github.com/apache/nifi/pull/5602#issuecomment-993584430


   Specify the precise BouncyCastle version in pom.xml to prevent BouncyCastle 
version conflict.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (NIFI-9485) BouncyCastle conflicted when compiled and running nifi-1.16.0 on windows11

[David Handermann (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459184#comment-17459184
 ] 

David Handermann commented on NIFI-9485:


Thank you for describing the issue and providing a screenshot [~sunyd1224], 
that is helpful for troubleshooting.  As mentioned in the comment in PR 5602, 
the root NiFi Maven pom.xml already specifies the version of Bouncy Castle.

It looks like you are running an old version of Java 8, and recent versions of 
NiFi require at least Java 8 Update 251.  Can you try upgrading to a more 
recent version of Java 8 and see if that works?

> BouncyCastle conflicted when compiled and running nifi-1.16.0 on windows11 
> ---
>
> Key: NIFI-9485
> URL: https://issues.apache.org/jira/browse/NIFI-9485
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Affects Versions: 1.16.0
> Environment: os: windows 11
> java version: 1.8.0_171
> maven version: 3.8.4
>Reporter: sunyd1224
>Priority: Minor
>  Labels: beginner
> Fix For: 1.16.0
>
> Attachments: error.png
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I built nifi-1.16.0 from the source code for a long time. When finished 
> successfully,I run the run-nifi.bat.  Hower,the error occurs:
> !error.png!
> The reason is that there is another lower version of BouncyCastle than needed 
> in the java classpath on my computer and the BouncyCastle is not specified 
> with the precise version in some of the pom.xml of nifi.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1225: MINIFICPP-1688: When storing time durations we should use std::chrono…

[GitBox]

szaszm commented on a change in pull request #1225:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1225#discussion_r768710368



##
File path: extensions/sftp/processors/ListSFTP.cpp
##
@@ -359,21 +360,21 @@ bool ListSFTP::filterFile(const std::string& parent_path, 
const std::string& fil
 
   /* Age */
   time_t now = time(nullptr);
-  uint64_t file_age = (now - attrs.mtime) * 1000;
+  std::chrono::milliseconds file_age = std::chrono::seconds(now - attrs.mtime);

Review comment:
   Good point. I would go for `system_clock` in that case, for simplicity. 
I think that these timestamps are just system timestamps anyway.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1225: MINIFICPP-1688: When storing time durations we should use std::chrono…

[GitBox]

martinzink commented on a change in pull request #1225:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1225#discussion_r768718262



##
File path: extensions/sftp/processors/ListSFTP.cpp
##
@@ -261,20 +263,19 @@ void ListSFTP::onSchedule(const 
std::shared_ptr &context,
   }
   context->getProperty(TargetSystemTimestampPrecision.getName(), 
target_system_timestamp_precision_);
   context->getProperty(EntityTrackingInitialListingTarget.getName(), 
entity_tracking_initial_listing_target_);
-  if (!context->getProperty(MinimumFileAge.getName(), value)) {
-logger_->log_error("Minimum File Age attribute is missing or invalid");
+
+  if (auto minimum_file_age = 
context->getProperty(MinimumFileAge)) {
+minimum_file_age_ = minimum_file_age->getMilliseconds();
   } else {
-core::TimeUnit unit;
-if (!core::Property::StringToTime(value, minimum_file_age_, unit) || 
!core::Property::ConvertTimeUnitToMS(minimum_file_age_, unit, 
minimum_file_age_)) {
-  logger_->log_error("Minimum File Age attribute is invalid");
-}
+logger_->log_error("Minimum File Age attribute is missing or invalid");
   }
-  if (context->getProperty(MaximumFileAge.getName(), value)) {
-core::TimeUnit unit;
-if (!core::Property::StringToTime(value, maximum_file_age_, unit) || 
!core::Property::ConvertTimeUnitToMS(maximum_file_age_, unit, 
maximum_file_age_)) {
-  logger_->log_error("Maximum File Age attribute is invalid");
-}
+
+  if (auto maximum_file_age = 
context->getProperty(MaximumFileAge)) {
+maximum_file_age_ = maximum_file_age->getMilliseconds();
+  } else {
+logger_->log_error("Maximum File Age attribute is missing or invalid");

Review comment:
   This one calls the
   ```
   template
   std::enable_if_t::value, std::optional>
   getProperty(const Property& property) const {
 T value;
 try {
   if (!getProperty(property.getName(), value)) return std::nullopt;
 } catch (const utils::internal::ValueException&) {
   return std::nullopt;
 }
 return value;
   }
   ```
   which doesnt throw just returns a nullopt if the value is invalid




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1225: MINIFICPP-1688: When storing time durations we should use std::chrono…

[GitBox]

szaszm commented on a change in pull request #1225:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1225#discussion_r768719679



##
File path: extensions/sftp/processors/ListSFTP.cpp
##
@@ -261,20 +263,19 @@ void ListSFTP::onSchedule(const 
std::shared_ptr &context,
   }
   context->getProperty(TargetSystemTimestampPrecision.getName(), 
target_system_timestamp_precision_);
   context->getProperty(EntityTrackingInitialListingTarget.getName(), 
entity_tracking_initial_listing_target_);
-  if (!context->getProperty(MinimumFileAge.getName(), value)) {
-logger_->log_error("Minimum File Age attribute is missing or invalid");
+
+  if (auto minimum_file_age = 
context->getProperty(MinimumFileAge)) {
+minimum_file_age_ = minimum_file_age->getMilliseconds();
   } else {
-core::TimeUnit unit;
-if (!core::Property::StringToTime(value, minimum_file_age_, unit) || 
!core::Property::ConvertTimeUnitToMS(minimum_file_age_, unit, 
minimum_file_age_)) {
-  logger_->log_error("Minimum File Age attribute is invalid");
-}
+logger_->log_error("Minimum File Age attribute is missing or invalid");
   }
-  if (context->getProperty(MaximumFileAge.getName(), value)) {
-core::TimeUnit unit;
-if (!core::Property::StringToTime(value, maximum_file_age_, unit) || 
!core::Property::ConvertTimeUnitToMS(maximum_file_age_, unit, 
maximum_file_age_)) {
-  logger_->log_error("Maximum File Age attribute is invalid");
-}
+
+  if (auto maximum_file_age = 
context->getProperty(MaximumFileAge)) {
+maximum_file_age_ = maximum_file_age->getMilliseconds();
+  } else {
+logger_->log_error("Maximum File Age attribute is missing or invalid");

Review comment:
   Sorry, I assumed that all of them throw on validation errors. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-9483) Evaluate Addition of log4j-to-slf4j

[David Handermann (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-9483:
---
Summary: Evaluate Addition of log4j-to-slf4j  (was: Evaluate Addition of 
log4j-slf4j-impl)

> Evaluate Addition of log4j-to-slf4j
> ---
>
> Key: NIFI-9483
> URL: https://issues.apache.org/jira/browse/NIFI-9483
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>
> Several extension components include transitive dependencies on Log4j 2. NiFi 
> uses the SLF4J API for logging in application code, and Logback as the 
> runtime implementation. 
> The Log4j 2 system supports SLF4J for runtime message routing, which would 
> ensure that all log messages route through SLF4J and Logback, as opposed to 
> falling back to the default Log4j 2 appender.  Current extension components 
> should be evaluated to determine whether an explicit dependency on 
> log4j-slf4j-impl is necessary to ensure proper runtime log handling for 
> libraries that rely on the Log4j 2 API.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9483) Evaluate Addition of log4j-to-slf4j

[David Handermann (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-9483:
---
Description: 
Several extension components include transitive dependencies on Log4j 2. NiFi 
uses the SLF4J API for logging in application code, and Logback as the runtime 
implementation. 

The Log4j 2 system supports SLF4J for runtime message routing, which would 
ensure that all log messages route through SLF4J and Logback, as opposed to 
falling back to the default Log4j 2 appender.  Current extension components 
should be evaluated to determine whether an explicit dependency on 
log4j-to-slf4j is necessary to ensure proper runtime log handling for libraries 
that rely on the Log4j 2 API.

  was:
Several extension components include transitive dependencies on Log4j 2. NiFi 
uses the SLF4J API for logging in application code, and Logback as the runtime 
implementation. 

The Log4j 2 system supports SLF4J for runtime message routing, which would 
ensure that all log messages route through SLF4J and Logback, as opposed to 
falling back to the default Log4j 2 appender.  Current extension components 
should be evaluated to determine whether an explicit dependency on 
log4j-slf4j-impl is necessary to ensure proper runtime log handling for 
libraries that rely on the Log4j 2 API.


> Evaluate Addition of log4j-to-slf4j
> ---
>
> Key: NIFI-9483
> URL: https://issues.apache.org/jira/browse/NIFI-9483
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>
> Several extension components include transitive dependencies on Log4j 2. NiFi 
> uses the SLF4J API for logging in application code, and Logback as the 
> runtime implementation. 
> The Log4j 2 system supports SLF4J for runtime message routing, which would 
> ensure that all log messages route through SLF4J and Logback, as opposed to 
> falling back to the default Log4j 2 appender.  Current extension components 
> should be evaluated to determine whether an explicit dependency on 
> log4j-to-slf4j is necessary to ensure proper runtime log handling for 
> libraries that rely on the Log4j 2 API.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9485) BouncyCastle conflicted when compiled and running nifi-1.16.0 on windows11

[Pierre Villard (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-9485:
-
Fix Version/s: (was: 1.16.0)

> BouncyCastle conflicted when compiled and running nifi-1.16.0 on windows11 
> ---
>
> Key: NIFI-9485
> URL: https://issues.apache.org/jira/browse/NIFI-9485
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Affects Versions: 1.16.0
> Environment: os: windows 11
> java version: 1.8.0_171
> maven version: 3.8.4
>Reporter: sunyd1224
>Priority: Minor
>  Labels: beginner
> Attachments: error.png
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I built nifi-1.16.0 from the source code for a long time. When finished 
> successfully,I run the run-nifi.bat.  Hower,the error occurs:
> !error.png!
> The reason is that there is another lower version of BouncyCastle than needed 
> in the java classpath on my computer and the BouncyCastle is not specified 
> with the precise version in some of the pom.xml of nifi.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [nifi] gresockj opened a new pull request #5603: NIFI-9461: Allowing Parameter descriptions to be changed

[GitBox]

gresockj opened a new pull request #5603:
URL: https://github.com/apache/nifi/pull/5603


   This fixes a bug that prevented parameter descriptions from being updated, 
if that was the only change.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [ ] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [ ] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [ ] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [ ] Have you verified that the full build is successful on JDK 8?
   - [ ] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-9399) Apply Secure Processing to TransformXml XSLT Sources

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9399:
---
Fix Version/s: 1.15.1

> Apply Secure Processing to TransformXml XSLT Sources
> 
>
> Key: NIFI-9399
> URL: https://issues.apache.org/jira/browse/NIFI-9399
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions, Security
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The {{TransformXml}} processor supports Secure Processing as of NiFi 1.3.0, 
> which prevents external access attempts using XML entity references. The 
> {{Secure Processing}} property applies to input FlowFiles, but does not apply 
> to the XSLT source that the processor uses during transformation. 
> {{TransformXml}} should be updated to apply Secure Processing to XSLT sources.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9399) Apply Secure Processing to TransformXml XSLT Sources

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459229#comment-17459229
 ] 

ASF subversion and git services commented on NIFI-9399:
---

Commit 200538a2903be98567ecfa1981967e714920d8cd in nifi's branch 
refs/heads/support/nifi-1.15 from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=200538a ]

NIFI-9399 Apply Secure Processing to TransformXml XSLT

- Added XML Stream Reader processing for XSLT with external entities disabled
- Removed unused XsltValidator
- Upgraded Saxon-HE from 9.6.0-5 to 10.6


> Apply Secure Processing to TransformXml XSLT Sources
> 
>
> Key: NIFI-9399
> URL: https://issues.apache.org/jira/browse/NIFI-9399
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions, Security
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The {{TransformXml}} processor supports Secure Processing as of NiFi 1.3.0, 
> which prevents external access attempts using XML entity references. The 
> {{Secure Processing}} property applies to input FlowFiles, but does not apply 
> to the XSLT source that the processor uses during transformation. 
> {{TransformXml}} should be updated to apply Secure Processing to XSLT sources.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9477) data loss doing Incremental load based on a timestamp from SQL server using QueryDatabaseTableRecord

[Peter Turcsanyi (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459267#comment-17459267
 ] 

Peter Turcsanyi commented on NIFI-9477:
---

[~abduq] Timestamp accuracy and having multiple records with the same 
timestamps is one aspect of this problem but there is another one: due to 
concurrent execution of transactions in RDBMS, it is not guaranteed that 
timestamps arrive in monotonically increasing way (that is a record with an 
older timestamp can appear).

Example scenario:
 # tx1 starts
 # tx1 creates a record with ts1
 # tx2 starts
 # tx2 creates a record with ts2 (ts2 > ts1)
 # tx2 commits and the record with ts2 becomes visible for others
 # QueryDatabaseTableRecord executes its query and stores ts2 as the max 
timestamp
 # tx1 commits, the record with ts1 becomes visible for others but 
QueryDatabaseTableRecord will never load it because ts1 < max ts (== ts2)

The recommended solution is to configure "Additional WHERE Clause" property 
with a condition that skips the most recent records. In Oracle it would look 
like {{"my_ts_column < systimestamp - interval 'N' second"}} (I believe MS SQL 
Server has something similar). It loads only those records that are older than 
'N' seconds (the interval should be longer than the longest data loading 
transaction).

It is similar to your solution but it does not generate duplicates and there is 
no need for a custom processor.

The other option would be to configure your data loading transactions to use 
stricter transaction isolation level and/or locks to achieve serial and ordered 
execution of transactions but in most cases it is not possible  or desirable 
(affects the loading side, performance reasons, etc.).

I hope this helps.

> data loss doing Incremental load based on a timestamp from SQL server using 
> QueryDatabaseTableRecord 
> -
>
> Key: NIFI-9477
> URL: https://issues.apache.org/jira/browse/NIFI-9477
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.13.2
> Environment: ubuntu 20 
> 2-node cluster using an embedded zk
>Reporter: Abdullah Aqeeli
>Priority: Critical
>
> hello all,
>  
> We are using Nifi to do an incremental load from SQL Server using 
> QueryDatabaseTableRecord where max-value column is updated_at column with a 
> [Datetime2 
> |https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
>  type. The source tables get updated in milliseconds. We noticed that we are 
> losing few events doing with each QueryDatabaseTableRecord run. As a 
> work-around we created a custom processor that subtracts few seconds from the 
> max-value in the state fetching overlapping events and then we handle any 
> duplicates later on in the pipeline and it seems to solve the issue. However, 
> this is not optimal. We'd like to know what's causing the skips. Our 
> assumption so far is that timestamp datatype accuracy is causing the skips in 
> the incremental load however we're not sure.  
>  
> Thanks!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9477) data loss doing Incremental load based on a timestamp from SQL server using QueryDatabaseTableRecord

[Peter Turcsanyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Turcsanyi updated NIFI-9477:
--
Component/s: Extensions
 (was: Core Framework)

> data loss doing Incremental load based on a timestamp from SQL server using 
> QueryDatabaseTableRecord 
> -
>
> Key: NIFI-9477
> URL: https://issues.apache.org/jira/browse/NIFI-9477
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.13.2
> Environment: ubuntu 20 
> 2-node cluster using an embedded zk
>Reporter: Abdullah Aqeeli
>Priority: Critical
>
> hello all,
>  
> We are using Nifi to do an incremental load from SQL Server using 
> QueryDatabaseTableRecord where max-value column is updated_at column with a 
> [Datetime2 
> |https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
>  type. The source tables get updated in milliseconds. We noticed that we are 
> losing few events doing with each QueryDatabaseTableRecord run. As a 
> work-around we created a custom processor that subtracts few seconds from the 
> max-value in the state fetching overlapping events and then we handle any 
> duplicates later on in the pipeline and it seems to solve the issue. However, 
> this is not optimal. We'd like to know what's causing the skips. Our 
> assumption so far is that timestamp datatype accuracy is causing the skips in 
> the incremental load however we're not sure.  
>  
> Thanks!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1221: MINIFICPP-1630 Create FetchAzureDataLakeStorage processor

[GitBox]

lordgamez commented on a change in pull request #1221:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1221#discussion_r768808670



##
File path: extensions/azure/storage/AzureDataLakeStorage.cpp
##
@@ -61,4 +61,29 @@ bool AzureDataLakeStorage::deleteFile(const 
storage::DeleteAzureDataLakeStorageP
   }
 }
 
+std::optional AzureDataLakeStorage::fetchFile(const 
FetchAzureDataLakeStorageParameters& params, io::BaseStream& stream) {
+  try {
+auto fetch_res = data_lake_storage_client_->fetchFile(params);
+
+std::vector buffer(4096);

Review comment:
   Updated in 172d2a4f893238d7a9818b812cc0b7b671c3cd0b

##
File path: extensions/azure/storage/AzureDataLakeStorage.cpp
##
@@ -61,4 +61,29 @@ bool AzureDataLakeStorage::deleteFile(const 
storage::DeleteAzureDataLakeStorageP
   }
 }
 
+std::optional AzureDataLakeStorage::fetchFile(const 
FetchAzureDataLakeStorageParameters& params, io::BaseStream& stream) {
+  try {
+auto fetch_res = data_lake_storage_client_->fetchFile(params);
+
+std::vector buffer(4096);
+size_t write_size = 0;
+if (fetch_res.FileSize < 0) return 0;
+while (write_size < gsl::narrow(fetch_res.FileSize)) {
+  const auto next_write_size = 
(std::min)(gsl::narrow(fetch_res.FileSize) - write_size, size_t{4096});

Review comment:
   Updated in 172d2a4f893238d7a9818b812cc0b7b671c3cd0b

##
File path: extensions/azure/processors/FetchAzureDataLakeStorage.cpp
##
@@ -0,0 +1,122 @@
+/**
+ * @file FetchAzureDataLakeStorage.cpp
+ * FetchAzureDataLakeStorage class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "FetchAzureDataLakeStorage.h"
+
+#include "core/Resource.h"
+
+namespace org::apache::nifi::minifi::azure::processors {
+
+const core::Property FetchAzureDataLakeStorage::RangeStart(
+core::PropertyBuilder::createProperty("Range Start")
+  ->withDescription("The byte position at which to start reading from the 
object. An empty value or a value of zero will start reading at the beginning 
of the object.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+
+const core::Property FetchAzureDataLakeStorage::RangeLength(
+core::PropertyBuilder::createProperty("Range Length")
+  ->withDescription("The number of bytes to download from the object, 
starting from the Range Start. "
+"An empty value or a value that extends beyond the end 
of the object will read to the end of the object.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+
+const core::Property FetchAzureDataLakeStorage::NumberOfRetries(
+core::PropertyBuilder::createProperty("Number of Retries")
+  ->withDescription("The number of automatic retries to perform if the 
download fails.")
+  ->withDefaultValue(0)
+  ->supportsExpressionLanguage(true)
+  ->build());
+
+const core::Relationship FetchAzureDataLakeStorage::Success("success", "Files 
that have been successfully fetched from Azure storage are transferred to this 
relationship");
+const core::Relationship FetchAzureDataLakeStorage::Failure("failure", "In 
case of fetch failure flowfiles are transferred to this relationship");
+
+void FetchAzureDataLakeStorage::initialize() {
+  // Add new supported properties
+  setSupportedProperties({
+AzureStorageCredentialsService,
+FilesystemName,
+DirectoryName,
+FileName,
+RangeStart,
+RangeLength,
+NumberOfRetries
+  });
+  // Set the supported relationships
+  setSupportedRelationships({
+Success,
+Failure
+  });
+}
+
+std::optional 
FetchAzureDataLakeStorage::buildFetchParameters(
+const std::shared_ptr& context, const 
std::shared_ptr& flow_file) {
+  storage::FetchAzureDataLakeStorageParameters params;
+  if (!setCommonParameters(params, *context, flow_file)) {
+return std::nullopt;
+  }
+
+  std::string value;
+  if (context->getProperty(RangeStart, value, flow_file)) {
+params.range_start = std::stoull(value);
+logger_->log_debug("Range Start property set to %llu", 
*params.range_start);
+  }
+
+  if (context->getProperty(RangeLength, value, flow_file)) {
+params.range_length = std::stoull(value);
+logger_->log_debug("Range L

[jira] [Updated] (NIFI-9468) Bump kafka client to 2.6.3

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9468:
---
Fix Version/s: 1.15.1

> Bump kafka client to 2.6.3
> --
>
> Key: NIFI-9468
> URL: https://issues.apache.org/jira/browse/NIFI-9468
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> See release notes here:
> [https://downloads.apache.org/kafka/2.6.3/RELEASE_NOTES.html]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9473) Upgrade Jackson to 2.12.5

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9473:
---
Fix Version/s: 1.15.1

> Upgrade Jackson to 2.12.5
> -
>
> Key: NIFI-9473
> URL: https://issues.apache.org/jira/browse/NIFI-9473
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Upgrade Jackson from 2.12.3 to 2.12.5



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9420) Upgrade Guava in MiNiFi to 31.0.1

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9420:
---
Fix Version/s: 1.15.1

> Upgrade Guava in MiNiFi to 31.0.1
> -
>
> Key: NIFI-9420
> URL: https://issues.apache.org/jira/browse/NIFI-9420
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: MiNiFi
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
>  Labels: dependency-upgrade, security
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> MiNiFi depends on Guava 26.0 to support caching in the Config Service REST 
> Resource. Although MiNiFi does not use the feature, Guava versions 30 and 
> below have a vulnerability associated with temporary directory creation as 
> described in [CVE-2020-8908|https://nvd.nist.gov/vuln/detail/CVE-2020-8908]. 
> Upgrading Guava to 31.0.1 addresses the associated vulnerability.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9426) Remove jackson-mapper-asl from MiNiFi

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9426:
---
Fix Version/s: 1.15.1

> Remove jackson-mapper-asl from MiNiFi
> -
>
> Key: NIFI-9426
> URL: https://issues.apache.org/jira/browse/NIFI-9426
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: MiNiFi
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
>  Labels: security
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The {{minifi-framework-core}} module has a dependency on 
> {{org.codehaus.jackson:jackson-mapper-asl}} version 1.9.13 that is no longer 
> used. MiNiFi has current dependencies on Jackson 2, so the 
> {{jackson-mapper-asl}} dependency should removed.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9419) ParseCEF - bump dependency to 2.1.0 and allow null extensions

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9419:
---
Fix Version/s: 1.15.1

> ParseCEF - bump dependency to 2.1.0 and allow null extensions
> -
>
> Key: NIFI-9419
> URL: https://issues.apache.org/jira/browse/NIFI-9419
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The version 2.1.0 of the parser has just been released. It allows for null 
> values to be set for extensions. Bumping the dependency version and adding a 
> property/unit test to cover this new feature.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9408) MiNiFi - Remove Ignite dependencies

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9408:
---
Fix Version/s: 1.15.1

> MiNiFi - Remove Ignite dependencies
> ---
>
> Key: NIFI-9408
> URL: https://issues.apache.org/jira/browse/NIFI-9408
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: MiNiFi
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Ignite is not used in MiNiFi Java and the dependencies against it should be 
> removed.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9396) Upgrade JUnit for MiNiFi to 4.13.2

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9396:
---
Fix Version/s: 1.15.1

> Upgrade JUnit for MiNiFi to 4.13.2
> --
>
> Key: NIFI-9396
> URL: https://issues.apache.org/jira/browse/NIFI-9396
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Bump junit to 4.13.2 (latest for 4.x)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9394) Remove RequestLogger Filter and TimerFilter

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9394:
---
Fix Version/s: 1.15.1

> Remove RequestLogger Filter and TimerFilter
> ---
>
> Key: NIFI-9394
> URL: https://issues.apache.org/jira/browse/NIFI-9394
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI, Security
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> With the introduction of standardized HTTP Request Logging in NIFI-9291, the 
> custom {{RequestLogger}} and {{TimerFilter}} classes should be removed.
> The {{RequestLogger}} filter logs informational messages for unencrypted HTTP 
> access attempts. With the default NiFi configuration enabling HTTPS, this use 
> case is no longer common, and HTTP Request Logging in nifi-request.log can 
> provide the same information plus additional request details.
> The {{TimerFilter}} runs for every HTTP request and writes debug log messages 
> that include millisecond timing information as well as the contents of 
> {{X-RequestTransactionId}} and {{X-Request-Replicated}} HTTP headers. This 
> debug logging is not enabled in the default Logback configuration. The same 
> header and request timing information can be captured using a custom 
> {{nifi.web.request.log.format}} with the following message format:
> {noformat}
> %{X-RequestTransactionId}i %{X-Request-Replicated}i %T 
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9395) Upgrade httpclient for MiNiFi

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9395:
---
Fix Version/s: 1.15.1

> Upgrade httpclient for MiNiFi
> -
>
> Key: NIFI-9395
> URL: https://issues.apache.org/jira/browse/NIFI-9395
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: MiNiFi
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Bump httpclient to 4.5.13 (latest).



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9393) Set Scheduled State for Ports with Flow Definitions

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9393:
---
Fix Version/s: 1.15.1

> Set Scheduled State for Ports with Flow Definitions
> ---
>
> Key: NIFI-9393
> URL: https://issues.apache.org/jira/browse/NIFI-9393
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The {{Download Flow Definition}} feature for exporting Process Groups as JSON 
> does not include the Scheduled State for Input Ports, Output Ports, or Remote 
> Ports. As a result of this behavior, downloading a Process Group with 
> {{DISABLED}} ports changes the initial status to {{STOPPED}} when uploading 
> the flow definition.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request #1227: MINIFICPP-1222 Remove engine queue for ExecutePythonProcessor

[GitBox]

lordgamez opened a new pull request #1227:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1227


   Originally the Jira ticket was about reowrking the engine queue to only 
allow the `maximum concurrent tasks` number of script engines to run at the 
same time. The problem with that was that even if the script engines were 
executed concurrently they use the same python interpreter with the global 
interpreter lock in place so no concurrency can take place.
   
   The only possibility for concurrency to use sub-interpreters which would be 
also problematic. First of all pybind11 does not support an interface for 
sub-interpreters, it is only available through the CPython API, but the 
documentation also warns about the caveats of the sub-interpreters here: 
https://pybind11.readthedocs.io/en/stable/advanced/embedding.html
   Secondly using the CPython API for sub-interpreters would require us to 
always know which thread we are running on and create separate sub-interpreters 
and thread states for each thread (in this case it would possibly not even 
allow us to run the python scripts from the main thread anymore). Managing the 
thread states and the sub-interpreters would be cumbersome in our architecture.
   
   Because all of the above problems I removed the python script queue from the 
ExecutePythonProcessor which will run the python scripts on a single thread.
   
   --
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI 
results for build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-9260) Making "write and rename" behaviour optional in PutHDFS

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9260:
---
Fix Version/s: 1.15.1

> Making "write and rename" behaviour optional in PutHDFS
> ---
>
> Key: NIFI-9260
> URL: https://issues.apache.org/jira/browse/NIFI-9260
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> In some cases the current "write and rename" behaviour of PutHDFS 
> undesirable. Particularly if the the server is backed with AWS S3 which has 
> no "native" rename but copies file content when renaming, serious perfomance 
> issues might arise.
> In order to avoid this, I propose to make this behaviour optional and make it 
> possible for the PutHDFS to write directly to the final destination. Of 
> course, depending on the use case, there is the risk of reading 
> "half-written" files in this manner, but in some scenarios this is not a 
> danger. Also, in case of S3 this should not be an issue as files in S3 are 
> visible only after the "write" operation is finished



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9394) Remove RequestLogger Filter and TimerFilter

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9394:
---
Fix Version/s: (was: 1.15.1)

> Remove RequestLogger Filter and TimerFilter
> ---
>
> Key: NIFI-9394
> URL: https://issues.apache.org/jira/browse/NIFI-9394
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI, Security
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.16.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> With the introduction of standardized HTTP Request Logging in NIFI-9291, the 
> custom {{RequestLogger}} and {{TimerFilter}} classes should be removed.
> The {{RequestLogger}} filter logs informational messages for unencrypted HTTP 
> access attempts. With the default NiFi configuration enabling HTTPS, this use 
> case is no longer common, and HTTP Request Logging in nifi-request.log can 
> provide the same information plus additional request details.
> The {{TimerFilter}} runs for every HTTP request and writes debug log messages 
> that include millisecond timing information as well as the contents of 
> {{X-RequestTransactionId}} and {{X-Request-Replicated}} HTTP headers. This 
> debug logging is not enabled in the default Logback configuration. The same 
> header and request timing information can be captured using a custom 
> {{nifi.web.request.log.format}} with the following message format:
> {noformat}
> %{X-RequestTransactionId}i %{X-Request-Replicated}i %T 
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9477) data loss doing Incremental load based on a timestamp from SQL server using QueryDatabaseTableRecord

[Abdullah Aqeeli (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abdullah Aqeeli updated NIFI-9477:
--
Description: 
hello all,

 

We are using Nifi to do an incremental load from SQL Server using 
QueryDatabaseTableRecord where max-value column is updated_at column with a 
[Datetime2 
|https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
 type. The source tables get updated in milliseconds. We noticed that we are 
losing few events with each QueryDatabaseTableRecord run. As a work-around we 
created a custom processor that subtracts few seconds from the max-value in the 
state fetching overlapping events and then we handle any duplicates later on in 
the pipeline and it seems to solve the issue. However, this is not optimal. 
We'd like to know what's causing the skips. Our assumption so far is that 
timestamp datatype accuracy is causing the skips in the incremental load 
however we're not sure.  

 

Thanks!

  was:
hello all,

 

We are using Nifi to do an incremental load from SQL Server using 
QueryDatabaseTableRecord where max-value column is updated_at column with a 
[Datetime2 
|https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
 type. The source tables get updated in milliseconds. We noticed that we are 
losing few events doing with each QueryDatabaseTableRecord run. As a 
work-around we created a custom processor that subtracts few seconds from the 
max-value in the state fetching overlapping events and then we handle any 
duplicates later on in the pipeline and it seems to solve the issue. However, 
this is not optimal. We'd like to know what's causing the skips. Our assumption 
so far is that timestamp datatype accuracy is causing the skips in the 
incremental load however we're not sure.  

 

Thanks!


> data loss doing Incremental load based on a timestamp from SQL server using 
> QueryDatabaseTableRecord 
> -
>
> Key: NIFI-9477
> URL: https://issues.apache.org/jira/browse/NIFI-9477
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.13.2
> Environment: ubuntu 20 
> 2-node cluster using an embedded zk
>Reporter: Abdullah Aqeeli
>Priority: Critical
>
> hello all,
>  
> We are using Nifi to do an incremental load from SQL Server using 
> QueryDatabaseTableRecord where max-value column is updated_at column with a 
> [Datetime2 
> |https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
>  type. The source tables get updated in milliseconds. We noticed that we are 
> losing few events with each QueryDatabaseTableRecord run. As a work-around we 
> created a custom processor that subtracts few seconds from the max-value in 
> the state fetching overlapping events and then we handle any duplicates later 
> on in the pipeline and it seems to solve the issue. However, this is not 
> optimal. We'd like to know what's causing the skips. Our assumption so far is 
> that timestamp datatype accuracy is causing the skips in the incremental load 
> however we're not sure.  
>  
> Thanks!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9369) JoltTransformJSON removes unicode escape sequence after output

[David Handermann (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-9369:
---
Fix Version/s: (was: 1.16.0)

> JoltTransformJSON removes unicode escape sequence after output
> --
>
> Key: NIFI-9369
> URL: https://issues.apache.org/jira/browse/NIFI-9369
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
> Environment: Linux 64-bit, Ubuntu distro, default OS encoding is UTF-8
>Reporter: Önder Ezdi
>Priority: Major
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Problem: 
> If a JSON comes with unicode escape sequence, processor accepts it and after 
> transformation, exports it with native characters instead unicode sequence. 
> Later InvokeHTTP requests to any server breaks json due character problem and 
> server won't accept if standard encoding configurations applied.
> Behavior:
> JoltTransformJSON processor input:
> {code:java}
> {"value": 
> "\u011f\u00fc\u015f\u0131\u00f6\u00e7\u011e\u00dc\u015e\u0130\u00d6\u00c7"}{code}
> JoltTransformJSON processor output:
> {code:java}
> {"value": "ğüşıöçĞÜŞİÖÇ"}{code}
> Expected:
> If its not a bug, "Encode Output" or "No encoding" output options should be 
> supplied as in dropdown selection or something like that on UI.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9194) Upsert for Oracle12+

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9194:
---
Fix Version/s: 1.15.1

> Upsert for Oracle12+
> 
>
> Key: NIFI-9194
> URL: https://issues.apache.org/jira/browse/NIFI-9194
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
> Environment: All OS
>Reporter: ROBERTO SANTOS
>Priority: Major
>  Labels: features
> Fix For: 1.16.0, 1.15.1
>
>   Original Estimate: 168h
>  Time Spent: 1h
>  Remaining Estimate: 23h 10m
>
> I have implemented Upsert capability for Oracle12+ in PutDatabaseRecord and 
> its Unit tests.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-5821) ExecuteScript should say Python is really Jython running

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-5821:
---
Fix Version/s: 1.15.1

> ExecuteScript should say Python is really Jython running
> 
>
> Key: NIFI-5821
> URL: https://issues.apache.org/jira/browse/NIFI-5821
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1, 1.15.0
>Reporter: Ryan Hendrickson
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
> Attachments: image-2018-11-15-00-37-05-004.png
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> ExecuteScript drop-down should say "jython" or "Python (jython)" ... 
> Something to indicate it is running jython.
> Code executed in the ExecuteScript processor, when Python is selected, is 
> actually running as Jython.  This should be made far more clear on the UI as 
> a user is selecting the Script Language.  The only place python is made 
> reference to is in the tags for the processor, which also makes reference to 
> python.
> Jython datetime.datetime is not handled the same way that Python 
> datetime.datetime is because of mapping datetime back to Java objects in 
> Jython.  This can cause plenty of issues, and cause Python code to need to be 
> modified to jython supported code.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9185) Add Avro logical type to SelectHive3QL processor

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9185:
---
Fix Version/s: 1.15.1

> Add Avro logical type to SelectHive3QL processor
> 
>
> Key: NIFI-9185
> URL: https://issues.apache.org/jira/browse/NIFI-9185
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Timea Barna
>Assignee: Timea Barna
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Add a similar property as ExecuteSQL and default it to false, add logical 
> type support for SelectHive3QL processor.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9385) Add a metrics provider in NiFi for Cloudera Manager

[David Handermann (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-9385:
---
Fix Version/s: (was: 1.16.0)

> Add a metrics provider in NiFi for Cloudera Manager
> ---
>
> Key: NIFI-9385
> URL: https://issues.apache.org/jira/browse/NIFI-9385
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Timea Barna
>Assignee: Timea Barna
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Add a metrics provider in NiFi for Cloudera Manager



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9408) MiNiFi - Remove Ignite dependencies

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459327#comment-17459327
 ] 

ASF subversion and git services commented on NIFI-9408:
---

Commit bd16b93da6ee5911872f3e868d43464d70da0621 in nifi's branch 
refs/heads/support/nifi-1.15 from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=bd16b93 ]

NIFI-9408 - MiNiFi - remove Ignite dependencies

Signed-off-by: Matthew Burgess 

This closes #5546


> MiNiFi - Remove Ignite dependencies
> ---
>
> Key: NIFI-9408
> URL: https://issues.apache.org/jira/browse/NIFI-9408
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: MiNiFi
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Ignite is not used in MiNiFi Java and the dependencies against it should be 
> removed.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9396) Upgrade JUnit for MiNiFi to 4.13.2

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459328#comment-17459328
 ] 

ASF subversion and git services commented on NIFI-9396:
---

Commit b79714c6f6e1273c7a83511d639389779c6506ef in nifi's branch 
refs/heads/support/nifi-1.15 from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=b79714c ]

NIFI-9396 - MiNiFi - bump junit to 4.13.2

This closes #5538

Signed-off-by: David Handermann 


> Upgrade JUnit for MiNiFi to 4.13.2
> --
>
> Key: NIFI-9396
> URL: https://issues.apache.org/jira/browse/NIFI-9396
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Bump junit to 4.13.2 (latest for 4.x)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9185) Add Avro logical type to SelectHive3QL processor

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459335#comment-17459335
 ] 

ASF subversion and git services commented on NIFI-9185:
---

Commit 2f9963a534d55df2608fe4efaa0841bee7bba393 in nifi's branch 
refs/heads/support/nifi-1.15 from Timea Barna
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2f9963a ]

NIFI-9185 Add Avro logical type to SelectHive3QL processor

Modifying unit test to avoid systemdefault timezone usuage

NIFI-9185 Applying review recommendations removing duplicate dependency from 
pom.xml

Signed-off-by: Matthew Burgess 

This closes #5358


> Add Avro logical type to SelectHive3QL processor
> 
>
> Key: NIFI-9185
> URL: https://issues.apache.org/jira/browse/NIFI-9185
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Timea Barna
>Assignee: Timea Barna
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Add a similar property as ExecuteSQL and default it to false, add logical 
> type support for SelectHive3QL processor.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9393) Set Scheduled State for Ports with Flow Definitions

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459331#comment-17459331
 ] 

ASF subversion and git services commented on NIFI-9393:
---

Commit 3955b88302b5f60e104c12b656fed64b5169b8b1 in nifi's branch 
refs/heads/support/nifi-1.15 from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3955b88 ]

NIFI-9393 Set Port Scheduled State for Flow Definitions

- Set Scheduled State for Versioned Port and Versioned Remote Port when mapping 
Flow Definition
- Updated StandardProcessGroup to set disable Port based on Scheduled State of 
DISABLED
- Updated StandardProcessGroup to set Remote Port transmitting based on 
Scheduled State of ENABLED

Signed-off-by: Nathan Gough 

This closes #5534.


> Set Scheduled State for Ports with Flow Definitions
> ---
>
> Key: NIFI-9393
> URL: https://issues.apache.org/jira/browse/NIFI-9393
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The {{Download Flow Definition}} feature for exporting Process Groups as JSON 
> does not include the Scheduled State for Input Ports, Output Ports, or Remote 
> Ports. As a result of this behavior, downloading a Process Group with 
> {{DISABLED}} ports changes the initial status to {{STOPPED}} when uploading 
> the flow definition.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9194) Upsert for Oracle12+

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459334#comment-17459334
 ] 

ASF subversion and git services commented on NIFI-9194:
---

Commit 2273fe57660cad05943e018abcbeb2fec8d4325c in nifi's branch 
refs/heads/support/nifi-1.15 from Roberto Santos
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2273fe5 ]

NIFI-9194: Upsert for Oracle12+

Fixes pr #5366.

Fixes pr #5366. Replace tabchars fot whitespaces.

Fixes pr #5366. Replaced tabchars for whitespaces. Removed unnecessary comments.

Signed-off-by: Matthew Burgess 

This closes #5366


> Upsert for Oracle12+
> 
>
> Key: NIFI-9194
> URL: https://issues.apache.org/jira/browse/NIFI-9194
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
> Environment: All OS
>Reporter: ROBERTO SANTOS
>Priority: Major
>  Labels: features
> Fix For: 1.16.0, 1.15.1
>
>   Original Estimate: 168h
>  Time Spent: 1h
>  Remaining Estimate: 23h 10m
>
> I have implemented Upsert capability for Oracle12+ in PutDatabaseRecord and 
> its Unit tests.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9395) Upgrade httpclient for MiNiFi

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459329#comment-17459329
 ] 

ASF subversion and git services commented on NIFI-9395:
---

Commit 94fee5b289ad86d8d7d56f4f3e52c23dab00d5e7 in nifi's branch 
refs/heads/support/nifi-1.15 from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=94fee5b ]

NIFI-9395 - MiNiFi - bump httpclient to 4.5.13

Signed-off-by: Matthew Burgess 

This closes #5537


> Upgrade httpclient for MiNiFi
> -
>
> Key: NIFI-9395
> URL: https://issues.apache.org/jira/browse/NIFI-9395
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: MiNiFi
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Bump httpclient to 4.5.13 (latest).



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-5821) ExecuteScript should say Python is really Jython running

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-5821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459336#comment-17459336
 ] 

ASF subversion and git services commented on NIFI-5821:
---

Commit 4e88943e60817aac6d779cf0bf2c970e9cfdb16b in nifi's branch 
refs/heads/support/nifi-1.15 from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4e88943 ]

NIFI-5821 Added Engine Name to Script Engine property descriptions

Signed-off-by: Matthew Burgess 

This closes #5529


> ExecuteScript should say Python is really Jython running
> 
>
> Key: NIFI-5821
> URL: https://issues.apache.org/jira/browse/NIFI-5821
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1, 1.15.0
>Reporter: Ryan Hendrickson
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
> Attachments: image-2018-11-15-00-37-05-004.png
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> ExecuteScript drop-down should say "jython" or "Python (jython)" ... 
> Something to indicate it is running jython.
> Code executed in the ExecuteScript processor, when Python is selected, is 
> actually running as Jython.  This should be made far more clear on the UI as 
> a user is selecting the Script Language.  The only place python is made 
> reference to is in the tags for the processor, which also makes reference to 
> python.
> Jython datetime.datetime is not handled the same way that Python 
> datetime.datetime is because of mapping datetime back to Java objects in 
> Jython.  This can cause plenty of issues, and cause Python code to need to be 
> modified to jython supported code.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9394) Remove RequestLogger Filter and TimerFilter

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459330#comment-17459330
 ] 

ASF subversion and git services commented on NIFI-9394:
---

Commit 73b32464303bb75e89b01a7c817f059e52f03d5b in nifi's branch 
refs/heads/support/nifi-1.15 from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=73b3246 ]

NIFI-9394 Removed RequestLogger and TimerFilter

- Removed logger references from logback.xml

Signed-off-by: Joe Gresock 

This closes #5543.


> Remove RequestLogger Filter and TimerFilter
> ---
>
> Key: NIFI-9394
> URL: https://issues.apache.org/jira/browse/NIFI-9394
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI, Security
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.16.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> With the introduction of standardized HTTP Request Logging in NIFI-9291, the 
> custom {{RequestLogger}} and {{TimerFilter}} classes should be removed.
> The {{RequestLogger}} filter logs informational messages for unencrypted HTTP 
> access attempts. With the default NiFi configuration enabling HTTPS, this use 
> case is no longer common, and HTTP Request Logging in nifi-request.log can 
> provide the same information plus additional request details.
> The {{TimerFilter}} runs for every HTTP request and writes debug log messages 
> that include millisecond timing information as well as the contents of 
> {{X-RequestTransactionId}} and {{X-Request-Replicated}} HTTP headers. This 
> debug logging is not enabled in the default Logback configuration. The same 
> header and request timing information can be captured using a custom 
> {{nifi.web.request.log.format}} with the following message format:
> {noformat}
> %{X-RequestTransactionId}i %{X-Request-Replicated}i %T 
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9394) Remove RequestLogger Filter and TimerFilter

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459332#comment-17459332
 ] 

ASF subversion and git services commented on NIFI-9394:
---

Commit 7208cc8b2287927a58bc16262114365e1fbe1900 in nifi's branch 
refs/heads/support/nifi-1.15 from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7208cc8 ]

Revert "NIFI-9394 Removed RequestLogger and TimerFilter"

This reverts commit 73b32464303bb75e89b01a7c817f059e52f03d5b.


> Remove RequestLogger Filter and TimerFilter
> ---
>
> Key: NIFI-9394
> URL: https://issues.apache.org/jira/browse/NIFI-9394
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI, Security
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.16.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> With the introduction of standardized HTTP Request Logging in NIFI-9291, the 
> custom {{RequestLogger}} and {{TimerFilter}} classes should be removed.
> The {{RequestLogger}} filter logs informational messages for unencrypted HTTP 
> access attempts. With the default NiFi configuration enabling HTTPS, this use 
> case is no longer common, and HTTP Request Logging in nifi-request.log can 
> provide the same information plus additional request details.
> The {{TimerFilter}} runs for every HTTP request and writes debug log messages 
> that include millisecond timing information as well as the contents of 
> {{X-RequestTransactionId}} and {{X-Request-Replicated}} HTTP headers. This 
> debug logging is not enabled in the default Logback configuration. The same 
> header and request timing information can be captured using a custom 
> {{nifi.web.request.log.format}} with the following message format:
> {noformat}
> %{X-RequestTransactionId}i %{X-Request-Replicated}i %T 
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9260) Making "write and rename" behaviour optional in PutHDFS

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459333#comment-17459333
 ] 

ASF subversion and git services commented on NIFI-9260:
---

Commit 534e6eafe7b8622e6d141802a40d7c6e01a09361 in nifi's branch 
refs/heads/support/nifi-1.15 from Bence Simon
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=534e6ea ]

NIFI-9260 Making the 'write and rename' behaviour optional for PutHDFS

This closes #5423.

Signed-off-by: Peter Turcsanyi 


> Making "write and rename" behaviour optional in PutHDFS
> ---
>
> Key: NIFI-9260
> URL: https://issues.apache.org/jira/browse/NIFI-9260
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Simon Bence
>Assignee: Simon Bence
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> In some cases the current "write and rename" behaviour of PutHDFS 
> undesirable. Particularly if the the server is backed with AWS S3 which has 
> no "native" rename but copies file content when renaming, serious perfomance 
> issues might arise.
> In order to avoid this, I propose to make this behaviour optional and make it 
> possible for the PutHDFS to write directly to the final destination. Of 
> course, depending on the use case, there is the risk of reading 
> "half-written" files in this manner, but in some scenarios this is not a 
> danger. Also, in case of S3 this should not be an issue as files in S3 are 
> visible only after the "write" operation is finished



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9448) Potential IllegalStateException when S2S HTTP Client Shutdown

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9448:
---
Fix Version/s: 1.15.1

> Potential IllegalStateException when S2S HTTP Client Shutdown
> -
>
> Key: NIFI-9448
> URL: https://issues.apache.org/jira/browse/NIFI-9448
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> NiFi Site-to-Site communication over HTTP relies on 
> {{SiteToSiteRestApiClient}} to handle requests and responses using the Apache 
> HttpComponents library. The HttpComponents library maintains a connection 
> pool for processing HTTP transactions.
> In the course of sending or receiving files, NiFi starts a background thread 
> to make periodic requests to the remote NiFi system in order to extend the 
> current transaction. The {{SiteToSiteRestApiClient}} stops the background 
> thread after completing a transaction. In some cases, the request to extend 
> the transaction can occur after the HttpComponents connection pool is 
> shutdown, resulting in an {{{}IllegalStateException{}}}. The background 
> thread treats all exceptions as failure conditions and then attempts to close 
> the S2S HTTP client itself. This causes subsequent retry requests using the 
> same {{SiteToSiteRestApiClient}} to fail with the same 
> {{IllegalStateException}} indicating that the connection pool is shutdown.
> The behavior of the extend transaction command should be changed to avoid 
> closing the S2S HTTP client when encountering an 
> {{{}IllegalStateException{}}}. This approach will support the potential for 
> subsequent retries to work or fail based on existing timeout configuration 
> settings.
> The following log messages provide a stack trace of the 
> {{IllegalStateException}} in the extend transaction command and subsequent 
> exception in the Remote Process Group Port connection.
> {noformat}
> WARN org.apache.nifi.remote.util.SiteToSiteRestApiClient: Failed to extend 
> transaction ttl
> java.lang.IllegalStateException: Connection pool shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at 
> org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191)
> at 
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:267)
> at 
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
> at 
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at 
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
> at 
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.extendTransaction(SiteToSiteRestApiClient.java:1028)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.lambda$startExtendingTtl$0(SiteToSiteRestApiClient.java:990)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> INFO org.apache.http.impl.execchain.RetryExec: I/O exception 
> (org.apache.http.NoHttpResponseException) caught when processing request: The 
> target server failed to respond
> INFO org.apache.http.impl.execchain.RetryExec: Retrying request
> ERROR org.apache.nifi.remote.StandardRemoteGroupPort: 
> RemoteGroupPort[name=REMOTE_PORT,targets=URL] failed to communicate with 
> remote NiFi instance due to java.lang.IllegalStateException: Connection pool 
> shut down
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9448) Potential IllegalStateException when S2S HTTP Client Shutdown

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459337#comment-17459337
 ] 

ASF subversion and git services commented on NIFI-9448:
---

Commit 7cedb9b8a89f2d3e8e258be24c99f70a34198d71 in nifi's branch 
refs/heads/support/nifi-1.15 from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7cedb9b ]

NIFI-9448 Improved S2S HTTP Extend Transaction Exception Handling

- Refactor background transaction extension to ExtendTransactionCommand
- Avoid closing S2S HTTP client for IllegalStateExceptions
- Avoid creating additional S2S HTTP client instance for transaction extension 
commands
- Add check for extend transaction requests received in client test class
- Add null check for Peer Persistence implementation in PeerSelector

Signed-off-by: Joe Gresock 

This closes #5577.


> Potential IllegalStateException when S2S HTTP Client Shutdown
> -
>
> Key: NIFI-9448
> URL: https://issues.apache.org/jira/browse/NIFI-9448
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> NiFi Site-to-Site communication over HTTP relies on 
> {{SiteToSiteRestApiClient}} to handle requests and responses using the Apache 
> HttpComponents library. The HttpComponents library maintains a connection 
> pool for processing HTTP transactions.
> In the course of sending or receiving files, NiFi starts a background thread 
> to make periodic requests to the remote NiFi system in order to extend the 
> current transaction. The {{SiteToSiteRestApiClient}} stops the background 
> thread after completing a transaction. In some cases, the request to extend 
> the transaction can occur after the HttpComponents connection pool is 
> shutdown, resulting in an {{{}IllegalStateException{}}}. The background 
> thread treats all exceptions as failure conditions and then attempts to close 
> the S2S HTTP client itself. This causes subsequent retry requests using the 
> same {{SiteToSiteRestApiClient}} to fail with the same 
> {{IllegalStateException}} indicating that the connection pool is shutdown.
> The behavior of the extend transaction command should be changed to avoid 
> closing the S2S HTTP client when encountering an 
> {{{}IllegalStateException{}}}. This approach will support the potential for 
> subsequent retries to work or fail based on existing timeout configuration 
> settings.
> The following log messages provide a stack trace of the 
> {{IllegalStateException}} in the extend transaction command and subsequent 
> exception in the Remote Process Group Port connection.
> {noformat}
> WARN org.apache.nifi.remote.util.SiteToSiteRestApiClient: Failed to extend 
> transaction ttl
> java.lang.IllegalStateException: Connection pool shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at 
> org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191)
> at 
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:267)
> at 
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
> at 
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at 
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
> at 
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.extendTransaction(SiteToSiteRestApiClient.java:1028)
> at 
> org.apache.nifi.remote.util.SiteToSiteRestApiClient.lambda$startExtendingTtl$0(SiteToSiteRestApiClient.java:990)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu

[jira] [Updated] (NIFI-9355) Upgrade Apache Curator from 4.2.0 to 5.2.0

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9355:
---
Fix Version/s: 1.15.1

> Upgrade Apache Curator from 4.2.0 to 5.2.0
> --
>
> Key: NIFI-9355
> URL: https://issues.apache.org/jira/browse/NIFI-9355
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Tools and Build
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>  Labels: dependency-upgrade
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Apache Curator 5.2.0 includes several bug fixes and improvements including 
> CURATOR-526, which corrects connection string logging status in 
> EnsembleTracker from error to debug when used in reference to certain 
> versions of ZooKeeper.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9457) Add support to microseconds in PutKudu

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9457:
---
Fix Version/s: 1.15.1

> Add support to microseconds in PutKudu
> --
>
> Key: NIFI-9457
> URL: https://issues.apache.org/jira/browse/NIFI-9457
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Assuming the field to push into Kudu is of type String, we're currently using 
> the DataTypeUtils.toTimestamp method to parse this String and convert it into 
> a timestamp. We, however, rely on classes that would only keep milliseconds 
> precision. We should provide a way to retain microseconds precision since 
> it's supported by Kudu. Until we refactor NiFi to leverage recent classes for 
> Date/Time handling, the best approach is to have an alternative path specific 
> to Kudu to support this specific case.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9457) Add support to microseconds in PutKudu

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459341#comment-17459341
 ] 

ASF subversion and git services commented on NIFI-9457:
---

Commit 4aefc67ccefe4b018483c89ac018798c35871c2c in nifi's branch 
refs/heads/support/nifi-1.15 from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4aefc67 ]

NIFI-9457 Support microseconds for String Timestamps in PutKudu

- Implemented override for Timestamp Record Field Type format handling to add 
support for optional microseconds
- Added FieldConverter and ObjectTimestampFieldConverter implementation for 
generalized Timestamp parsing using DateTimeFormatter
- Updated PutKudu unit tests for standard Timestamp and Timestamp with 
microseconds

Signed-off-by: Pierre Villard 

This closes #5589.


> Add support to microseconds in PutKudu
> --
>
> Key: NIFI-9457
> URL: https://issues.apache.org/jira/browse/NIFI-9457
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Assuming the field to push into Kudu is of type String, we're currently using 
> the DataTypeUtils.toTimestamp method to parse this String and convert it into 
> a timestamp. We, however, rely on classes that would only keep milliseconds 
> precision. We should provide a way to retain microseconds precision since 
> it's supported by Kudu. Until we refactor NiFi to leverage recent classes for 
> Date/Time handling, the best approach is to have an alternative path specific 
> to Kudu to support this specific case.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9375) Refactor Registry using OSUtils from nifi-bootstrap-utils

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9375:
---
Fix Version/s: 1.15.1

> Refactor Registry using OSUtils from nifi-bootstrap-utils
> -
>
> Key: NIFI-9375
> URL: https://issues.apache.org/jira/browse/NIFI-9375
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: NiFi Registry
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> NiFi Registry has a version of {{OSUtils}} that warns of illegal reflective 
> access when running under Java 11. The version of {{OSUtils}} in 
> {{nifi-bootstrap-utils}} avoids illegal reflective access and provides the 
> same functionality, so NiFi Registry should be updated to use the shared 
> class.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9355) Upgrade Apache Curator from 4.2.0 to 5.2.0

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459340#comment-17459340
 ] 

ASF subversion and git services commented on NIFI-9355:
---

Commit 488ca1bcfde22597e9d25038864ddbbb22fbf844 in nifi's branch 
refs/heads/support/nifi-1.15 from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=488ca1b ]

NIFI-9355 Upgraded Apache Curator from 4.2.0 to 5.2.0

- Removed unused Curator dependency declaration from MiNiFi

Signed-off-by: Nathan Gough 

This closes #5503.


> Upgrade Apache Curator from 4.2.0 to 5.2.0
> --
>
> Key: NIFI-9355
> URL: https://issues.apache.org/jira/browse/NIFI-9355
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Tools and Build
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
>  Labels: dependency-upgrade
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Apache Curator 5.2.0 includes several bug fixes and improvements including 
> CURATOR-526, which corrects connection string logging status in 
> EnsembleTracker from error to debug when used in reference to certain 
> versions of ZooKeeper.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9375) Refactor Registry using OSUtils from nifi-bootstrap-utils

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459343#comment-17459343
 ] 

ASF subversion and git services commented on NIFI-9375:
---

Commit 4cfa7f7eeb768c624380b9f05271638a39fc8ad5 in nifi's branch 
refs/heads/support/nifi-1.15 from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4cfa7f7 ]

NIFI-9375 Refactored Registry using OSUtils from nifi-bootstrap-utils

Signed-off-by: Pierre Villard 

This closes #5520.


> Refactor Registry using OSUtils from nifi-bootstrap-utils
> -
>
> Key: NIFI-9375
> URL: https://issues.apache.org/jira/browse/NIFI-9375
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: NiFi Registry
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Minor
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> NiFi Registry has a version of {{OSUtils}} that warns of illegal reflective 
> access when running under Java 11. The version of {{OSUtils}} in 
> {{nifi-bootstrap-utils}} avoids illegal reflective access and provides the 
> same functionality, so NiFi Registry should be updated to use the shared 
> class.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9376) Update Documentation for Encrypted Passwords in Flows

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459345#comment-17459345
 ] 

ASF subversion and git services commented on NIFI-9376:
---

Commit 5e2df44dcde4969244c66752e7740e2834071f45 in nifi's branch 
refs/heads/support/nifi-1.15 from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5e2df44 ]

NIFI-9376 Updated Admin Guide removing reference to default password

Signed-off-by: Pierre Villard 

This closes #5519.


> Update Documentation for Encrypted Passwords in Flows
> -
>
> Key: NIFI-9376
> URL: https://issues.apache.org/jira/browse/NIFI-9376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Trivial
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The NiFi Administrator's Guide section {{Encrypted Passwords in Flows}} 
> indicates that a default hard-coded password will be used when 
> {{nifi.sensitive.props.key}} is empty. The is no longer the case as of NiFi 
> 1.14.0, so this statement should be removed.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9376) Update Documentation for Encrypted Passwords in Flows

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9376:
---
Fix Version/s: 1.15.1

> Update Documentation for Encrypted Passwords in Flows
> -
>
> Key: NIFI-9376
> URL: https://issues.apache.org/jira/browse/NIFI-9376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.15.0
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Trivial
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The NiFi Administrator's Guide section {{Encrypted Passwords in Flows}} 
> indicates that a default hard-coded password will be used when 
> {{nifi.sensitive.props.key}} is empty. The is no longer the case as of NiFi 
> 1.14.0, so this statement should be removed.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9480) Create Apache NiFi 1.15.1 release

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459352#comment-17459352
 ] 

ASF subversion and git services commented on NIFI-9480:
---

Commit 32a65332caa321b0b9f40da38a29a800b577a32e in nifi's branch 
refs/heads/support/nifi-1.15 from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=32a6533 ]

NIFI-9480 fixing version ref


> Create Apache NiFi 1.15.1 release
> -
>
> Key: NIFI-9480
> URL: https://issues.apache.org/jira/browse/NIFI-9480
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 1.15.1
>
>
> Due to NIFI-9474 which addresses any potential exposure for log4shell we will 
> create an urgent release.
> Very few - if any other bugs - will be included but I will quickly review.
> This will be a 24 hour vote type release.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9417) Flaky BulletinMergerTest.java

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459390#comment-17459390
 ] 

ASF subversion and git services commented on NIFI-9417:
---

Commit 81819495408fd69f370af9ddbb33317d15f79944 in nifi's branch 
refs/heads/support/nifi-1.15 from Yiming Li
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8181949 ]

NIFI-9417: Update BulletinMergerTest.java

Signed-off-by: Matthew Burgess 

This closes #5572


> Flaky BulletinMergerTest.java
> -
>
> Key: NIFI-9417
> URL: https://issues.apache.org/jira/browse/NIFI-9417
> Project: Apache NiFi
>  Issue Type: Bug
> Environment: Maven 3.6.0, Java 1.8.0_292
>Reporter: Yiming Li
>Priority: Trivial
> Fix For: 1.16.0
>
>   Original Estimate: 1h
>  Time Spent: 0.5h
>  Remaining Estimate: 0.5h
>
> SHA: 5aced2b4bc66d2dd85b1507755fff2b1fb8c4a64
> The hashmap does not have a guaranteed iteration order, in the process of 
> BulletinMerger.mergeBulletins() (line-78), the order of [copyOfBulletin1, 
> bulletinEntity1] in the list can be eithter copyOfBulletin1 first or 
> bulletinEntity1 first. The mergeBulletins() will keep the most recent 
> bulletin with the larger time, and because all BulletinEntity are created 
> from the function {*}{*}createBulletin()(line-38) and have the same time 
> property in this test, mergeBulletins() will just keep the first in the list. 
> So sometimes bulletinEntity1 will be returned instead of copyOfBulletin1, 
> which results in the failure (line-80): 
> assertTrue(bulletinEntities.contains(copyOfBulletin1));
> The proposed fix is replacing LinkedHashMap with Hashmap to insure the 
> determinism. After this change, the order will always be bulletinEntity1 
> before the copyOfBulletin1 and the merge results will always contain 
> bulletinEntity1 instead of copyOfBulletin1. So I also change the assertion to 
> assertTrue(bulletinEntities.contains(bulletinEntity1);
> Tentative Pull Request: [https://github.com/LeoYimingLi/nifi/pull/1/files]
>  
> The way to reproduce the flaky test failure:
>  0.Environment setup: Maven 3.6.0 and Java 1.8.0_292
>  1.clone the repo:
>   git clone https://github.com/spring-projects/spring-data-cassandra
>   cd spring-data-cassandra
>   git checkout 5aced2b4bc66d2dd85b1507755fff2b1fb8c4a64
>  2.run with [Nondex tool](https://github.com/TestingResearchIllinois/NonDex) 
> (which explores different behaviors of under-determined APIs and reports test 
> failures)
>   mvn install -pl 
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster 
> -am -DskipTests
>   mvn -pl 
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster 
> edu.illinois:nondex-maven-plugin:1.1.2:debug 
> -Dtest=org.apache.nifi.cluster.manager.BulletinMergerTest#mergeBulletins
>  
>   3.then we can get the test failure:
> --
> [INFO] Results:
> [INFO] 
> [ERROR] Failures: 
> [ERROR]   BulletinMergerTest.mergeBulletins:80 expected [true] but found 
> [false]
> [INFO] 
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> ---
> And BulletinMergerTest.mergeBulletins line-80 is :
> assertTrue(bulletinEntities.contains(copyOfBulletin1));
> in this case, the bulletinEntities did not contain copyOfBulletin1 but 
> bulletinEntity1.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9417) Flaky BulletinMergerTest.java

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9417:
---
Fix Version/s: 1.15.1

> Flaky BulletinMergerTest.java
> -
>
> Key: NIFI-9417
> URL: https://issues.apache.org/jira/browse/NIFI-9417
> Project: Apache NiFi
>  Issue Type: Bug
> Environment: Maven 3.6.0, Java 1.8.0_292
>Reporter: Yiming Li
>Priority: Trivial
> Fix For: 1.16.0, 1.15.1
>
>   Original Estimate: 1h
>  Time Spent: 0.5h
>  Remaining Estimate: 0.5h
>
> SHA: 5aced2b4bc66d2dd85b1507755fff2b1fb8c4a64
> The hashmap does not have a guaranteed iteration order, in the process of 
> BulletinMerger.mergeBulletins() (line-78), the order of [copyOfBulletin1, 
> bulletinEntity1] in the list can be eithter copyOfBulletin1 first or 
> bulletinEntity1 first. The mergeBulletins() will keep the most recent 
> bulletin with the larger time, and because all BulletinEntity are created 
> from the function {*}{*}createBulletin()(line-38) and have the same time 
> property in this test, mergeBulletins() will just keep the first in the list. 
> So sometimes bulletinEntity1 will be returned instead of copyOfBulletin1, 
> which results in the failure (line-80): 
> assertTrue(bulletinEntities.contains(copyOfBulletin1));
> The proposed fix is replacing LinkedHashMap with Hashmap to insure the 
> determinism. After this change, the order will always be bulletinEntity1 
> before the copyOfBulletin1 and the merge results will always contain 
> bulletinEntity1 instead of copyOfBulletin1. So I also change the assertion to 
> assertTrue(bulletinEntities.contains(bulletinEntity1);
> Tentative Pull Request: [https://github.com/LeoYimingLi/nifi/pull/1/files]
>  
> The way to reproduce the flaky test failure:
>  0.Environment setup: Maven 3.6.0 and Java 1.8.0_292
>  1.clone the repo:
>   git clone https://github.com/spring-projects/spring-data-cassandra
>   cd spring-data-cassandra
>   git checkout 5aced2b4bc66d2dd85b1507755fff2b1fb8c4a64
>  2.run with [Nondex tool](https://github.com/TestingResearchIllinois/NonDex) 
> (which explores different behaviors of under-determined APIs and reports test 
> failures)
>   mvn install -pl 
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster 
> -am -DskipTests
>   mvn -pl 
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster 
> edu.illinois:nondex-maven-plugin:1.1.2:debug 
> -Dtest=org.apache.nifi.cluster.manager.BulletinMergerTest#mergeBulletins
>  
>   3.then we can get the test failure:
> --
> [INFO] Results:
> [INFO] 
> [ERROR] Failures: 
> [ERROR]   BulletinMergerTest.mergeBulletins:80 expected [true] but found 
> [false]
> [INFO] 
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> ---
> And BulletinMergerTest.mergeBulletins line-80 is :
> assertTrue(bulletinEntities.contains(copyOfBulletin1));
> in this case, the bulletinEntities did not contain copyOfBulletin1 but 
> bulletinEntity1.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9417) Flaky BulletinMergerTest.java

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9417:
---
Fix Version/s: (was: 1.15.1)

> Flaky BulletinMergerTest.java
> -
>
> Key: NIFI-9417
> URL: https://issues.apache.org/jira/browse/NIFI-9417
> Project: Apache NiFi
>  Issue Type: Bug
> Environment: Maven 3.6.0, Java 1.8.0_292
>Reporter: Yiming Li
>Priority: Trivial
> Fix For: 1.16.0
>
>   Original Estimate: 1h
>  Time Spent: 0.5h
>  Remaining Estimate: 0.5h
>
> SHA: 5aced2b4bc66d2dd85b1507755fff2b1fb8c4a64
> The hashmap does not have a guaranteed iteration order, in the process of 
> BulletinMerger.mergeBulletins() (line-78), the order of [copyOfBulletin1, 
> bulletinEntity1] in the list can be eithter copyOfBulletin1 first or 
> bulletinEntity1 first. The mergeBulletins() will keep the most recent 
> bulletin with the larger time, and because all BulletinEntity are created 
> from the function {*}{*}createBulletin()(line-38) and have the same time 
> property in this test, mergeBulletins() will just keep the first in the list. 
> So sometimes bulletinEntity1 will be returned instead of copyOfBulletin1, 
> which results in the failure (line-80): 
> assertTrue(bulletinEntities.contains(copyOfBulletin1));
> The proposed fix is replacing LinkedHashMap with Hashmap to insure the 
> determinism. After this change, the order will always be bulletinEntity1 
> before the copyOfBulletin1 and the merge results will always contain 
> bulletinEntity1 instead of copyOfBulletin1. So I also change the assertion to 
> assertTrue(bulletinEntities.contains(bulletinEntity1);
> Tentative Pull Request: [https://github.com/LeoYimingLi/nifi/pull/1/files]
>  
> The way to reproduce the flaky test failure:
>  0.Environment setup: Maven 3.6.0 and Java 1.8.0_292
>  1.clone the repo:
>   git clone https://github.com/spring-projects/spring-data-cassandra
>   cd spring-data-cassandra
>   git checkout 5aced2b4bc66d2dd85b1507755fff2b1fb8c4a64
>  2.run with [Nondex tool](https://github.com/TestingResearchIllinois/NonDex) 
> (which explores different behaviors of under-determined APIs and reports test 
> failures)
>   mvn install -pl 
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster 
> -am -DskipTests
>   mvn -pl 
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster 
> edu.illinois:nondex-maven-plugin:1.1.2:debug 
> -Dtest=org.apache.nifi.cluster.manager.BulletinMergerTest#mergeBulletins
>  
>   3.then we can get the test failure:
> --
> [INFO] Results:
> [INFO] 
> [ERROR] Failures: 
> [ERROR]   BulletinMergerTest.mergeBulletins:80 expected [true] but found 
> [false]
> [INFO] 
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> ---
> And BulletinMergerTest.mergeBulletins line-80 is :
> assertTrue(bulletinEntities.contains(copyOfBulletin1));
> in this case, the bulletinEntities did not contain copyOfBulletin1 but 
> bulletinEntity1.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (NIFI-9486) Upgrade to logback 1.2.8

[Bryan Bende (Jira)]

Bryan Bende created NIFI-9486:
-

 Summary: Upgrade to logback 1.2.8
 Key: NIFI-9486
 URL: https://issues.apache.org/jira/browse/NIFI-9486
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Bryan Bende


Logback just released version 1.2.8 to address a vulnerability described in 
[https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Assigned] (NIFI-9486) Upgrade to logback 1.2.8

[Bryan Bende (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende reassigned NIFI-9486:
-

Assignee: Bryan Bende

> Upgrade to logback 1.2.8
> 
>
> Key: NIFI-9486
> URL: https://issues.apache.org/jira/browse/NIFI-9486
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
>
> Logback just released version 1.2.8 to address a vulnerability described in 
> [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9486) Upgrade to logback 1.2.8

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9486:
---
Fix Version/s: 1.16.0
   1.15.1

> Upgrade to logback 1.2.8
> 
>
> Key: NIFI-9486
> URL: https://issues.apache.org/jira/browse/NIFI-9486
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>
> Logback just released version 1.2.8 to address a vulnerability described in 
> [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9486) Upgrade to logback 1.2.8

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9486:
---
Description: 
Logback just released version 1.2.8 to address a vulnerability described in 
[https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.

And another vulnerability arguably from 1.2.7

  was:Logback just released version 1.2.8 to address a vulnerability described 
in [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 
1.2.6.


> Upgrade to logback 1.2.8
> 
>
> Key: NIFI-9486
> URL: https://issues.apache.org/jira/browse/NIFI-9486
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>
> Logback just released version 1.2.8 to address a vulnerability described in 
> [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.
> And another vulnerability arguably from 1.2.7



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [nifi] bbende opened a new pull request #5604: NIFI-9486 Upgrade to logback 1.2.8, remove unnecessary dependencyMana…

[GitBox]

bbende opened a new pull request #5604:
URL: https://github.com/apache/nifi/pull/5604


   …gement entries in minifi and nifi-registry in favor of root pom
   
   
   Thank you for submitting a contribution to Apache NiFi.
   
   Please provide a short description of the PR here:
   
    Description of PR
   
   _Enables X functionality; fixes bug NIFI-._
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [ ] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [ ] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [ ] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [ ] Have you verified that the full build is successful on JDK 8?
   - [ ] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (NIFI-9477) data loss doing Incremental load based on a timestamp from SQL server using QueryDatabaseTableRecord

[Abdullah Aqeeli (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459416#comment-17459416
 ] 

Abdullah Aqeeli commented on NIFI-9477:
---

[~turcsanyip] 

Thank you for the detailed answer, much appreciated! This makes total sense I 
can see this solving our issue.

 

 

 

 

 

> data loss doing Incremental load based on a timestamp from SQL server using 
> QueryDatabaseTableRecord 
> -
>
> Key: NIFI-9477
> URL: https://issues.apache.org/jira/browse/NIFI-9477
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.13.2
> Environment: ubuntu 20 
> 2-node cluster using an embedded zk
>Reporter: Abdullah Aqeeli
>Priority: Critical
>
> hello all,
>  
> We are using Nifi to do an incremental load from SQL Server using 
> QueryDatabaseTableRecord where max-value column is updated_at column with a 
> [Datetime2 
> |https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
>  type. The source tables get updated in milliseconds. We noticed that we are 
> losing few events with each QueryDatabaseTableRecord run. As a work-around we 
> created a custom processor that subtracts few seconds from the max-value in 
> the state fetching overlapping events and then we handle any duplicates later 
> on in the pipeline and it seems to solve the issue. However, this is not 
> optimal. We'd like to know what's causing the skips. Our assumption so far is 
> that timestamp datatype accuracy is causing the skips in the incremental load 
> however we're not sure.  
>  
> Thanks!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (NIFI-9477) data loss doing Incremental load based on a timestamp from SQL server using QueryDatabaseTableRecord

[Abdullah Aqeeli (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abdullah Aqeeli resolved NIFI-9477.
---
Resolution: Resolved

> data loss doing Incremental load based on a timestamp from SQL server using 
> QueryDatabaseTableRecord 
> -
>
> Key: NIFI-9477
> URL: https://issues.apache.org/jira/browse/NIFI-9477
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.13.2
> Environment: ubuntu 20 
> 2-node cluster using an embedded zk
>Reporter: Abdullah Aqeeli
>Priority: Critical
>
> hello all,
>  
> We are using Nifi to do an incremental load from SQL Server using 
> QueryDatabaseTableRecord where max-value column is updated_at column with a 
> [Datetime2 
> |https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime2-transact-sql]data
>  type. The source tables get updated in milliseconds. We noticed that we are 
> losing few events with each QueryDatabaseTableRecord run. As a work-around we 
> created a custom processor that subtracts few seconds from the max-value in 
> the state fetching overlapping events and then we handle any duplicates later 
> on in the pipeline and it seems to solve the issue. However, this is not 
> optimal. We'd like to know what's causing the skips. Our assumption so far is 
> that timestamp datatype accuracy is causing the skips in the incremental load 
> however we're not sure.  
>  
> Thanks!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [nifi] asfgit closed pull request #5604: NIFI-9486 Upgrade to logback 1.2.8, remove unnecessary dependencyMana…

[GitBox]

asfgit closed pull request #5604:
URL: https://github.com/apache/nifi/pull/5604


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (NIFI-9486) Upgrade to logback 1.2.8

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459438#comment-17459438
 ] 

ASF subversion and git services commented on NIFI-9486:
---

Commit 09f254c6113c4c0747201eed67a0e0101424cd12 in nifi's branch 
refs/heads/main from Bryan Bende
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=09f254c ]

NIFI-9486 This closes #5604. Upgrade to logback 1.2.8, remove unnecessary 
dependencyManagement entries in minifi and nifi-registry in favor of root pom
Remove unnecessary use of slf4j-simple


> Upgrade to logback 1.2.8
> 
>
> Key: NIFI-9486
> URL: https://issues.apache.org/jira/browse/NIFI-9486
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Logback just released version 1.2.8 to address a vulnerability described in 
> [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.
> And another vulnerability arguably from 1.2.7



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (NIFI-9486) Upgrade to logback 1.2.8

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-9486.

Resolution: Fixed

> Upgrade to logback 1.2.8
> 
>
> Key: NIFI-9486
> URL: https://issues.apache.org/jira/browse/NIFI-9486
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Logback just released version 1.2.8 to address a vulnerability described in 
> [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.
> And another vulnerability arguably from 1.2.7



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9486) Upgrade to logback 1.2.8

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459439#comment-17459439
 ] 

ASF subversion and git services commented on NIFI-9486:
---

Commit 0cb61f63ccd0eb500d1edfba0ef3f021404f60d7 in nifi's branch 
refs/heads/support/nifi-1.15 from Bryan Bende
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=0cb61f6 ]

NIFI-9486 This closes #5604. Upgrade to logback 1.2.8, remove unnecessary 
dependencyManagement entries in minifi and nifi-registry in favor of root pom
Remove unnecessary use of slf4j-simple


> Upgrade to logback 1.2.8
> 
>
> Key: NIFI-9486
> URL: https://issues.apache.org/jira/browse/NIFI-9486
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Logback just released version 1.2.8 to address a vulnerability described in 
> [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.
> And another vulnerability arguably from 1.2.7



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Reopened] (NIFI-9486) Upgrade to logback 1.2.8

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt reopened NIFI-9486:


> Upgrade to logback 1.2.8
> 
>
> Key: NIFI-9486
> URL: https://issues.apache.org/jira/browse/NIFI-9486
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Logback just released version 1.2.8 to address a vulnerability described in 
> [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.
> And another vulnerability arguably from 1.2.7



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [nifi] joewitt commented on pull request #5604: NIFI-9486 Upgrade to logback 1.2.8, remove unnecessary dependencyMana…

[GitBox]

joewitt commented on pull request #5604:
URL: https://github.com/apache/nifi/pull/5604#issuecomment-993927221


   registry doesnt log properly.  need to validate registry and minifi java.  
nifi works great


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] asfgit closed pull request #5604: NIFI-9486 Upgrade to logback 1.2.8, remove unnecessary dependencyMana…

[GitBox]

asfgit closed pull request #5604:
URL: https://github.com/apache/nifi/pull/5604


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (NIFI-9486) Upgrade to logback 1.2.8

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459466#comment-17459466
 ] 

ASF subversion and git services commented on NIFI-9486:
---

Commit e7449bf0d3fa1612f2019603f09b8bc99d0b26a7 in nifi's branch 
refs/heads/main from Bryan Bende
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=e7449bf ]

NIFI-9486 This closes #5604. Add compile scope to logging deps in 
nifi-registry-assembly
NIFI-9486 Add compile scope to logging deps in minifi-assembly

This commit is 2 of 2.

Signed-off-by: Joe Witt 


> Upgrade to logback 1.2.8
> 
>
> Key: NIFI-9486
> URL: https://issues.apache.org/jira/browse/NIFI-9486
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Logback just released version 1.2.8 to address a vulnerability described in 
> [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.
> And another vulnerability arguably from 1.2.7



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (NIFI-9486) Upgrade to logback 1.2.8

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-9486.

Resolution: Fixed

> Upgrade to logback 1.2.8
> 
>
> Key: NIFI-9486
> URL: https://issues.apache.org/jira/browse/NIFI-9486
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Logback just released version 1.2.8 to address a vulnerability described in 
> [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.
> And another vulnerability arguably from 1.2.7



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9486) Upgrade to logback 1.2.8

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459468#comment-17459468
 ] 

ASF subversion and git services commented on NIFI-9486:
---

Commit d9715d7aff26ebaf55e5277a8b5db33f908df14c in nifi's branch 
refs/heads/support/nifi-1.15 from Bryan Bende
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d9715d7 ]

NIFI-9486 This closes #5604. Add compile scope to logging deps in 
nifi-registry-assembly
NIFI-9486 Add compile scope to logging deps in minifi-assembly

This commit is 2 of 2.

Signed-off-by: Joe Witt 


> Upgrade to logback 1.2.8
> 
>
> Key: NIFI-9486
> URL: https://issues.apache.org/jira/browse/NIFI-9486
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 1.16.0, 1.15.1
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Logback just released version 1.2.8 to address a vulnerability described in 
> [https://jira.qos.ch/browse/LOGBACK-1591] . We are currently on logback 1.2.6.
> And another vulnerability arguably from 1.2.7



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (NIFI-9487) Tests are failing on Android 11

[Ivan Serdyuk (Jira)]

Ivan Serdyuk created NIFI-9487:
--

 Summary: Tests are failing on Android 11
 Key: NIFI-9487
 URL: https://issues.apache.org/jira/browse/NIFI-9487
 Project: Apache NiFi
  Issue Type: Bug
  Components: MiNiFi
Reporter: Ivan Serdyuk


[INFO] Running 
org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeCoordinatorTest
            [ERROR] Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time 
elapsed: 1.009 s <<< FAILURE! - in 
org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeCoordinatorTest
                                     [ERROR] 
org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeCoordinatorTest.testNotifyListeners
  Time elapsed: 0.916 s  <<< FAILURE!                
org.opentest4j.MultipleFailuresError:                 Multiple Failures (2 
failures)                                java.lang.ExceptionInInitializerError: 
                                                         
java.lang.NullPointerException: Cannot invoke 
"org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeCoordinator.close()"
 because "this.coordinatorSpy" is null                                          
                                                 [ERROR] 
org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeCoordinatorTest.testInit
  Time elapsed: 0.004 s  <<< FAILURE!                           
org.opentest4j.MultipleFailuresError:                 Multiple Failures (2 
failures)                                java.lang.NoClassDefFoundError: Could 
not initialize class org.mockito.internal.creation.cglib.ClassImposterizer$3    
                                            java.lang.NullPointerException: 
Cannot invoke 
"org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeCoordinator.close()"
 because "this.coordinatorSpy" is null                                          
                                                 [ERROR] 
org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeCoordinatorTest.testRegisterListener
  Time elapsed: 0.004 s  <<< FAILURE!               
org.opentest4j.MultipleFailuresError:                 Multiple Failures (2 
failures)                                java.lang.NoClassDefFoundError: Could 
not initialize class org.mockito.internal.creation.cglib.ClassImposterizer$3    
                                            java.lang.NullPointerException: 
Cannot invoke 
"org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeCoordinator.close()"
 because "this.coordinatorSpy" is null                                          
                                                 [INFO] Running 
org.apache.nifi.minifi.bootstrap.configuration.differentiators.WholeConfigDifferentiatorTest
 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.211 
s <<< FAILURE! - in 
org.apache.nifi.minifi.bootstrap.configuration.differentiators.WholeConfigDifferentiatorTest
                          [ERROR] 
org.apache.nifi.minifi.bootstrap.configuration.differentiators.WholeConfigDifferentiatorTest
  Time elapsed: 0.211 s  <<< ERROR!                           
java.lang.NoClassDefFoundError: Could not initialize class 
org.mockito.internal.creation.cglib.ClassImposterizer$3                         
                               at 
org.apache.nifi.minifi.bootstrap.configuration.differentiators.WholeConfigDifferentiatorTest.beforeClass(WholeConfigDifferentiatorTest.java:64)
                                                              [INFO] Running 
org.apache.nifi.minifi.bootstrap.configuration.ingestors.FileChangeIngestorTest 
             [ERROR] Tests run: 7, Failures: 7, Errors: 0, Skipped: 0, Time 
elapsed: 0.058 s <<< FAILURE! - in 
org.apache.nifi.minifi.bootstrap.configuration.ingestors.FileChangeIngestorTest 
                                      [ERROR] 
org.apache.nifi.minifi.bootstrap.configuration.ingestors.FileChangeIngestorTest.testInitialize_invalidPollingPeriod
  Time elapsed: 0.004 s  <<< FAILURE!  org.opentest4j.MultipleFailuresError:    
             Multiple Failures (2 failures)                                
java.lang.NoClassDefFoundError: Could not initialize class 
org.mockito.internal.creation.cglib.ClassImposterizer$3                         
                       java.lang.NullPointerException: Cannot invoke 
"org.apache.nifi.minifi.bootstrap.configuration.ingestors.FileChangeIngestor.close()"
 because "this.notifierSpy" is null                                             
                                                   [ERROR] 
org.apache.nifi.minifi.bootstrap.configuration.ingestors.FileChangeIngestorTest.testInitialize_useDefaultPolling
  Time elapsed: 0.004 s  <<< FAILURE!     org.opentest4j.MultipleFailuresError: 
                Multiple Failures (2 failures)                                
java.lang.NoClassDefFoundError: Could not initialize 

[GitHub] [nifi] github-actions[bot] closed pull request #5098: WIP:NIFI-8248 Modify PutAzureDataLakeStorage processor to use temp file

[GitBox]

github-actions[bot] closed pull request #5098:
URL: https://github.com/apache/nifi/pull/5098


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] MikeThomsen commented on pull request #5301: NIFI-8209 Added Neo4J 4.X and 3.X clients by splitting the current co…

[GitBox]

MikeThomsen commented on pull request #5301:
URL: https://github.com/apache/nifi/pull/5301#issuecomment-994205454


   @mattyb149 I'll split off the SSL stuff into a separate tech spike and 
ticket if @exceptionfactory is right about the SSL (probably is AFAICT).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (NIFI-9488) Make bootstrap configurable to not restart NiFi automatically

[Bill Nickless (Jira)]

Bill Nickless created NIFI-9488:
---

 Summary: Make bootstrap configurable to not restart NiFi 
automatically
 Key: NIFI-9488
 URL: https://issues.apache.org/jira/browse/NIFI-9488
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Configuration, Docker
Affects Versions: 1.15.0
 Environment: container runtimes like Docker or containerd, driven by 
an orchestrator like Kubernetes
Reporter: Bill Nickless


When running NiFi under a container runtime and orchestrator like Kubernetes, 
it is sometimes useful for the orchestrator to know when the NiFi process is 
stopping (e.g. killed by the Linux OOMkiller).  When the bootstrap 
automatically restarts NiFi it obscures that fact from the orchestrator.  
Having the bootstrap simply terminate when NiFi terminates would expose the 
fact of the main NiFi process crashing or terminating to the container runtime, 
which would in turn notify the orchestrator.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9480) Create Apache NiFi 1.15.1 release

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459621#comment-17459621
 ] 

ASF subversion and git services commented on NIFI-9480:
---

Commit 2a756372fc7097ece6258c2af47b9a5f26384b02 in nifi's branch 
refs/heads/NIFI-9480-RC1 from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2a75637 ]

NIFI-9480-RC1 prepare release nifi-1.15.1-RC1


> Create Apache NiFi 1.15.1 release
> -
>
> Key: NIFI-9480
> URL: https://issues.apache.org/jira/browse/NIFI-9480
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 1.15.1
>
>
> Due to NIFI-9474 which addresses any potential exposure for log4shell we will 
> create an urgent release.
> Very few - if any other bugs - will be included but I will quickly review.
> This will be a 24 hour vote type release.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (NIFI-9480) Create Apache NiFi 1.15.1 release

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459622#comment-17459622
 ] 

ASF subversion and git services commented on NIFI-9480:
---

Commit b09619d3ea7c73c3f6058de0f5cb567e00e945ce in nifi's branch 
refs/heads/NIFI-9480-RC1 from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=b09619d ]

NIFI-9480-RC1 prepare for next development iteration


> Create Apache NiFi 1.15.1 release
> -
>
> Key: NIFI-9480
> URL: https://issues.apache.org/jira/browse/NIFI-9480
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Joe Witt
>Assignee: Joe Witt
>Priority: Critical
> Fix For: 1.15.1
>
>
> Due to NIFI-9474 which addresses any potential exposure for log4shell we will 
> create an urgent release.
> Very few - if any other bugs - will be included but I will quickly review.
> This will be a 24 hour vote type release.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (NIFI-9194) Upsert for Oracle12+

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt updated NIFI-9194:
---
Issue Type: Improvement  (was: New Feature)

> Upsert for Oracle12+
> 
>
> Key: NIFI-9194
> URL: https://issues.apache.org/jira/browse/NIFI-9194
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
> Environment: All OS
>Reporter: ROBERTO SANTOS
>Priority: Major
>  Labels: features
> Fix For: 1.16.0, 1.15.1
>
>   Original Estimate: 168h
>  Time Spent: 1h
>  Remaining Estimate: 23h 10m
>
> I have implemented Upsert capability for Oracle12+ in PutDatabaseRecord and 
> its Unit tests.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)