[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378095180
 
 

 ##
 File path: extensions/sql/processors/QueryDatabaseTable.cpp
 ##
 @@ -0,0 +1,475 @@
+/**
+ * @file QueryDatabaseTable.cpp
+ * PutSQL class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "QueryDatabaseTable.h"
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+
+#include "io/DataStream.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "Exception.h"
+#include "utils/OsUtils.h"
+#include "data/DatabaseConnectors.h"
+#include "data/JSONSQLWriter.h"
+#include "data/SQLRowsetProcessor.h"
+#include "data/WriteCallback.h"
+#include "data/MaxCollector.h"
+#include "data/Utils.h"
+#include "utils/file/FileUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+const std::string QueryDatabaseTable::ProcessorName("QueryDatabaseTable");
+
+const core::Property QueryDatabaseTable::s_tableName(
+  core::PropertyBuilder::createProperty("Table 
Name")->isRequired(true)->withDescription("The name of the database table to be 
queried.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_columnNames(
+  core::PropertyBuilder::createProperty("Columns to 
Return")->isRequired(false)->withDescription(
+"A comma-separated list of column names to be used in the query. If your 
database requires special treatment of the names (quoting, e.g.), each name 
should include such treatment. "
+"If no column names are supplied, all columns in the specified table will 
be returned. "
+"NOTE: It is important to use consistent column names for a given table 
for incremental fetch to work 
properly.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_maxValueColumnNames(
+  core::PropertyBuilder::createProperty("Maximum-value 
Columns")->isRequired(false)->withDescription(
+"A comma-separated list of column names. The processor will keep track of 
the maximum value for each column that has been returned since the processor 
started running. "
+"Using multiple columns implies an order to the column list, and each 
column's values are expected to increase more slowly than the previous columns' 
values. "
+"Thus, using multiple columns implies a hierarchical structure of columns, 
which is usually used for partitioning tables. "
+"This processor can be used to retrieve only those rows that have been 
added/updated since the last retrieval. "
+"Note that some ODBC types such as bit/boolean are not conducive to 
maintaining maximum value, so columns of these types should not be listed in 
this property, and will result in error(s) during processing. "
+"If no columns are provided, all rows from the table will be considered, 
which could have a performance impact. "
+"NOTE: It is important to use consistent max-value column names for a 
given table for incremental fetch to work properly. "
+"NOTE: Because of a limitation of database access library 'soci', which 
doesn't support milliseconds in it's 'dt_date', "
+"there is a possibility that flowfiles might have duplicated records, if a 
max-value column with 'dt_date' type has value with milliseconds.")->
+supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_whereClause(
+  
core::PropertyBuilder::createProperty("db-fetch-where-clause")->isRequired(false)->withDescription(
+"A custom clause to be added in the WHERE condition when building SQL 
queries.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_sqlQuery(
+  
core::PropertyBuilder::createProperty("db-fetch-sql-query")->isRequired(false)->withDescription(
+"A custom SQL query used to retrieve data. Instead of building a SQL query 
from other properties, this query will be wrapped as a sub-query. "
+"Query must have no ORDER BY 
statement.")->supportsExpressionLanguage(true)->build());
+
+const 

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

szaszm commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r377584537
 
 

 ##
 File path: extensions/sql/data/MaxCollector.h
 ##
 @@ -0,0 +1,172 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+
+#include "SQLRowSubscriber.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MaxCollector: public SQLRowSubscriber {
+  void beginProcessRow() override {}
+
+  void endProcessRow() override {
+if (columnsVerified_) {
+  return;
+}
+
+if (countColumns_ != mapState_.size())
+  throw minifi::Exception(PROCESSOR_EXCEPTION, "MaxCollector: Column(s) '" 
+ maxValueColumnNames_ + "' are not found in the columns of '" + selectQuery_ + 
"' result.");
+
+columnsVerified_ = true;
+  }
+
+  void processColumnName(const std::string& name) override {
+if (columnsVerified_) {
+  return;
+}
+
+if (mapState_.count(name)) {
+  countColumns_++;
+}
+  }
+
+  void processColumn(const std::string& name, const std::string& value)  
override {
+updateMaxValue(name, '\'' + value + '\'');
+  }
+
+  void processColumn(const std::string& name, double value) override {
+updateMaxValue(name, value);
+  }
+
+  void processColumn(const std::string& name, int value) override {
+updateMaxValue(name, value);
+  }
+
+  void processColumn(const std::string& name, long long value) override {
+updateMaxValue(name, value);
+  }
+
+  void processColumn(const std::string& name, unsigned long long value) 
override {
+updateMaxValue(name, value);
+  }
+
+  void processColumn(const std::string& name, const char* value) override {}
+
+  template 
+  struct MaxValue {
+void updateMaxValue(const std::string& name, const T& value) {
+  const auto it = mapColumnNameValue_.find(name);
+  if (it == mapColumnNameValue_.end()) {
+mapColumnNameValue_.insert({ name, value });
+  } else {
+if (value > it->second) {
+  it->second = value;
+}
+  }
+}
+
+std::unordered_map mapColumnNameValue_;
+  };
+
+  template 
+  struct TupleIndexByType {
+constexpr static int index() {
+  using tupleElType = typename std::decay(Tuple()))>::type;
+
+  return TupleIndexByType>::value>::index();
+}
+  };
+
+  template 
+  struct TupleIndexByType {
+constexpr static int index() {
+  return Index;
+}
+  };
+
+  template 
+  struct UpdateMapState {
+UpdateMapState(const Tuple& tpl, std::unordered_map& mapState) {
+  for (auto& el : mapState) {
+const auto& maxVal = std::get(tpl);
+
+const auto it = maxVal.mapColumnNameValue_.find(el.first);
+if (it != maxVal.mapColumnNameValue_.end()) {
+  std::stringstream ss;
+  ss << it->second;
+  el.second = ss.str();
+}
+  }
+
+  UpdateMapState(tpl, mapState);
+}
+  };
+
+  template 
+  struct UpdateMapState {
+UpdateMapState(const Tuple&, std::unordered_map&) {}
+  };
+
+  template 
+  struct MaxValues : public std::tuple...> {
+constexpr static size_t size = sizeof...(Ts);
+  };
+
+ public:
+  MaxCollector(const std::string& selectQuery, const std::string& 
maxValueColumnNames, std::unordered_map& mapState)
+:selectQuery_(selectQuery), maxValueColumnNames_(maxValueColumnNames), 
mapState_(mapState) {
+  }
+
+  template 
+  void updateMaxValue(const std::string& columnName, const T& value) {
+if (mapState_.count(columnName)) {
+  constexpr auto index = TupleIndexByType::index();
+  std::get(maxValues_).updateMaxValue(columnName, value);
+}
+  }
+
+  bool updateMapState() {
+auto mapState = mapState_;
+UpdateMapState(maxValues_, mapState_);
+
+return mapState != mapState_;
 
 Review comment:
   The fact that we copy the whole map here to see if there were changes while 
make the code way more verbose to avoid additional lookups/writes in `MaxValue` 
is contradictory and feels wrong.
   
   Can we either optimize properly for performance if it matters or reduce 
verbosity if it doesn't?

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

szaszm commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378208898
 
 

 ##
 File path: extensions/sql/data/WriteCallback.h
 ##
 @@ -0,0 +1,47 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include "FlowFileRecord.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+
+class WriteCallback : public OutputStreamCallback {
+public:
+  WriteCallback(const char *data, uint64_t size)
+: _data(const_cast(data)),
 
 Review comment:
   If there a reason for taking `const char*` and casting away `const` instead 
of taking mutable `char*`? If so, please add code comment, otherwise please fix 
it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

szaszm commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378213435
 
 

 ##
 File path: win_build_vs.bat
 ##
 @@ -77,4 +81,4 @@ goto :eof
 
 :usage
 @echo "Usage: %0  options"
-exit /B 1
+exit /B 1
 
 Review comment:
   No newline at the end of the file


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378130935
 
 

 ##
 File path: extensions/sql/services/ODBCConnector.h
 ##
 @@ -0,0 +1,127 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once 
+
+#include "core/logging/LoggerConfiguration.h"
+#include "core/controller/ControllerService.h"
+
+#include "DatabaseService.h"
+#include "core/Resource.h"
+#include "data/DatabaseConnectors.h"
+#include 
+#include 
+
+#include 
+#include 
+
+#include 
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+namespace controllers {
+
+class ODBCConnection : public sql::Connection {
+ public:
+  explicit ODBCConnection(const std::string& connectionString)
+: connection_string_(connectionString) {
+  session_ = std::make_unique(getSessionParameters());
+  }
+
+  virtual ~ODBCConnection() {
+  }
+
+  bool connected(std::string& exception) const override {
+try {
+  exception.clear();
+  // According to 
https://stackoverflow.com/questions/3668506/efficient-sql-test-query-or-validation-query-that-will-work-across-all-or-most
 by Rob Hruska, 
+  // 'select 1' works for: H2, MySQL, Microsoft SQL Server, PostgreSQL, 
SQLite. For Orcale 'SELECT 1 FROM DUAL' works.
+  prepareStatement("select 1")->execute();
+  return true;
+} catch (std::exception& e) {
+  exception = e.what();
+  return false;
+}
+  }
+
+  std::unique_ptr prepareStatement(const std::string& query) 
const override {
+return std::make_unique(session_, query);
+  }
+
+  std::unique_ptr getSession() const override {
+return std::make_unique(session_);
+  }
+
+ private:
+   const soci::connection_parameters getSessionParameters() const {
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378131050
 
 

 ##
 File path: extensions/sql/processors/QueryDatabaseTable.cpp
 ##
 @@ -0,0 +1,475 @@
+/**
+ * @file QueryDatabaseTable.cpp
+ * PutSQL class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "QueryDatabaseTable.h"
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+
+#include "io/DataStream.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "Exception.h"
+#include "utils/OsUtils.h"
+#include "data/DatabaseConnectors.h"
+#include "data/JSONSQLWriter.h"
+#include "data/SQLRowsetProcessor.h"
+#include "data/WriteCallback.h"
+#include "data/MaxCollector.h"
+#include "data/Utils.h"
+#include "utils/file/FileUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+const std::string QueryDatabaseTable::ProcessorName("QueryDatabaseTable");
+
+const core::Property QueryDatabaseTable::s_tableName(
+  core::PropertyBuilder::createProperty("Table 
Name")->isRequired(true)->withDescription("The name of the database table to be 
queried.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_columnNames(
+  core::PropertyBuilder::createProperty("Columns to 
Return")->isRequired(false)->withDescription(
+"A comma-separated list of column names to be used in the query. If your 
database requires special treatment of the names (quoting, e.g.), each name 
should include such treatment. "
+"If no column names are supplied, all columns in the specified table will 
be returned. "
+"NOTE: It is important to use consistent column names for a given table 
for incremental fetch to work 
properly.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_maxValueColumnNames(
+  core::PropertyBuilder::createProperty("Maximum-value 
Columns")->isRequired(false)->withDescription(
+"A comma-separated list of column names. The processor will keep track of 
the maximum value for each column that has been returned since the processor 
started running. "
+"Using multiple columns implies an order to the column list, and each 
column's values are expected to increase more slowly than the previous columns' 
values. "
+"Thus, using multiple columns implies a hierarchical structure of columns, 
which is usually used for partitioning tables. "
+"This processor can be used to retrieve only those rows that have been 
added/updated since the last retrieval. "
+"Note that some ODBC types such as bit/boolean are not conducive to 
maintaining maximum value, so columns of these types should not be listed in 
this property, and will result in error(s) during processing. "
+"If no columns are provided, all rows from the table will be considered, 
which could have a performance impact. "
+"NOTE: It is important to use consistent max-value column names for a 
given table for incremental fetch to work properly. "
+"NOTE: Because of a limitation of database access library 'soci', which 
doesn't support milliseconds in it's 'dt_date', "
+"there is a possibility that flowfiles might have duplicated records, if a 
max-value column with 'dt_date' type has value with milliseconds.")->
+supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_whereClause(
+  
core::PropertyBuilder::createProperty("db-fetch-where-clause")->isRequired(false)->withDescription(
+"A custom clause to be added in the WHERE condition when building SQL 
queries.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_sqlQuery(
+  
core::PropertyBuilder::createProperty("db-fetch-sql-query")->isRequired(false)->withDescription(
+"A custom SQL query used to retrieve data. Instead of building a SQL query 
from other properties, this query will be wrapped as a sub-query. "
+"Query must have no ORDER BY 
statement.")->supportsExpressionLanguage(true)->build());
+
+const 

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

szaszm commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378203621
 
 

 ##
 File path: extensions/sql/processors/QueryDatabaseTable.cpp
 ##
 @@ -0,0 +1,475 @@
+/**
+ * @file QueryDatabaseTable.cpp
+ * PutSQL class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "QueryDatabaseTable.h"
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+
+#include "io/DataStream.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "Exception.h"
+#include "utils/OsUtils.h"
+#include "data/DatabaseConnectors.h"
+#include "data/JSONSQLWriter.h"
+#include "data/SQLRowsetProcessor.h"
+#include "data/WriteCallback.h"
+#include "data/MaxCollector.h"
+#include "data/Utils.h"
+#include "utils/file/FileUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+const std::string QueryDatabaseTable::ProcessorName("QueryDatabaseTable");
+
+const core::Property QueryDatabaseTable::s_tableName(
+  core::PropertyBuilder::createProperty("Table 
Name")->isRequired(true)->withDescription("The name of the database table to be 
queried.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_columnNames(
+  core::PropertyBuilder::createProperty("Columns to 
Return")->isRequired(false)->withDescription(
+"A comma-separated list of column names to be used in the query. If your 
database requires special treatment of the names (quoting, e.g.), each name 
should include such treatment. "
+"If no column names are supplied, all columns in the specified table will 
be returned. "
+"NOTE: It is important to use consistent column names for a given table 
for incremental fetch to work 
properly.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_maxValueColumnNames(
+  core::PropertyBuilder::createProperty("Maximum-value 
Columns")->isRequired(false)->withDescription(
+"A comma-separated list of column names. The processor will keep track of 
the maximum value for each column that has been returned since the processor 
started running. "
+"Using multiple columns implies an order to the column list, and each 
column's values are expected to increase more slowly than the previous columns' 
values. "
+"Thus, using multiple columns implies a hierarchical structure of columns, 
which is usually used for partitioning tables. "
+"This processor can be used to retrieve only those rows that have been 
added/updated since the last retrieval. "
+"Note that some ODBC types such as bit/boolean are not conducive to 
maintaining maximum value, so columns of these types should not be listed in 
this property, and will result in error(s) during processing. "
+"If no columns are provided, all rows from the table will be considered, 
which could have a performance impact. "
+"NOTE: It is important to use consistent max-value column names for a 
given table for incremental fetch to work properly. "
+"NOTE: Because of a limitation of database access library 'soci', which 
doesn't support milliseconds in it's 'dt_date', "
+"there is a possibility that flowfiles might have duplicated records, if a 
max-value column with 'dt_date' type has value with milliseconds.")->
+supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_whereClause(
+  
core::PropertyBuilder::createProperty("db-fetch-where-clause")->isRequired(false)->withDescription(
+"A custom clause to be added in the WHERE condition when building SQL 
queries.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_sqlQuery(
+  
core::PropertyBuilder::createProperty("db-fetch-sql-query")->isRequired(false)->withDescription(
+"A custom SQL query used to retrieve data. Instead of building a SQL query 
from other properties, this query will be wrapped as a sub-query. "
+"Query must have no ORDER BY 
statement.")->supportsExpressionLanguage(true)->build());
+
+const 

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378128633
 
 

 ##
 File path: extensions/sql/SQLLoader.h
 ##
 @@ -0,0 +1,81 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef EXTENSION_SQLLOADER_H
+#define EXTENSION_SQLLOADER_H
+
+#include "core/ClassLoader.h"
+#include "processors/ExecuteSQL.h"
+#include "processors/PutSQL.h"
+#include "processors/QueryDatabaseTable.h"
+#include "services/ODBCConnector.h"
+
+class SQLFactory : public core::ObjectFactory {
+ public:
+  SQLFactory() {
+
+  }
+
+  /**
+   * Gets the name of the object.
+   * @return class name of processor
+   */
+  virtual std::string getName() override {
+return "SQLFactory";
+  }
+
+  virtual std::string getClassName() override{
+return "SQLFactory";
+  }
+  /**
+   * Gets the class name for the object
+   * @return class name for the processor.
+   */
+  virtual std::vector getClassNames() override{
+std::vector class_names = {"ExecuteSQL", "PutSQL", 
"QueryDatabaseTable", "ODBCService"};
+return class_names;
+  }
+
+  template 
+  static std::unique_ptr getObjectFactory() {
+return std::unique_ptr(new core::DefautObjectFactory());
+  }
+
+  virtual std::unique_ptr assign(const std::string _name) 
override {
+if (utils::StringUtils::equalsIgnoreCase(class_name, "ExecuteSQL")) {
+  return getObjectFactory();
+}
+if (utils::StringUtils::equalsIgnoreCase(class_name, "PutSQL")) {
+  return getObjectFactory();
+}
+if (utils::StringUtils::equalsIgnoreCase(class_name, 
"QueryDatabaseTable")) {
+  return getObjectFactory();
+}
+if (utils::StringUtils::equalsIgnoreCase(class_name, "ODBCService")) {
+  return getObjectFactory();
+}
+
+return nullptr;
+  }
+
+  static bool added;
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378129066
 
 

 ##
 File path: extensions/sql/data/JSONSQLWriter.h
 ##
 @@ -0,0 +1,65 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include "rapidjson/document.h"
+
+#include "SQLWriter.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class JSONSQLWriter: public SQLWriter {
+ public:
+  JSONSQLWriter(bool pretty);
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378128845
 
 

 ##
 File path: extensions/sql/SQLLoader.h
 ##
 @@ -0,0 +1,81 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef EXTENSION_SQLLOADER_H
+#define EXTENSION_SQLLOADER_H
+
+#include "core/ClassLoader.h"
+#include "processors/ExecuteSQL.h"
+#include "processors/PutSQL.h"
+#include "processors/QueryDatabaseTable.h"
+#include "services/ODBCConnector.h"
+
+class SQLFactory : public core::ObjectFactory {
+ public:
+  SQLFactory() {
+
+  }
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378128964
 
 

 ##
 File path: extensions/sql/data/DatabaseConnectors.h
 ##
 @@ -0,0 +1,106 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef EXTENSIONS_SQL_SERVICES_DATABASECONNECTORS_H_
+#define EXTENSIONS_SQL_SERVICES_DATABASECONNECTORS_H_
+
+#include 
+#include 
+#include 
+#include 
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378128742
 
 

 ##
 File path: extensions/sql/SQLLoader.h
 ##
 @@ -0,0 +1,81 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef EXTENSION_SQLLOADER_H
+#define EXTENSION_SQLLOADER_H
+
+#include "core/ClassLoader.h"
+#include "processors/ExecuteSQL.h"
+#include "processors/PutSQL.h"
+#include "processors/QueryDatabaseTable.h"
+#include "services/ODBCConnector.h"
+
+class SQLFactory : public core::ObjectFactory {
+ public:
+  SQLFactory() {
+
+  }
+
+  /**
+   * Gets the name of the object.
+   * @return class name of processor
+   */
+  virtual std::string getName() override {
+return "SQLFactory";
+  }
+
+  virtual std::string getClassName() override{
+return "SQLFactory";
+  }
+  /**
+   * Gets the class name for the object
+   * @return class name for the processor.
+   */
+  virtual std::vector getClassNames() override{
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #733: MINIFICPP-1155 - Rocksdb repos print invalid stats

[GitBox]

arpadboda closed pull request #733: MINIFICPP-1155 - Rocksdb repos print 
invalid stats
URL: https://github.com/apache/nifi-minifi-cpp/pull/733
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

arpadboda commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378143395
 
 

 ##
 File path: libminifi/src/FlowController.cpp
 ##
 @@ -931,6 +942,35 @@ int16_t 
FlowController::getMetricsNodes(std::vector>&
 manifest_vector, uint16_t metricsClass) {
+std::lock_guard lock(metrics_mutex_);
+(void)metricsClass;
 
 Review comment:
   What's this, why do we need this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

arpadboda commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378143506
 
 

 ##
 File path: libminifi/src/FlowController.cpp
 ##
 @@ -931,6 +942,35 @@ int16_t 
FlowController::getMetricsNodes(std::vector>&
 manifest_vector, uint16_t metricsClass) {
+std::lock_guard lock(metrics_mutex_);
+(void)metricsClass;
+for (auto metric : agent_information_) {
 
 Review comment:
   const auto& to avoid copy? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

arpadboda commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378143151
 
 

 ##
 File path: libminifi/src/FlowController.cpp
 ##
 @@ -931,6 +942,35 @@ int16_t 
FlowController::getMetricsNodes(std::vector>&
 manifest_vector, uint16_t metricsClass) {
 
 Review comment:
   I think this func should be const


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

arpadboda commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378146683
 
 

 ##
 File path: libminifi/src/c2/C2Agent.cpp
 ##
 @@ -315,16 +317,35 @@ void C2Agent::performHeartBeat() {
 payload.addPayload(std::move(deviceInfo));
   }
 
-  if (!root_response_nodes_.empty()) {
-for (auto metric : root_response_nodes_) {
-  C2Payload child_metric_payload(Operation::HEARTBEAT);
-  child_metric_payload.setLabel(metric.first);
-  if (metric.second->isArray()) {
-child_metric_payload.setContainer(true);
-  }
-  serializeMetrics(child_metric_payload, metric.first, 
metric.second->serialize(), metric.second->isArray());
-  payload.addPayload(std::move(child_metric_payload));
+  for (auto metric : root_response_nodes_) {
+C2Payload child_metric_payload(Operation::HEARTBEAT);
+bool isArray{false};
+std::string metricName;
+std::vector metrics;
+std::shared_ptr reporter;
+std::shared_ptr agentInfoManifest;
+
+//Send agent manifest in first heartbeat
+if (!manifest_sent_
+&& (reporter = 
std::dynamic_pointer_cast(update_sink_))
+&& (agentInfoManifest = reporter->getAgentInformationWithManifest())
+&& metric.first == agentInfoManifest->getName()) {
+
+metricName = agentInfoManifest->getName();
+isArray = agentInfoManifest->isArray();
+metrics = std::move(agentInfoManifest->serialize());
+manifest_sent_ = true;
 
 Review comment:
   What if we fail to send?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

arpadboda commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378147512
 
 

 ##
 File path: libminifi/src/c2/C2Agent.cpp
 ##
 @@ -553,6 +577,20 @@ void C2Agent::handle_describe(const C2ContentResponse 
) {
 }
 response.addPayload(std::move(options));
 
+auto reporter = 
std::dynamic_pointer_cast(update_sink_);
+if (reporter != nullptr) {
+std::vector> 
metrics_vec;
+
+C2Payload agentInfo(Operation::ACKNOWLEDGE, resp.ident, false, true);
+agentInfo.setLabel("agentInfo");
+
+reporter->getManifestNodes(metrics_vec, 0);
+for (auto metric : metrics_vec) {
 
 Review comment:
   const auto& pls


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378133280
 
 

 ##
 File path: win_build_vs.bat
 ##
 @@ -22,10 +22,12 @@ if [%1]==[] goto usage
 set builddir=%1
 set skiptests=OFF
 set cmake_build_type=Release
+set build_type=Release
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378188116
 
 

 ##
 File path: extensions/sql/data/MaxCollector.h
 ##
 @@ -0,0 +1,172 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+
+#include "SQLRowSubscriber.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class MaxCollector: public SQLRowSubscriber {
+  void beginProcessRow() override {}
+
+  void endProcessRow() override {
+if (columnsVerified_) {
+  return;
+}
+
+if (countColumns_ != mapState_.size())
+  throw minifi::Exception(PROCESSOR_EXCEPTION, "MaxCollector: Column(s) '" 
+ maxValueColumnNames_ + "' are not found in the columns of '" + selectQuery_ + 
"' result.");
+
+columnsVerified_ = true;
+  }
+
+  void processColumnName(const std::string& name) override {
+if (columnsVerified_) {
+  return;
+}
+
+if (mapState_.count(name)) {
+  countColumns_++;
+}
+  }
+
+  void processColumn(const std::string& name, const std::string& value)  
override {
+updateMaxValue(name, '\'' + value + '\'');
+  }
+
+  void processColumn(const std::string& name, double value) override {
+updateMaxValue(name, value);
+  }
+
+  void processColumn(const std::string& name, int value) override {
+updateMaxValue(name, value);
+  }
+
+  void processColumn(const std::string& name, long long value) override {
+updateMaxValue(name, value);
+  }
+
+  void processColumn(const std::string& name, unsigned long long value) 
override {
+updateMaxValue(name, value);
+  }
+
+  void processColumn(const std::string& name, const char* value) override {}
+
+  template 
+  struct MaxValue {
+void updateMaxValue(const std::string& name, const T& value) {
+  const auto it = mapColumnNameValue_.find(name);
+  if (it == mapColumnNameValue_.end()) {
+mapColumnNameValue_.insert({ name, value });
+  } else {
+if (value > it->second) {
+  it->second = value;
+}
+  }
+}
+
+std::unordered_map mapColumnNameValue_;
+  };
+
+  template 
+  struct TupleIndexByType {
+constexpr static int index() {
+  using tupleElType = typename std::decay(Tuple()))>::type;
+
+  return TupleIndexByType>::value>::index();
+}
+  };
+
+  template 
+  struct TupleIndexByType {
+constexpr static int index() {
+  return Index;
+}
+  };
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378188245
 
 

 ##
 File path: extensions/sql/data/SQLRowsetProcessor.h
 ##
 @@ -0,0 +1,62 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+
+#include 
+
+#include "SQLRowSubscriber.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+
+class SQLRowsetProcessor
+{
+ public:
+  SQLRowsetProcessor(const soci::rowset& rowset, const 
std::vector& rowSubscribers);
+
+  size_t process(size_t max = 0);
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

szaszm commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378203621
 
 

 ##
 File path: extensions/sql/processors/QueryDatabaseTable.cpp
 ##
 @@ -0,0 +1,475 @@
+/**
+ * @file QueryDatabaseTable.cpp
+ * PutSQL class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "QueryDatabaseTable.h"
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+
+#include "io/DataStream.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "Exception.h"
+#include "utils/OsUtils.h"
+#include "data/DatabaseConnectors.h"
+#include "data/JSONSQLWriter.h"
+#include "data/SQLRowsetProcessor.h"
+#include "data/WriteCallback.h"
+#include "data/MaxCollector.h"
+#include "data/Utils.h"
+#include "utils/file/FileUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+const std::string QueryDatabaseTable::ProcessorName("QueryDatabaseTable");
+
+const core::Property QueryDatabaseTable::s_tableName(
+  core::PropertyBuilder::createProperty("Table 
Name")->isRequired(true)->withDescription("The name of the database table to be 
queried.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_columnNames(
+  core::PropertyBuilder::createProperty("Columns to 
Return")->isRequired(false)->withDescription(
+"A comma-separated list of column names to be used in the query. If your 
database requires special treatment of the names (quoting, e.g.), each name 
should include such treatment. "
+"If no column names are supplied, all columns in the specified table will 
be returned. "
+"NOTE: It is important to use consistent column names for a given table 
for incremental fetch to work 
properly.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_maxValueColumnNames(
+  core::PropertyBuilder::createProperty("Maximum-value 
Columns")->isRequired(false)->withDescription(
+"A comma-separated list of column names. The processor will keep track of 
the maximum value for each column that has been returned since the processor 
started running. "
+"Using multiple columns implies an order to the column list, and each 
column's values are expected to increase more slowly than the previous columns' 
values. "
+"Thus, using multiple columns implies a hierarchical structure of columns, 
which is usually used for partitioning tables. "
+"This processor can be used to retrieve only those rows that have been 
added/updated since the last retrieval. "
+"Note that some ODBC types such as bit/boolean are not conducive to 
maintaining maximum value, so columns of these types should not be listed in 
this property, and will result in error(s) during processing. "
+"If no columns are provided, all rows from the table will be considered, 
which could have a performance impact. "
+"NOTE: It is important to use consistent max-value column names for a 
given table for incremental fetch to work properly. "
+"NOTE: Because of a limitation of database access library 'soci', which 
doesn't support milliseconds in it's 'dt_date', "
+"there is a possibility that flowfiles might have duplicated records, if a 
max-value column with 'dt_date' type has value with milliseconds.")->
+supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_whereClause(
+  
core::PropertyBuilder::createProperty("db-fetch-where-clause")->isRequired(false)->withDescription(
+"A custom clause to be added in the WHERE condition when building SQL 
queries.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_sqlQuery(
+  
core::PropertyBuilder::createProperty("db-fetch-sql-query")->isRequired(false)->withDescription(
+"A custom SQL query used to retrieve data. Instead of building a SQL query 
from other properties, this query will be wrapped as a sub-query. "
+"Query must have no ORDER BY 
statement.")->supportsExpressionLanguage(true)->build());
+
+const 

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378188342
 
 

 ##
 File path: extensions/sql/processors/QueryDatabaseTable.cpp
 ##
 @@ -0,0 +1,475 @@
+/**
+ * @file QueryDatabaseTable.cpp
+ * PutSQL class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "QueryDatabaseTable.h"
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+
+#include "io/DataStream.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "Exception.h"
+#include "utils/OsUtils.h"
+#include "data/DatabaseConnectors.h"
+#include "data/JSONSQLWriter.h"
+#include "data/SQLRowsetProcessor.h"
+#include "data/WriteCallback.h"
+#include "data/MaxCollector.h"
+#include "data/Utils.h"
+#include "utils/file/FileUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+const std::string QueryDatabaseTable::ProcessorName("QueryDatabaseTable");
+
+const core::Property QueryDatabaseTable::s_tableName(
+  core::PropertyBuilder::createProperty("Table 
Name")->isRequired(true)->withDescription("The name of the database table to be 
queried.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_columnNames(
+  core::PropertyBuilder::createProperty("Columns to 
Return")->isRequired(false)->withDescription(
+"A comma-separated list of column names to be used in the query. If your 
database requires special treatment of the names (quoting, e.g.), each name 
should include such treatment. "
+"If no column names are supplied, all columns in the specified table will 
be returned. "
+"NOTE: It is important to use consistent column names for a given table 
for incremental fetch to work 
properly.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_maxValueColumnNames(
+  core::PropertyBuilder::createProperty("Maximum-value 
Columns")->isRequired(false)->withDescription(
+"A comma-separated list of column names. The processor will keep track of 
the maximum value for each column that has been returned since the processor 
started running. "
+"Using multiple columns implies an order to the column list, and each 
column's values are expected to increase more slowly than the previous columns' 
values. "
+"Thus, using multiple columns implies a hierarchical structure of columns, 
which is usually used for partitioning tables. "
+"This processor can be used to retrieve only those rows that have been 
added/updated since the last retrieval. "
+"Note that some ODBC types such as bit/boolean are not conducive to 
maintaining maximum value, so columns of these types should not be listed in 
this property, and will result in error(s) during processing. "
+"If no columns are provided, all rows from the table will be considered, 
which could have a performance impact. "
+"NOTE: It is important to use consistent max-value column names for a 
given table for incremental fetch to work properly. "
+"NOTE: Because of a limitation of database access library 'soci', which 
doesn't support milliseconds in it's 'dt_date', "
+"there is a possibility that flowfiles might have duplicated records, if a 
max-value column with 'dt_date' type has value with milliseconds.")->
+supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_whereClause(
+  
core::PropertyBuilder::createProperty("db-fetch-where-clause")->isRequired(false)->withDescription(
+"A custom clause to be added in the WHERE condition when building SQL 
queries.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_sqlQuery(
+  
core::PropertyBuilder::createProperty("db-fetch-sql-query")->isRequired(false)->withDescription(
+"A custom SQL query used to retrieve data. Instead of building a SQL query 
from other properties, this query will be wrapped as a sub-query. "
+"Query must have no ORDER BY 
statement.")->supportsExpressionLanguage(true)->build());
+
+const 

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378188464
 
 

 ##
 File path: extensions/sql/SQLLoader.h
 ##
 @@ -0,0 +1,81 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef EXTENSION_SQLLOADER_H
+#define EXTENSION_SQLLOADER_H
+
+#include "core/ClassLoader.h"
+#include "processors/ExecuteSQL.h"
+#include "processors/PutSQL.h"
+#include "processors/QueryDatabaseTable.h"
+#include "services/ODBCConnector.h"
+
+class SQLFactory : public core::ObjectFactory {
+ public:
+  SQLFactory() {
+
+  }
+
+  /**
+   * Gets the name of the object.
+   * @return class name of processor
+   */
+  virtual std::string getName() override {
+return "SQLFactory";
+  }
+
+  virtual std::string getClassName() override{
+return "SQLFactory";
+  }
+  /**
+   * Gets the class name for the object
+   * @return class name for the processor.
+   */
+  virtual std::vector getClassNames() override{
+std::vector class_names = {"ExecuteSQL", "PutSQL", 
"QueryDatabaseTable", "ODBCService"};
+return class_names;
+  }
+
+  template 
+  static std::unique_ptr getObjectFactory() {
+return std::unique_ptr(new core::DefautObjectFactory());
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378188642
 
 

 ##
 File path: extensions/sql/services/ODBCConnector.h
 ##
 @@ -0,0 +1,127 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once 
+
+#include "core/logging/LoggerConfiguration.h"
+#include "core/controller/ControllerService.h"
+
+#include "DatabaseService.h"
+#include "core/Resource.h"
+#include "data/DatabaseConnectors.h"
+#include 
+#include 
+
+#include 
+#include 
+
+#include 
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace sql {
+namespace controllers {
+
+class ODBCConnection : public sql::Connection {
+ public:
+  explicit ODBCConnection(const std::string& connectionString)
+: connection_string_(connectionString) {
+  session_ = std::make_unique(getSessionParameters());
+  }
+
+  virtual ~ODBCConnection() {
+  }
+
+  bool connected(std::string& exception) const override {
+try {
+  exception.clear();
+  // According to 
https://stackoverflow.com/questions/3668506/efficient-sql-test-query-or-validation-query-that-will-work-across-all-or-most
 by Rob Hruska, 
+  // 'select 1' works for: H2, MySQL, Microsoft SQL Server, PostgreSQL, 
SQLite. For Orcale 'SELECT 1 FROM DUAL' works.
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (MINIFICPP-1159) Fix unstable processor integration tests and clean up code duplication

[Arpad Boda (Jira)]

Arpad Boda created MINIFICPP-1159:
-

 Summary: Fix unstable processor integration tests and clean up 
code duplication
 Key: MINIFICPP-1159
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1159
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Arpad Boda
Assignee: Arpad Boda
 Fix For: 0.8.0


TailFileTest and TailFileCronTest sometimes fail due to misconfiguration of 
LogAttribute proc, which makes it too long to log the incoming flowfiles. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] arpadboda opened a new pull request #736: MINIFICPP-1159 - Fix unstable processor integration tests and clean u…

[GitBox]

arpadboda opened a new pull request #736: MINIFICPP-1159 - Fix unstable 
processor integration tests and clean u…
URL: https://github.com/apache/nifi-minifi-cpp/pull/736
 
 
   …p code duplication
   
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-registry] bbende opened a new pull request #258: NIFIREG-358 Refactoring proxy authorization to be part of Authorizables

[GitBox]

bbende opened a new pull request #258: NIFIREG-358 Refactoring proxy 
authorization to be part of Authorizables
URL: https://github.com/apache/nifi-registry/pull/258
 
 
   - Remove framework authorizer
   - Introduce ProxyChainAuthorizable to wrap all Authorizables
   - Introduce PublicResourceAuthorizable to wrap bucket Authorizable


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (MINIFICPP-1158) Event driven processors can starve each other

[Arpad Boda (Jira)]

Arpad Boda created MINIFICPP-1158:
-

 Summary: Event driven processors can starve each other
 Key: MINIFICPP-1158
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1158
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Arpad Boda
Assignee: Arpad Boda
 Fix For: 0.8.0


The task (that wraps onTrigger call) for event driven processors executes a 
loop and waits 1 sec for work to do.
In case the frequency of the incoming work is higher than that, the task never 
exists, just triggers the processor in an endless loop. 
In case the workload is high and there are multiple event driven processors 
(actually more than the number of threads configured for the flow), some simply 
gets starved. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378223401
 
 

 ##
 File path: libminifi/include/core/state/nodes/AgentInformation.h
 ##
 @@ -623,18 +632,70 @@ class AgentManifest : public DeviceInformation {
 };
 
 /**
- * Purpose and Justification: Prints classes along with their properties for 
the current agent.
+ * This class is used for regular heartbeat without manifest
+ * A light weight heartbeat
  */
-class AgentInformation : public DeviceInformation, public AgentMonitor, public 
AgentIdentifier {
+class AgentInformationWithoutManifest : public DeviceInformation, public 
AgentMonitor, public AgentIdentifier {
+public:
+
+AgentInformationWithoutManifest(std::string name, utils::Identifier & uuid)
+: DeviceInformation(name, uuid) {
 
 Review comment:
   Constructor initializer list should be indented +4 spaces to the right.
   
   Use `std::move(name)` to avoid unnecessary copying of `name`, since 
`DeviceInformation` takes it by-value, too.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378230298
 
 

 ##
 File path: libminifi/src/c2/C2Agent.cpp
 ##
 @@ -315,16 +317,35 @@ void C2Agent::performHeartBeat() {
 payload.addPayload(std::move(deviceInfo));
   }
 
-  if (!root_response_nodes_.empty()) {
-for (auto metric : root_response_nodes_) {
-  C2Payload child_metric_payload(Operation::HEARTBEAT);
-  child_metric_payload.setLabel(metric.first);
-  if (metric.second->isArray()) {
-child_metric_payload.setContainer(true);
-  }
-  serializeMetrics(child_metric_payload, metric.first, 
metric.second->serialize(), metric.second->isArray());
-  payload.addPayload(std::move(child_metric_payload));
+  for (auto metric : root_response_nodes_) {
+C2Payload child_metric_payload(Operation::HEARTBEAT);
+bool isArray{false};
+std::string metricName;
+std::vector metrics;
+std::shared_ptr reporter;
+std::shared_ptr agentInfoManifest;
+
+//Send agent manifest in first heartbeat
+if (!manifest_sent_
+&& (reporter = 
std::dynamic_pointer_cast(update_sink_))
+&& (agentInfoManifest = reporter->getAgentInformationWithManifest())
+&& metric.first == agentInfoManifest->getName()) {
+
+metricName = agentInfoManifest->getName();
+isArray = agentInfoManifest->isArray();
+metrics = std::move(agentInfoManifest->serialize());
+manifest_sent_ = true;
 
 Review comment:
   These lines have one extra level of indentation


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378226346
 
 

 ##
 File path: libminifi/include/core/state/nodes/AgentInformation.h
 ##
 @@ -643,16 +704,7 @@ class AgentInformation : public DeviceInformation, public 
AgentMonitor, public A
   }
 
   std::vector serialize() {
-std::vector serialized;
-
-SerializedResponseNode ident;
-
-ident.name = "identifier";
-ident.value = identifier_;
-
-SerializedResponseNode agentClass;
-agentClass.name = "agentClass";
-agentClass.value = agent_class_;
+std::vector 
serialized(std::move(AgentInformationWithoutManifest::serialize()));
 
 Review comment:
   `std::move` is redundant, since the result of a function call expression is 
already an rvalue. It will prevent copy/move elision, thus making the end 
result even slower than without `std::move`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378222396
 
 

 ##
 File path: libminifi/include/core/state/nodes/AgentInformation.h
 ##
 @@ -623,18 +632,70 @@ class AgentManifest : public DeviceInformation {
 };
 
 /**
- * Purpose and Justification: Prints classes along with their properties for 
the current agent.
+ * This class is used for regular heartbeat without manifest
+ * A light weight heartbeat
  */
-class AgentInformation : public DeviceInformation, public AgentMonitor, public 
AgentIdentifier {
+class AgentInformationWithoutManifest : public DeviceInformation, public 
AgentMonitor, public AgentIdentifier {
+public:
+
+AgentInformationWithoutManifest(std::string name, utils::Identifier & uuid)
+: DeviceInformation(name, uuid) {
+  setArray(false);
+ }
+
+AgentInformationWithoutManifest(const std::string )
 
 Review comment:
   Single parameter constructors should either be marked `explicit` or have a 
comment explaining why implicit conversion from the parameter type is desired.
   
   Same with `AgentInformationWithManifest`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378225328
 
 

 ##
 File path: libminifi/include/core/state/nodes/AgentInformation.h
 ##
 @@ -623,18 +632,70 @@ class AgentManifest : public DeviceInformation {
 };
 
 /**
- * Purpose and Justification: Prints classes along with their properties for 
the current agent.
+ * This class is used for regular heartbeat without manifest
+ * A light weight heartbeat
  */
-class AgentInformation : public DeviceInformation, public AgentMonitor, public 
AgentIdentifier {
+class AgentInformationWithoutManifest : public DeviceInformation, public 
AgentMonitor, public AgentIdentifier {
 
 Review comment:
   Public inheritance means is-a relationship. By looking at the names, this 
class fits none of its base classes.
   Use composition (preferred) or private inheritance for 
implemented-in-terms-of relationship.
   
   Couldn't this class become just a 
`serialize_agent_information_without_manifest(...)` free function? OOP is a 
toolbox, not an obligation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

szaszm commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378230978
 
 

 ##
 File path: libminifi/include/core/state/nodes/AgentInformation.h
 ##
 @@ -643,16 +704,7 @@ class AgentInformation : public DeviceInformation, public 
AgentMonitor, public A
   }
 
   std::vector serialize() {
 
 Review comment:
   Can the `serialize()` member functions be marked as `const`? They don't seem 
to mutate the object.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (NIFI-7109) Unit tests should be able to determine if item validator was called

[Matt Burgess (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-7109:
---
Status: Patch Available  (was: Open)

> Unit tests should be able to determine if item validator was called
> ---
>
> Key: NIFI-7109
> URL: https://issues.apache.org/jira/browse/NIFI-7109
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> In cases where there is a 'per item' validator provided for a list type 
> validator, we cannot tell if an item is invalid because it failed the outer 
> validator or the passed validator.
> We should be able to do this, with a mock, delighting validator that counts 
> the calls.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] arpadboda opened a new pull request #735: MINIFICPP-1158 - Event driven processors can starve each other

[GitBox]

arpadboda opened a new pull request #735: MINIFICPP-1158 - Event driven 
processors can starve each other
URL: https://github.com/apache/nifi-minifi-cpp/pull/735
 
 
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (NIFI-7109) Unit tests should be able to determine if item validator was called

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035494#comment-17035494
 ] 

ASF subversion and git services commented on NIFI-7109:
---

Commit 6e8f10c4f66f0167b9a43c91c549f1d36208d9cc in nifi's branch 
refs/heads/master from Otto Fowler
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6e8f10c ]

NIFI-7109 Unit tests should be able to determine if item validator was called
- Create a mock Validator to track count of calls to validate().
We cannot use Mockito for this, because it can't mock all the
StandardValidators

refactor based on review comments

fix naming in comments

moved to main based on review

Signed-off-by: Matthew Burgess 

This closes #4043


> Unit tests should be able to determine if item validator was called
> ---
>
> Key: NIFI-7109
> URL: https://issues.apache.org/jira/browse/NIFI-7109
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> In cases where there is a 'per item' validator provided for a list type 
> validator, we cannot tell if an item is invalid because it failed the outer 
> validator or the passed validator.
> We should be able to do this, with a mock, delighting validator that counts 
> the calls.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] asfgit closed pull request #4043: NIFI-7109 Unit tests should be able to determine if item validator wa…

[GitBox]

asfgit closed pull request #4043: NIFI-7109 Unit tests should be able to 
determine if item validator wa…
URL: https://github.com/apache/nifi/pull/4043
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] mattyb149 commented on issue #4043: NIFI-7109 Unit tests should be able to determine if item validator wa…

[GitBox]

mattyb149 commented on issue #4043: NIFI-7109 Unit tests should be able to 
determine if item validator wa…
URL: https://github.com/apache/nifi/pull/4043#issuecomment-585289520
 
 
   +1 LGTM (Travis failures are unrelated), ran contrib-check, everything looks 
good. Thanks for the improvement! Merging to master


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (NIFI-7109) Unit tests should be able to determine if item validator was called

[Matt Burgess (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-7109:
---
Fix Version/s: 1.12.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Unit tests should be able to determine if item validator was called
> ---
>
> Key: NIFI-7109
> URL: https://issues.apache.org/jira/browse/NIFI-7109
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
> Fix For: 1.12.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> In cases where there is a 'per item' validator provided for a list type 
> validator, we cannot tell if an item is invalid because it failed the outer 
> validator or the passed validator.
> We should be able to do this, with a mock, delighting validator that counts 
> the calls.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-registry] bbende commented on a change in pull request #258: NIFIREG-358 Refactoring proxy authorization to be part of Authorizables

[GitBox]

bbende commented on a change in pull request #258: NIFIREG-358 Refactoring 
proxy authorization to be part of Authorizables
URL: https://github.com/apache/nifi-registry/pull/258#discussion_r378295031
 
 

 ##
 File path: 
nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/resource/Authorizable.java
 ##
 @@ -211,18 +205,10 @@ default void authorize(Authorizer authorizer, 
RequestAction action, NiFiUser use
 userContext = null;
 }
 
-final List proxyChain = new ArrayList<>();
-NiFiUser proxyUser = user.getChain();
-while (proxyUser  != null) {
-proxyChain.add(proxyUser.getIdentity());
-proxyUser = proxyUser.getChain();
-}
 
 Review comment:
   Do we consider this a breaking change?
   
   In 0.5.0 we added the proxyIdentities field and build methods to 
AuthorizationRequest. I left those there, but stopped populating them since we 
don't want to use them anymore. So it won't break compilation if some was using 
those, but they won't have values anymore. I can put this back if we want to 
leave it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] phrocker commented on issue #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

phrocker commented on issue #734: MINIFICPP-1157 Implement lightweight C2 
heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#issuecomment-585261190
 
 
   @msharee9 taking a look, thanks. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] phrocker edited a comment on issue #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

phrocker edited a comment on issue #734: MINIFICPP-1157 Implement lightweight 
C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#issuecomment-585281331
 
 
   @msharee9 Could you have a different endpoint that is used for a full 
posting and then another endpoint for a lightweight HB. The period for those 
could be different or the full posting could be sent only once ( with success 
validation ).
   
   Another alternative is to have the C2 server request a full posting and then 
the registration occur in that case and otherwise always be a light weight HB. 
That may alleviate some of the complexity per @arpadboda 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] phrocker commented on issue #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

phrocker commented on issue #734: MINIFICPP-1157 Implement lightweight C2 
heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#issuecomment-585281331
 
 
   @msharee9 Why not have a different endpoint that is used for a full posting 
and then another endpoint for a lightweight HB. The period for those could be 
different or the full posting could be sent only once ( with success validation 
).
   
   Another alternative is to have the C2 server request a full posting and then 
the registration occur in that case and otherwise always be a light weight HB. 
That may alleviate some of the complexity per @arpadboda 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378458847
 
 

 ##
 File path: libminifi/include/core/state/nodes/AgentInformation.h
 ##
 @@ -623,18 +632,70 @@ class AgentManifest : public DeviceInformation {
 };
 
 /**
- * Purpose and Justification: Prints classes along with their properties for 
the current agent.
+ * This class is used for regular heartbeat without manifest
+ * A light weight heartbeat
  */
-class AgentInformation : public DeviceInformation, public AgentMonitor, public 
AgentIdentifier {
+class AgentInformationWithoutManifest : public DeviceInformation, public 
AgentMonitor, public AgentIdentifier {
+public:
+
+AgentInformationWithoutManifest(std::string name, utils::Identifier & uuid)
+: DeviceInformation(name, uuid) {
+  setArray(false);
+ }
+
+AgentInformationWithoutManifest(const std::string )
 
 Review comment:
   fair point. Thanks for catching it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378443443
 
 

 ##
 File path: libminifi/src/c2/C2Agent.cpp
 ##
 @@ -553,6 +577,20 @@ void C2Agent::handle_describe(const C2ContentResponse 
) {
 }
 response.addPayload(std::move(options));
 
+auto reporter = 
std::dynamic_pointer_cast(update_sink_);
+if (reporter != nullptr) {
+std::vector> 
metrics_vec;
+
+C2Payload agentInfo(Operation::ACKNOWLEDGE, resp.ident, false, true);
+agentInfo.setLabel("agentInfo");
+
+reporter->getManifestNodes(metrics_vec, 0);
+for (auto metric : metrics_vec) {
 
 Review comment:
   fair enough


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 commented on issue #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 commented on issue #734: MINIFICPP-1157 Implement lightweight C2 
heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#issuecomment-585369988
 
 
   > @msharee9 Could you have a different endpoint that is used for a full 
posting and then another endpoint for a lightweight HB. The period for those 
could be different or the full posting could be sent only once ( with success 
validation ).
   > 
   > Another alternative is to have the C2 server request a full posting and 
then the registration occur in that case and otherwise always be a light weight 
HB. That may alleviate some of the complexity per @arpadboda
   
   The full posting is done only once in this code change and that is during C2 
initialization and after that it is lightweight both going to the same endpoint 
though. I don't think it would work best from C2 server standpoint if we want 
to send to two different endpoints.
   
   Sending the heartbeat during initialization is an optimization (thanks to 
@szaszm suggestion), otherwise, the original idea was to only send manifest 
when C2 server requests for it in DESCRIBE and always send lightweight 
heartbeats always.
   
   In case the first heavyweight heartbeat is lost due to some reason, the C2 
server will request manifest anyhow upon first successful reception of 
lightweight heartbeat.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 commented on issue #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 commented on issue #734: MINIFICPP-1157 Implement lightweight C2 
heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#issuecomment-585365334
 
 
   > This seems to be a bit complicated and introduce some code duplication.
   > 
   > Why can't we simply extend agentinformation (for eg. with a ctor 
parameter) to include/exclude manifest?
   
   To give some background first:
   The existing minifi property nifi.c2.root.classes has "AgentInformation" as 
one of the root classes. And this class has all the information including 
(agent status, agent id, class and the manifest).
   For backward compatibility we want to keep that class as it is.
   
   Going forward, you have a choice to specify if you need a heartbeat with or 
without manifest.
   AgentInformation = (With manifest as it was before this change)
   AgentInformationWithoutManifest = (Everything as before except manifest)
   
   Because of the above reason (configurable class name), we cannot introduce a 
parameter to include/exclude manifest unless we modify the class loader code 
   ```
   template 
   std::shared_ptr ClassLoader::instantiate(const std::string _name, 
const std::string );
   ```
   with some if/else and I don't think that is a good choice.
   
   I agree that there is some code duplication and I will make some improvement 
there.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 edited a comment on issue #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 edited a comment on issue #734: MINIFICPP-1157 Implement lightweight 
C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#issuecomment-585369988
 
 
   > @msharee9 Could you have a different endpoint that is used for a full 
posting and then another endpoint for a lightweight HB. The period for those 
could be different or the full posting could be sent only once ( with success 
validation ).
   > 
   > Another alternative is to have the C2 server request a full posting and 
then the registration occur in that case and otherwise always be a light weight 
HB. That may alleviate some of the complexity per @arpadboda
   
   The full posting is done only once in this code change and that is during C2 
initialization and after that it is lightweight both going to the same endpoint 
though. I don't think it would work best from C2 server standpoint if we want 
to send to two different endpoints.
   
   Sending the full heartbeat with manifest during initialization is an 
optimization (thanks to @szaszm suggestion), otherwise, the original idea was 
to only send manifest when C2 server requests for it in DESCRIBE and always 
send lightweight heartbeats always.
   
   In case the first heavyweight heartbeat is lost due to some reason, the C2 
server will request manifest anyhow upon first successful reception of 
lightweight heartbeat.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378462016
 
 

 ##
 File path: libminifi/src/c2/C2Agent.cpp
 ##
 @@ -315,16 +317,35 @@ void C2Agent::performHeartBeat() {
 payload.addPayload(std::move(deviceInfo));
   }
 
-  if (!root_response_nodes_.empty()) {
-for (auto metric : root_response_nodes_) {
-  C2Payload child_metric_payload(Operation::HEARTBEAT);
-  child_metric_payload.setLabel(metric.first);
-  if (metric.second->isArray()) {
-child_metric_payload.setContainer(true);
-  }
-  serializeMetrics(child_metric_payload, metric.first, 
metric.second->serialize(), metric.second->isArray());
-  payload.addPayload(std::move(child_metric_payload));
+  for (auto metric : root_response_nodes_) {
+C2Payload child_metric_payload(Operation::HEARTBEAT);
+bool isArray{false};
+std::string metricName;
+std::vector metrics;
+std::shared_ptr reporter;
+std::shared_ptr agentInfoManifest;
+
+//Send agent manifest in first heartbeat
+if (!manifest_sent_
+&& (reporter = 
std::dynamic_pointer_cast(update_sink_))
+&& (agentInfoManifest = reporter->getAgentInformationWithManifest())
+&& metric.first == agentInfoManifest->getName()) {
+
+metricName = agentInfoManifest->getName();
+isArray = agentInfoManifest->isArray();
+metrics = std::move(agentInfoManifest->serialize());
+manifest_sent_ = true;
 
 Review comment:
   If C2 server does not receive the manifest in the first heartbeat, it 
requests manifest in DESCRIBE command when it next receives lightweight 
heartbeat. Sending the manifest during the first heartbeat is a slight 
optimization.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378462225
 
 

 ##
 File path: libminifi/src/FlowController.cpp
 ##
 @@ -931,6 +942,35 @@ int16_t 
FlowController::getMetricsNodes(std::vector>&
 manifest_vector, uint16_t metricsClass) {
+std::lock_guard lock(metrics_mutex_);
+(void)metricsClass;
+for (auto metric : agent_information_) {
 
 Review comment:
   ok


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378441353
 
 

 ##
 File path: libminifi/include/core/state/nodes/AgentInformation.h
 ##
 @@ -623,18 +632,70 @@ class AgentManifest : public DeviceInformation {
 };
 
 /**
- * Purpose and Justification: Prints classes along with their properties for 
the current agent.
+ * This class is used for regular heartbeat without manifest
+ * A light weight heartbeat
  */
-class AgentInformation : public DeviceInformation, public AgentMonitor, public 
AgentIdentifier {
+class AgentInformationWithoutManifest : public DeviceInformation, public 
AgentMonitor, public AgentIdentifier {
 
 Review comment:
   Did not understand why it does not fit is-a relationship to any of its base 
classes. Can you give example?
   
   No this cannot become a free function unless we change the way we register 
and load classes. 
   This instantiation of this class and other class (AgentInformation) depends 
on configuration of root node classes in "nifi.c2.root.classes" property that 
are loaded during C2 initialization in FlowController. I want to keep it that 
way and be consistent with the existing code base.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] mattyb149 commented on issue #4026: NIFI-7087: Use FlowManager.findAllConnections() when available

[GitBox]

mattyb149 commented on issue #4026: NIFI-7087: Use 
FlowManager.findAllConnections() when available
URL: https://github.com/apache/nifi/pull/4026#issuecomment-585427244
 
 
   Sorry for the force-push override of the commit, but with the merge conflict 
it was just easier. All changes should be the same except now there are no 
changes to StandardProcessGroup


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378529132
 
 

 ##
 File path: libminifi/include/core/state/nodes/AgentInformation.h
 ##
 @@ -643,16 +704,7 @@ class AgentInformation : public DeviceInformation, public 
AgentMonitor, public A
   }
 
   std::vector serialize() {
-std::vector serialized;
-
-SerializedResponseNode ident;
-
-ident.name = "identifier";
-ident.value = identifier_;
-
-SerializedResponseNode agentClass;
-agentClass.name = "agentClass";
-agentClass.value = agent_class_;
+std::vector 
serialized(std::move(AgentInformationWithoutManifest::serialize()));
 
 Review comment:
   good point. Will remove it


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (NIFI-7139) Add Error Message To Flow File Attribute for Record Processors

[Shawn Weeks (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shawn Weeks updated NIFI-7139:
--
Description: Add an attribute to flow files that fail during record 
processing containing the error message. This is primarily useful when that 
error message may be used in downstream processing and is dependent on the 
Record Reader or Writer raising a meaningful exception. This would be similar 
to what the Execute SQL Processors does and would add an attribute 
record.error.message to the flow file on the failure relationship of 
AbstractRecordProcessor.  (was: Add an attribute to flow files that fail during 
record processing contain the error message. This is primarily useful when that 
error message may effect downstream processing and is dependent on the Record 
Reader or Writer raising a meaningful exception. This would be similar to what 
the Execute SQL Processors do and would add an attribute record.error.message 
to the flow file on the failure relationship of AbstractRecordProcessor.)

> Add Error Message To Flow File Attribute for Record Processors
> --
>
> Key: NIFI-7139
> URL: https://issues.apache.org/jira/browse/NIFI-7139
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Shawn Weeks
>Assignee: Shawn Weeks
>Priority: Trivial
>
> Add an attribute to flow files that fail during record processing containing 
> the error message. This is primarily useful when that error message may be 
> used in downstream processing and is dependent on the Record Reader or Writer 
> raising a meaningful exception. This would be similar to what the Execute SQL 
> Processors does and would add an attribute record.error.message to the flow 
> file on the failure relationship of AbstractRecordProcessor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7123) onPropertyModified() is called on nifi start up even when properties have never been modified

[Nissim Shiman (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035736#comment-17035736
 ] 

Nissim Shiman commented on NIFI-7123:
-

More details as to where this is biting us:

Currently on startup  controller services are getting NullPointerException when 
this code is run through at nifi startup.

These are custom controller services, whose properties still have the default 
values, that are overriding/implementing the onPropertyModified()  

The issue is, that on startup, when onPropertyModified() is run, the oldValue 
[1] is null, even when using default Values

Left as is, the only option is to do a notNull check for the oldValue in each 
of the onPropertyModified()'s to avoid the exception


[1] 
https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/components/ConfigurableComponent.java#L68

> onPropertyModified() is called on nifi start up even when properties have 
> never been modified
> -
>
> Key: NIFI-7123
> URL: https://issues.apache.org/jira/browse/NIFI-7123
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Nissim Shiman
>Assignee: Nissim Shiman
>Priority: Major
>
> Processors and Controller Services inherit the onPropertyModified() method 
> from ConfigurableComponent. java [1]
> This method is called when nifi starts for all processors and controller 
> services, even for properties that are set to their defaults (i.e. have never 
> been modified).
> [1]  
> https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/components/ConfigurableComponent.java#L68



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378537631
 
 

 ##
 File path: libminifi/src/FlowController.cpp
 ##
 @@ -931,6 +942,35 @@ int16_t 
FlowController::getMetricsNodes(std::vector>&
 manifest_vector, uint16_t metricsClass) {
+std::lock_guard lock(metrics_mutex_);
+(void)metricsClass;
 
 Review comment:
   This is a cache of manifest object (not the actual manifest data). Needed in 
response to DESCRIBE manifest.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] markap14 commented on a change in pull request #4026: NIFI-7087: Use FlowManager.findAllConnections() when available

[GitBox]

markap14 commented on a change in pull request #4026: NIFI-7087: Use 
FlowManager.findAllConnections() when available
URL: https://github.com/apache/nifi/pull/4026#discussion_r378495549
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java
 ##
 @@ -4894,7 +4894,7 @@ public void verifyCanUpdate(final VersionedFlowSnapshot 
updatedFlow, final boole
 // any connection that does exist in the proposed flow. This 
results in us having a Map whose values are those Connections
 // that were removed. We can then check for any connections 
that have data in them. If any Connection is to be removed but
 // has data, then we should throw an IllegalStateException.
-findAllConnections().forEach(conn -> 
removedConnectionByVersionedId.put(conn.getVersionedComponentId().orElse(conn.getIdentifier()),
 conn));
+flowManager.findAllConnections().forEach(conn -> 
removedConnectionByVersionedId.put(conn.getVersionedComponentId().orElse(conn.getIdentifier()),
 conn));
 
 Review comment:
   I think this call is actually changing the logic. `findAllConnections()` 
here is going to return all Connections in this ProcessGroup and any 
children/descendant groups. This changes it from "all connections here and 
below" to "all connections in the canvas," which I don't think is what we want.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] markap14 commented on a change in pull request #4026: NIFI-7087: Use FlowManager.findAllConnections() when available

[GitBox]

markap14 commented on a change in pull request #4026: NIFI-7087: Use 
FlowManager.findAllConnections() when available
URL: https://github.com/apache/nifi/pull/4026#discussion_r378495890
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java
 ##
 @@ -5018,9 +5018,11 @@ public void verifyCanUpdate(final VersionedFlowSnapshot 
updatedFlow, final boole
 final Map proposedConnections = new 
HashMap<>();
 findAllConnections(updatedFlow.getFlowContents(), 
proposedConnections);
 
-findAllConnections().stream()
-.filter(conn -> conn.getVersionedComponentId().isPresent())
-.forEach(conn -> 
proposedConnections.remove(conn.getVersionedComponentId().get()));
+for (Connection conn : flowManager.findAllConnections()) {
 
 Review comment:
   Same comment as above. Here, I think we need to use the existing 
`findAllConnections()` method. We only want to change to 
`flowManager.findAllConnections()` when calling it on the root process group.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (NIFI-7139) Add Error Message To Flow File Attribute for Record Processors

[Shawn Weeks (Jira)]

Shawn Weeks created NIFI-7139:
-

 Summary: Add Error Message To Flow File Attribute for Record 
Processors
 Key: NIFI-7139
 URL: https://issues.apache.org/jira/browse/NIFI-7139
 Project: Apache NiFi
  Issue Type: New Feature
Reporter: Shawn Weeks
Assignee: Shawn Weeks


Add an attribute to flow files that fail during record processing contain the 
error message. This is primarily useful when that error message may effect 
downstream processing and is dependent on the Record Reader or Writer raising a 
meaningful exception. This would be similar to what the Execute SQL Processors 
do and would add an attribute record.error.message to the flow file on the 
failure relationship of AbstractRecordProcessor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement lightweight C2 heartbeat.

[GitBox]

msharee9 commented on a change in pull request #734: MINIFICPP-1157 Implement 
lightweight C2 heartbeat.
URL: https://github.com/apache/nifi-minifi-cpp/pull/734#discussion_r378537631
 
 

 ##
 File path: libminifi/src/FlowController.cpp
 ##
 @@ -931,6 +942,35 @@ int16_t 
FlowController::getMetricsNodes(std::vector>&
 manifest_vector, uint16_t metricsClass) {
+std::lock_guard lock(metrics_mutex_);
+(void)metricsClass;
 
 Review comment:
   Are you talking about the "metricsClass" variable there?
   If you are asking about the function, it returns manifest information needed 
while responding to DESCRIBE manifest.
   
   metricsClass has been removed from this function in the updated code.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378603638
 
 

 ##
 File path: extensions/sql/processors/QueryDatabaseTable.cpp
 ##
 @@ -0,0 +1,475 @@
+/**
+ * @file QueryDatabaseTable.cpp
+ * PutSQL class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "QueryDatabaseTable.h"
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+
+#include "io/DataStream.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "Exception.h"
+#include "utils/OsUtils.h"
+#include "data/DatabaseConnectors.h"
+#include "data/JSONSQLWriter.h"
+#include "data/SQLRowsetProcessor.h"
+#include "data/WriteCallback.h"
+#include "data/MaxCollector.h"
+#include "data/Utils.h"
+#include "utils/file/FileUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+const std::string QueryDatabaseTable::ProcessorName("QueryDatabaseTable");
+
+const core::Property QueryDatabaseTable::s_tableName(
+  core::PropertyBuilder::createProperty("Table 
Name")->isRequired(true)->withDescription("The name of the database table to be 
queried.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_columnNames(
+  core::PropertyBuilder::createProperty("Columns to 
Return")->isRequired(false)->withDescription(
+"A comma-separated list of column names to be used in the query. If your 
database requires special treatment of the names (quoting, e.g.), each name 
should include such treatment. "
+"If no column names are supplied, all columns in the specified table will 
be returned. "
+"NOTE: It is important to use consistent column names for a given table 
for incremental fetch to work 
properly.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_maxValueColumnNames(
+  core::PropertyBuilder::createProperty("Maximum-value 
Columns")->isRequired(false)->withDescription(
+"A comma-separated list of column names. The processor will keep track of 
the maximum value for each column that has been returned since the processor 
started running. "
+"Using multiple columns implies an order to the column list, and each 
column's values are expected to increase more slowly than the previous columns' 
values. "
+"Thus, using multiple columns implies a hierarchical structure of columns, 
which is usually used for partitioning tables. "
+"This processor can be used to retrieve only those rows that have been 
added/updated since the last retrieval. "
+"Note that some ODBC types such as bit/boolean are not conducive to 
maintaining maximum value, so columns of these types should not be listed in 
this property, and will result in error(s) during processing. "
+"If no columns are provided, all rows from the table will be considered, 
which could have a performance impact. "
+"NOTE: It is important to use consistent max-value column names for a 
given table for incremental fetch to work properly. "
+"NOTE: Because of a limitation of database access library 'soci', which 
doesn't support milliseconds in it's 'dt_date', "
+"there is a possibility that flowfiles might have duplicated records, if a 
max-value column with 'dt_date' type has value with milliseconds.")->
+supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_whereClause(
+  
core::PropertyBuilder::createProperty("db-fetch-where-clause")->isRequired(false)->withDescription(
+"A custom clause to be added in the WHERE condition when building SQL 
queries.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_sqlQuery(
+  
core::PropertyBuilder::createProperty("db-fetch-sql-query")->isRequired(false)->withDescription(
+"A custom SQL query used to retrieve data. Instead of building a SQL query 
from other properties, this query will be wrapped as a sub-query. "
+"Query must have no ORDER BY 
statement.")->supportsExpressionLanguage(true)->build());
+
+const 

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378603638
 
 

 ##
 File path: extensions/sql/processors/QueryDatabaseTable.cpp
 ##
 @@ -0,0 +1,475 @@
+/**
+ * @file QueryDatabaseTable.cpp
+ * PutSQL class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "QueryDatabaseTable.h"
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+
+#include "io/DataStream.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "Exception.h"
+#include "utils/OsUtils.h"
+#include "data/DatabaseConnectors.h"
+#include "data/JSONSQLWriter.h"
+#include "data/SQLRowsetProcessor.h"
+#include "data/WriteCallback.h"
+#include "data/MaxCollector.h"
+#include "data/Utils.h"
+#include "utils/file/FileUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+const std::string QueryDatabaseTable::ProcessorName("QueryDatabaseTable");
+
+const core::Property QueryDatabaseTable::s_tableName(
+  core::PropertyBuilder::createProperty("Table 
Name")->isRequired(true)->withDescription("The name of the database table to be 
queried.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_columnNames(
+  core::PropertyBuilder::createProperty("Columns to 
Return")->isRequired(false)->withDescription(
+"A comma-separated list of column names to be used in the query. If your 
database requires special treatment of the names (quoting, e.g.), each name 
should include such treatment. "
+"If no column names are supplied, all columns in the specified table will 
be returned. "
+"NOTE: It is important to use consistent column names for a given table 
for incremental fetch to work 
properly.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_maxValueColumnNames(
+  core::PropertyBuilder::createProperty("Maximum-value 
Columns")->isRequired(false)->withDescription(
+"A comma-separated list of column names. The processor will keep track of 
the maximum value for each column that has been returned since the processor 
started running. "
+"Using multiple columns implies an order to the column list, and each 
column's values are expected to increase more slowly than the previous columns' 
values. "
+"Thus, using multiple columns implies a hierarchical structure of columns, 
which is usually used for partitioning tables. "
+"This processor can be used to retrieve only those rows that have been 
added/updated since the last retrieval. "
+"Note that some ODBC types such as bit/boolean are not conducive to 
maintaining maximum value, so columns of these types should not be listed in 
this property, and will result in error(s) during processing. "
+"If no columns are provided, all rows from the table will be considered, 
which could have a performance impact. "
+"NOTE: It is important to use consistent max-value column names for a 
given table for incremental fetch to work properly. "
+"NOTE: Because of a limitation of database access library 'soci', which 
doesn't support milliseconds in it's 'dt_date', "
+"there is a possibility that flowfiles might have duplicated records, if a 
max-value column with 'dt_date' type has value with milliseconds.")->
+supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_whereClause(
+  
core::PropertyBuilder::createProperty("db-fetch-where-clause")->isRequired(false)->withDescription(
+"A custom clause to be added in the WHERE condition when building SQL 
queries.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_sqlQuery(
+  
core::PropertyBuilder::createProperty("db-fetch-sql-query")->isRequired(false)->withDescription(
+"A custom SQL query used to retrieve data. Instead of building a SQL query 
from other properties, this query will be wrapped as a sub-query. "
+"Query must have no ORDER BY 
statement.")->supportsExpressionLanguage(true)->build());
+
+const 

[GitHub] [nifi-minifi-cpp] am-c-p-p commented on a change in pull request #732: MINIFICPP-1013

[GitBox]

am-c-p-p commented on a change in pull request #732: MINIFICPP-1013
URL: https://github.com/apache/nifi-minifi-cpp/pull/732#discussion_r378603638
 
 

 ##
 File path: extensions/sql/processors/QueryDatabaseTable.cpp
 ##
 @@ -0,0 +1,475 @@
+/**
+ * @file QueryDatabaseTable.cpp
+ * PutSQL class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "QueryDatabaseTable.h"
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+
+#include "io/DataStream.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "Exception.h"
+#include "utils/OsUtils.h"
+#include "data/DatabaseConnectors.h"
+#include "data/JSONSQLWriter.h"
+#include "data/SQLRowsetProcessor.h"
+#include "data/WriteCallback.h"
+#include "data/MaxCollector.h"
+#include "data/Utils.h"
+#include "utils/file/FileUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+const std::string QueryDatabaseTable::ProcessorName("QueryDatabaseTable");
+
+const core::Property QueryDatabaseTable::s_tableName(
+  core::PropertyBuilder::createProperty("Table 
Name")->isRequired(true)->withDescription("The name of the database table to be 
queried.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_columnNames(
+  core::PropertyBuilder::createProperty("Columns to 
Return")->isRequired(false)->withDescription(
+"A comma-separated list of column names to be used in the query. If your 
database requires special treatment of the names (quoting, e.g.), each name 
should include such treatment. "
+"If no column names are supplied, all columns in the specified table will 
be returned. "
+"NOTE: It is important to use consistent column names for a given table 
for incremental fetch to work 
properly.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_maxValueColumnNames(
+  core::PropertyBuilder::createProperty("Maximum-value 
Columns")->isRequired(false)->withDescription(
+"A comma-separated list of column names. The processor will keep track of 
the maximum value for each column that has been returned since the processor 
started running. "
+"Using multiple columns implies an order to the column list, and each 
column's values are expected to increase more slowly than the previous columns' 
values. "
+"Thus, using multiple columns implies a hierarchical structure of columns, 
which is usually used for partitioning tables. "
+"This processor can be used to retrieve only those rows that have been 
added/updated since the last retrieval. "
+"Note that some ODBC types such as bit/boolean are not conducive to 
maintaining maximum value, so columns of these types should not be listed in 
this property, and will result in error(s) during processing. "
+"If no columns are provided, all rows from the table will be considered, 
which could have a performance impact. "
+"NOTE: It is important to use consistent max-value column names for a 
given table for incremental fetch to work properly. "
+"NOTE: Because of a limitation of database access library 'soci', which 
doesn't support milliseconds in it's 'dt_date', "
+"there is a possibility that flowfiles might have duplicated records, if a 
max-value column with 'dt_date' type has value with milliseconds.")->
+supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_whereClause(
+  
core::PropertyBuilder::createProperty("db-fetch-where-clause")->isRequired(false)->withDescription(
+"A custom clause to be added in the WHERE condition when building SQL 
queries.")->supportsExpressionLanguage(true)->build());
+
+const core::Property QueryDatabaseTable::s_sqlQuery(
+  
core::PropertyBuilder::createProperty("db-fetch-sql-query")->isRequired(false)->withDescription(
+"A custom SQL query used to retrieve data. Instead of building a SQL query 
from other properties, this query will be wrapped as a sub-query. "
+"Query must have no ORDER BY 
statement.")->supportsExpressionLanguage(true)->build());
+
+const 

[jira] [Created] (NIFI-7140) PutSql support database transaction rollback

[ZhangCheng (Jira)]

ZhangCheng created NIFI-7140:


 Summary: PutSql support database transaction rollback
 Key: NIFI-7140
 URL: https://issues.apache.org/jira/browse/NIFI-7140
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.11.1
Reporter: ZhangCheng






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-7140) PutSql support database transaction rollback when is false

[ZhangCheng (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhangCheng updated NIFI-7140:
-
Description: 
For PutSQL processor,if the 'Support Fragmented Transactions' is true, the 
processor will will process all FlowFiles with that fragment.identifier as a 
single transaction;
In actuality,it works. But when some sql of the transaction failed and 
 is 
 false , the database transaction will not roll back.
Sometimes,we need the  database transaction rollback and do not want  the 
flowfile rollback, we need that the failed database  transaction route  to 
REL_FAILURE.

If he is true and  is 
 false , I think it should still support the capability  of database 
transaction rollback, for example it should add a property (like )  which can indicate that whether the 
processor support  database transaction rollback when the 'Support Fragmented 
Transactions' is true ,and  only manage the flowfile 
rollback.

  was:
For PutSQL processor,if the 'Support Fragmented Transactions' is true, the 
processor will will process all FlowFiles with that fragment.identifier as a 
single transaction;
In actuality,it works. But when some sql of the transaction failed,the 
transaction will not roll back.
If he 'Support Fragmented Transactions' is true,I think it should be support 
transaction rollback,  Or it should add a property (eg. 'Support Fragmented 
Transactions RollBack' )  which can indicate that whether the processor support 
 database transaction rollback when the 'Support Fragmented Transactions' is 
true.

Summary: PutSql support database transaction rollback when 
is false  (was: PutSql support database transaction 
rollback)

> PutSql support database transaction rollback when is 
> false
> 
>
> Key: NIFI-7140
> URL: https://issues.apache.org/jira/browse/NIFI-7140
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.11.1
>Reporter: ZhangCheng
>Priority: Major
>
> For PutSQL processor,if the 'Support Fragmented Transactions' is true, the 
> processor will will process all FlowFiles with that fragment.identifier as a 
> single transaction;
> In actuality,it works. But when some sql of the transaction failed and 
>  is 
>  false , the database transaction will not roll back.
> Sometimes,we need the  database transaction rollback and do not want  the 
> flowfile rollback, we need that the failed database  transaction route  to 
> REL_FAILURE.
> If he is true and  is 
>  false , I think it should still support the capability  of database 
> transaction rollback, for example it should add a property (like  Fragmented Transactions RollBack>)  which can indicate that whether the 
> processor support  database transaction rollback when the 'Support Fragmented 
> Transactions' is true ,and  only manage the flowfile 
> rollback.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-7140) PutSql support database transaction rollback when is false

[ZhangCheng (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhangCheng updated NIFI-7140:
-
Description: 
For PutSQL processor,if the 'Support Fragmented Transactions' is true, the 
processor will process all FlowFiles with that fragment.identifier as a single 
transaction;
In actuality,it works. 
But when some sql of the transaction failed and  is false 
, the database transaction will not roll back.
Sometimes,we need the  database transaction rollback and do not want  the 
flowfile rollback, we need that the failed database  transaction route  to 
REL_FAILURE.
If the is true and  is 
false , I think it should still support the capability  of database transaction 
rollback, for example :it should add a property (like )  which can indicate that whether the processor support  
database transaction rollback when the 'Support Fragmented Transactions' is 
true ,and  only manage the flowfiles rollback.

  was:
For PutSQL processor,if the 'Support Fragmented Transactions' is true, the 
processor will will process all FlowFiles with that fragment.identifier as a 
single transaction;
In actuality,it works. But when some sql of the transaction failed and 
 is 
 false , the database transaction will not roll back.
Sometimes,we need the  database transaction rollback and do not want  the 
flowfile rollback, we need that the failed database  transaction route  to 
REL_FAILURE.

If he is true and  is 
 false , I think it should still support the capability  of database 
transaction rollback, for example it should add a property (like )  which can indicate that whether the 
processor support  database transaction rollback when the 'Support Fragmented 
Transactions' is true ,and  only manage the flowfile 
rollback.


> PutSql support database transaction rollback when is 
> false
> 
>
> Key: NIFI-7140
> URL: https://issues.apache.org/jira/browse/NIFI-7140
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.11.1
>Reporter: ZhangCheng
>Priority: Major
>
> For PutSQL processor,if the 'Support Fragmented Transactions' is true, the 
> processor will process all FlowFiles with that fragment.identifier as a 
> single transaction;
> In actuality,it works. 
> But when some sql of the transaction failed and  is 
> false , the database transaction will not roll back.
> Sometimes,we need the  database transaction rollback and do not want  the 
> flowfile rollback, we need that the failed database  transaction route  to 
> REL_FAILURE.
> If the is true and  is 
> false , I think it should still support the capability  of database 
> transaction rollback, for example :it should add a property (like  Fragmented Transactions RollBack>)  which can indicate that whether the 
> processor support  database transaction rollback when the 'Support Fragmented 
> Transactions' is true ,and  only manage the flowfiles 
> rollback.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-7140) PutSql support database transaction rollback

[ZhangCheng (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhangCheng updated NIFI-7140:
-
Description: 
For PutSQL processor,if the 'Support Fragmented Transactions' is true, the 
processor will will process all FlowFiles with that fragment.identifier as a 
single transaction;
In actuality,it works. But when some sql of the transaction failed,the 
transaction will not roll back.
If he 'Support Fragmented Transactions' is true,I think it should be support 
transaction rollback,  Or it should add a property (eg. 'Support Fragmented 
Transactions RollBack' )  which can indicate that whether the processor support 
 database transaction rollback when the 'Support Fragmented Transactions' is 
true.

> PutSql support database transaction rollback
> 
>
> Key: NIFI-7140
> URL: https://issues.apache.org/jira/browse/NIFI-7140
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.11.1
>Reporter: ZhangCheng
>Priority: Major
>
> For PutSQL processor,if the 'Support Fragmented Transactions' is true, the 
> processor will will process all FlowFiles with that fragment.identifier as a 
> single transaction;
> In actuality,it works. But when some sql of the transaction failed,the 
> transaction will not roll back.
> If he 'Support Fragmented Transactions' is true,I think it should be support 
> transaction rollback,  Or it should add a property (eg. 'Support Fragmented 
> Transactions RollBack' )  which can indicate that whether the processor 
> support  database transaction rollback when the 'Support Fragmented 
> Transactions' is true.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)