[GitHub] nifi pull request #3113: NIFI-5724 making the database connection autocommit...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3113#discussion_r232166089
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutSQL.java
---
@@ -134,6 +138,14 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
+static final PropertyDescriptor AUTO_COMMIT = new
PropertyDescriptor.Builder()
+.name("database-session-autocommit")
+.displayName("Database session autocommit value")
--- End diff --
I'm going to change this displayName to "Database Session AutoCommit" to
align capitalization with other properties.
---
[GitHub] nifi pull request #3113: NIFI-5724 making the database connection autocommit...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3113#discussion_r232159628
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutSQL.java
---
@@ -189,13 +201,40 @@
properties.add(CONNECTION_POOL);
properties.add(SQL_STATEMENT);
properties.add(SUPPORT_TRANSACTIONS);
+properties.add(AUTO_COMMIT);
properties.add(TRANSACTION_TIMEOUT);
properties.add(BATCH_SIZE);
properties.add(OBTAIN_GENERATED_KEYS);
properties.add(RollbackOnFailure.ROLLBACK_ON_FAILURE);
return properties;
}
+@Override
+protected final Collection
customValidate(ValidationContext context) {
+final Collection results = new ArrayList<>();
+final String support_transactions =
context.getProperty(SUPPORT_TRANSACTIONS).getValue();
+final String rollback_on_failure =
context.getProperty(RollbackOnFailure.ROLLBACK_ON_FAILURE).getValue();
+final String auto_commit =
context.getProperty(AUTO_COMMIT).getValue();
+
+if(auto_commit.equalsIgnoreCase("true")) {
+if(support_transactions.equalsIgnoreCase("true")) {
+results.add(new ValidationResult.Builder()
+
.subject(SUPPORT_TRANSACTIONS.getDisplayName())
+.explanation(format("'%s' cannot be set to
'true' when '%s' is also set to 'true'."
++ "Transactions for batch updates
cannot be supported when auto commit is set to 'true'",
SUPPORT_TRANSACTIONS.getDisplayName(), AUTO_COMMIT.getDisplayName()))
+.build());
+}
+if(rollback_on_failure.equalsIgnoreCase("true")) {
+results.add(new ValidationResult.Builder()
+
.subject(RollbackOnFailure.ROLLBACK_ON_FAILURE.getDisplayName())
+.explanation(format("'%s' cannot be set to 'true'
when '%s' is also set to 'true'."
++ "Transaction rollbacks for batch updates
cannot be supported when auto commit is set to 'true'",
RollbackOnFailure.ROLLBACK_ON_FAILURE.getDisplayName(),
AUTO_COMMIT.getDisplayName()))
--- End diff --
This line causes following check-style error. Please build with
`-Pcontrib-check` to confirm styles locally.
```
src/main/java/org/apache/nifi/processors/standard/PutSQL.java:[231] (sizes)
LineLength: Line is longer than 200 characters (found 217).
```
---
[jira] [Commented] (NIFI-5724) Make the autocommit value in the PutSQL processor configurable
[
https://issues.apache.org/jira/browse/NIFI-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681047#comment-16681047
]
ASF GitHub Bot commented on NIFI-5724:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3113#discussion_r232159628
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutSQL.java
---
@@ -189,13 +201,40 @@
properties.add(CONNECTION_POOL);
properties.add(SQL_STATEMENT);
properties.add(SUPPORT_TRANSACTIONS);
+properties.add(AUTO_COMMIT);
properties.add(TRANSACTION_TIMEOUT);
properties.add(BATCH_SIZE);
properties.add(OBTAIN_GENERATED_KEYS);
properties.add(RollbackOnFailure.ROLLBACK_ON_FAILURE);
return properties;
}
+@Override
+protected final Collection
customValidate(ValidationContext context) {
+final Collection results = new ArrayList<>();
+final String support_transactions =
context.getProperty(SUPPORT_TRANSACTIONS).getValue();
+final String rollback_on_failure =
context.getProperty(RollbackOnFailure.ROLLBACK_ON_FAILURE).getValue();
+final String auto_commit =
context.getProperty(AUTO_COMMIT).getValue();
+
+if(auto_commit.equalsIgnoreCase("true")) {
+if(support_transactions.equalsIgnoreCase("true")) {
+results.add(new ValidationResult.Builder()
+
.subject(SUPPORT_TRANSACTIONS.getDisplayName())
+.explanation(format("'%s' cannot be set to
'true' when '%s' is also set to 'true'."
++ "Transactions for batch updates
cannot be supported when auto commit is set to 'true'",
SUPPORT_TRANSACTIONS.getDisplayName(), AUTO_COMMIT.getDisplayName()))
+.build());
+}
+if(rollback_on_failure.equalsIgnoreCase("true")) {
+results.add(new ValidationResult.Builder()
+
.subject(RollbackOnFailure.ROLLBACK_ON_FAILURE.getDisplayName())
+.explanation(format("'%s' cannot be set to 'true'
when '%s' is also set to 'true'."
++ "Transaction rollbacks for batch updates
cannot be supported when auto commit is set to 'true'",
RollbackOnFailure.ROLLBACK_ON_FAILURE.getDisplayName(),
AUTO_COMMIT.getDisplayName()))
--- End diff --
This line causes following check-style error. Please build with
`-Pcontrib-check` to confirm styles locally.
```
src/main/java/org/apache/nifi/processors/standard/PutSQL.java:[231] (sizes)
LineLength: Line is longer than 200 characters (found 217).
```
> Make the autocommit value in the PutSQL processor configurable
> --
>
> Key: NIFI-5724
> URL: https://issues.apache.org/jira/browse/NIFI-5724
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: vish uma
>Priority: Minor
>
> The PutSQL processor currently always sets the autocommit value on the
> database session to false before the SQL statement is run and resets it back
> to the original value after.
> i am not sure if the autocommit value is hardcoded to false for a reason, if
> it is, please let me know.
> This is causing an issue with the snowflake DB where abruptly disconnected
> sessions do not release the locks they have taken.
> i would like to make this autocommit value configurable. I can submit a patch
> for this if there is no objections.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5724) Make the autocommit value in the PutSQL processor configurable
[
https://issues.apache.org/jira/browse/NIFI-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681048#comment-16681048
]
ASF GitHub Bot commented on NIFI-5724:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3113#discussion_r232166089
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutSQL.java
---
@@ -134,6 +138,14 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
+static final PropertyDescriptor AUTO_COMMIT = new
PropertyDescriptor.Builder()
+.name("database-session-autocommit")
+.displayName("Database session autocommit value")
--- End diff --
I'm going to change this displayName to "Database Session AutoCommit" to
align capitalization with other properties.
> Make the autocommit value in the PutSQL processor configurable
> --
>
> Key: NIFI-5724
> URL: https://issues.apache.org/jira/browse/NIFI-5724
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: vish uma
>Priority: Minor
>
> The PutSQL processor currently always sets the autocommit value on the
> database session to false before the SQL statement is run and resets it back
> to the original value after.
> i am not sure if the autocommit value is hardcoded to false for a reason, if
> it is, please let me know.
> This is causing an issue with the snowflake DB where abruptly disconnected
> sessions do not release the locks they have taken.
> i would like to make this autocommit value configurable. I can submit a patch
> for this if there is no objections.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5724) Make the autocommit value in the PutSQL processor configurable
[ https://issues.apache.org/jira/browse/NIFI-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681051#comment-16681051 ] ASF subversion and git services commented on NIFI-5724: --- Commit 63f55d05b4f3f83b9e9f2206f4129ae7a4ade569 in nifi's branch refs/heads/master from Vish Uma [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=63f55d0 ] NIFI-5724 make database connection autocommit configurable making the database session autocommit value a configurable property adding custom validation to PutSQL processor so as to disallow 'supports transaction' and 'rollback on failure' to be true when the autocommit value has been set to true fixing some style issues to conform to standards This closes #3113. Signed-off-by: Koji Kawamura > Make the autocommit value in the PutSQL processor configurable > -- > > Key: NIFI-5724 > URL: https://issues.apache.org/jira/browse/NIFI-5724 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: vish uma >Priority: Minor > > The PutSQL processor currently always sets the autocommit value on the > database session to false before the SQL statement is run and resets it back > to the original value after. > i am not sure if the autocommit value is hardcoded to false for a reason, if > it is, please let me know. > This is causing an issue with the snowflake DB where abruptly disconnected > sessions do not release the locks they have taken. > i would like to make this autocommit value configurable. I can submit a patch > for this if there is no objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3113: NIFI-5724 making the database connection autocommit...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3113 ---
[jira] [Commented] (NIFI-5724) Make the autocommit value in the PutSQL processor configurable
[ https://issues.apache.org/jira/browse/NIFI-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681052#comment-16681052 ] ASF GitHub Bot commented on NIFI-5724: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3113 > Make the autocommit value in the PutSQL processor configurable > -- > > Key: NIFI-5724 > URL: https://issues.apache.org/jira/browse/NIFI-5724 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: vish uma >Priority: Minor > > The PutSQL processor currently always sets the autocommit value on the > database session to false before the SQL statement is run and resets it back > to the original value after. > i am not sure if the autocommit value is hardcoded to false for a reason, if > it is, please let me know. > This is causing an issue with the snowflake DB where abruptly disconnected > sessions do not release the locks they have taken. > i would like to make this autocommit value configurable. I can submit a patch > for this if there is no objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5724) Make the autocommit value in the PutSQL processor configurable
[ https://issues.apache.org/jira/browse/NIFI-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura reassigned NIFI-5724: --- Assignee: vish uma > Make the autocommit value in the PutSQL processor configurable > -- > > Key: NIFI-5724 > URL: https://issues.apache.org/jira/browse/NIFI-5724 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: vish uma >Assignee: vish uma >Priority: Minor > > The PutSQL processor currently always sets the autocommit value on the > database session to false before the SQL statement is run and resets it back > to the original value after. > i am not sure if the autocommit value is hardcoded to false for a reason, if > it is, please let me know. > This is causing an issue with the snowflake DB where abruptly disconnected > sessions do not release the locks they have taken. > i would like to make this autocommit value configurable. I can submit a patch > for this if there is no objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5724) Make the autocommit value in the PutSQL processor configurable
[ https://issues.apache.org/jira/browse/NIFI-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura resolved NIFI-5724. - Resolution: Fixed Fix Version/s: 1.9.0 [~vishcious] I've added you to NiFi JIRA as a contributor. You can assign yourself to JIRAs now. Thanks again! > Make the autocommit value in the PutSQL processor configurable > -- > > Key: NIFI-5724 > URL: https://issues.apache.org/jira/browse/NIFI-5724 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: vish uma >Assignee: vish uma >Priority: Minor > Fix For: 1.9.0 > > > The PutSQL processor currently always sets the autocommit value on the > database session to false before the SQL statement is run and resets it back > to the original value after. > i am not sure if the autocommit value is hardcoded to false for a reason, if > it is, please let me know. > This is causing an issue with the snowflake DB where abruptly disconnected > sessions do not release the locks they have taken. > i would like to make this autocommit value configurable. I can submit a patch > for this if there is no objections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4026) SiteToSite Partitioning
[ https://issues.apache.org/jira/browse/NIFI-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681065#comment-16681065 ] Koji Kawamura commented on NIFI-4026: - [~pvillard] Just for checking. Do we have anything to add to the load-balancing capability we released with 1.8.0? Or can this JIRA be closed now? I believe the cluster topology change scenario is handled by node offloading and consistent hashing. > SiteToSite Partitioning > --- > > Key: NIFI-4026 > URL: https://issues.apache.org/jira/browse/NIFI-4026 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Pierre Villard >Priority: Major > > To answer some use cases and to always provide more flexibility to the > Site-to-Site mechanism it would be interesting to introduce a S2S > Partitioning Key. > The idea would be to add a parameter in the S2S configuration to compute the > destination node based on the attribute of a flow file. The user would set > the attribute to read from the incoming flow files and a hashing function > would be applied on this attribute value to get a number between 1 and N (N > being the number of nodes on the remote cluster) to select the destination > node. > It could even be possible to let the user code a custom hashing function in a > scripting language. > This approach would potentially force the “batching” to 1, or it could be > necessary to create bins to batch together flow files that are supposed to go > to the same node. > Obviously, it comes the question regarding how to handle cluster scale > up/down. However, I believe this is an edge case and should not be blocking > this feature. > Some of the use cases could be: > - better load balancing of the flow files when using the List/Fetch pattern > (example: ListHDFS/FetchHDFS and load balance based on the size of the remote > file to fetch) > - being able to keep on the same node the data related to the same element > (based on business requirements, example: all the logs from a given host > should be merged in the same file and not have one file per NiFi node) > - give the possibility to send all the data back to the primary node (we > could say that if the hash function returns 0, then the destination node is > the primary node) in case this is required for specific operations. This > would avoid the need to do the full workflow on the primary node only when > some parts can be load balanced. > I also think that this work would be a good foundation for the "node > labeling" stuff that has been discussed on the mailing lists in the past. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4026) SiteToSite Partitioning
[ https://issues.apache.org/jira/browse/NIFI-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681087#comment-16681087 ] Pierre Villard commented on NIFI-4026: -- Yes, you're absolutely right, closing it. > SiteToSite Partitioning > --- > > Key: NIFI-4026 > URL: https://issues.apache.org/jira/browse/NIFI-4026 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Pierre Villard >Priority: Major > > To answer some use cases and to always provide more flexibility to the > Site-to-Site mechanism it would be interesting to introduce a S2S > Partitioning Key. > The idea would be to add a parameter in the S2S configuration to compute the > destination node based on the attribute of a flow file. The user would set > the attribute to read from the incoming flow files and a hashing function > would be applied on this attribute value to get a number between 1 and N (N > being the number of nodes on the remote cluster) to select the destination > node. > It could even be possible to let the user code a custom hashing function in a > scripting language. > This approach would potentially force the “batching” to 1, or it could be > necessary to create bins to batch together flow files that are supposed to go > to the same node. > Obviously, it comes the question regarding how to handle cluster scale > up/down. However, I believe this is an edge case and should not be blocking > this feature. > Some of the use cases could be: > - better load balancing of the flow files when using the List/Fetch pattern > (example: ListHDFS/FetchHDFS and load balance based on the size of the remote > file to fetch) > - being able to keep on the same node the data related to the same element > (based on business requirements, example: all the logs from a given host > should be merged in the same file and not have one file per NiFi node) > - give the possibility to send all the data back to the primary node (we > could say that if the hash function returns 0, then the destination node is > the primary node) in case this is required for specific operations. This > would avoid the need to do the full workflow on the primary node only when > some parts can be load balanced. > I also think that this work would be a good foundation for the "node > labeling" stuff that has been discussed on the mailing lists in the past. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-4026) SiteToSite Partitioning
[ https://issues.apache.org/jira/browse/NIFI-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard resolved NIFI-4026. -- Resolution: Duplicate Fix Version/s: 1.8.0 > SiteToSite Partitioning > --- > > Key: NIFI-4026 > URL: https://issues.apache.org/jira/browse/NIFI-4026 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Pierre Villard >Priority: Major > Fix For: 1.8.0 > > > To answer some use cases and to always provide more flexibility to the > Site-to-Site mechanism it would be interesting to introduce a S2S > Partitioning Key. > The idea would be to add a parameter in the S2S configuration to compute the > destination node based on the attribute of a flow file. The user would set > the attribute to read from the incoming flow files and a hashing function > would be applied on this attribute value to get a number between 1 and N (N > being the number of nodes on the remote cluster) to select the destination > node. > It could even be possible to let the user code a custom hashing function in a > scripting language. > This approach would potentially force the “batching” to 1, or it could be > necessary to create bins to batch together flow files that are supposed to go > to the same node. > Obviously, it comes the question regarding how to handle cluster scale > up/down. However, I believe this is an edge case and should not be blocking > this feature. > Some of the use cases could be: > - better load balancing of the flow files when using the List/Fetch pattern > (example: ListHDFS/FetchHDFS and load balance based on the size of the remote > file to fetch) > - being able to keep on the same node the data related to the same element > (based on business requirements, example: all the logs from a given host > should be merged in the same file and not have one file per NiFi node) > - give the possibility to send all the data back to the primary node (we > could say that if the hash function returns 0, then the destination node is > the primary node) in case this is required for specific operations. This > would avoid the need to do the full workflow on the primary node only when > some parts can be load balanced. > I also think that this work would be a good foundation for the "node > labeling" stuff that has been discussed on the mailing lists in the past. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #399: NIFI-1833 - Adding Azure Storage processors
Github user joaocc commented on the issue: https://github.com/apache/nifi/pull/399 @simonellistonball Hi. Was the work on Azure Storage Table dropped? I tried to find it in the current (1.8) version but only Storage Blob seems to be there. Thx ---
[jira] [Commented] (NIFI-1833) Add support for Azure Blob Storage and Table Storage
[ https://issues.apache.org/jira/browse/NIFI-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1668#comment-1668 ] ASF GitHub Bot commented on NIFI-1833: -- Github user joaocc commented on the issue: https://github.com/apache/nifi/pull/399 @simonellistonball Hi. Was the work on Azure Storage Table dropped? I tried to find it in the current (1.8) version but only Storage Blob seems to be there. Thx > Add support for Azure Blob Storage and Table Storage > > > Key: NIFI-1833 > URL: https://issues.apache.org/jira/browse/NIFI-1833 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 0.6.1 >Reporter: Simon Elliston Ball >Assignee: Jeff Storck >Priority: Major > Fix For: 1.2.0 > > > It would be useful to have an Azure equivalent of the current S3 capability. > Azure also provides a Table storage mechanism, providing simple key value > storage. Since the Azure SDKs are Apache Licensed, this should be reasonably > straightforward. A first cut is available as an addition to the existing > azure bundle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5806) SetSNMP processor HOST:PORT not changing
[ https://issues.apache.org/jira/browse/NIFI-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard reassigned NIFI-5806: Assignee: Pierre Villard > SetSNMP processor HOST:PORT not changing > > > Key: NIFI-5806 > URL: https://issues.apache.org/jira/browse/NIFI-5806 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Hemantha kumara M S >Assignee: Pierre Villard >Priority: Major > > When changing the "Host Name" or "Port" on SetSNMP processor the > configuration does not take effect and the communication continues to be > active for the previous defined IP:PORT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3162: NIFI-5806 - remove SNMP target when stopping SNMP p...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3162 NIFI-5806 - remove SNMP target when stopping SNMP processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5806 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3162.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3162 commit 3ce309c9a85f716751c61ea07b7b75d26688f325 Author: Pierre Villard Date: 2018-11-09T09:24:05Z NIFI-5806 - remove SNMP target when stopping SNMP processors ---
[jira] [Commented] (NIFI-5806) SetSNMP processor HOST:PORT not changing
[ https://issues.apache.org/jira/browse/NIFI-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681128#comment-16681128 ] ASF GitHub Bot commented on NIFI-5806: -- GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3162 NIFI-5806 - remove SNMP target when stopping SNMP processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5806 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3162.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3162 commit 3ce309c9a85f716751c61ea07b7b75d26688f325 Author: Pierre Villard Date: 2018-11-09T09:24:05Z NIFI-5806 - remove SNMP target when stopping SNMP processors > SetSNMP processor HOST:PORT not changing > > > Key: NIFI-5806 > URL: https://issues.apache.org/jira/browse/NIFI-5806 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Hemantha kumara M S >Assignee: Pierre Villard >Priority: Major > > When changing the "Host Name" or "Port" on SetSNMP processor the > configuration does not take effect and the communication continues to be > active for the previous defined IP:PORT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5806) SetSNMP processor HOST:PORT not changing
[ https://issues.apache.org/jira/browse/NIFI-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5806: - Status: Patch Available (was: Open) > SetSNMP processor HOST:PORT not changing > > > Key: NIFI-5806 > URL: https://issues.apache.org/jira/browse/NIFI-5806 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Hemantha kumara M S >Assignee: Pierre Villard >Priority: Major > > When changing the "Host Name" or "Port" on SetSNMP processor the > configuration does not take effect and the communication continues to be > active for the previous defined IP:PORT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5806) SetSNMP processor HOST:PORT not changing
[ https://issues.apache.org/jira/browse/NIFI-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5806: - Component/s: Extensions > SetSNMP processor HOST:PORT not changing > > > Key: NIFI-5806 > URL: https://issues.apache.org/jira/browse/NIFI-5806 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Hemantha kumara M S >Assignee: Pierre Villard >Priority: Major > > When changing the "Host Name" or "Port" on SetSNMP processor the > configuration does not take effect and the communication continues to be > active for the previous defined IP:PORT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3162: NIFI-5806 - remove SNMP target when stopping SNMP p...
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3162#discussion_r232194884
--- Diff:
nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/processors/AbstractSNMPProcessor.java
---
@@ -252,6 +252,7 @@ public void close() {
this.getLogger().warn("Failure while closing UDP transport
mapping", e);
}
this.snmp = null;
+this.snmpTarget = null;
--- End diff --
along with this change, I was wondering move below code block from
ontrigger to onscheduled?
synchronized (this) {
this.buildTargetResource(context);
}
---
[jira] [Commented] (NIFI-5806) SetSNMP processor HOST:PORT not changing
[
https://issues.apache.org/jira/browse/NIFI-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681179#comment-16681179
]
ASF GitHub Bot commented on NIFI-5806:
--
Github user hemantha-kumara commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3162#discussion_r232194884
--- Diff:
nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/main/java/org/apache/nifi/snmp/processors/AbstractSNMPProcessor.java
---
@@ -252,6 +252,7 @@ public void close() {
this.getLogger().warn("Failure while closing UDP transport
mapping", e);
}
this.snmp = null;
+this.snmpTarget = null;
--- End diff --
along with this change, I was wondering move below code block from
ontrigger to onscheduled?
synchronized (this) {
this.buildTargetResource(context);
}
> SetSNMP processor HOST:PORT not changing
>
>
> Key: NIFI-5806
> URL: https://issues.apache.org/jira/browse/NIFI-5806
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.7.1
>Reporter: Hemantha kumara M S
>Assignee: Pierre Villard
>Priority: Major
>
> When changing the "Host Name" or "Port" on SetSNMP processor the
> configuration does not take effect and the communication continues to be
> active for the previous defined IP:PORT
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5333) Create GetMongoRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681303#comment-16681303 ] ASF GitHub Bot commented on NIFI-5333: -- Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3011#discussion_r232223940 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/pom.xml --- @@ -101,6 +101,7 @@ org.apache.nifi nifi-mongodb-client-service-api +1.8.0-SNAPSHOT --- End diff -- Has to be changed. > Create GetMongoRecord processor > --- > > Key: NIFI-5333 > URL: https://issues.apache.org/jira/browse/NIFI-5333 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > A processor similar to GetMongo that uses the record API should be created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3011: NIFI-5333 Added GetMongoRecord.
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3011#discussion_r232223940 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/pom.xml --- @@ -101,6 +101,7 @@ org.apache.nifi nifi-mongodb-client-service-api +1.8.0-SNAPSHOT --- End diff -- Has to be changed. ---
[jira] [Created] (NIFI-5807) ListFTP uncaught exception
Radim Kučera created NIFI-5807:
--
Summary: ListFTP uncaught exception
Key: NIFI-5807
URL: https://issues.apache.org/jira/browse/NIFI-5807
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.7.1
Environment: Centos7, OpenJDK Runtime Environment (build 1.8.0_181-b13)
Reporter: Radim Kučera
Using ListFTP / FetchFTP processors to interact with a slightly bizzare FTP,
which identifies itself as a "Dopra version: 1.6" but otherwise acts as a
pretty standard FTP causes following exception.
By all means this should be at least user-overwritable exception, in a terms of
"I really don't care what system that is until it works as expected," which is
exactly this case.
As you may expect, it is a vendor stripped down *nix OS Made in China. But then
abain, interacting with it by CLI utilities is functioning as expected. is
there anything that can be fixed in this matter?
{noformat}
2018-11-09 12:38:04,111 INFO [NiFi Web Server-97213]
o.a.n.c.s.StandardProcessScheduler Starting
ListFTP[id=1c1418b7-342c-1e52-6f52-a9a083e8fff7]
2018-11-09 12:38:04,113 INFO [Timer-Driven Process Thread-8]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
ListFTP[id=1c1418b7-342c-1e52-6f52-a9a083e8fff7] to run with 1 threads
2018-11-09 12:38:04,169 ERROR [Timer-Driven Process Thread-1]
o.a.nifi.processors.standard.ListFTP
ListFTP[id=1c1418b7-342c-1e52-6f52-a9a083e8fff7]
ListFTP[id=1c1418b7-342c-1e52-6f52-a9a083e8fff7] failed to process session due
to org.apache.commons.net.ftp.parser.ParserInitializationException: Unknown
parser type: Dopra version: 1.6; Processor Administratively Yielded for 1 sec:
org.apache.commons.net.ftp.parser.ParserInitializationException: Unknown parser
type: Dopra version: 1.6
org.apache.commons.net.ftp.parser.ParserInitializationException: Unknown parser
type: Dopra version: 1.6
at
org.apache.commons.net.ftp.parser.DefaultFTPFileEntryParserFactory.createFileEntryParser(DefaultFTPFileEntryParserFactory.java:170)
at
org.apache.commons.net.ftp.parser.DefaultFTPFileEntryParserFactory.createFileEntryParser(DefaultFTPFileEntryParserFactory.java:94)
at org.apache.commons.net.ftp.FTPClient.__createParser(FTPClient.java:3381)
at
org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3338)
at org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:3016)
at
org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:235)
at
org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:191)
at
org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:106)
at
org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:405)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-11-09 12:38:04,169 WARN [Timer-Driven Process Thread-1]
o.a.n.controller.tasks.ConnectableTask Administratively Yielding
ListFTP[id=1c1418b7-342c-1e52-6f52-a9a083e8fff7] due to uncaught Exception:
org.apache.commons.net.ftp.parser.ParserInitializationException: Unknown parser
type: Dopra version: 1.6
org.apache.commons.net.ftp.parser.ParserInitializationException: Unknown parser
type: Dopra version: 1.6
at
org.apache.commons.net.ftp.parser.DefaultFTPFileEntryParserFactory.createFileEntryParser(DefaultFTPFileEntryParserFactory.java:170)
at
org.apache.commons.net.ftp.parser.DefaultFTPFileEntryParserFactory.createFileEntryParser(DefaultFTPFileEntryParserFactory.java:94)
at org.apache.commons.net.ftp.FTPClient.__createParser(FTPClient.java:3381)
at
org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3338)
at org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:3016)
at
org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:235)
at
org.apache.nifi.processors.standa
[jira] [Updated] (NIFI-5807) ListFTP uncaught exception
[
https://issues.apache.org/jira/browse/NIFI-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Radim Kučera updated NIFI-5807:
---
Description:
Using ListFTP / FetchFTP processors to interact with a slightly bizzare FTP,
which identifies itself as a "Dopra version: 1.6" but otherwise acts as a
pretty standard FTP causes following exception.
By all means this should be at least user-overwritable exception, in a terms of
"I really don't care what system that is until it works as expected," which is
exactly this case.
As you may expect, it is a vendor stripped down *nix OS Made in China. But
then again, interacting with it by CLI utilities is fully functional. Is there
anything that can be fixed in this matter?
{noformat}
2018-11-09 12:38:04,111 INFO [NiFi Web Server-97213]
o.a.n.c.s.StandardProcessScheduler Starting
ListFTP[id=1c1418b7-342c-1e52-6f52-a9a083e8fff7]
2018-11-09 12:38:04,113 INFO [Timer-Driven Process Thread-8]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
ListFTP[id=1c1418b7-342c-1e52-6f52-a9a083e8fff7] to run with 1 threads
2018-11-09 12:38:04,169 ERROR [Timer-Driven Process Thread-1]
o.a.nifi.processors.standard.ListFTP
ListFTP[id=1c1418b7-342c-1e52-6f52-a9a083e8fff7]
ListFTP[id=1c1418b7-342c-1e52-6f52-a9a083e8fff7] failed to process session due
to org.apache.commons.net.ftp.parser.ParserInitializationException: Unknown
parser type: Dopra version: 1.6; Processor Administratively Yielded for 1 sec:
org.apache.commons.net.ftp.parser.ParserInitializationException: Unknown parser
type: Dopra version: 1.6
org.apache.commons.net.ftp.parser.ParserInitializationException: Unknown parser
type: Dopra version: 1.6
at
org.apache.commons.net.ftp.parser.DefaultFTPFileEntryParserFactory.createFileEntryParser(DefaultFTPFileEntryParserFactory.java:170)
at
org.apache.commons.net.ftp.parser.DefaultFTPFileEntryParserFactory.createFileEntryParser(DefaultFTPFileEntryParserFactory.java:94)
at org.apache.commons.net.ftp.FTPClient.__createParser(FTPClient.java:3381)
at
org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3338)
at org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:3016)
at
org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:235)
at
org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:191)
at
org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:106)
at
org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:405)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-11-09 12:38:04,169 WARN [Timer-Driven Process Thread-1]
o.a.n.controller.tasks.ConnectableTask Administratively Yielding
ListFTP[id=1c1418b7-342c-1e52-6f52-a9a083e8fff7] due to uncaught Exception:
org.apache.commons.net.ftp.parser.ParserInitializationException: Unknown parser
type: Dopra version: 1.6
org.apache.commons.net.ftp.parser.ParserInitializationException: Unknown parser
type: Dopra version: 1.6
at
org.apache.commons.net.ftp.parser.DefaultFTPFileEntryParserFactory.createFileEntryParser(DefaultFTPFileEntryParserFactory.java:170)
at
org.apache.commons.net.ftp.parser.DefaultFTPFileEntryParserFactory.createFileEntryParser(DefaultFTPFileEntryParserFactory.java:94)
at org.apache.commons.net.ftp.FTPClient.__createParser(FTPClient.java:3381)
at
org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3338)
at org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:3016)
at
org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:235)
at
org.apache.nifi.processors.standard.util.FTPTransfer.getListing(FTPTransfer.java:191)
at
org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:106)
at
org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:4
[GitHub] nifi pull request #3100: NIFI-5718: Implemented LineDemarcator and removed N...
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3100#discussion_r232267315
--- Diff:
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/RepeatingInputStream.java
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Objects;
+
+public class RepeatingInputStream extends InputStream {
--- End diff --
@patricker sorry - thanks for pinging! We don't know that we are trying to
avoid `@Ignore`d tests in general necessarily. What we are trying to avoid is
marking tests as `@Ignore` as a "solution" to tests that intermittently fail.
Instead, those tests need to be fixed or deleted. Having a test that is
intended to be run manually to test performance before & after a change is a
common pattern that we follow throughout the codebase. Some people also have
marked these types of tests as Integration Tests instead of unit tests, but I
think that's a bad idea, as it's not truly an integration test. I'm sure there
are better ways to handle this, but for the time being I would assert that it
is common practice and don't believe it's unfavorable.
re: the presence of the RepeatingInputStream. I do understand your
temptation to remove it. The reason that I suggest we keep it, though, is that
there have been many times that it would have been helpful but was not
available so I used only a small bit of data in tests. I do think there are
others tests in the codebase that could be updated to use this, as well. I'm
not hell-bent on this, though, so if you still feel it should be removed, say
the word and i'll push a commit that removes it.
Thanks for the review!
---
[jira] [Commented] (NIFI-5718) Performance degraded in ReplaceText processor
[
https://issues.apache.org/jira/browse/NIFI-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681496#comment-16681496
]
ASF GitHub Bot commented on NIFI-5718:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3100#discussion_r232267315
--- Diff:
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/RepeatingInputStream.java
---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Objects;
+
+public class RepeatingInputStream extends InputStream {
--- End diff --
@patricker sorry - thanks for pinging! We don't know that we are trying to
avoid `@Ignore`d tests in general necessarily. What we are trying to avoid is
marking tests as `@Ignore` as a "solution" to tests that intermittently fail.
Instead, those tests need to be fixed or deleted. Having a test that is
intended to be run manually to test performance before & after a change is a
common pattern that we follow throughout the codebase. Some people also have
marked these types of tests as Integration Tests instead of unit tests, but I
think that's a bad idea, as it's not truly an integration test. I'm sure there
are better ways to handle this, but for the time being I would assert that it
is common practice and don't believe it's unfavorable.
re: the presence of the RepeatingInputStream. I do understand your
temptation to remove it. The reason that I suggest we keep it, though, is that
there have been many times that it would have been helpful but was not
available so I used only a small bit of data in tests. I do think there are
others tests in the codebase that could be updated to use this, as well. I'm
not hell-bent on this, though, so if you still feel it should be removed, say
the word and i'll push a commit that removes it.
Thanks for the review!
> Performance degraded in ReplaceText processor
> -
>
> Key: NIFI-5718
> URL: https://issues.apache.org/jira/browse/NIFI-5718
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Attachments: Screen Shot 2018-10-17 at 10.55.53 AM.png
>
>
> NIFI-5711 addresses some licensing concerns in the NLKBufferedReader class.
> In doing so, however, it results in lower performance. The ReplaceText
> processor is affected if the Evaluation Mode is set to Line-by-Line, and the
> RouteText processor will also be affected. We should be able to match the
> performance of the previous version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3160: NIFI-5805: Pool the BinaryEncoders used by the WriteAvroRe...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3160 @ijokarumawak thanks for the review - and all great points! New commit coming momentarily. ---
[jira] [Commented] (NIFI-5805) Avro Record Writer service creates byte buffer for every Writer created
[ https://issues.apache.org/jira/browse/NIFI-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681499#comment-16681499 ] ASF GitHub Bot commented on NIFI-5805: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3160 @ijokarumawak thanks for the review - and all great points! New commit coming momentarily. > Avro Record Writer service creates byte buffer for every Writer created > --- > > Key: NIFI-5805 > URL: https://issues.apache.org/jira/browse/NIFI-5805 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > When we use the Avro RecordSet Writer, and do not embed the schema, the > Writer uses the Avro BinaryEncoder object to serialize the data. This object > can be initialized, but instead we create a new one for each writer. This > results in creating a new 64 KB byte[] each time. When we are writing many > records to a given FlowFile, this is not a big deal. However, when used in > PublishKafkaRecord or similar processors, where a new writer must be created > for every Record, this can have a very significant performance impact. > An improvement would be to have the user configure the maximum number of > BinaryEncoder objects to pool and then use a simple pooling mechanism to > reuse these objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3160: NIFI-5805: Pool the BinaryEncoders used by the WriteAvroRe...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3160 New commit pushed. ---
[jira] [Commented] (NIFI-5805) Avro Record Writer service creates byte buffer for every Writer created
[ https://issues.apache.org/jira/browse/NIFI-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681519#comment-16681519 ] ASF GitHub Bot commented on NIFI-5805: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3160 New commit pushed. > Avro Record Writer service creates byte buffer for every Writer created > --- > > Key: NIFI-5805 > URL: https://issues.apache.org/jira/browse/NIFI-5805 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > When we use the Avro RecordSet Writer, and do not embed the schema, the > Writer uses the Avro BinaryEncoder object to serialize the data. This object > can be initialized, but instead we create a new one for each writer. This > results in creating a new 64 KB byte[] each time. When we are writing many > records to a given FlowFile, this is not a big deal. However, when used in > PublishKafkaRecord or similar processors, where a new writer must be created > for every Record, this can have a very significant performance impact. > An improvement would be to have the user configure the maximum number of > BinaryEncoder objects to pool and then use a simple pooling mechanism to > reuse these objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5805) Avro Record Writer service creates byte buffer for every Writer created
[ https://issues.apache.org/jira/browse/NIFI-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-5805: - Fix Version/s: 1.9.0 Status: Patch Available (was: Open) > Avro Record Writer service creates byte buffer for every Writer created > --- > > Key: NIFI-5805 > URL: https://issues.apache.org/jira/browse/NIFI-5805 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > When we use the Avro RecordSet Writer, and do not embed the schema, the > Writer uses the Avro BinaryEncoder object to serialize the data. This object > can be initialized, but instead we create a new one for each writer. This > results in creating a new 64 KB byte[] each time. When we are writing many > records to a given FlowFile, this is not a big deal. However, when used in > PublishKafkaRecord or similar processors, where a new writer must be created > for every Record, this can have a very significant performance impact. > An improvement would be to have the user configure the maximum number of > BinaryEncoder objects to pool and then use a simple pooling mechanism to > reuse these objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3161: Increase timeouts in unit tests to avoid frequent failures
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3161 Thanks @ijokarumawak ---
[GitHub] nifi pull request #3161: Increase timeouts in unit tests to avoid frequent f...
Github user markap14 closed the pull request at: https://github.com/apache/nifi/pull/3161 ---
[GitHub] nifi issue #3111: NIFI-5757 AvroRecordSetWriter - Fix for slow synchronized ...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3111 @arkadius thanks for the update! Code review looks good. All unit tests and contrib-check pass. This does touch some super heavily used processors, though, like UpdateAttribute, so I'll want to take my time testing. This could certainly be a great improvement though. Thanks again! ---
[jira] [Commented] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
[
https://issues.apache.org/jira/browse/NIFI-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681557#comment-16681557
]
ASF GitHub Bot commented on NIFI-5757:
--
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/3111
@arkadius thanks for the update! Code review looks good. All unit tests and
contrib-check pass. This does touch some super heavily used processors, though,
like UpdateAttribute, so I'll want to take my time testing. This could
certainly be a great improvement though. Thanks again!
> AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
> ---
>
> Key: NIFI-5757
> URL: https://issues.apache.org/jira/browse/NIFI-5757
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Arek Burdach
>Priority: Major
>
> Avro record serialization is a quite expensive operation.
> This stack trace I very often see in thread dumps:
> {noformat}
> Thread 48583: (state = BLOCKED)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
> @bci=9, line=124 (Compiled frame)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=96, line=92 (Compiled frame)
> - sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
> java.lang.Object[]) @bci=56 (Compiled frame)
> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
> java.lang.Object[]) @bci=6, line=43 (Compiled frame)
> - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
> @bci=56, line=498 (Compiled frame)
> -
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
> java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
> frame)
> - com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=24 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
> org.apache.nifi.serialization.record.RecordSet,
> org.apache.nifi.serialization.RecordSetWriterFactory,
> org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
> java.lang.String) @bci=71, line=169 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
> @bci=94, line=412 (Compiled frame)
> {noformat}
> The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
> every access to cache of compiled schemas.
> I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
> instead: https://github.com/apache/nifi/pull/3111
> It is not a perfect fix because it removes cache size limitation which BTW
> was hardcoded to {{20}}. Services can be reusable by many flows so such a
> hard limit is not a good choice.
> What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3111: NIFI-5757 AvroRecordSetWriter - Fix for slow synchronized ...
Github user arkadius commented on the issue: https://github.com/apache/nifi/pull/3111 @markap14 No problem, looking forward for conclusions from this testing. ---
[jira] [Commented] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
[
https://issues.apache.org/jira/browse/NIFI-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681566#comment-16681566
]
ASF GitHub Bot commented on NIFI-5757:
--
Github user arkadius commented on the issue:
https://github.com/apache/nifi/pull/3111
@markap14 No problem, looking forward for conclusions from this testing.
> AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
> ---
>
> Key: NIFI-5757
> URL: https://issues.apache.org/jira/browse/NIFI-5757
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Arek Burdach
>Priority: Major
>
> Avro record serialization is a quite expensive operation.
> This stack trace I very often see in thread dumps:
> {noformat}
> Thread 48583: (state = BLOCKED)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
> @bci=9, line=124 (Compiled frame)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=96, line=92 (Compiled frame)
> - sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
> java.lang.Object[]) @bci=56 (Compiled frame)
> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
> java.lang.Object[]) @bci=6, line=43 (Compiled frame)
> - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
> @bci=56, line=498 (Compiled frame)
> -
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
> java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
> frame)
> - com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=24 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
> org.apache.nifi.serialization.record.RecordSet,
> org.apache.nifi.serialization.RecordSetWriterFactory,
> org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
> java.lang.String) @bci=71, line=169 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
> @bci=94, line=412 (Compiled frame)
> {noformat}
> The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
> every access to cache of compiled schemas.
> I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
> instead: https://github.com/apache/nifi/pull/3111
> It is not a perfect fix because it removes cache size limitation which BTW
> was hardcoded to {{20}}. Services can be reusable by many flows so such a
> hard limit is not a good choice.
> What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232292846
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
Sounds great to me.
I'll also modify Hive description and squash the commit.
---
[GitHub] nifi issue #3100: NIFI-5718: Implemented LineDemarcator and removed NLKBuffe...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3100 @markap14, I agree that the class would be useful. Can you include a unit test that references it? As long as some piece of the PR legitimately references the class then I'm completely happy keeping it in and merging it, even with the `@Ignore` attribute, now that it's been explained to me. ---
[jira] [Commented] (NIFI-5780) Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
[
https://issues.apache.org/jira/browse/NIFI-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681596#comment-16681596
]
ASF GitHub Bot commented on NIFI-5780:
--
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232292846
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
Sounds great to me.
I'll also modify Hive description and squash the commit.
> Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
> --
>
> Key: NIFI-5780
> URL: https://issues.apache.org/jira/browse/NIFI-5780
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Deon Huang
>Assignee: Deon Huang
>Priority: Minor
>
> Sometimes we might need to set up session relate configuration before or
> after query.
> For example:
> Pre query can be used for session relate setting like our use case Teradata
> Query Banding.
> Same feature (pre query and post query) is added to SelectHiveQL in
> https://issues.apache.org/jira/browse/NIFI-5044
> Planning to add this feature to ExecuteSQL and ExecuteSQLRecord processors.
> If pre or post statement fail, will not produce resultset flowfile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232293044
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
++ "Note, the results/outputs of these queries will be
suppressed if successfully executed.")
--- End diff --
Thanks for the suggestion, will omit it at next commit.
---
[jira] [Commented] (NIFI-5718) Performance degraded in ReplaceText processor
[ https://issues.apache.org/jira/browse/NIFI-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681597#comment-16681597 ] ASF GitHub Bot commented on NIFI-5718: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3100 @markap14, I agree that the class would be useful. Can you include a unit test that references it? As long as some piece of the PR legitimately references the class then I'm completely happy keeping it in and merging it, even with the `@Ignore` attribute, now that it's been explained to me. > Performance degraded in ReplaceText processor > - > > Key: NIFI-5718 > URL: https://issues.apache.org/jira/browse/NIFI-5718 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Attachments: Screen Shot 2018-10-17 at 10.55.53 AM.png > > > NIFI-5711 addresses some licensing concerns in the NLKBufferedReader class. > In doing so, however, it results in lower performance. The ReplaceText > processor is affected if the Evaluation Mode is set to Line-by-Line, and the > RouteText processor will also be affected. We should be able to match the > performance of the previous version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5780) Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
[
https://issues.apache.org/jira/browse/NIFI-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681598#comment-16681598
]
ASF GitHub Bot commented on NIFI-5780:
--
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232293044
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
++ "Note, the results/outputs of these queries will be
suppressed if successfully executed.")
--- End diff --
Thanks for the suggestion, will omit it at next commit.
> Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
> --
>
> Key: NIFI-5780
> URL: https://issues.apache.org/jira/browse/NIFI-5780
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Deon Huang
>Assignee: Deon Huang
>Priority: Minor
>
> Sometimes we might need to set up session relate configuration before or
> after query.
> For example:
> Pre query can be used for session relate setting like our use case Teradata
> Query Banding.
> Same feature (pre query and post query) is added to SelectHiveQL in
> https://issues.apache.org/jira/browse/NIFI-5044
> Planning to add this feature to ExecuteSQL and ExecuteSQLRecord processors.
> If pre or post statement fail, will not produce resultset flowfile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5808) MergeRecord doesn't honor "Max Bin Age" property
Davide Sessi created NIFI-5808:
--
Summary: MergeRecord doesn't honor "Max Bin Age" property
Key: NIFI-5808
URL: https://issues.apache.org/jira/browse/NIFI-5808
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework, Core UI
Affects Versions: 1.8.0
Environment: Both Ubuntu and Windows 10
Reporter: Davide Sessi
Attachments: Test_MergeRecord_max-bin-age.xml
MergeRecord ignores "Max Bin Age" property and does not merge records at all.
Only NiFi 1.8.0 is affetcted.
The attached test case should generate 2 types of CSV files:
{code:java}
column1;column2
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
...and so on for many rows...{code}
and
{code:java}
column1;column2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
...and so on for many rows...{code}
It works as expected for NiFi 1.6.0 and 1.7.1.
But using NiFi 1.8.0 I only get hundreds of files with just 1 record like the
following:
{code:java}
column1;column2
A1;B1
{code}
and
{code:java}
column1;column2
A2;B2
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3156: NIFI-5780 Add pre and post statements to ExecuteSQL and Ex...
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/3156 Thanks for the suggestion. Update description to be more precise in ExecuteSQL, ExecuteSQLRecord, SelectHiveQL. Squash the this commit. ---
[jira] [Commented] (NIFI-5780) Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
[ https://issues.apache.org/jira/browse/NIFI-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681601#comment-16681601 ] ASF GitHub Bot commented on NIFI-5780: -- Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/3156 Thanks for the suggestion. Update description to be more precise in ExecuteSQL, ExecuteSQLRecord, SelectHiveQL. Squash the this commit. > Add pre and post statements to ExecuteSQL and ExecuteSQLRecord > -- > > Key: NIFI-5780 > URL: https://issues.apache.org/jira/browse/NIFI-5780 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Deon Huang >Assignee: Deon Huang >Priority: Minor > > Sometimes we might need to set up session relate configuration before or > after query. > For example: > Pre query can be used for session relate setting like our use case Teradata > Query Banding. > Same feature (pre query and post query) is added to SelectHiveQL in > https://issues.apache.org/jira/browse/NIFI-5044 > Planning to add this feature to ExecuteSQL and ExecuteSQLRecord processors. > If pre or post statement fail, will not produce resultset flowfile. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3133 @colindean I am getting compile errors. I had to add an import, `import org.apache.nifi.components.PropertyValue;`, otherwise your method `extractMillisWithInfinite` would not build. This is building OK for you? Maybe something got lost in the squash? ---
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681618#comment-16681618
]
ASF GitHub Bot commented on NIFI-5790:
--
Github user patricker commented on the issue:
https://github.com/apache/nifi/pull/3133
@colindean I am getting compile errors. I had to add an import, `import
org.apache.nifi.components.PropertyValue;`, otherwise your method
`extractMillisWithInfinite` would not build.
This is building OK for you? Maybe something got lost in the squash?
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5808) MergeRecord doesn't honor "Max Bin Age" property
[
https://issues.apache.org/jira/browse/NIFI-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Sessi updated NIFI-5808:
---
Description:
MergeRecord ignores "Max Bin Age" property and does not merge records at all.
Only NiFi 1.8.0 is affected.
The attached test case should generate 2 types of CSV files:
{code:java}
column1;column2
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
...and so on for many rows...{code}
and
{code:java}
column1;column2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
...and so on for many rows...{code}
It works as expected for NiFi 1.6.0 and 1.7.1.
But using NiFi 1.8.0 I only get hundreds of files with just 1 record like the
following:
{code:java}
column1;column2
A1;B1
{code}
and
{code:java}
column1;column2
A2;B2
{code}
was:
MergeRecord ignores "Max Bin Age" property and does not merge records at all.
Only NiFi 1.8.0 is affetcted.
The attached test case should generate 2 types of CSV files:
{code:java}
column1;column2
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
A1;B1
...and so on for many rows...{code}
and
{code:java}
column1;column2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
A2;B2
...and so on for many rows...{code}
It works as expected for NiFi 1.6.0 and 1.7.1.
But using NiFi 1.8.0 I only get hundreds of files with just 1 record like the
following:
{code:java}
column1;column2
A1;B1
{code}
and
{code:java}
column1;column2
A2;B2
{code}
> MergeRecord doesn't honor "Max Bin Age" property
>
>
> Key: NIFI-5808
> URL: https://issues.apache.org/jira/browse/NIFI-5808
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Core UI
>Affects Versions: 1.8.0
> Environment: Both Ubuntu and Windows 10
>Reporter: Davide Sessi
>Priority: Major
> Attachments: Test_MergeRecord_max-bin-age.xml
>
>
> MergeRecord ignores "Max Bin Age" property and does not merge records at all.
> Only NiFi 1.8.0 is affected.
> The attached test case should generate 2 types of CSV files:
> {code:java}
> column1;column2
> A1;B1
> A1;B1
> A1;B1
> A1;B1
> A1;B1
> A1;B1
> A1;B1
> A1;B1
> A1;B1
> A1;B1
> A1;B1
> A1;B1
> A1;B1
> ...and so on for many rows...{code}
> and
> {code:java}
> column1;column2
> A2;B2
> A2;B2
> A2;B2
> A2;B2
> A2;B2
> A2;B2
> A2;B2
> A2;B2
> A2;B2
> ...and so on for many rows...{code}
> It works as expected for NiFi 1.6.0 and 1.7.1.
> But using NiFi 1.8.0 I only get hundreds of files with just 1 record like the
> following:
> {code:java}
> column1;column2
> A1;B1
> {code}
> and
> {code:java}
> column1;column2
> A2;B2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5809) QueryRecord stops processing data if selecting only a single field and that field is null
Mark Payne created NIFI-5809:
Summary: QueryRecord stops processing data if selecting only a
single field and that field is null
Key: NIFI-5809
URL: https://issues.apache.org/jira/browse/NIFI-5809
Project: Apache NiFi
Issue Type: Bug
Reporter: Mark Payne
>From the users@ mailing list, Mandeep Gill reported an issue where the
>following input to QueryRecord:
\{"id": "129984bf31e025599c0e9232df5c7b7c", "price": 19.47}
{"id": "a6cfcb7c7178b9d18c50f2f2dc41dab3", "price": null}
Can cause problems with the following 2 queries:
select count( *) from flowfile where price is null
select price from flowfile
I believe both of these queries are affected by the same issue: the query only
looks at the "price" field and as soon as it encountered a null value there, it
mistook that as a null record, meaning that it immediately throws out the
record and thinks it's the end of the stream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user colindean commented on the issue: https://github.com/apache/nifi/pull/3133 Very strange⦠it's there in the PR's commit and my local copy and Travis built fine: https://github.com/apache/nifi/blob/a3868928e278dc6797367f97d1d84eebcf1aad38/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java#L27 ---
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3133 @colindean Disregard. Git was misbehaving. ---
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681648#comment-16681648
]
ASF GitHub Bot commented on NIFI-5790:
--
Github user colindean commented on the issue:
https://github.com/apache/nifi/pull/3133
Very strange… it's there in the PR's commit and my local copy and Travis
built fine:
https://github.com/apache/nifi/blob/a3868928e278dc6797367f97d1d84eebcf1aad38/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java#L27
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-2575) HiveQL Processors Fail due to invalid JDBC URI resolution when using Zookeeper URI
[
https://issues.apache.org/jira/browse/NIFI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681658#comment-16681658
]
Zack Atkinson commented on NIFI-2575:
-
[~mattyb149], I see HDF NiFi 1.5-1.7.1 supports Hive2 queries using Zookeeper.
Is there any plans to update Apache NiFi with this fix?
> HiveQL Processors Fail due to invalid JDBC URI resolution when using
> Zookeeper URI
> --
>
> Key: NIFI-2575
> URL: https://issues.apache.org/jira/browse/NIFI-2575
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Yolanda M. Davis
>Priority: Major
>
> When configuring a HiveQL processor using the Zookeeper URL (e.g.
> jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
> it appears that the JDBC driver does not properly build the the uri in the
> expected format. This is because HS2 is storing JDBC parameters in ZK
> (https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the
> driver to be able to parse and use those values to configure the connection.
> However it appears the driver is expecting zookeeper to simply return the
> host:port and subsequently building an invalid URI.
> This problem has result in two variation of errors. The following was
> experienced by [~mattyb149]
> {noformat}
> 2016-08-15 12:45:12,918 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.Utils Resolved authority:
> hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC
> Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,836 INFO [Timer-Driven Process Thread-2]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 12:45:14,064 INFO [Timer-Driven Process Thread-2-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 12:45:14,182 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 12:45:14,337 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df]
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due
> to java.lang.reflect.UndeclaredThrowableException; rolling back session:
> java.lang.reflect.UndeclaredThrowableException
> 2016-08-15 12:45:14,346 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> java.lang.reflect.UndeclaredThrowableException: null
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:255)
> ~[na:na]
> at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown
> Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_65]
> at java.lang.reflect.Method.invoke(Method.java:497)
> ~[na:1.8.0_65]
> at
> org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy81.getConnection(Unknown Source) ~[na:na]
> at
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681650#comment-16681650
]
ASF GitHub Bot commented on NIFI-5790:
--
Github user patricker commented on the issue:
https://github.com/apache/nifi/pull/3133
@colindean Disregard. Git was misbehaving.
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPCo...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3133#discussion_r232317090
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
---
@@ -59,6 +61,31 @@
+ "Note that no flow file input (attributes, e.g.) is
available for use in Expression Language constructs for these properties.")
public class DBCPConnectionPool extends AbstractControllerService
implements DBCPService {
+/**
+ * Copied from {@link GenericObjectPoolConfig.DEFAULT_MIN_IDLE} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MIN_IDLE = "0";
+/**
+ * Copied from {@link GenericObjectPoolConfig.DEFAULT_MAX_IDLE} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MAX_IDLE = "8";
+/**
+ * Copied from private variable {@link
BasicDataSource.maxConnLifetimeMillis} in Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MAX_CONN_LIFETIME = "-1";
+/**
+ * Copied from {@link
GenericObjectPoolConfig.DEFAULT_TIME_BETWEEN_EVICTION_RUNS_MILLIS} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_EVICTION_RUN_PERIOD =
String.valueOf(-1L);
+/**
+ * Copied from {@link
GenericObjectPoolConfig.DEFAULT_MIN_EVICTABLE_IDLE_TIME_MILLIS} in Commons-DBCP
2.5.0
+ */
+private static final String DEFAULT_MIN_EVICTABLE_IDLE_TIME =
String.valueOf(180L) + " millis";
--- End diff --
Testing on my local instance is looking good so far. One request, can you
change this to `30 min`? A bit easier to read as a default value that way :)
---
[jira] [Commented] (NIFI-5809) QueryRecord stops processing data if selecting only a single field and that field is null
[
https://issues.apache.org/jira/browse/NIFI-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681683#comment-16681683
]
ASF GitHub Bot commented on NIFI-5809:
--
GitHub user markap14 opened a pull request:
https://github.com/apache/nifi/pull/3163
NIFI-5809: If QueryRecord has a single-column projection and that res…
…ults in a null value, do not confuse that with a null value being returned
from the Record Reader
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markap14/nifi NIFI-5809
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3163.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3163
commit 5b86fc10589cfb9484738e6d339ccfa16a46cae5
Author: Mark Payne
Date: 2018-11-09T16:40:59Z
NIFI-5809: If QueryRecord has a single-column projection and that results
in a null value, do not confuse that with a null value being returned from the
Record Reader
> QueryRecord stops processing data if selecting only a single field and that
> field is null
> -
>
> Key: NIFI-5809
> URL: https://issues.apache.org/jira/browse/NIFI-5809
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Mark Payne
>Priority: Major
>
> From the users@ mailing list, Mandeep Gill reported an issue where the
> following input to QueryRecord:
> \{"id": "129984bf31e025599c0e9232df5c7b7c", "price": 19.47}
> {"id": "a6cfcb7c7178b9d18c50f2f2dc41dab3", "price": null}
> Can cause problems with the following 2 queries:
> select count( *) from flowfile where price is null
> select price from flowfile
> I believe both of these queries are affected by the same issue: the query
> only looks at the "price" field and as soon as it encountered a null value
> there, it mistook that as a null record, meaning that it immediately throws
> out the record and thinks it's the end of the stream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3163: NIFI-5809: If QueryRecord has a single-column proje...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3163 NIFI-5809: If QueryRecord has a single-column projection and that res⦠â¦ults in a null value, do not confuse that with a null value being returned from the Record Reader Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5809 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3163.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3163 commit 5b86fc10589cfb9484738e6d339ccfa16a46cae5 Author: Mark Payne Date: 2018-11-09T16:40:59Z NIFI-5809: If QueryRecord has a single-column projection and that results in a null value, do not confuse that with a null value being returned from the Record Reader ---
[jira] [Updated] (NIFI-5809) QueryRecord stops processing data if selecting only a single field and that field is null
[
https://issues.apache.org/jira/browse/NIFI-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-5809:
-
Assignee: Mark Payne
Affects Version/s: 1.9.0
Status: Patch Available (was: Open)
> QueryRecord stops processing data if selecting only a single field and that
> field is null
> -
>
> Key: NIFI-5809
> URL: https://issues.apache.org/jira/browse/NIFI-5809
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> From the users@ mailing list, Mandeep Gill reported an issue where the
> following input to QueryRecord:
> \{"id": "129984bf31e025599c0e9232df5c7b7c", "price": 19.47}
> {"id": "a6cfcb7c7178b9d18c50f2f2dc41dab3", "price": null}
> Can cause problems with the following 2 queries:
> select count( *) from flowfile where price is null
> select price from flowfile
> I believe both of these queries are affected by the same issue: the query
> only looks at the "price" field and as soon as it encountered a null value
> there, it mistook that as a null record, meaning that it immediately throws
> out the record and thinks it's the end of the stream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681681#comment-16681681
]
ASF GitHub Bot commented on NIFI-5790:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3133#discussion_r232317090
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
---
@@ -59,6 +61,31 @@
+ "Note that no flow file input (attributes, e.g.) is
available for use in Expression Language constructs for these properties.")
public class DBCPConnectionPool extends AbstractControllerService
implements DBCPService {
+/**
+ * Copied from {@link GenericObjectPoolConfig.DEFAULT_MIN_IDLE} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MIN_IDLE = "0";
+/**
+ * Copied from {@link GenericObjectPoolConfig.DEFAULT_MAX_IDLE} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MAX_IDLE = "8";
+/**
+ * Copied from private variable {@link
BasicDataSource.maxConnLifetimeMillis} in Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MAX_CONN_LIFETIME = "-1";
+/**
+ * Copied from {@link
GenericObjectPoolConfig.DEFAULT_TIME_BETWEEN_EVICTION_RUNS_MILLIS} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_EVICTION_RUN_PERIOD =
String.valueOf(-1L);
+/**
+ * Copied from {@link
GenericObjectPoolConfig.DEFAULT_MIN_EVICTABLE_IDLE_TIME_MILLIS} in Commons-DBCP
2.5.0
+ */
+private static final String DEFAULT_MIN_EVICTABLE_IDLE_TIME =
String.valueOf(180L) + " millis";
--- End diff --
Testing on my local instance is looking good so far. One request, can you
change this to `30 min`? A bit easier to read as a default value that way :)
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3100: NIFI-5718: Implemented LineDemarcator and removed NLKBuffe...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3100 @patricker done. On a 3-year-old Macbook Pro, looks like the LineDemarcator is able to demarcate 10 MM lines in about 350 millis. Just as a point of reference :) ---
[jira] [Commented] (NIFI-5718) Performance degraded in ReplaceText processor
[ https://issues.apache.org/jira/browse/NIFI-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681715#comment-16681715 ] ASF GitHub Bot commented on NIFI-5718: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3100 @patricker done. On a 3-year-old Macbook Pro, looks like the LineDemarcator is able to demarcate 10 MM lines in about 350 millis. Just as a point of reference :) > Performance degraded in ReplaceText processor > - > > Key: NIFI-5718 > URL: https://issues.apache.org/jira/browse/NIFI-5718 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Attachments: Screen Shot 2018-10-17 at 10.55.53 AM.png > > > NIFI-5711 addresses some licensing concerns in the NLKBufferedReader class. > In doing so, however, it results in lower performance. The ReplaceText > processor is affected if the Evaluation Mode is set to Line-by-Line, and the > RouteText processor will also be affected. We should be able to match the > performance of the previous version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (NIFI-2575) HiveQL Processors Fail due to invalid JDBC URI resolution when using Zookeeper URI
[
https://issues.apache.org/jira/browse/NIFI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681658#comment-16681658
]
Zack Atkinson edited comment on NIFI-2575 at 11/9/18 5:50 PM:
--
[~mattyb149], I see HDF NiFi 1.5-1.7.1 supports Hive2 queries using Zookeeper.
Is there any plans to update Apache NiFi with this fix? I am still seeing this
error all the way through Apache NiFi v1.8 when querying Hive against Zookeeper
using Kerberos.
was (Author: zack08):
[~mattyb149], I see HDF NiFi 1.5-1.7.1 supports Hive2 queries using Zookeeper.
Is there any plans to update Apache NiFi with this fix?
> HiveQL Processors Fail due to invalid JDBC URI resolution when using
> Zookeeper URI
> --
>
> Key: NIFI-2575
> URL: https://issues.apache.org/jira/browse/NIFI-2575
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Yolanda M. Davis
>Priority: Major
>
> When configuring a HiveQL processor using the Zookeeper URL (e.g.
> jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
> it appears that the JDBC driver does not properly build the the uri in the
> expected format. This is because HS2 is storing JDBC parameters in ZK
> (https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the
> driver to be able to parse and use those values to configure the connection.
> However it appears the driver is expecting zookeeper to simply return the
> host:port and subsequently building an invalid URI.
> This problem has result in two variation of errors. The following was
> experienced by [~mattyb149]
> {noformat}
> 2016-08-15 12:45:12,918 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.Utils Resolved authority:
> hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC
> Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,836 INFO [Timer-Driven Process Thread-2]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 12:45:14,064 INFO [Timer-Driven Process Thread-2-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 12:45:14,182 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 12:45:14,337 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df]
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due
> to java.lang.reflect.UndeclaredThrowableException; rolling back session:
> java.lang.reflect.UndeclaredThrowableException
> 2016-08-15 12:45:14,346 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> java.lang.reflect.UndeclaredThrowableException: null
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:255)
> ~[na:na]
> at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown
> Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_65]
> at java.lang.reflect.Method.invoke(Method.java:497)
> ~[na:1.8.0_65]
> at
> org.apache.nifi.contro
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681806#comment-16681806
]
ASF GitHub Bot commented on NIFI-5790:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3133
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPCo...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3133 ---
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681804#comment-16681804
]
ASF subversion and git services commented on NIFI-5790:
---
Commit a628aced6b32b16eb4509094d214266938a915b9 in nifi's branch
refs/heads/master from [~colindean]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=a628ace ]
NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectionPool
Signed-off-by: Peter Wicks
This Closes #3133
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3131: NIFI-3229 When a queue contains only Penalized FlowFile's ...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3131 @patricker thanks for the PR! I had gone down this path before but I backed out the changes. The changes in this PR will result in obtaining a Write Lock on the queue of every incoming connection for every running processor in the graph. This can become quite expensive for a complex flow that is made up of thousands (or even 10's of thousands) of processors) and result in overall system performance suffering. This is why we are so care in the FlowFile Queue's implementation to ensure that isActiveQueueEmpty() never obtains a lock but instead only references AtomicReference variables. We should be able to do better, though. For example, when we pull a FlowFile from the queue, we check if it's penalized. If so, we throw it back on. Since the queue is ordered, we could do some smart things like looking at the FlowFile expiration date, then keeping track of the fact that we know all FlowFiles are penalized until that time is reached - or until a non-penalized FlowFile is added to the queue. ---
[jira] [Commented] (NIFI-3229) When a queue contains only Penalized FlowFile's the next processor Tasks/Time statistics becomes extremely large
[ https://issues.apache.org/jira/browse/NIFI-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681881#comment-16681881 ] ASF GitHub Bot commented on NIFI-3229: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3131 @patricker thanks for the PR! I had gone down this path before but I backed out the changes. The changes in this PR will result in obtaining a Write Lock on the queue of every incoming connection for every running processor in the graph. This can become quite expensive for a complex flow that is made up of thousands (or even 10's of thousands) of processors) and result in overall system performance suffering. This is why we are so care in the FlowFile Queue's implementation to ensure that isActiveQueueEmpty() never obtains a lock but instead only references AtomicReference variables. We should be able to do better, though. For example, when we pull a FlowFile from the queue, we check if it's penalized. If so, we throw it back on. Since the queue is ordered, we could do some smart things like looking at the FlowFile expiration date, then keeping track of the fact that we know all FlowFiles are penalized until that time is reached - or until a non-penalized FlowFile is added to the queue. > When a queue contains only Penalized FlowFile's the next processor Tasks/Time > statistics becomes extremely large > > > Key: NIFI-3229 > URL: https://issues.apache.org/jira/browse/NIFI-3229 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Dmitry Lukyanov >Assignee: Peter Wicks >Priority: Minor > Attachments: flow.xml.gz, nifi-stats.png, nifi-stats2.png > > > fetchfile on `not.found` produces penalized flow file > in this case i'm expecting the next processor will do one task execution when > flow file penalize time over. > but according to stats it executes approximately 1-6 times. > i understand that it could be a feature but stats became really unclear... > maybe there should be two columns? > `All Task/Times` and `Committed Task/Times` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3111: NIFI-5757 AvroRecordSetWriter - Fix for slow synchronized ...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3111 @arkadius I did a good bit of testing and wasn't able to break anything. Thanks for the improvements! +1, I have merged to master. ---
[jira] [Commented] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
[
https://issues.apache.org/jira/browse/NIFI-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681887#comment-16681887
]
ASF subversion and git services commented on NIFI-5757:
---
Commit 765df6781782d0988e958fb69d9825797a086fe7 in nifi's branch
refs/heads/master from [~arkadiusx]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=765df67 ]
NIFI-5757 Using Caffeine instead of slow synchronization on LinkedHashMap for
caches - mainly avro schema caches
This closes #3111.
Signed-off-by: Mark Payne
> AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
> ---
>
> Key: NIFI-5757
> URL: https://issues.apache.org/jira/browse/NIFI-5757
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Arek Burdach
>Priority: Major
>
> Avro record serialization is a quite expensive operation.
> This stack trace I very often see in thread dumps:
> {noformat}
> Thread 48583: (state = BLOCKED)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
> @bci=9, line=124 (Compiled frame)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=96, line=92 (Compiled frame)
> - sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
> java.lang.Object[]) @bci=56 (Compiled frame)
> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
> java.lang.Object[]) @bci=6, line=43 (Compiled frame)
> - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
> @bci=56, line=498 (Compiled frame)
> -
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
> java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
> frame)
> - com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=24 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
> org.apache.nifi.serialization.record.RecordSet,
> org.apache.nifi.serialization.RecordSetWriterFactory,
> org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
> java.lang.String) @bci=71, line=169 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
> @bci=94, line=412 (Compiled frame)
> {noformat}
> The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
> every access to cache of compiled schemas.
> I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
> instead: https://github.com/apache/nifi/pull/3111
> It is not a perfect fix because it removes cache size limitation which BTW
> was hardcoded to {{20}}. Services can be reusable by many flows so such a
> hard limit is not a good choice.
> What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3111: NIFI-5757 AvroRecordSetWriter - Fix for slow synchr...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3111 ---
[jira] [Resolved] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
[
https://issues.apache.org/jira/browse/NIFI-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne resolved NIFI-5757.
--
Resolution: Fixed
Fix Version/s: 1.9.0
> AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
> ---
>
> Key: NIFI-5757
> URL: https://issues.apache.org/jira/browse/NIFI-5757
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Arek Burdach
>Priority: Major
> Fix For: 1.9.0
>
>
> Avro record serialization is a quite expensive operation.
> This stack trace I very often see in thread dumps:
> {noformat}
> Thread 48583: (state = BLOCKED)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
> @bci=9, line=124 (Compiled frame)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=96, line=92 (Compiled frame)
> - sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
> java.lang.Object[]) @bci=56 (Compiled frame)
> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
> java.lang.Object[]) @bci=6, line=43 (Compiled frame)
> - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
> @bci=56, line=498 (Compiled frame)
> -
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
> java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
> frame)
> - com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=24 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
> org.apache.nifi.serialization.record.RecordSet,
> org.apache.nifi.serialization.RecordSetWriterFactory,
> org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
> java.lang.String) @bci=71, line=169 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
> @bci=94, line=412 (Compiled frame)
> {noformat}
> The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
> every access to cache of compiled schemas.
> I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
> instead: https://github.com/apache/nifi/pull/3111
> It is not a perfect fix because it removes cache size limitation which BTW
> was hardcoded to {{20}}. Services can be reusable by many flows so such a
> hard limit is not a good choice.
> What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
[
https://issues.apache.org/jira/browse/NIFI-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681888#comment-16681888
]
ASF GitHub Bot commented on NIFI-5757:
--
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/3111
@arkadius I did a good bit of testing and wasn't able to break anything.
Thanks for the improvements! +1, I have merged to master.
> AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
> ---
>
> Key: NIFI-5757
> URL: https://issues.apache.org/jira/browse/NIFI-5757
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Arek Burdach
>Priority: Major
> Fix For: 1.9.0
>
>
> Avro record serialization is a quite expensive operation.
> This stack trace I very often see in thread dumps:
> {noformat}
> Thread 48583: (state = BLOCKED)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
> @bci=9, line=124 (Compiled frame)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=96, line=92 (Compiled frame)
> - sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
> java.lang.Object[]) @bci=56 (Compiled frame)
> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
> java.lang.Object[]) @bci=6, line=43 (Compiled frame)
> - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
> @bci=56, line=498 (Compiled frame)
> -
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
> java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
> frame)
> - com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=24 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
> org.apache.nifi.serialization.record.RecordSet,
> org.apache.nifi.serialization.RecordSetWriterFactory,
> org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
> java.lang.String) @bci=71, line=169 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
> @bci=94, line=412 (Compiled frame)
> {noformat}
> The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
> every access to cache of compiled schemas.
> I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
> instead: https://github.com/apache/nifi/pull/3111
> It is not a perfect fix because it removes cache size limitation which BTW
> was hardcoded to {{20}}. Services can be reusable by many flows so such a
> hard limit is not a good choice.
> What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
[
https://issues.apache.org/jira/browse/NIFI-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681889#comment-16681889
]
ASF GitHub Bot commented on NIFI-5757:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3111
> AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
> ---
>
> Key: NIFI-5757
> URL: https://issues.apache.org/jira/browse/NIFI-5757
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Arek Burdach
>Priority: Major
> Fix For: 1.9.0
>
>
> Avro record serialization is a quite expensive operation.
> This stack trace I very often see in thread dumps:
> {noformat}
> Thread 48583: (state = BLOCKED)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
> @bci=9, line=124 (Compiled frame)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=96, line=92 (Compiled frame)
> - sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
> java.lang.Object[]) @bci=56 (Compiled frame)
> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
> java.lang.Object[]) @bci=6, line=43 (Compiled frame)
> - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
> @bci=56, line=498 (Compiled frame)
> -
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
> java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
> frame)
> - com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=24 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
> org.apache.nifi.serialization.record.RecordSet,
> org.apache.nifi.serialization.RecordSetWriterFactory,
> org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
> java.lang.String) @bci=71, line=169 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
> @bci=94, line=412 (Compiled frame)
> {noformat}
> The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
> every access to cache of compiled schemas.
> I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
> instead: https://github.com/apache/nifi/pull/3111
> It is not a perfect fix because it removes cache size limitation which BTW
> was hardcoded to {{20}}. Services can be reusable by many flows so such a
> hard limit is not a good choice.
> What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3051: NIFI-5642: QueryCassandra processor : output FlowFi...
Github user aglotero commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3051#discussion_r232374856
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
---
@@ -192,15 +205,17 @@ public void onScheduled(final ProcessContext context)
{
@Override
public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
FlowFile fileToProcess = null;
+FlowFile inputFlowFile = null;
if (context.hasIncomingConnection()) {
-fileToProcess = session.get();
+inputFlowFile = session.get();
// If we have no FlowFile, and all incoming connections are
self-loops then we can continue on.
// However, if we have no FlowFile and we have connections
coming from other Processors, then
// we know that we should run only if we have a FlowFile.
-if (fileToProcess == null && context.hasNonLoopConnection()) {
+if (inputFlowFile == null && context.hasNonLoopConnection()) {
return;
}
+session.remove(inputFlowFile);
--- End diff --
I've created a "original" relationship to maintain the logic from other
processors.
---
[GitHub] nifi pull request #3051: NIFI-5642: QueryCassandra processor : output FlowFi...
Github user aglotero commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3051#discussion_r232375017
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
---
@@ -191,76 +203,110 @@ public void onScheduled(final ProcessContext
context) {
@Override
public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
+FlowFile inputFlowFile = null;
FlowFile fileToProcess = null;
+
+Map attributes = null;
+
if (context.hasIncomingConnection()) {
-fileToProcess = session.get();
+inputFlowFile = session.get();
// If we have no FlowFile, and all incoming connections are
self-loops then we can continue on.
// However, if we have no FlowFile and we have connections
coming from other Processors, then
// we know that we should run only if we have a FlowFile.
-if (fileToProcess == null && context.hasNonLoopConnection()) {
+if (inputFlowFile == null && context.hasNonLoopConnection()) {
return;
}
+
+attributes = inputFlowFile.getAttributes();
}
final ComponentLog logger = getLogger();
-final String selectQuery =
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(fileToProcess).getValue();
-final long queryTimeout =
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(fileToProcess).asTimePeriod(TimeUnit.MILLISECONDS);
+final String selectQuery =
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(inputFlowFile).getValue();
+final long queryTimeout =
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(inputFlowFile).asTimePeriod(TimeUnit.MILLISECONDS);
final String outputFormat =
context.getProperty(OUTPUT_FORMAT).getValue();
-final Charset charset =
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(fileToProcess).getValue());
+final long maxRowsPerFlowFile =
context.getProperty(MAX_ROWS_PER_FLOW_FILE).evaluateAttributeExpressions().asInteger();
+final Charset charset =
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(inputFlowFile).getValue());
final StopWatch stopWatch = new StopWatch(true);
-if (fileToProcess == null) {
-fileToProcess = session.create();
+if(inputFlowFile != null){
+session.transfer(inputFlowFile, REL_ORIGINAL);
}
try {
// The documentation for the driver recommends the session
remain open the entire time the processor is running
// and states that it is thread-safe. This is why
connectionSession is not in a try-with-resources.
final Session connectionSession = cassandraSession.get();
-final ResultSetFuture queryFuture =
connectionSession.executeAsync(selectQuery);
+final ResultSet resultSet;
+
+if (queryTimeout > 0) {
+resultSet = connectionSession.execute(selectQuery,
queryTimeout, TimeUnit.MILLISECONDS);
+}else{
+resultSet = connectionSession.execute(selectQuery);
+}
+
final AtomicLong nrOfRows = new AtomicLong(0L);
-fileToProcess = session.write(fileToProcess, new
OutputStreamCallback() {
-@Override
-public void process(final OutputStream out) throws
IOException {
-try {
-logger.debug("Executing CQL query {}", new
Object[]{selectQuery});
-final ResultSet resultSet;
-if (queryTimeout > 0) {
-resultSet =
queryFuture.getUninterruptibly(queryTimeout, TimeUnit.MILLISECONDS);
-if (AVRO_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToAvroStream(resultSet, out, queryTimeout,
TimeUnit.MILLISECONDS));
-} else if (JSON_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToJsonStream(resultSet, out, charset, queryTimeout,
TimeUnit.MILLISECONDS));
-}
-} else {
-resultSet = queryFuture.getUninterruptibly();
-if (AVRO_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToAvroStream(resultSet, out, 0, null));
-}
[jira] [Commented] (NIFI-5642) QueryCassandra processor : output FlowFiles as soon fetch_size is reached
[
https://issues.apache.org/jira/browse/NIFI-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681897#comment-16681897
]
ASF GitHub Bot commented on NIFI-5642:
--
Github user aglotero commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3051#discussion_r232375017
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
---
@@ -191,76 +203,110 @@ public void onScheduled(final ProcessContext
context) {
@Override
public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
+FlowFile inputFlowFile = null;
FlowFile fileToProcess = null;
+
+Map attributes = null;
+
if (context.hasIncomingConnection()) {
-fileToProcess = session.get();
+inputFlowFile = session.get();
// If we have no FlowFile, and all incoming connections are
self-loops then we can continue on.
// However, if we have no FlowFile and we have connections
coming from other Processors, then
// we know that we should run only if we have a FlowFile.
-if (fileToProcess == null && context.hasNonLoopConnection()) {
+if (inputFlowFile == null && context.hasNonLoopConnection()) {
return;
}
+
+attributes = inputFlowFile.getAttributes();
}
final ComponentLog logger = getLogger();
-final String selectQuery =
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(fileToProcess).getValue();
-final long queryTimeout =
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(fileToProcess).asTimePeriod(TimeUnit.MILLISECONDS);
+final String selectQuery =
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(inputFlowFile).getValue();
+final long queryTimeout =
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(inputFlowFile).asTimePeriod(TimeUnit.MILLISECONDS);
final String outputFormat =
context.getProperty(OUTPUT_FORMAT).getValue();
-final Charset charset =
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(fileToProcess).getValue());
+final long maxRowsPerFlowFile =
context.getProperty(MAX_ROWS_PER_FLOW_FILE).evaluateAttributeExpressions().asInteger();
+final Charset charset =
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(inputFlowFile).getValue());
final StopWatch stopWatch = new StopWatch(true);
-if (fileToProcess == null) {
-fileToProcess = session.create();
+if(inputFlowFile != null){
+session.transfer(inputFlowFile, REL_ORIGINAL);
}
try {
// The documentation for the driver recommends the session
remain open the entire time the processor is running
// and states that it is thread-safe. This is why
connectionSession is not in a try-with-resources.
final Session connectionSession = cassandraSession.get();
-final ResultSetFuture queryFuture =
connectionSession.executeAsync(selectQuery);
+final ResultSet resultSet;
+
+if (queryTimeout > 0) {
+resultSet = connectionSession.execute(selectQuery,
queryTimeout, TimeUnit.MILLISECONDS);
+}else{
+resultSet = connectionSession.execute(selectQuery);
+}
+
final AtomicLong nrOfRows = new AtomicLong(0L);
-fileToProcess = session.write(fileToProcess, new
OutputStreamCallback() {
-@Override
-public void process(final OutputStream out) throws
IOException {
-try {
-logger.debug("Executing CQL query {}", new
Object[]{selectQuery});
-final ResultSet resultSet;
-if (queryTimeout > 0) {
-resultSet =
queryFuture.getUninterruptibly(queryTimeout, TimeUnit.MILLISECONDS);
-if (AVRO_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToAvroStream(resultSet, out, queryTimeout,
TimeUnit.MILLISECONDS));
-} else if (JSON_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToJsonStream(resultSet, out, charset, queryTimeout,
TimeUnit.MILLISECONDS));
-}
-} else {
-
[jira] [Commented] (NIFI-5642) QueryCassandra processor : output FlowFiles as soon fetch_size is reached
[
https://issues.apache.org/jira/browse/NIFI-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681894#comment-16681894
]
ASF GitHub Bot commented on NIFI-5642:
--
Github user aglotero commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3051#discussion_r232374856
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
---
@@ -192,15 +205,17 @@ public void onScheduled(final ProcessContext context)
{
@Override
public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
FlowFile fileToProcess = null;
+FlowFile inputFlowFile = null;
if (context.hasIncomingConnection()) {
-fileToProcess = session.get();
+inputFlowFile = session.get();
// If we have no FlowFile, and all incoming connections are
self-loops then we can continue on.
// However, if we have no FlowFile and we have connections
coming from other Processors, then
// we know that we should run only if we have a FlowFile.
-if (fileToProcess == null && context.hasNonLoopConnection()) {
+if (inputFlowFile == null && context.hasNonLoopConnection()) {
return;
}
+session.remove(inputFlowFile);
--- End diff --
I've created a "original" relationship to maintain the logic from other
processors.
> QueryCassandra processor : output FlowFiles as soon fetch_size is reached
> -
>
> Key: NIFI-5642
> URL: https://issues.apache.org/jira/browse/NIFI-5642
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.1
>Reporter: André Gomes Lamas Otero
>Priority: Major
>
> When I'm using QueryCassandra alongside with fetch_size parameter I expected
> that as soon my reader reaches the fetch_size the processor outputs some data
> to be processed by the next processor, but QueryCassandra reads all the data,
> then output the flow files.
> I'll start to work on a patch for this situation, I'll appreciate any
> suggestion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3131: NIFI-3229 When a queue contains only Penalized FlowFile's ...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3131 @markap14 I was worried about the same thing, which is why the `if` statement is structured as it is. First, we do the standard check on `isActiveQueueEmpty`. This happens in the code now as you mentioned, and right now if this passes we create a writelock update the queue and call the processor. All my change does is add one additional check, but only if the queue is not empty. So as far as I can tell, I'm locking one extra time for a queue that is already going to get locked, but not locking any queues that would not already get locked. Also, because I'm updating the queue during my check, when the processor does get called the lock should not last as long as it would otherwise, as there is less work to do. So overall lock time should be affected only minimally. Thoughts? ---
[jira] [Commented] (NIFI-3229) When a queue contains only Penalized FlowFile's the next processor Tasks/Time statistics becomes extremely large
[ https://issues.apache.org/jira/browse/NIFI-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681908#comment-16681908 ] ASF GitHub Bot commented on NIFI-3229: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3131 @markap14 I was worried about the same thing, which is why the `if` statement is structured as it is. First, we do the standard check on `isActiveQueueEmpty`. This happens in the code now as you mentioned, and right now if this passes we create a writelock update the queue and call the processor. All my change does is add one additional check, but only if the queue is not empty. So as far as I can tell, I'm locking one extra time for a queue that is already going to get locked, but not locking any queues that would not already get locked. Also, because I'm updating the queue during my check, when the processor does get called the lock should not last as long as it would otherwise, as there is less work to do. So overall lock time should be affected only minimally. Thoughts? > When a queue contains only Penalized FlowFile's the next processor Tasks/Time > statistics becomes extremely large > > > Key: NIFI-3229 > URL: https://issues.apache.org/jira/browse/NIFI-3229 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Dmitry Lukyanov >Assignee: Peter Wicks >Priority: Minor > Attachments: flow.xml.gz, nifi-stats.png, nifi-stats2.png > > > fetchfile on `not.found` produces penalized flow file > in this case i'm expecting the next processor will do one task execution when > flow file penalize time over. > but according to stats it executes approximately 1-6 times. > i understand that it could be a feature but stats became really unclear... > maybe there should be two columns? > `All Task/Times` and `Committed Task/Times` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5810) Add EL support to User Name property in ConsumeJMS and PublishJMS
Ed Berezitsky created NIFI-5810: --- Summary: Add EL support to User Name property in ConsumeJMS and PublishJMS Key: NIFI-5810 URL: https://issues.apache.org/jira/browse/NIFI-5810 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.8.0 Reporter: Ed Berezitsky Assignee: Ed Berezitsky ConsumeJMS and PublishJMS don't support EL for user name property. As a result, username should be hard coded, so updating this property per environment makes flows not version-able. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5810) Add EL support to User Name property in ConsumeJMS and PublishJMS
[ https://issues.apache.org/jira/browse/NIFI-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681926#comment-16681926 ] Ed Berezitsky commented on NIFI-5810: - Not adding GetJMSQueue, GetJMSTopic and PutJMS is intentional. These processors are deprecated anyway. > Add EL support to User Name property in ConsumeJMS and PublishJMS > - > > Key: NIFI-5810 > URL: https://issues.apache.org/jira/browse/NIFI-5810 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Ed Berezitsky >Assignee: Ed Berezitsky >Priority: Major > > ConsumeJMS and PublishJMS don't support EL for user name property. > As a result, username should be hard coded, so updating this property per > environment makes flows not version-able. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user colindean commented on the issue: https://github.com/apache/nifi/pull/3133 ð @patricker @mattyb149 How often and for what reasons does NiFi issue patch releases? Looking through the release history, it seems pretty rare and saved for regressions only. More directly, I'd love to get this into an official release sooner than later, in order to avoid the burdens of building NiFi internally or pulling in patched NARs as a part of our build process (we ship NiFi as a part of a product in development, but we're like _days_ from shipping 1.0â¦). I understand if there are rules/guidelines that would prevent this. ---
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681947#comment-16681947
]
ASF GitHub Bot commented on NIFI-5790:
--
Github user colindean commented on the issue:
https://github.com/apache/nifi/pull/3133
🎉
@patricker @mattyb149 How often and for what reasons does NiFi issue patch
releases? Looking through the release history, it seems pretty rare and saved
for regressions only. More directly, I'd love to get this into an official
release sooner than later, in order to avoid the burdens of building NiFi
internally or pulling in patched NARs as a part of our build process (we ship
NiFi as a part of a product in development, but we're like _days_ from shipping
1.0…). I understand if there are rules/guidelines that would prevent this.
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3100: NIFI-5718: Implemented LineDemarcator and removed NLKBuffe...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3100 +1 LGTM Ran all of the unit tests, looks good. Built it and ran it locally, had no issues. I also ran a unit test that never got put in from the ticket that started this whole journey, https://issues.apache.org/jira/browse/NIFI-5689?focusedCommentId=16652020&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16652020, no issues. ---
[jira] [Commented] (NIFI-5718) Performance degraded in ReplaceText processor
[ https://issues.apache.org/jira/browse/NIFI-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681951#comment-16681951 ] ASF GitHub Bot commented on NIFI-5718: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3100 +1 LGTM Ran all of the unit tests, looks good. Built it and ran it locally, had no issues. I also ran a unit test that never got put in from the ticket that started this whole journey, https://issues.apache.org/jira/browse/NIFI-5689?focusedCommentId=16652020&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16652020, no issues. > Performance degraded in ReplaceText processor > - > > Key: NIFI-5718 > URL: https://issues.apache.org/jira/browse/NIFI-5718 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Attachments: Screen Shot 2018-10-17 at 10.55.53 AM.png > > > NIFI-5711 addresses some licensing concerns in the NLKBufferedReader class. > In doing so, however, it results in lower performance. The ReplaceText > processor is affected if the Evaluation Mode is set to Line-by-Line, and the > RouteText processor will also be affected. We should be able to match the > performance of the previous version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3100: NIFI-5718: Implemented LineDemarcator and removed N...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3100 ---
[jira] [Commented] (NIFI-5718) Performance degraded in ReplaceText processor
[ https://issues.apache.org/jira/browse/NIFI-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681956#comment-16681956 ] ASF subversion and git services commented on NIFI-5718: --- Commit 830f7aa84d2f61781422daa43648a09c3a08f392 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=830f7aa ] NIFI-5718: Added performance-based unit test (Ignored) for LineDemarcator Signed-off-by: Peter Wicks This closes #3100. > Performance degraded in ReplaceText processor > - > > Key: NIFI-5718 > URL: https://issues.apache.org/jira/browse/NIFI-5718 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Attachments: Screen Shot 2018-10-17 at 10.55.53 AM.png > > > NIFI-5711 addresses some licensing concerns in the NLKBufferedReader class. > In doing so, however, it results in lower performance. The ReplaceText > processor is affected if the Evaluation Mode is set to Line-by-Line, and the > RouteText processor will also be affected. We should be able to match the > performance of the previous version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5718) Performance degraded in ReplaceText processor
[ https://issues.apache.org/jira/browse/NIFI-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681957#comment-16681957 ] ASF GitHub Bot commented on NIFI-5718: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3100 > Performance degraded in ReplaceText processor > - > > Key: NIFI-5718 > URL: https://issues.apache.org/jira/browse/NIFI-5718 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Attachments: Screen Shot 2018-10-17 at 10.55.53 AM.png > > > NIFI-5711 addresses some licensing concerns in the NLKBufferedReader class. > In doing so, however, it results in lower performance. The ReplaceText > processor is affected if the Evaluation Mode is set to Line-by-Line, and the > RouteText processor will also be affected. We should be able to match the > performance of the previous version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5718) Performance degraded in ReplaceText processor
[ https://issues.apache.org/jira/browse/NIFI-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681955#comment-16681955 ] ASF subversion and git services commented on NIFI-5718: --- Commit 564ad0cd71e56b426e02757c7a0a70e28ccbea12 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=564ad0c ] NIFI-5718: Implemented LineDemarcator and removed NLKBufferedReader in order to improve performance > Performance degraded in ReplaceText processor > - > > Key: NIFI-5718 > URL: https://issues.apache.org/jira/browse/NIFI-5718 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Attachments: Screen Shot 2018-10-17 at 10.55.53 AM.png > > > NIFI-5711 addresses some licensing concerns in the NLKBufferedReader class. > In doing so, however, it results in lower performance. The ReplaceText > processor is affected if the Evaluation Mode is set to Line-by-Line, and the > RouteText processor will also be affected. We should be able to match the > performance of the previous version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3133 @colindean Your correct, patch releases are for very special circumstances. I don't believe you have any option except to build this yourself until 1.9.0 gets released; and since 1.8.0 just came out, that is going to be a bit. To keep things stable, you can checkout the 1.8.0 tag and apply your change as a cherry pick so you don't accidentally include any unexpected changes, but still an internal build. ---
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682007#comment-16682007
]
ASF GitHub Bot commented on NIFI-5790:
--
Github user patricker commented on the issue:
https://github.com/apache/nifi/pull/3133
@colindean Your correct, patch releases are for very special circumstances.
I don't believe you have any option except to build this yourself until 1.9.0
gets released; and since 1.8.0 just came out, that is going to be a bit.
To keep things stable, you can checkout the 1.8.0 tag and apply your change
as a cherry pick so you don't accidentally include any unexpected changes, but
still an internal build.
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user colindean commented on the issue: https://github.com/apache/nifi/pull/3133 Thank you for confirming. Looks like my team will have some work to do next week! ---
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682019#comment-16682019
]
ASF GitHub Bot commented on NIFI-5790:
--
Github user colindean commented on the issue:
https://github.com/apache/nifi/pull/3133
Thank you for confirming. Looks like my team will have some work to do
next week!
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3164: NIFI-5810 Add UserName EL support to JMS processors
GitHub user bdesert opened a pull request: https://github.com/apache/nifi/pull/3164 NIFI-5810 Add UserName EL support to JMS processors Adding EL support to a property "User Name" to ConsumeJMS and PublishJSM Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bdesert/nifi NIFI-5810_JMS_EL Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3164.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3164 commit 5013dfff04bff154f8e1fc150e096a3ab6098a49 Author: Ed B Date: 2018-11-10T00:27:10Z NIFI-5810 Add UserName EL support to JMS processors Adding EL support to a property "User Name" to ConsumeJMS and PublishJSM ---
[jira] [Commented] (NIFI-5810) Add EL support to User Name property in ConsumeJMS and PublishJMS
[ https://issues.apache.org/jira/browse/NIFI-5810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682110#comment-16682110 ] ASF GitHub Bot commented on NIFI-5810: -- GitHub user bdesert opened a pull request: https://github.com/apache/nifi/pull/3164 NIFI-5810 Add UserName EL support to JMS processors Adding EL support to a property "User Name" to ConsumeJMS and PublishJSM Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bdesert/nifi NIFI-5810_JMS_EL Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3164.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3164 commit 5013dfff04bff154f8e1fc150e096a3ab6098a49 Author: Ed B Date: 2018-11-10T00:27:10Z NIFI-5810 Add UserName EL support to JMS processors Adding EL support to a property "User Name" to ConsumeJMS and PublishJSM > Add EL support to User Name property in ConsumeJMS and PublishJMS > - > > Key: NIFI-5810 > URL: https://issues.apache.org/jira/browse/NIFI-5810 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Ed Berezitsky >Assignee: Ed Berezitsky >Priority: Major > > ConsumeJMS and PublishJMS don't support EL for user name property. > As a result, username should be hard coded, so updating this property per > environment makes flows not version-able. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
