[GitHub] nifi issue #3218: NIFI-5895 - Add a processor to start an Oozie workflow
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3218 This PR has been tested against an Oozie 4.2.0 server instance in the following conditions: - Plaintext - Plaintext + SSL - Kerberos - Kerberos + SSL A very basic examples has been used: https://github.com/apache/oozie/tree/branch-4.3/examples/src/main/apps/shell Testing workflow: https://gist.github.com/pvillard31/4a0a7b33dc27c11538cec0b5e80c0324 The workflow.xml from the Oozie repo must be pushed in HDFS in a directory and that directory is used in the configuration of the processor. All the properties given in the job.properties file from the Oozie repo must be passed through the configuration of the processor or the attributes of the flow files. ---
[GitHub] nifi pull request #3218: NIFI-5895 - Add a processor to start an Oozie workf...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3218 NIFI-5895 - Add a processor to start an Oozie workflow Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi oozie Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3218.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3218 commit bbcdf5dc87e7dfb15f6d48a3d32b5f73540d6427 Author: Pierre Villard Date: 2018-12-11T16:25:01Z NIFI-5895 - Add a processor to start an Oozie workflow ---
[GitHub] nifi pull request #3215: NIFI-5889 - changed the wording about Destination U...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3215 NIFI-5889 - changed the wording about Destination URL in S2S RTs Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5889 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3215.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3215 commit e9503171ede6b874648251ceefe85ef88e6388dd Author: Pierre Villard Date: 2018-12-11T16:36:45Z NIFI-5889 - changed the wording about Destination URL in S2S RTs ---
[GitHub] nifi issue #3212: NIFI-5885 ArrayOutOfBoundsException at EL 'or' and 'and' f...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3212 +1, merging to master, thanks @ijokarumawak ---
[GitHub] nifi issue #3176: NIFI-5830 RedisConnectionPoolService does not work with St...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3176 Changes make sense to me, reproduced the issue with a Docker instance and confirmed this PR fixes the issue when using a standalone Redis instance. +1, merging to master, thanks @javajefe ---
[GitHub] nifi issue #3208: NIFI-5872 - Added compression option to JsonRecordSetWrite...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3208 Ran a test on my laptop with a workflow generating a lot of data: - without compression, I reached 129M events / 11.15GB out per 5 minutes - with compression (Snappy), I reached 161M events / 1GB out per 5 minutes It improves the number of processed eps while reducing the amount of data on disk. ---
[GitHub] nifi pull request #3208: NIFI-5872 - Added compression option to JsonRecordS...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3208 NIFI-5872 - Added compression option to JsonRecordSetWriter Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5872 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3208.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3208 commit 2a71d632f75c704b07f04a2a4fe9596e2a56cb74 Author: Pierre Villard Date: 2018-12-06T08:50:20Z NIFI-5872 - Added compression option to JsonRecordSetWriter ---
[GitHub] nifi issue #3206: NIFI-5877 Protecting against null values when obtaining ma...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3206 +1, merging to master, thanks @bbende ---
[GitHub] nifi pull request #3198: NIFI-5867 - add thread() EL function to get thread ...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3198#discussion_r239017060
--- Diff:
nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/evaluation/functions/ThreadEvaluator.java
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.attribute.expression.language.evaluation.functions;
+
+import java.util.Map;
+
+import org.apache.nifi.attribute.expression.language.evaluation.Evaluator;
+import
org.apache.nifi.attribute.expression.language.evaluation.QueryResult;
+import
org.apache.nifi.attribute.expression.language.evaluation.StringEvaluator;
+import
org.apache.nifi.attribute.expression.language.evaluation.StringQueryResult;
+
+public class ThreadEvaluator extends StringEvaluator {
+
+private final StringQueryResult thread;
+
+public ThreadEvaluator() {
+// See org.apache.nifi.engine.FlowEngine
+thread = new StringQueryResult(Thread.currentThread().getName());
--- End diff --
yeah... good point, pushed a commit to address that. Thanks!
---
[GitHub] nifi issue #3199: NIFI-5863 Fixed AbstractFlowFileQueue logging
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3199 +1, merged to master, thanks @ijokarumawak ---
[GitHub] nifi pull request #3198: NIFI-5867 - add thread() EL function to get thread ...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3198 NIFI-5867 - add thread() EL function to get thread name Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi thread-id Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3198.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3198 commit aa48caacac303b6d418796e3025f9756fc7f0458 Author: Pierre Villard Date: 2018-12-04T21:28:33Z NIFI-5867 - add thread() EL function to get thread name ---
[GitHub] nifi issue #3182: NIFI-5838 - Improve the schema validation method in Kite p...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3182 Hey @ijokarumawak - just had the opportunity to test this on the cluster where I originally faced the issue. Only properly closing the `FileSystem` object didn't help. It's is necessary to also have the invalidation mechanism to prevent NiFi from being unavailable with the BLOCKED threads. ---
[GitHub] nifi issue #3176: NIFI-5830 RedisConnectionPoolService does not work with St...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3176 Hey @javajefe - I just merged #3135, could you rebase this PR against master? ---
[GitHub] nifi issue #3135: [NIFI-5795] RedisDistributedMapCacheClientService put miss...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3135 Confirmed the reported issue and confirmed the fix. Merging to master, thanks @luup2k ---
[GitHub] nifi issue #3187: NIFI-5847 Added GovCloud-East region to enum list.
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3187 Closing this PR as the original intent has been resolved with #3190. @jmlogan, if you feel like there is something to be done for additional regions that are not exposed by the AWS SDK, feel free to open a JIRA / submit a PR for that. ---
[GitHub] nifi issue #3190: NIFI-5850: Replaced custom AWS regions enum with the one f...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3190 Thanks @zenfenan - merged to master. ---
[GitHub] nifi issue #3194: NIFI-5860 - Add GCPReportingTask
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3194 Example of dashboard created in Stackdriver once the reporting task is configured: https://user-images.githubusercontent.com/11541012/49339198-9f84a900-f62e-11e8-9a6f-88785bf223ca.png";> ---
[GitHub] nifi pull request #3194: NIFI-5860 - Add GCPReportingTask
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3194 NIFI-5860 - Add GCPReportingTask Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi gcp-reporting-task Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3194.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3194 commit 857db5cca1c62fa46ed3b448b58db1954824afc3 Author: Pierre Villard Date: 2018-11-02T13:52:33Z NIFI-5860 - Add GCPReportingTask ---
[GitHub] nifi issue #3190: NIFI-5850: Replaced custom AWS regions enum with the one f...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3190 @zenfenan agree that bumping the version just before a release is a good idea (and we should try to do it for multiple bundles) but I still suggest to bump the version in this PR so that we can also close #3187 and allow the community to use NiFi or the AWS NAR (built from master) with the new region. ---
[GitHub] nifi issue #3191: NIFI-5855: Remove unnecessary ORDER BY clause in GenerateT...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3191 +1, merging to master, thanks @mattyb149 ---
[GitHub] nifi issue #3189: NIFI-5849: ListXXX can lose cluster state on processor res...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3189 +1, code change LGTM, merging to master, thanks @ijokarumawak ---
[GitHub] nifi pull request #3182: NIFI-5838 - Improve the schema validation method in...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3182#discussion_r237407789
--- Diff:
nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteProcessor.java
---
@@ -101,38 +100,38 @@ protected static Schema getSchema(String
uriOrLiteral, Configuration conf) {
return parseSchema(uriOrLiteral);
}
+if(uri.getScheme() == null) {
+throw new SchemaNotFoundException("If the schema is not a JSON
string, a scheme must be specified in the URI "
++ "(ex: dataset:, view:, resource:, file:, hdfs:,
etc).");
+}
+
try {
if ("dataset".equals(uri.getScheme()) ||
"view".equals(uri.getScheme())) {
return
Datasets.load(uri).getDataset().getDescriptor().getSchema();
} else if ("resource".equals(uri.getScheme())) {
-try (InputStream in =
Resources.getResource(uri.getSchemeSpecificPart())
-.openStream()) {
+try (InputStream in =
Resources.getResource(uri.getSchemeSpecificPart()).openStream()) {
return parseSchema(uri, in);
}
} else {
// try to open the file
Path schemaPath = new Path(uri);
-FileSystem fs = schemaPath.getFileSystem(conf);
-try (InputStream in = fs.open(schemaPath)) {
+try (InputStream in =
schemaPath.getFileSystem(conf).open(schemaPath)) {
--- End diff --
Done - thanks @ijokarumawak
---
[GitHub] nifi issue #3182: NIFI-5838 - Improve the schema validation method in Kite p...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3182 Oh that's a good point, just pushed a commit to address that. Not sure I'll be able to test that change to confirm that's enough for the reported issue (at least in the comings days). ---
[GitHub] nifi pull request #3182: NIFI-5838 - Improve the schema validation method in...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3182 NIFI-5838 - Improve the schema validation method in Kite processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5838 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3182.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3182 commit 1628dd452f0322f88c811d3b0563a8df7d21dd40 Author: Pierre Villard Date: 2018-11-22T17:50:11Z NIFI-5838 - Improve the schema validation method in Kite processors ---
[GitHub] nifi issue #3167: NIFI-5812: Marked Database processors as 'PrimaryNodeOnly'
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3167 +1, merged to master, thanks @zenfenan ---
[GitHub] nifi issue #3135: [NIFI-5795] RedisDistributedMapCacheClientService put miss...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3135 Agree, I'd prefer to merge this one first, and then take care of the other one. ---
[GitHub] nifi issue #3181: NIFI-5836 changed the brittle timing based tests to integr...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3181 It LGTM, it the Travis build is OK, I'm a +1 ---
[GitHub] nifi issue #3019: [NIFI-4731][NIFI-4933] Big Query processor
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3019 rebased against master if anyone wants to give it a try ;) ---
[GitHub] nifi issue #3135: [NIFI-5795] RedisDistributedMapCacheClientService put miss...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3135 @zenfenan - did you make progress on this one? I see this change has also been added in #3176 ---
[GitHub] nifi issue #3168: NIFI-5814: Addressed issue in DatabaseReader class that wa...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3168 +1, reproduced the issue and confirmed the fix, merging to master, thanks @markap14 ---
[GitHub] nifi pull request #3176: NIFI-5830 RedisConnectionPoolService does not work ...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3176#discussion_r234457408
--- Diff:
nifi-nar-bundles/nifi-redis-bundle/nifi-redis-extensions/src/main/java/org/apache/nifi/redis/service/RedisDistributedMapCacheClientService.java
---
@@ -188,7 +189,7 @@ public void onDisabled() {
public void put(final K key, final V value, final Serializer
keySerializer, final Serializer valueSerializer) throws IOException {
withConnection(redisConnection -> {
final Tuple kv = serialize(key, value,
keySerializer, valueSerializer);
-redisConnection.set(kv.getKey(), kv.getValue(),
Expiration.seconds(ttl), null);
+redisConnection.set(kv.getKey(), kv.getValue(),
Expiration.seconds(ttl), RedisStringCommands.SetOption.UPSERT);
--- End diff --
Not sure that change is related to the reported issue, no?
---
[GitHub] nifi issue #3175: Support for generic fixed when using decimal logical type
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3175 +1, confirmed the issue and the unit tests, merging to master, thanks @gideonkorir ---
[GitHub] nifi issue #1953: NIFI-4130 Add lookup controller service in TransformXML to...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1953 Hey @bdesert - thanks a lot for the review, I just pushed a commit that should address your comments. ---
[GitHub] nifi pull request #3169: NIFI-5815 - PutORC processor 'Restricted' still req...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3169#discussion_r232937372
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/orc/PutORC.java
---
@@ -63,7 +65,11 @@
+ "the path is the directory that contains this ORC file
on HDFS. For example, this processor can send flow files downstream to
ReplaceText to set the content "
+ "to this DDL (plus the LOCATION clause as described),
then to PutHiveQL processor to create the table if it doesn't exist.")
})
-@Restricted("Provides operator the ability to write to any file that NiFi
has access to in HDFS or the local filesystem.")
+@Restricted(restrictions = {
+@Restriction(
+requiredPermission = RequiredPermission.WRITE_FILESYSTEM,
+explanation = "Provides operator the ability to delete any
file that NiFi has access to in HDFS or the local filesystem.")
--- End diff --
copy/paste stupid error... thanks @ijokarumawak !
---
[GitHub] nifi pull request #3169: NIFI-5815 - PutORC processor 'Restricted' still req...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3169 NIFI-5815 - PutORC processor 'Restricted' still requires access to re⦠â¦stricted components regardless of restriction Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5815 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3169.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3169 commit 1f211c37d63ef50dbfecb3596306d118cc3f8865 Author: Pierre Villard Date: 2018-11-12T19:22:03Z NIFI-5815 - PutORC processor 'Restricted' still requires access to restricted components regardless of restriction ---
[GitHub] nifi issue #1953: NIFI-4130 Add lookup controller service in TransformXML to...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1953 Hey @bdesert - I just rebased the PR. I'll probably add a warning about adding large XSLTs in a StringLookupService as it'll be in memory. ---
[GitHub] nifi issue #3167: NIFI-5812: Marked Database processors as 'PrimaryNodeOnly'
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3167 I'd definitely like (a lot) to see something at framework level to make a processor invalid if it is set on Primary Node Only with an incoming connection. ---
[GitHub] nifi issue #3164: NIFI-5810 Add UserName EL support to JMS processors
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3164 +1, merged to master, thanks @bdesert ---
[GitHub] nifi pull request #3162: NIFI-5806 - remove SNMP target when stopping SNMP p...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3162 NIFI-5806 - remove SNMP target when stopping SNMP processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5806 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3162.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3162 commit 3ce309c9a85f716751c61ea07b7b75d26688f325 Author: Pierre Villard Date: 2018-11-09T09:24:05Z NIFI-5806 - remove SNMP target when stopping SNMP processors ---
[GitHub] nifi issue #3159: NIFI-5794 Allow empty string demarcator in Consume/Publish...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3159 In the end, I only changed the ConsumeKafka processors since it does not make sense in PublishKafka (one would just have to not set any demarcator to get the same result: all messages in the flow files are sent as one single message in Kafka). The main interest of this PR is to increase efficiency when we consume Avro data from Kafka: there is no need to set a demarcator and we do want to have multiples messages in the same flow file. I've used this [template](https://gist.github.com/pvillard31/57e627d206e7965cfb87cc27c09caf44) to test the change and to confirm the behavior: when pulling Avro data from Kafka with "empty string" as demarcator to have multiple messages in the same flow file, we are able to process the records in the next processor using a ConvertRecord for instance. ---
[GitHub] nifi pull request #3159: NIFI-5794 Allow empty string demarcator in Consume/...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3159 NIFI-5794 Allow empty string demarcator in Consume/PublishKafka proce⦠â¦ssors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5794 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3159.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3159 commit 88fbcffdc3fa6aadac14c23af9a5012edaa21e0f Author: Pierre Villard Date: 2018-11-07T09:55:10Z NIFI-5794 Allow empty string demarcator in Consume/PublishKafka processors ---
[GitHub] nifi issue #3158: NIFI-5802: Add QueryRecord nullable field support
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3158 +1, merging to master, thanks @ijokarumawak ---
[GitHub] nifi issue #3157: NIFI-5800: Do not recursively call hashCode on child schem...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3157 +1, merging to master, thanks! ---
[GitHub] nifi issue #3155: NIFI-5784 Edit Admin Guide to remove duplicate content tha...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3155 +1, merging to master, thanks! ---
[GitHub] nifi issue #3137: NIFI-5797 : EscapedJava for FlattenJson
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3137 I believe this has been fixed by #3138. If you agree, can you close this PR? ---
[GitHub] nifi issue #3138: NIFI-5798:Fixed bug in FlattenJson that was escaping text ...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3138 +1, merging to master, thanks! ---
[GitHub] nifi issue #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3124 Hey @andrewmlim - it looks good to me, thanks for taking care of it, it's going to be very useful! @bbende - do you think it still makes sense to keep the CLI README file? we don't want to remove it until 1.9.0 is released (with the doc changes) but I think it'd be better to keep things in one place after that. I can file a JIRA if you agree with it. ---
[GitHub] nifi issue #3130: NIFI-5791: Add Apache Daffodil (incubating) bundle
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3130 Not related to the naming discussions but would it make sense to consider a DFDLReader and a DFDLWriter implementation? Didn't look into the code but since it can parse/unparse into JSON, it should be doable? It would be a nice addition. ---
[GitHub] nifi pull request #3122: NIFI-5777: Update the tag and the property of LogMe...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3122#discussion_r230709356
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestLogMessage.java
---
@@ -57,7 +57,7 @@ MockComponentLog getMockComponentLog() {
public void before() throws InitializationException {
testableLogMessage = new TestableLogMessage();
runner = TestRunners.newTestRunner(testableLogMessage);
-
+runner.setValidateExpressionUsage(false);
--- End diff --
Yeah sounds good to me to have EL support ;)
---
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Github user pvillard31 commented on a diff in the pull request: https://github.com/apache/nifi/pull/3124#discussion_r230325510 --- Diff: nifi-docs/src/main/asciidoc/toolkit-guide.adoc --- @@ -0,0 +1,1257 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// += Apache NiFi Toolkit Guide +Apache NiFi Team +:homepage: http://nifi.apache.org +:linkattrs: + +== Overview +The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include: + +* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes. +* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance. +* File Manager -- The `file-manager` tool enables administrators to backup, install or restore a NiFi installation from backup. +* Flow Analyzer -- The `flow-analyzer` tool produces a report that helps administrators understand the max amount of data which can be stored in backpressure for a given flow. +* Node Manager -- The `node-manager` tool enables administrators to perform status checks on nodes as well as the ability to connect, disconnect, or remove nodes from the cluster. +* Notify -- The `notify` tool enables administrators to send bulletins to the NiFi UI. +* S2S -- The `s2s` tool enables administrators to send data into or out of NiFi flows over site-to-site. +* TLS Toolkit -- The `tls-toolkit` utility generates the required keystores, truststore, and relevant configuration files to facilitate the setup of a secure NiFi instance. +* ZooKeeper Migrator -- The `zk-migrator` tool enables administrators to: +** move ZooKeeper information from one ZooKeeper cluster to another +** migrate ZooKeeper node ownership + +The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation. + +NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi downloads page^]). + +=== Prerequisites for Running in a Secure Environment +For secured nodes and clusters, two policies should be configured in advance: + +* Access the controller â A user that will have access to these utilities should be authorized in NiFi by creating an âaccess the controllerâ policy (`/controller`) with both view and modify rights +* Proxy user request â If not previously set, nodeâs identity (the DN value of the nodeâs certificate) should be authorized to proxy requests on behalf of a user + +When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored. + +== NiFi CLI +This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes. + +=== Usage +The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell. + +To execute a single command: + + ./bin/cli.sh + +To launch the interactive shell: + + ./bin/cli.sh + +To show help: + + cli.sh -h + +The following are available options: + + demo quick-import + nifi current-user + nifi cluster-summary + nifi connect-node + nifi delete-node + nifi disconnect-node + nifi get-root-id + nifi get-node + nifi get-nodes + nifi offload-node + nifi list-reg-clients + nifi create-reg-client + nifi update-reg-client + nifi get-reg-client-id
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Github user pvillard31 commented on a diff in the pull request: https://github.com/apache/nifi/pull/3124#discussion_r230322139 --- Diff: nifi-docs/src/main/asciidoc/toolkit-guide.adoc --- @@ -0,0 +1,1257 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// += Apache NiFi Toolkit Guide +Apache NiFi Team +:homepage: http://nifi.apache.org +:linkattrs: + +== Overview +The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include: + +* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes. +* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance. +* File Manager -- The `file-manager` tool enables administrators to backup, install or restore a NiFi installation from backup. +* Flow Analyzer -- The `flow-analyzer` tool produces a report that helps administrators understand the max amount of data which can be stored in backpressure for a given flow. +* Node Manager -- The `node-manager` tool enables administrators to perform status checks on nodes as well as the ability to connect, disconnect, or remove nodes from the cluster. +* Notify -- The `notify` tool enables administrators to send bulletins to the NiFi UI. +* S2S -- The `s2s` tool enables administrators to send data into or out of NiFi flows over site-to-site. +* TLS Toolkit -- The `tls-toolkit` utility generates the required keystores, truststore, and relevant configuration files to facilitate the setup of a secure NiFi instance. +* ZooKeeper Migrator -- The `zk-migrator` tool enables administrators to: +** move ZooKeeper information from one ZooKeeper cluster to another +** migrate ZooKeeper node ownership + +The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation. + +NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi downloads page^]). + +=== Prerequisites for Running in a Secure Environment +For secured nodes and clusters, two policies should be configured in advance: + +* Access the controller â A user that will have access to these utilities should be authorized in NiFi by creating an âaccess the controllerâ policy (`/controller`) with both view and modify rights +* Proxy user request â If not previously set, nodeâs identity (the DN value of the nodeâs certificate) should be authorized to proxy requests on behalf of a user + +When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored. + +== NiFi CLI +This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes. + +=== Usage +The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell. + +To execute a single command: + + ./bin/cli.sh + +To launch the interactive shell: + + ./bin/cli.sh + +To show help: + + cli.sh -h --- End diff -- ```suggestion ./bin/cli.sh -h ``` ---
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Github user pvillard31 commented on a diff in the pull request: https://github.com/apache/nifi/pull/3124#discussion_r230324180 --- Diff: nifi-docs/src/main/asciidoc/toolkit-guide.adoc --- @@ -0,0 +1,1257 @@ +// +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// += Apache NiFi Toolkit Guide +Apache NiFi Team +:homepage: http://nifi.apache.org +:linkattrs: + +== Overview +The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include: + +* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes. +* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance. +* File Manager -- The `file-manager` tool enables administrators to backup, install or restore a NiFi installation from backup. +* Flow Analyzer -- The `flow-analyzer` tool produces a report that helps administrators understand the max amount of data which can be stored in backpressure for a given flow. +* Node Manager -- The `node-manager` tool enables administrators to perform status checks on nodes as well as the ability to connect, disconnect, or remove nodes from the cluster. +* Notify -- The `notify` tool enables administrators to send bulletins to the NiFi UI. +* S2S -- The `s2s` tool enables administrators to send data into or out of NiFi flows over site-to-site. +* TLS Toolkit -- The `tls-toolkit` utility generates the required keystores, truststore, and relevant configuration files to facilitate the setup of a secure NiFi instance. +* ZooKeeper Migrator -- The `zk-migrator` tool enables administrators to: +** move ZooKeeper information from one ZooKeeper cluster to another +** migrate ZooKeeper node ownership + +The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation. + +NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi downloads page^]). + +=== Prerequisites for Running in a Secure Environment +For secured nodes and clusters, two policies should be configured in advance: + +* Access the controller â A user that will have access to these utilities should be authorized in NiFi by creating an âaccess the controllerâ policy (`/controller`) with both view and modify rights +* Proxy user request â If not previously set, nodeâs identity (the DN value of the nodeâs certificate) should be authorized to proxy requests on behalf of a user + +When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored. + +== NiFi CLI +This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes. + +=== Usage +The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell. + +To execute a single command: + + ./bin/cli.sh + +To launch the interactive shell: + + ./bin/cli.sh + +To show help: + + cli.sh -h + +The following are available options: + + demo quick-import + nifi current-user + nifi cluster-summary + nifi connect-node + nifi delete-node + nifi disconnect-node + nifi get-root-id + nifi get-node + nifi get-nodes + nifi offload-node + nifi list-reg-clients + nifi create-reg-client + nifi update-reg-client + nifi get-reg-client-id
[GitHub] nifi issue #3098: NIFI-5728 XML Writer to populate record tag name properly
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3098 +1, merging to master ---
[GitHub] nifi pull request #3122: NIFI-5777: Update the tag and the property of LogMe...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3122#discussion_r230300635
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestLogMessage.java
---
@@ -57,7 +57,7 @@ MockComponentLog getMockComponentLog() {
public void before() throws InitializationException {
testableLogMessage = new TestableLogMessage();
runner = TestRunners.newTestRunner(testableLogMessage);
-
+runner.setValidateExpressionUsage(false);
--- End diff --
In that case I'm not sure to understand the intent of this PR. Initially I
thought it was about removing
``.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)``
on ``LOG_LEVEL`` since the property only accepts a list of allowed values. If
we do that, then we should remove the
``.evaluateAttributeExpressions(flowFile)``, no?
On the other hand, if we do want to allow EL on this property, then the
property and code should be changed to allow free text value that is validated
against the list of allowed values only if EL is not used. And then if EL is
used, there should be a default behavior in case the value, after EL
evaluation, is not an allowed value.
Am I missing something?
---
[GitHub] nifi pull request #3123: NIFI-5781 - Incorrect schema for provenance events ...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3123 NIFI-5781 - Incorrect schema for provenance events in SiteToSiteProve⦠â¦nanceReportingTask Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5781 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3123.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3123 commit 15ba89ccae4c01cc5130869a5619dde9881dee16 Author: Pierre Villard Date: 2018-11-01T14:19:06Z NIFI-5781 - Incorrect schema for provenance events in SiteToSiteProvenanceReportingTask ---
[GitHub] nifi pull request #3098: NIFI-5728 XML Writer to populate record tag name pr...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3098#discussion_r230024436
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/test/java/org/apache/nifi/xml/TestXMLRecordSetWriter.java
---
@@ -116,6 +124,34 @@ public void testRootAndRecordNaming() throws
IOException, InitializationExceptio
assertThat(expected,
CompareMatcher.isSimilarTo(actual).ignoreWhitespace().withNodeMatcher(new
DefaultNodeMatcher(ElementSelectors.byNameAndText)));
}
+@Test
+public void testSchemaRootRecordNaming() throws IOException,
InitializationException {
+String avroSchemaText = new
String(Files.readAllBytes(Paths.get("src/test/resources/xml/testschema3")));;
+Schema avroSchema = new Schema.Parser().parse(avroSchemaText);
+
+SchemaIdentifier schemaId =
SchemaIdentifier.builder().name("schemaName").build();
+RecordSchema recordSchema = AvroTypeUtil.createSchema(avroSchema,
avroSchemaText, schemaId);
+
+XMLRecordSetWriter writer = new _XMLRecordSetWriter(recordSchema);
+TestRunner runner = setup(writer);
+
+runner.setProperty(writer, XMLRecordSetWriter.ROOT_TAG_NAME,
"ROOT_NODE");
+//runner.setProperty(writer, XMLRecordSetWriter.RECORD_TAG_NAME,
"RECORD_NODE");
--- End diff --
to remove?
---
[GitHub] nifi pull request #3098: NIFI-5728 XML Writer to populate record tag name pr...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3098#discussion_r227674603
--- Diff:
nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/SimpleRecordSchema.java
---
@@ -213,4 +215,30 @@ public String toString() {
public SchemaIdentifier getIdentifier() {
return schemaIdentifier;
}
+
+/**
+ * Set schema name.
+ * @param schemaName schema name as defined in a root record.
+ */
+public void setSchemaName(String schemaName) {
+this.schemaName = schemaName;
+}
+
+@Override
+public Optional getSchemaName() {
+return Optional.ofNullable(schemaName);
+}
+
+/**
+ * Set schema namespace.
+ * @param schemaNamespace schema name as defined in a root record.
--- End diff --
schema namespace
---
[GitHub] nifi pull request #3122: NIFI-5777: Update the tag and the property of LogMe...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3122#discussion_r230022659
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestLogMessage.java
---
@@ -57,7 +57,7 @@ MockComponentLog getMockComponentLog() {
public void before() throws InitializationException {
testableLogMessage = new TestableLogMessage();
runner = TestRunners.newTestRunner(testableLogMessage);
-
+runner.setValidateExpressionUsage(false);
--- End diff --
I disagree with this change: disabling the expression usage validation
should be avoided as much as possible and, if I recall correctly, is only
necessary in very specific edge cases. Instead of disabling the validation, I'd
fix the original issue by removing ``.evaluateAttributeExpressions(flowFile)``
on L129 in LogMessage since it does not make sense. Once it's fixed, I'm a +1
and can merge to master. Feel free to squash your commits. Thanks!
---
[GitHub] nifi issue #3120: NIFI-5776: ListSFTP should log SSH_FXP_STATUS id value
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3120 Thanks @ijokarumawak - merged to master ! ---
[GitHub] nifi issue #3121: NIFI-1505: Add processor description for LogAttribute
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3121 Thanks @kotarot - merged to master! ---
[GitHub] nifi issue #3103: NIFI-5739: Maintain CaptureChangeMySQL JDBC connection aut...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3103 +1, merging to master, thanks @ijokarumawak ---
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3086 Hey @mattyb149 - I gave up on the unit test and added the Ignore annotation. I confirmed in my environment that the controller service won't be enabled if Kerberos authentication fails and that we are not logging the "authentication successful" message anymore. ---
[GitHub] nifi issue #3092: NIFI-5525 - CSVRecordReader fails with StringIndexOutOfBou...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3092 @patricker - thanks for the review, pushed a commit to address your comment ---
[GitHub] nifi issue #2882: NIFI-4914
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2882 Hi @david-streamlio - sorry it took so long, I've been busy with a lot of things... I've not been able to complete all my tests successfully but I wanted to give a status with my tests so far. All the code changes I did are available here: 8ca2db6712b9fc843a3d2ee7917ecc6db2dd54f2 Most of the changes are just to be consistent with the code base but some have been required to make things work. In addition to that, I have the following comments: if we leave the ack timeout to 0sec, we get the following error 2018-10-22 08:23:18,998 WARN [Timer-Driven Process Thread-8] o.a.n.controller.tasks.ConnectableTask Administratively Yielding ConsumePulsar[id=1a70a923-0166-1000-e698-63a0160cc922] due to uncaught Exception: java.lang.IllegalArgumentException: Ack timeout should be should be greater than 1000 ms java.lang.IllegalArgumentException: Ack timeout should be should be greater than 1000 ms at org.apache.pulsar.shade.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) at org.apache.pulsar.client.impl.ConsumerBuilderImpl.ackTimeout(ConsumerBuilderImpl.java:147) at org.apache.nifi.processors.pulsar.AbstractPulsarConsumerProcessor.getConsumerBulder(AbstractPulsarConsumerProcessor.java:361) at org.apache.nifi.processors.pulsar.AbstractPulsarConsumerProcessor.getConsumer(AbstractPulsarConsumerProcessor.java:338) at org.apache.nifi.processors.pulsar.pubsub.ConsumePulsar.onTrigger(ConsumePulsar.java:46) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Is it something that changed with the latest version of Pulsar? In the consume processors, I'd add the topic name as attribute of flowfile and possibly other attributes that could be useful for later routing/filtering. When stopping the processors or stopping NiFi, some processors does not stop cleanly and it requires a process kill for NiFi to shutdown. During my last test, it was on ConsumePulsar but it could be on other processors as well. ---
[GitHub] nifi pull request #3093: NIFI-5726 - Update README with Slack Workspace info...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3093 NIFI-5726 - Update README with Slack Workspace information Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5726 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3093.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3093 commit cc7756e33f7d63e018b232180e42b8a94556e84a Author: Pierre Villard Date: 2018-10-19T09:08:08Z NIFI-5726 - Update README with Slack Workspace information ---
[GitHub] nifi pull request #3092: NIFI-5525 - CSVRecordReader fails with StringIndexO...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3092 NIFI-5525 - CSVRecordReader fails with StringIndexOutOfBoundsExceptio⦠â¦n when field is a double quote Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5525 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3092.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3092 commit 97ae7f8d7f5c03bf63ebcc7fee74780f9abf7954 Author: Pierre Villard Date: 2018-10-19T08:44:00Z NIFI-5525 - CSVRecordReader fails with StringIndexOutOfBoundsException when field is a double quote ---
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3086 tried to squash everything and force push, but same result - tried to look in the code if something is different and should be initialized differently but didn't find anything obvious :( ---
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3086 Tried few things... no luck :( ---
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3086 The unit test is working fine locally on my side but does not work in Travis. I assume it could be because I've Kerberos related configs on my laptop. Didn't find any similar testing in other processors? Not sure how to address it (tried few things). What do you think? Just adding @Ignore on the test? =/ ---
[GitHub] nifi issue #2022: NIFI-4200 - Initial commit for a ControlNiFi processor
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2022 Hey @patricker - I'd need to revisit the PR to add some minor things like EL scope and similar things. Regarding the processor itself, I believe it adds some interesting possibilities and still think it could be an interesting addition. However, I want to be sure the concerns raised in the JIRA are addressed before moving on. If people are interested by this feature, feel free to drop a comment on the JIRA: https://issues.apache.org/jira/browse/NIFI-4200 ---
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3086 hold on - just realized that the unit test does not work in the Travis env for the kerb stuff ---
[GitHub] nifi issue #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authenticatio...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3086 Hey @mattyb149 - could you let me know if this LGTY ;) ---
[GitHub] nifi pull request #3086: NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authen...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3086 NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authentication issue/mis⦠â¦leading Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5714 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3086.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3086 commit 0fc99a19057bbbc7e1c8123e8cbc8e408c53873c Author: Pierre Villard Date: 2018-10-16T23:04:16Z NIFI-5714 - Hive[3]ConnectionPool - Kerberos Authentication issue/misleading ---
[GitHub] nifi issue #3061: NIFI-5681: Fix a locale-dependent test in TestVersionedFlo...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3061 +1, merging to master, thanks @kotarot ---
[GitHub] nifi issue #3058: NIFI-5675: Fix some locale-dependent tests in ConvertExcel...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3058 +1, merging to master, thanks @kotarot ---
[GitHub] nifi issue #3059: NIFI-5676: Fix a timezone-dependent test in PutORCTest
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3059 +1, merging to master, thanks @kotarot ---
[GitHub] nifi issue #3055: NIFI-5600: Fixing columns in queue listing and component s...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3055 LGTM, built the PR and tested the change, working as expected, merging to master. Thanks @mcgilman ---
[GitHub] nifi issue #3021: NIFI-5600 - Restore node location display on queue listing
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3021 Thanks much @mcgilman ! ---
[GitHub] nifi pull request #3021: NIFI-5600 - Restore node location display on queue ...
Github user pvillard31 closed the pull request at: https://github.com/apache/nifi/pull/3021 ---
[GitHub] nifi issue #2945: NIFI-4517: Added ExecuteSQLRecord and QueryDatabaseTableRe...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2945 Gave it another try, all LGTM, merging to master, thanks much @mattyb149 ---
[GitHub] nifi issue #1964: Nifi 4141 - Adding a processor for convert Json to Orc
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1964 @bbasosuho - do you mind closing this PR now we added the PutORC processor in NiFi? Thanks. ---
[GitHub] nifi issue #3032: NIFI-5612: Support JDBC drivers that return Long for unsig...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3032 Hey @colindean, I'll let @bbende & @mattyb149 have a look on the code but as long as 1 of the 3 Travis builds is green, you're good to go. We have to fix the intermittent issues on the builds but that's not related to your PR. ---
[GitHub] nifi issue #1015: NIFI-2767 Periodically reload properties from file-based v...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1015 Hey @jfrazee - I think we can close this PR with the new feature to define variables at Process Group level through the UI. ---
[GitHub] nifi issue #2945: NIFI-4517: Added ExecuteSQLRecord and QueryDatabaseTableRe...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2945 I've been trying the two processors in my usual test workflows replacing ExecuteSQL with the record version (and similarly with QueryDatabaseTable) and it works fine. Once Mike's comments are addressed and the PR rebased, I can have a final review before merging. ---
[GitHub] nifi issue #2987: NIFI-3344 Added property to JoltTransformJSON allowing the...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2987 Code LGTM, merging to master, thanks @nalewis ---
[GitHub] nifi pull request #3031: NIFI-5635 - Description PutEmail properties with mu...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3031 NIFI-5635 - Description PutEmail properties with multiple senders/rec⦠â¦ipients Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5635 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3031.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3031 commit 1c76f7826d60cb3dc644e9863c671aad1e1f76a9 Author: Pierre Villard Date: 2018-09-25T20:53:28Z NIFI-5635 - Description PutEmail properties with multiple senders/recipients ---
[GitHub] nifi pull request #3029: NIFI-5633 - allDelineatedValues can throw ArrayInde...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3029 NIFI-5633 - allDelineatedValues can throw ArrayIndexOutOfBoundsException Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5633 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3029.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3029 commit b9cbc23114a0078c47384a84e91c0c0d9b24c1b2 Author: Pierre Villard Date: 2018-09-25T12:57:49Z NIFI-5633 - allDelineatedValues can throw ArrayIndexOutOfBoundsException ---
[GitHub] nifi issue #3027: NIFI-5618: Avoid NPE when viewing Provenance Event details...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3027 Code change LGTM, taking care of the check style issue while merging, +1, merging to master, thanks @markap14 ---
[GitHub] nifi issue #3026: NIFI-5630: Ensure that we include counters in Status Histo...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3026 Build ok, tested with UpdateCounter processor to confirm that counter is available in Status History. +1, merging to master, thanks @markap14 ---
[GitHub] nifi issue #2954: NIFI-5514: Fixed bugs in MergeRecord around minimum thresh...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2954 +1, merging to master, thanks @markap14 ---
[GitHub] nifi pull request #3023: NIFI-5591 - Added avro compression format to Execut...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3023 NIFI-5591 - Added avro compression format to ExecuteSQL Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5591 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3023.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3023 commit 30ad5d306123b9f99a9327aeac77727ca6d6490d Author: Pierre Villard Date: 2018-09-23T19:42:26Z NIFI-5591 - Added avro compression format to ExecuteSQL ---
[GitHub] nifi pull request #3022: NIFI-5588 - Fix max wait time in DBCP Connection Po...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3022 NIFI-5588 - Fix max wait time in DBCP Connection Pool Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5588 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3022.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3022 commit b8978480c64a6e4d6242caba51a562defd290fd7 Author: Pierre Villard Date: 2018-09-23T20:00:36Z NIFI-5588 - Fix max wait time in DBCP Connection Pool ---
[GitHub] nifi issue #3021: NIFI-5600 - Restore node location display on queue listing
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3021 @mcgilman @scottyaslan - friendly ping to be sure I didn't do something against the purpose of NIFI-5208 (#2740) ---
[GitHub] nifi pull request #3021: NIFI-5600 - Restore node location display on queue ...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3021 NIFI-5600 - Restore node location display on queue listing Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5600 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3021.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3021 commit db9f85417b2f014572abea62d46a1a04537c5f37 Author: Pierre Villard Date: 2018-09-23T15:22:41Z NIFI-5600 - Restore node location display on queue listing ---
[GitHub] nifi issue #3018: NIFI-5622 Updated test resource keystores and truststores ...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3018 I'm a +1 on this PR but will wait a bit in case someone with a security background wants to double check. ---
[GitHub] nifi issue #3019: [NIFI-4731][NIFI-4933] Big Query processor
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3019 This PR supercedes #2682 and #2420. @ijokarumawak - would you mind having a look? (in particular in regards to the proxy settings) @zenfenan - a friendly ping since you looked at the one opened by @danieljimenez ---
[GitHub] nifi pull request #3019: [NIFI-4731][NIFI-4933] Big Query processor
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3019 [NIFI-4731][NIFI-4933] Big Query processor Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi PR2682 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3019.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3019 commit e08d5cbd2b972bc92c39ccb5d2c6054ac2c5f852 Author: Daniel Jimenez Date: 2018-04-29T08:01:34Z NIFI-4731: BQ Processors and GCP library update. commit 6d55066fa6ac3e6adf60e566ecaa7f629def11a1 Author: Pierre Villard Date: 2018-09-18T21:19:03Z [NIFI-4731][NIFI-4933] BigQuery PR Review ---
[GitHub] nifi pull request #2882: NIFI-4914
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2882#discussion_r218858965
--- Diff:
nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-processors/src/main/java/org/apache/nifi/processors/pulsar/AbstractPulsarConsumerProcessor.java
---
@@ -0,0 +1,412 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.pulsar;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.pulsar.PulsarClientService;
+import org.apache.nifi.pulsar.cache.LRUCache;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.ConsumerBuilder;
+import org.apache.pulsar.client.api.ConsumerCryptoFailureAction;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.client.api.SubscriptionType;
+
+public abstract class AbstractPulsarConsumerProcessor extends
AbstractProcessor {
+
+static final AllowableValue EXCLUSIVE = new
AllowableValue("Exclusive", "Exclusive", "There can be only 1 consumer on the
same topic with the same subscription name");
+static final AllowableValue SHARED = new AllowableValue("Shared",
"Shared", "Multiple consumer will be able to use the same subscription name and
the messages");
+static final AllowableValue FAILOVER = new AllowableValue("Failover",
"Failover", "Multiple consumer will be able to use the same subscription name
but only 1 consumer "
++ "will receive the messages. If that consumer
disconnects, one of the other connected consumers will start receiving
messages");
+
+static final AllowableValue CONSUME = new
AllowableValue(ConsumerCryptoFailureAction.CONSUME.name(), "Consume",
+"Mark the message as consumed despite being unable to decrypt
the contents");
+static final AllowableValue DISCARD = new
AllowableValue(ConsumerCryptoFailureAction.DISCARD.name(), "Discard",
+"Discard the message and don't perform any addtional
processing on the message");
+static final AllowableValue FAIL = new
AllowableValue(ConsumerCryptoFailureAction.FAIL.name(), "Fail",
+"Report a failure condition, and the route the message
contents to the FAILED relationship.");
--- End diff --
typo: "and then route the message content"
---
[GitHub] nifi pull request #2882: NIFI-4914
Github user pvillard31 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2882#discussion_r218857973 --- Diff: nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-processors/.gitignore --- @@ -0,0 +1 @@ +/target/ --- End diff -- same here ---
[GitHub] nifi pull request #2882: NIFI-4914
Github user pvillard31 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2882#discussion_r218858305 --- Diff: nifi-nar-bundles/nifi-pulsar-bundle/nifi-pulsar-processors/pom.xml --- @@ -0,0 +1,78 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + + +org.apache.nifi +nifi-pulsar-bundle +1.8.0-SNAPSHOT + + +nifi-pulsar-processors +jar + + + +org.apache.nifi +nifi-api + + +org.apache.nifi +nifi-record-serialization-service-api + + +org.apache.nifi +nifi-record + + +org.apache.nifi +nifi-utils +1.8.0-SNAPSHOT + + +org.apache.nifi +nifi-ssl-context-service-api + + +org.apache.nifi +nifi-pulsar-client-service-api +1.8.0-SNAPSHOT +provided + + +org.apache.pulsar +pulsar-client +2.0.0-rc1-incubating --- End diff -- can we set the pulsar version as a property in the root pom of the bundle and reference that version? Also upgrade to 2.1.1 if possible? ---
