[GitHub] nifi pull request #2968: NIFI-5456: AWS clients now work with private link e...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2968#discussion_r241044902
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
---
@@ -286,7 +286,7 @@ protected void
initializeRegionAndEndpoint(ProcessContext context) {
final String urlstr =
StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue());
if (!urlstr.isEmpty()) {
getLogger().info("Overriding endpoint with {}", new
Object[]{urlstr});
-this.client.setEndpoint(urlstr);
+this.client.setEndpoint(urlstr,
this.client.getServiceName(), this.region.getName());
--- End diff --
Thanks @amcdonaldccri for pointing this out. I'll work on fixing it.
---
[GitHub] nifi pull request #3105: NIFI-5621: Added Cassandra connection provider serv...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3105#discussion_r241005492 --- Diff: nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-services-nar/src/main/resources/META-INF/LICENSE --- @@ -0,0 +1,369 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to
[GitHub] nifi pull request #3105: NIFI-5621: Added Cassandra connection provider serv...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3105#discussion_r240655171 --- Diff: nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-services-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,291 @@ +nifi-cassandra-services-nar +Copyright 2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +** +Apache Software License v2 +** + +The following binary components are provided under the Apache Software License v2 + + (ASLv2) DataStax Java Driver for Apache Cassandra - Core + The following NOTICE information applies: +DataStax Java Driver for Apache Cassandra - Core +Copyright (C) 2012-2017 DataStax Inc. + + (ASLv2) Jackson JSON processor +The following NOTICE information applies: + # Jackson JSON processor + + Jackson is a high-performance, Free/Open Source JSON processing library. + It was originally written by Tatu Saloranta (tatu.salora...@iki.fi), and has + been in development since 2007. + It is currently developed by a community of developers, as well as supported + commercially by FasterXML.com. + + ## Licensing + + Jackson core and extension components may licensed under different licenses. + To find the details that apply to this artifact see the accompanying LICENSE file. + For more information, including possible other licensing options, contact + FasterXML.com (http://fasterxml.com). + + ## Credits + + A list of contributors may be found from CREDITS file, which is included + in some artifacts (usually source distributions); but is always available + from the source code management (SCM) system project uses. + + (ASLv2) Apache Commons Codec +The following NOTICE information applies: + Apache Commons Codec + Copyright 2002-2014 The Apache Software Foundation + + src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java + contains test data from http://aspell.net/test/orig/batch0.tab. + Copyright (C) 2002 Kevin Atkinson (kev...@gnu.org) + + === + + The content of package org.apache.commons.codec.language.bm has been translated + from the original php source code available at http://stevemorse.org/phoneticinfo.htm + with permission from the original authors. + Original source copyright: + Copyright (c) 2008 Alexander Beider & Stephen P. Morse. + + (ASLv2) Apache Commons Lang +The following NOTICE information applies: + Apache Commons Lang + Copyright 2001-2017 The Apache Software Foundation + + This product includes software from the Spring Framework, + under the Apache License 2.0 (see: StringUtils.containsWhitespace()) + + (ASLv2) Guava +The following NOTICE information applies: + Guava + Copyright 2015 The Guava Authors + + (ASLv2) JSON-SMART +The following NOTICE information applies: +Copyright 2011 JSON-SMART authors + + (ASLv2) Dropwizard Metrics +The following NOTICE information applies: + Copyright (c) 2010-2013 Coda Hale, Yammer.com + + (ASLv2) The Netty Project + The following NOTICE information applies: + Copyright 2014 The Netty Project + --- + This product contains the extensions to Java Collections Framework which has + been derived from the works by JSR-166 EG, Doug Lea, and Jason T. Greene: + + * LICENSE: + * license/LICENSE.jsr166y.txt (Public Domain) + * HOMEPAGE: + * http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/ + * http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbosscache/experimental/jsr166/ + + This product contains a modified version of Robert Harder's Public Domain + Base64 Encoder and Decoder, which can be obtained at: + + * LICENSE: + * license/LICENSE.base64.txt (Public Domain) + * HOMEPAGE: + * http://iharder.sourceforge.net/current/java/base64/ + + This product contains a modified portion of 'Webbit', an event based + WebSocket and HTTP server, which can be obtained at: + + * LICENSE: + * license/LICENSE.webbit.txt (BSD License) + * HOMEPAGE: + * https://github.com/joewalnes/webbit + + This product contains a modified portion of 'SLF4J', a
[GitHub] nifi pull request #3105: NIFI-5621: Added Cassandra connection provider serv...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3105#discussion_r240654783 --- Diff: nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-services-nar/src/main/resources/META-INF/LICENSE --- @@ -0,0 +1,369 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to
[GitHub] nifi issue #3105: NIFI-5621: Added Cassandra connection provider service
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3105 @MikeThomsen I have worked on the LICENSE & NOTICE files as suggested. Please take a look now. ---
[GitHub] nifi pull request #3105: NIFI-5621: Added Cassandra connection provider serv...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3105#discussion_r239164057 --- Diff: nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-services-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,327 @@ +nifi-cassandra-services-nar +Copyright 2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +** +Apache Software License v2 +** + +The following binary components are provided under the Apache Software License v2 + + (ASLv2) DataStax Java Driver for Apache Cassandra - Core + The following NOTICE information applies: +DataStax Java Driver for Apache Cassandra - Core +Copyright (C) 2012-2017 DataStax Inc. + + (ASLv2) Apache Avro + The following NOTICE information applies: +Apache Avro +Copyright 2009-2017 The Apache Software Foundation + + (ASLv2) Jackson JSON processor --- End diff -- Understood. ---
[GitHub] nifi pull request #3105: NIFI-5621: Added Cassandra connection provider serv...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3105#discussion_r239163988 --- Diff: nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-services-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,327 @@ +nifi-cassandra-services-nar --- End diff -- Fixed all. Please take a look. ---
[GitHub] nifi issue #3105: NIFI-5621: Added Cassandra connection provider service
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3105 @MikeThomsen Yes, I am. ---
[GitHub] nifi issue #3190: NIFI-5850: Replaced custom AWS regions enum with the one f...
Github user zenfenan commented on the issue:
https://github.com/apache/nifi/pull/3190
>
>
> @zenfenan agree that bumping the version just before a release is a good
idea (and we should try to do it for multiple bundles) but I still suggest to
bump the version in this PR so that we can also close #3187 and allow the
community to use NiFi or the AWS NAR (built from master) with the new region.
@pvillard31 Fair point. Updated the version to the latest stable one and
confirmed the new region comes up.
---
[GitHub] nifi issue #3187: NIFI-5847 Added GovCloud-East region to enum list.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3187 @jmlogan It is an enum on the AWS SDK side so whatever new regions that are added to AWS cloud services, the team managing the SDK project update the enum on their side. ---
[GitHub] nifi pull request #3190: NIFI-5850: Replaced custom AWS regions enum with th...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/3190 NIFI-5850: Replaced custom AWS regions enum with the one from AWS Jav⦠â¦a SDK Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5850 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3190.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3190 commit 0688425f971d14e258a4955cd2d2ed7046c28466 Author: zenfenan Date: 2018-11-29T09:29:23Z NIFI-5850: Replaced custom AWS regions enum with the one from AWS Java SDK ---
[GitHub] nifi issue #3167: NIFI-5812: Marked Database processors as 'PrimaryNodeOnly'
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3167 @mattyb149 Fixed the checkstyle violation and rebased. ---
[GitHub] nifi issue #3135: [NIFI-5795] RedisDistributedMapCacheClientService put miss...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3135 @pvillard31 I wasn't able to. What do you suggest? We'll close this one and cover the objective with #3176 ? or we would merge this in once we verify the changes and ask @javajefe to rebase #3176? The latter feels like a better one.. since this is the first contribution by @luup2k for NiFi ---
[GitHub] nifi issue #3167: NIFI-5812: Marked Database processors as 'PrimaryNodeOnly'
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3167 Since there is some thought process required on how the primary node only restriction be effectively made on the `GenerateTableFetch`, I'll change this PR to only have the annotation on the `QueryDatabaseTable`. Sounds good? ---
[GitHub] nifi pull request #3011: NIFI-5333 Added GetMongoRecord.
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3011#discussion_r234443725
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongoRecord.java
---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.nifi.processors.mongodb;
+
+import com.mongodb.client.FindIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoCursor;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.mongodb.MongoDBClientService;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.RecordSetWriter;
+import org.apache.nifi.serialization.RecordSetWriterFactory;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.bson.Document;
+import org.bson.types.ObjectId;
+
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@CapabilityDescription("A record-based version of GetMongo that uses the
Record writers to write the MongoDB result set.")
+@Tags({"mongo", "mongodb", "get", "fetch", "record", "json"})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@WritesAttributes({
+@WritesAttribute(attribute = GetMongo.DB_NAME, description = "The
database where the results came from."),
+@WritesAttribute(attribute = GetMongo.COL_NAME, description = "The
collection where the results came from.")
+})
+public class GetMongoRecord extends AbstractMongoQueryProcessor {
+public static final PropertyDescriptor CLIENT = new
PropertyDescriptor.Builder()
--- End diff --
We already have the `CLIENT_SERVICE` property descriptor in the
AbstractMongo processor. This has to be removed.
---
[GitHub] nifi pull request #3167: NIFI-5812: Marked Database processors as 'PrimaryNo...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/3167 NIFI-5812: Marked Database processors as 'PrimaryNodeOnly' Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5812 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3167.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3167 commit 1d848c036437b485bac0b58c636789cfc96a8f52 Author: zenfenan Date: 2018-11-12T10:16:26Z NIFI-5812: Marked Database processors as PrimaryNodeOnly ---
[GitHub] nifi pull request #3011: NIFI-5333 Added GetMongoRecord.
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3011#discussion_r232223940 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/pom.xml --- @@ -101,6 +101,7 @@ org.apache.nifi nifi-mongodb-client-service-api +1.8.0-SNAPSHOT --- End diff -- Has to be changed. ---
[GitHub] nifi issue #3105: NIFI-5621: Added Cassandra connection provider service
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3105 @mattyb149 Rebased. ---
[GitHub] nifi issue #3105: NIFI-5621: Added Cassandra connection provider service
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3105 Note to reviewers: Please take a close look from the L&N perspective, especially the NOTICE file. I have basically re-used the NOTICE from the Cassandra processor bundle but I have few doubts which I've left as comments. @joewitt Any input on the L&N would be greatly appreciated. Thanks. ---
[GitHub] nifi pull request #3105: NIFI-5621: Added Cassandra connection provider serv...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3105#discussion_r227436737 --- Diff: nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-services-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,327 @@ +nifi-cassandra-services-nar +Copyright 2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +** +Apache Software License v2 +** + +The following binary components are provided under the Apache Software License v2 + + (ASLv2) DataStax Java Driver for Apache Cassandra - Core + The following NOTICE information applies: +DataStax Java Driver for Apache Cassandra - Core +Copyright (C) 2012-2017 DataStax Inc. + + (ASLv2) Apache Avro + The following NOTICE information applies: +Apache Avro +Copyright 2009-2017 The Apache Software Foundation + + (ASLv2) Jackson JSON processor +The following NOTICE information applies: + # Jackson JSON processor + + Jackson is a high-performance, Free/Open Source JSON processing library. + It was originally written by Tatu Saloranta (tatu.salora...@iki.fi), and has + been in development since 2007. + It is currently developed by a community of developers, as well as supported + commercially by FasterXML.com. + + ## Licensing + + Jackson core and extension components may licensed under different licenses. + To find the details that apply to this artifact see the accompanying LICENSE file. + For more information, including possible other licensing options, contact + FasterXML.com (http://fasterxml.com). + + ## Credits + + A list of contributors may be found from CREDITS file, which is included + in some artifacts (usually source distributions); but is always available + from the source code management (SCM) system project uses. + + (ASLv2) Apache Commons Codec +The following NOTICE information applies: + Apache Commons Codec + Copyright 2002-2014 The Apache Software Foundation + + src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java + contains test data from http://aspell.net/test/orig/batch0.tab. + Copyright (C) 2002 Kevin Atkinson (kev...@gnu.org) + + === + + The content of package org.apache.commons.codec.language.bm has been translated + from the original php source code available at http://stevemorse.org/phoneticinfo.htm + with permission from the original authors. + Original source copyright: + Copyright (c) 2008 Alexander Beider & Stephen P. Morse. + + (ASLv2) Apache Commons Compress +The following NOTICE information applies: + Apache Commons Compress + Copyright 2002-2017 The Apache Software Foundation + + The files in the package org.apache.commons.compress.archivers.sevenz + were derived from the LZMA SDK, version 9.20 (C/ and CPP/7zip/), + which has been placed in the public domain: + + "LZMA SDK is placed in the public domain." (http://www.7-zip.org/sdk.html) + + (ASLv2) Apache Commons IO +The following NOTICE information applies: + Apache Commons IO + Copyright 2002-2016 The Apache Software Foundation + +(ASLv2) Apache Commons Lang +The following NOTICE information applies: + Apache Commons Lang + Copyright 2001-2017 The Apache Software Foundation + + This product includes software from the Spring Framework, + under the Apache License 2.0 (see: StringUtils.containsWhitespace()) + + (ASLv2) Guava +The following NOTICE information applies: + Guava + Copyright 2015 The Guava Authors + + (ASLv2) Dropwizard Metrics --- End diff -- This one as well. ---
[GitHub] nifi pull request #3105: NIFI-5621: Added Cassandra connection provider serv...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3105#discussion_r227436607 --- Diff: nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-services-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,327 @@ +nifi-cassandra-services-nar +Copyright 2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +** +Apache Software License v2 +** + +The following binary components are provided under the Apache Software License v2 + + (ASLv2) DataStax Java Driver for Apache Cassandra - Core + The following NOTICE information applies: +DataStax Java Driver for Apache Cassandra - Core +Copyright (C) 2012-2017 DataStax Inc. + + (ASLv2) Apache Avro + The following NOTICE information applies: +Apache Avro +Copyright 2009-2017 The Apache Software Foundation + + (ASLv2) Jackson JSON processor --- End diff -- Same case here. I don't think this dependency is used anywhere in the CS' POM. ---
[GitHub] nifi pull request #3105: NIFI-5621: Added Cassandra connection provider serv...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/3105#discussion_r227436263 --- Diff: nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-services-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,327 @@ +nifi-cassandra-services-nar +Copyright 2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +** +Apache Software License v2 +** + +The following binary components are provided under the Apache Software License v2 + + (ASLv2) DataStax Java Driver for Apache Cassandra - Core + The following NOTICE information applies: +DataStax Java Driver for Apache Cassandra - Core +Copyright (C) 2012-2017 DataStax Inc. + + (ASLv2) Apache Avro --- End diff -- Should this be referenced here? The controller service never uses any Avro dependency. ---
[GitHub] nifi pull request #3105: NIFI-5621: Added Cassandra connection provider serv...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/3105 NIFI-5621: Added Cassandra connection provider service Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5621 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3105.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3105 commit 5743140a28374e51c045df085e31ad0a6c7f6565 Author: zenfenan Date: 2018-10-21T17:41:07Z NIFI-5621: Added Cassandra connection provider service ---
[GitHub] nifi issue #3073: NIFI-5698: Fixed DeleteAzureBlobStorage bug
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3073 > > > Confirmed that the reported issue happened before applying fix. And resolved with this fix. +1, merging. Thanks again @zenfenan! Sorry for introducing this terrible bug.. @ijokarumawak No issues. Thanks for reviewing and merging. Appreciate the quick turnaround. :) ---
[GitHub] nifi pull request #3073: NIFI-5698: Fixed DeleteAzureBlobStorage bug
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/3073 NIFI-5698: Fixed DeleteAzureBlobStorage bug Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5698 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3073.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3073 commit 72530cefd81545ee86d4fdc0d5fcc1eebc77d755 Author: zenfenan Date: 2018-10-14T07:48:25Z NIFI-5698: Fixed DeleteAzureBlobStorage bug ---
[GitHub] nifi issue #3011: NIFI-5333 Added GetMongoRecord.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3011 @MikeThomsen Will try to do by this weekend. ---
[GitHub] nifi issue #3025: NIFI-5605 Added UpdateCassandra.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3025 I think this can go in with `PutCassandraQL`. `PutCassandraRecord` actually performs inserts only. ---
[GitHub] nifi issue #2896: NIFI-5239 Made a client service an optional source of conn...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2896 +1 from me. Thanks for the contribution. Merged to the master. ---
[GitHub] nifi issue #2992: NIFI-5510: Introducing PutCassandraRecord processor
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2992 I think we can merge this one, if everything checks out good instead of waiting. ---
[GitHub] nifi issue #2992: NIFI-5510: Introducing PutCassandraRecord processor
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2992 Yeah, I have actually started the implementation for a connection pool service. Wasn't able to open up a Jira. Got held up with some other things. I'll be doing it and will also wrap up the implementation and raise a PR. ---
[GitHub] nifi pull request #2992: NIFI-5510: Introducing PutCassandraRecord processor
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2992#discussion_r217948182
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/PutCassandraRecord.java
---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.cassandra;
+
+import com.datastax.driver.core.BatchStatement;
+import com.datastax.driver.core.ConsistencyLevel;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.exceptions.AuthenticationException;
+import com.datastax.driver.core.exceptions.NoHostAvailableException;
+import com.datastax.driver.core.querybuilder.Insert;
+import com.datastax.driver.core.querybuilder.QueryBuilder;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnShutdown;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+import org.apache.nifi.util.StopWatch;
+
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@Tags({"cassandra", "cql", "put", "insert", "update", "set", "record"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Writes the content of the incoming FlowFile as
individual records to Apache Cassandra using native protocol version 3 or
higher.")
+public class PutCassandraRecord extends AbstractCassandraProcessor {
+
+static final PropertyDescriptor RECORD_READER_FACTORY = new
PropertyDescriptor.Builder()
+.name("put-cassandra-record-reader")
+.displayName("Record Reader")
+.description("Specifies the type of Record Reader controller
service to use for parsing the incoming data " +
+"and determining the schema")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor TABLE = new
PropertyDescriptor.Builder()
+.name("put-cassandra-record-table")
+.displayName("Table name")
+.description("The name of the Cassandra table to which the
records have to be written.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_EL_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_
[GitHub] nifi pull request #2992: NIFI-5510: Introducing PutCassandraRecord processor
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2992#discussion_r217948138
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/PutCassandraRecord.java
---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.cassandra;
+
+import com.datastax.driver.core.BatchStatement;
+import com.datastax.driver.core.ConsistencyLevel;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.exceptions.AuthenticationException;
+import com.datastax.driver.core.exceptions.NoHostAvailableException;
+import com.datastax.driver.core.querybuilder.Insert;
+import com.datastax.driver.core.querybuilder.QueryBuilder;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnShutdown;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+import org.apache.nifi.util.StopWatch;
+
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@Tags({"cassandra", "cql", "put", "insert", "update", "set", "record"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Writes the content of the incoming FlowFile as
individual records to Apache Cassandra using native protocol version 3 or
higher.")
+public class PutCassandraRecord extends AbstractCassandraProcessor {
+
+static final PropertyDescriptor RECORD_READER_FACTORY = new
PropertyDescriptor.Builder()
+.name("put-cassandra-record-reader")
+.displayName("Record Reader")
+.description("Specifies the type of Record Reader controller
service to use for parsing the incoming data " +
+"and determining the schema")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor TABLE = new
PropertyDescriptor.Builder()
+.name("put-cassandra-record-table")
+.displayName("Table name")
+.description("The name of the Cassandra table to which the
records have to be written.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_EL_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_
[GitHub] nifi pull request #2992: NIFI-5510: Introducing PutCassandraRecord processor
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2992#discussion_r217947988
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/PutCassandraRecord.java
---
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.cassandra;
+
+import com.datastax.driver.core.BatchStatement;
+import com.datastax.driver.core.ConsistencyLevel;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.exceptions.AuthenticationException;
+import com.datastax.driver.core.exceptions.NoHostAvailableException;
+import com.datastax.driver.core.querybuilder.Insert;
+import com.datastax.driver.core.querybuilder.QueryBuilder;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnShutdown;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+import org.apache.nifi.util.StopWatch;
+
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@Tags({"cassandra", "cql", "put", "insert", "update", "set", "record"})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Writes the content of the incoming FlowFile as
individual records to Apache Cassandra using native protocol version 3 or
higher.")
--- End diff --
Understood. I'll update it.
---
[GitHub] nifi pull request #2896: NIFI-5239 Made a client service an optional source ...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2896#discussion_r217945557
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-services/src/main/java/org/apache/nifi/mongodb/MongoDBControllerService.java
---
@@ -17,156 +17,162 @@
package org.apache.nifi.mongodb;
-import com.mongodb.client.FindIterable;
+import com.mongodb.MongoClient;
+import com.mongodb.MongoClientOptions;
+import com.mongodb.MongoClientURI;
+import com.mongodb.WriteConcern;
import com.mongodb.client.MongoCollection;
-import com.mongodb.client.MongoCursor;
import com.mongodb.client.MongoDatabase;
-import com.mongodb.client.model.UpdateOptions;
-
+import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.Tags;
import org.apache.nifi.annotation.lifecycle.OnDisabled;
import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.authentication.exception.ProviderCreationException;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
import org.apache.nifi.controller.ConfigurationContext;
-import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.security.util.SslContextFactory;
+import org.apache.nifi.ssl.SSLContextService;
import org.bson.Document;
-import java.io.IOException;
+import javax.net.ssl.SSLContext;
import java.util.ArrayList;
import java.util.List;
@Tags({"mongo", "mongodb", "service"})
@CapabilityDescription(
"Provides a controller service that wraps most of the functionality of
the MongoDB driver."
)
-public class MongoDBControllerService extends
AbstractMongoDBControllerService implements MongoDBClientService {
+public class MongoDBControllerService extends AbstractControllerService
implements MongoDBClientService {
--- End diff --
If possible, I would be +1 for a controller service with a better name. I'm
not saying `MongoDBControllerService` is a bad name but a name that rhymes with
functionality that this controller service offers would sound even better.
something like `MongoDBClientProvider` or something.
---
[GitHub] nifi pull request #2896: NIFI-5239 Made a client service an optional source ...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2896#discussion_r217945280
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-services/src/main/java/org/apache/nifi/mongodb/MongoDBControllerService.java
---
@@ -17,156 +17,162 @@
package org.apache.nifi.mongodb;
-import com.mongodb.client.FindIterable;
+import com.mongodb.MongoClient;
+import com.mongodb.MongoClientOptions;
+import com.mongodb.MongoClientURI;
+import com.mongodb.WriteConcern;
import com.mongodb.client.MongoCollection;
-import com.mongodb.client.MongoCursor;
import com.mongodb.client.MongoDatabase;
-import com.mongodb.client.model.UpdateOptions;
-
+import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.Tags;
import org.apache.nifi.annotation.lifecycle.OnDisabled;
import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.authentication.exception.ProviderCreationException;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
import org.apache.nifi.controller.ConfigurationContext;
-import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.security.util.SslContextFactory;
+import org.apache.nifi.ssl.SSLContextService;
import org.bson.Document;
-import java.io.IOException;
+import javax.net.ssl.SSLContext;
import java.util.ArrayList;
import java.util.List;
@Tags({"mongo", "mongodb", "service"})
@CapabilityDescription(
"Provides a controller service that wraps most of the functionality of
the MongoDB driver."
)
-public class MongoDBControllerService extends
AbstractMongoDBControllerService implements MongoDBClientService {
+public class MongoDBControllerService extends AbstractControllerService
implements MongoDBClientService {
private MongoDatabase db;
private MongoCollection col;
+private String uri;
@OnEnabled
-public void onEnabled(final ConfigurationContext context) throws
InitializationException, IOException, InterruptedException {
+public void onEnabled(final ConfigurationContext context) {
+this.uri =
context.getProperty(URI).evaluateAttributeExpressions().getValue();
this.createClient(context);
-this.db =
this.mongoClient.getDatabase(context.getProperty(MongoDBControllerService.DATABASE_NAME).getValue());
-this.col =
this.db.getCollection(context.getProperty(MongoDBControllerService.COLLECTION_NAME).getValue());
-}
-
-@OnDisabled
-public void onDisable() {
-this.mongoClient.close();
-}
-
-@Override
-public long count(Document query) {
-return this.col.count(query);
-}
-
-@Override
-public void delete(Document query) {
-this.col.deleteMany(query);
}
-@Override
-public boolean exists(Document query) {
-return this.col.count(query) > 0;
-}
-
-@Override
-public Document findOne(Document query) {
-MongoCursor cursor =
this.col.find(query).limit(1).iterator();
-Document retVal = cursor.tryNext();
-cursor.close();
-
-return retVal;
-}
+static List descriptors = new ArrayList<>();
-@Override
-public Document findOne(Document query, Document projection) {
-MongoCursor cursor = projection != null
-?
this.col.find(query).projection(projection).limit(1).iterator()
-: this.col.find(query).limit(1).iterator();
-Document retVal = cursor.tryNext();
-cursor.close();
-
-return retVal;
-}
-
-@Override
-public List findMany(Document query) {
-return findMany(query, null, -1);
+static {
+descriptors.add(URI);
+descriptors.add(SSL_CONTEXT_SERVICE);
--- End diff --
Why are we only exposing `URI` alone? I thought this PR intends to offer a
controller service where we can configure a connection to a MongoDB database &
collection and use that controller service optionally instead of the processor
level `URI`, `Database Name`, `Collection Name`.
---
[GitHub] nifi pull request #2896: NIFI-5239 Made a client service an optional source ...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2896#discussion_r217945056
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-services/src/main/java/org/apache/nifi/mongodb/MongoDBControllerService.java
---
@@ -17,156 +17,162 @@
package org.apache.nifi.mongodb;
-import com.mongodb.client.FindIterable;
+import com.mongodb.MongoClient;
+import com.mongodb.MongoClientOptions;
+import com.mongodb.MongoClientURI;
+import com.mongodb.WriteConcern;
import com.mongodb.client.MongoCollection;
-import com.mongodb.client.MongoCursor;
import com.mongodb.client.MongoDatabase;
-import com.mongodb.client.model.UpdateOptions;
-
+import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.Tags;
import org.apache.nifi.annotation.lifecycle.OnDisabled;
import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.authentication.exception.ProviderCreationException;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
import org.apache.nifi.controller.ConfigurationContext;
-import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.security.util.SslContextFactory;
+import org.apache.nifi.ssl.SSLContextService;
import org.bson.Document;
-import java.io.IOException;
+import javax.net.ssl.SSLContext;
import java.util.ArrayList;
import java.util.List;
@Tags({"mongo", "mongodb", "service"})
@CapabilityDescription(
"Provides a controller service that wraps most of the functionality of
the MongoDB driver."
--- End diff --
`CapabilityDescription` has to be updated.
---
[GitHub] nifi issue #2993: NIFI-5333 Added GetMongoRecord.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2993 Will try to complete it by this weekend. Extremely sorry.. Got super busy for the past few days. ---
[GitHub] nifi issue #2992: NIFI-5510: Introducing PutCassandraRecord processor
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2992 @MikeThomsen No complex setup is required. A Cassandra cluster has to be up and running. That's all. ---
[GitHub] nifi pull request #2992: NIFI-5510: Introducing PutCassandraRecord processor
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2992 NIFI-5510: Introducing PutCassandraRecord processor Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5510 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2992.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2992 commit 4934a159fb769961c47d3809b522dfaa197fc1ac Author: zenfenan Date: 2018-08-17T09:56:47Z NIFI-5510: Introducing PutCassandraRecord processor ---
[GitHub] nifi pull request #2968: NIFI-5456: AWS clients now work with private link e...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2968#discussion_r215131280
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
---
@@ -286,7 +286,7 @@ protected void
initializeRegionAndEndpoint(ProcessContext context) {
final String urlstr =
StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue());
if (!urlstr.isEmpty()) {
getLogger().info("Overriding endpoint with {}", new
Object[]{urlstr});
-this.client.setEndpoint(urlstr);
+this.client.setEndpoint(urlstr,
this.client.getServiceName(), this.region.getName());
--- End diff --
@joewitt The patch has been tested by @Mermadi and it is working fine. Can
we close this one?
---
[GitHub] nifi issue #2968: NIFI-5456: AWS clients now work with private link endpoint...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2968 @Mermadi Had a chance to test it out? ---
[GitHub] nifi pull request #2896: NIFI-5239 Made a client service an optional source ...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2896#discussion_r214684778
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -244,22 +266,20 @@ public final void closeClient() {
}
}
-protected MongoDatabase getDatabase(final ProcessContext context) {
-return getDatabase(context, null);
-}
+//protected MongoDatabase getDatabase(final ProcessContext context) {
+//return getDatabase(context, null);
+//}
protected MongoDatabase getDatabase(final ProcessContext context,
final FlowFile flowFile) {
final String databaseName =
context.getProperty(DATABASE_NAME).evaluateAttributeExpressions(flowFile).getValue();
-if (StringUtils.isEmpty(databaseName)) {
-throw new ProcessException("Database name was empty after
expression language evaluation.");
-}
-return mongoClient.getDatabase(databaseName);
-}
-protected MongoCollection getCollection(final ProcessContext
context) {
-return getCollection(context, null);
+return clientService!= null ?
clientService.getDatabase(databaseName) : mongoClient.getDatabase(databaseName);
}
+//protected MongoCollection getCollection(final
ProcessContext context) {
+//return getCollection(context, null);
+//}
+
--- End diff --
Same here.
---
[GitHub] nifi pull request #2896: NIFI-5239 Made a client service an optional source ...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2896#discussion_r214684734
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -244,22 +266,20 @@ public final void closeClient() {
}
}
-protected MongoDatabase getDatabase(final ProcessContext context) {
-return getDatabase(context, null);
-}
+//protected MongoDatabase getDatabase(final ProcessContext context) {
+//return getDatabase(context, null);
+//}
--- End diff --
Comments to be removed.
---
[GitHub] nifi pull request #2958: NIFI-5544: GetMongo refactored
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2958#discussion_r214022342
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongo.java
---
@@ -204,144 +212,145 @@ private ObjectWriter getObjectWriter(ObjectMapper
mapper, String ppSetting) {
@Override
public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
FlowFile input = null;
+logger = getLogger();
+
if (context.hasIncomingConnection()) {
input = session.get();
-
if (input == null && context.hasNonLoopConnection()) {
return;
}
}
-final ComponentLog logger = getLogger();
+final Document query = getQuery(context, session, input );
-Map attributes = new HashMap<>();
-attributes.put(CoreAttributes.MIME_TYPE.key(), "application/json");
+if (query == null) {
+return;
+}
-final Document query;
+final String jsonTypeSetting =
context.getProperty(JSON_TYPE).getValue();
+final String usePrettyPrint =
context.getProperty(USE_PRETTY_PRINTING).getValue();
final Charset charset =
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(input).getValue());
+final Map attributes = new HashMap<>();
-String queryStr;
-if (context.getProperty(QUERY).isSet()) {
-queryStr =
context.getProperty(QUERY).evaluateAttributeExpressions(input).getValue();
-query = Document.parse(queryStr);
-} else if (!context.getProperty(QUERY).isSet() && input == null) {
-queryStr = "{}";
-query = Document.parse("{}");
-} else {
-try {
-ByteArrayOutputStream out = new ByteArrayOutputStream();
-session.exportTo(input, out);
-out.close();
-queryStr = new String(out.toByteArray());
-query = Document.parse(queryStr);
-} catch (Exception ex) {
-getLogger().error("Error reading flowfile", ex);
-if (input != null) { //Likely culprit is a bad query
-session.transfer(input, REL_FAILURE);
-return;
-} else {
-throw new ProcessException(ex);
-}
-}
-}
+attributes.put(CoreAttributes.MIME_TYPE.key(), "application/json");
if (context.getProperty(QUERY_ATTRIBUTE).isSet()) {
final String queryAttr =
context.getProperty(QUERY_ATTRIBUTE).evaluateAttributeExpressions(input).getValue();
-attributes.put(queryAttr, queryStr);
+attributes.put(queryAttr, query.toJson());
}
final Document projection = context.getProperty(PROJECTION).isSet()
?
Document.parse(context.getProperty(PROJECTION).evaluateAttributeExpressions(input).getValue())
: null;
final Document sort = context.getProperty(SORT).isSet()
?
Document.parse(context.getProperty(SORT).evaluateAttributeExpressions(input).getValue())
: null;
-final String jsonTypeSetting =
context.getProperty(JSON_TYPE).getValue();
-final String usePrettyPrint =
context.getProperty(USE_PRETTY_PRINTING).getValue();
-configureMapper(jsonTypeSetting);
+final MongoCollection collection =
getCollection(context, input);
+final FindIterable it = collection.find(query);
-try {
-final MongoCollection collection =
getCollection(context, input);
+attributes.put(DB_NAME,
collection.getNamespace().getDatabaseName());
+attributes.put(COL_NAME,
collection.getNamespace().getCollectionName());
-attributes.put(DB_NAME,
collection.getNamespace().getDatabaseName());
-attributes.put(COL_NAME,
collection.getNamespace().getCollectionName());
+if (projection != null) {
+it.projection(projection);
+}
+if (sort != null) {
+it.sort(sort);
+}
+if (context.getProperty(LIMIT).isSet()) {
+
it.limit(context.getProperty(LIMIT).evaluateAttributeExpressions(input).asInteger());
+}
+if (context.getProperty(BATCH_SIZE).isSet()) {
+
it.batchSize(context.getProperty(BATCH_SIZE).evaluateAttributeExpressions(input).asI
[GitHub] nifi pull request #2968: NIFI-5456: AWS clients now work with private link e...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2968#discussion_r213901647
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
---
@@ -286,7 +286,7 @@ protected void
initializeRegionAndEndpoint(ProcessContext context) {
final String urlstr =
StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue());
if (!urlstr.isEmpty()) {
getLogger().info("Overriding endpoint with {}", new
Object[]{urlstr});
-this.client.setEndpoint(urlstr);
+this.client.setEndpoint(urlstr,
this.client.getServiceName(), this.region.getName());
--- End diff --
Yep. They will be present always. We have made `Region` a `required`
parameter so it will be there. The service name is provided by the AWSClient
implementation in the AWS SDK. Gone through their implementation and verified
that, it takes care of undefined and null service names i.e, it computes the
service name again and returns, if it turns out to be null.
---
[GitHub] nifi pull request #2968: NIFI-5456: AWS clients now work with private link e...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2968 NIFI-5456: AWS clients now work with private link endpoints (VPC) Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5456 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2968.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2968 commit 7b390a9d0b003301f22a89cecf07a622bbc78f5e Author: zenfenan Date: 2018-08-26T13:36:45Z NIFI-5456: AWS clients now work with private link endpoints (VPC) ---
[GitHub] nifi pull request #2946: NIFI-5494: Added ExpressionLanguage support to 'Hiv...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2946#discussion_r211700721 --- Diff: nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/hive/PutHive3Streaming.java --- @@ -133,6 +133,7 @@ + "Please see the Hive documentation for more details.") .required(false) .addValidator(HiveUtils.createMultipleFilesExistValidator()) + .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) --- End diff -- It's fixed now. Appreciate if you could take a look. ---
[GitHub] nifi pull request #2958: NIFI-5544: GetMongo refactored
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2958 NIFI-5544: GetMongo refactored Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5544 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2958.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2958 commit 9477bfd7b65a943b4faced2259131c59c0051a4b Author: zenfenan Date: 2018-08-20T11:06:25Z NIFI-5544: GetMongo refactored ---
[GitHub] nifi pull request #2946: NIFI-5494: Added ExpressionLanguage support to 'Hiv...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2946#discussion_r209458600 --- Diff: nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/PutHiveStreaming.java --- @@ -182,6 +182,7 @@ + "Please see the Hive documentation for more details.") .required(false) .addValidator(HiveUtils.createMultipleFilesExistValidator()) + .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) --- End diff -- Oops.. Extremely sorry.. My mistake. Cherry-picked the wrong commit. Let me fix this real quick. ---
[GitHub] nifi pull request #2946: NIFI-5494: Added ExpressionLanguage support to 'Hiv...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2946 NIFI-5494: Added ExpressionLanguage support to 'Hive Configuration Re⦠â¦sources' Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI5494CLONE Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2946.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2946 commit 3688fead662d65826a36c820a1c5064324ec9bf0 Author: zenfenan Date: 2018-08-12T14:53:02Z NIFI-5494: Added ExpressionLanguage support to 'Hive Configuration Resources' ---
[GitHub] nifi pull request #2925: NIFI-5469 Additional italics and code formatting co...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2925#discussion_r209010726
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -3256,9 +3361,9 @@ stream {
image:s2s-rproxy-portnumber.svg["Port number to Node mapping"]
--- End diff --
Got it but I still prefer the image being loaded on the GitHub md files
though. Anyways, I'm good. I'll go ahead and merge it once the broken link for
BCrypt is fixed.
---
[GitHub] nifi issue #2914: NIFI-5448 Added failure relationship to UpdateAttributes t...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2914 +1 Built it and verified the changes with a sample flow. Everything works as expected. Thanks @pvillard31 and @mcgilman for the reviews. Thanks @MikeThomsen. Merged to the master. ---
[GitHub] nifi issue #2937: NIFI-4434 Fixed recursive listing with a custom regex filt...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2937 +1 Thanks @jtstorck. Merged to the master. Thanks @ottobackwards for reviewing. ---
[GitHub] nifi issue #2914: NIFI-5448 Added failure relationship to UpdateAttributes t...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2914 @MikeThomsen The suggestion made by @mcgilman sounds good to me. If we could rename `Stop Processing` to `Penalize`, that would make more sense. Thoughts? ---
[GitHub] nifi pull request #2925: NIFI-5469 Additional italics and code formatting co...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2925#discussion_r208232010
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -3256,9 +3361,9 @@ stream {
image:s2s-rproxy-portnumber.svg["Port number to Node mapping"]
--- End diff --
Does the screenshots load for you? It doesn't for me. Tried across browsers
and also confirmed that the URL is different.
---
[GitHub] nifi pull request #2925: NIFI-5469 Additional italics and code formatting co...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2925#discussion_r207975946 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -1486,35 +1588,35 @@ If no administrator action is taken, the configuration values remain unencrypted [[encrypt-config_tool]] === Encrypt-Config Tool -The `encrypt-config` command line tool (invoked as `./bin/encrypt-config.sh` or `bin\encrypt-config.bat`) reads from a 'nifi.properties' file with plaintext sensitive configuration values, prompts for a master password or raw hexadecimal key, and encrypts each value. It replaces the plain values with the protected value in the same file, or writes to a new 'nifi.properties' file if specified. +The `encrypt-config` command line tool (invoked as `./bin/encrypt-config.sh` or `bin\encrypt-config.bat`) reads from a _nifi.properties_ file with plaintext sensitive configuration values, prompts for a master password or raw hexadecimal key, and encrypts each value. It replaces the plain values with the protected value in the same file, or writes to a new _nifi.properties_ file if specified. --- End diff -- L1454 The link for `Bcrypt Spring Security` is broken. It has to be updated to https://docs.spring.io/spring-security/site/docs/current/api/org/springframework/security/crypto/bcrypt/BCrypt.html ---
[GitHub] nifi pull request #2925: NIFI-5469 Additional italics and code formatting co...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2925#discussion_r207899982
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -3256,9 +3361,9 @@ stream {
image:s2s-rproxy-portnumber.svg["Port number to Node mapping"]
--- End diff --
I think this is a good opportunity to fix the screenshots as well. They
aren't loading properly. The URL has changed, all the image URLs have to be
changed from `src/main/asciidoc/` to
`src/main/asciidoc/images/`
---
[GitHub] nifi pull request #2924: NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS proce...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2924 NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5450 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2924.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2924 commit 7ec2ac467f6a114037f4e2bfdd5df201cf2dd8d8 Author: zenfenan Date: 2018-07-30T17:48:37Z NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS processors ---
[GitHub] nifi issue #2906: NIFI-5440 Added db and collection attributes to GetMongo o...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2906 Reviewing.. ---
[GitHub] nifi pull request #2881: NIFI-5413 Added additional documentation for PutMon...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2881#discussion_r205535512 --- Diff: nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/resources/docs/org.apache.nifi.processors.mongodb.PutMongoRecord/additionalDetails.html --- @@ -0,0 +1,37 @@ + + + + + +PutMongoRecord + + + + + +Description: + +This processor is a record-aware processor for inserting data into MongoDB. It uses a configured record reader and +schema to read an incoming record set from the body of a flowfile and then inserts batches of those records into +a configured MongoDB collection. + + +This processor does not support updates, deletes or upserts. The number of documents to insert at a time is controlled +by the Insert Batch Size configuration property. This value should be set to a reasonable size to ensure --- End diff -- Nitpick: Since these are all text and we are not actually introducing any cool HTML formatting, would it not be enough if we just add it to the processor's `@CapabilityDescription` itself? ---
[GitHub] nifi issue #2881: NIFI-5413 Added additional documentation for PutMongoRecor...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2881 Reviewing.. ---
[GitHub] nifi issue #2913: NIFI-5083: enabled EL support for processor configuration ...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 @snagafritz Hey.. I forgot to close this PR while merging. It is now merged to the master. Can you manually close this PR? ---
[GitHub] nifi issue #2913: NIFI-5083: enabled EL support for processor configuration ...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 +1 LGTM. Merged to master. Thanks @snagafritz ---
[GitHub] nifi issue #2913: NIFI-5083: enabled EL support for processor configuration ...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 Reviewing.. ---
[GitHub] nifi pull request #2682: NIFI-4731: BQ Processors and GCP library update.
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2682#discussion_r204829727
--- Diff:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/BigQueryAttributes.java
---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.bigquery;
+
+/**
+ * Attributes associated with the BigQuery processors
+ */
+public class BigQueryAttributes {
+private BigQueryAttributes() {}
+
+public static final String DATASET_ATTR = "bq.dataset";
--- End diff --
It's purely a nitpick. Feel free to ignore. I just wanted to follow the
same approach that many cloud processor bundles use.
---
[GitHub] nifi issue #2619: NIFI-5059 Updated MongoDBLookupService to be able to detec...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2619 @MikeThomsen I'm actually traveling with limited access to mails and internet. I'll try to take a look as soon as I can, if someone doesn't get to already. ---
[GitHub] nifi issue #2809: NIFI-5316 Fixed array handling for Avro that comes from Pa...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2809 Forgot to add the 'This closes #PR' in the commit message so it is still shown as open. :| ---
[GitHub] nifi issue #2809: NIFI-5316 Fixed array handling for Avro that comes from Pa...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2809 Thanks @bbende Merged to master ---
[GitHub] nifi pull request #2818: NIFI-5343: Fix GetMongo to properly read charset fr...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2818 NIFI-5343: Fix GetMongo to properly read charset from FlowFile attri⦠â¦butes Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5343 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2818.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2818 commit 776e7df9fb971ffeb29c3539df3695cb4ca19d41 Author: zenfenan Date: 2018-06-27T10:00:11Z NIFI-5343: Fix GetMongo to properly reads charset from FlowFile attributes ---
[GitHub] nifi issue #2813: NIFI-5084 Added GenerateRecord processor.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2813 Sure @MikeThomsen I'll take a look ---
[GitHub] nifi pull request #2815: NIFI-5334: GetMongo keeps original attributes when ...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2815 NIFI-5334: GetMongo keeps original attributes when it has incoming co⦠â¦nnections Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5334 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2815.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2815 commit f554c143653f29ccb382fff5452685d51b47dff3 Author: zenfenan Date: 2018-06-26T14:31:59Z NIFI-5334: GetMongo keeps original attributes when it has incoming connections ---
[GitHub] nifi pull request #2808: NIFI-5332: Fix GetMongoIT test failures
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2808 NIFI-5332: Fix GetMongoIT test failures Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5332 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2808.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2808 commit 34511bc52fbf8915f38823921668f01ab11d7c8f Author: zenfenan Date: 2018-06-23T08:47:58Z NIFI-5332: Fix GetMongoIT test failures ---
[GitHub] nifi pull request #2760: NIFI-5266: Sanitize ES parameters in PutElasticsear...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2760#discussion_r196073711
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java
---
@@ -288,42 +290,23 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
session.read(file, in -> {
json.append(IOUtils.toString(in, charset).replace("\r\n",
" ").replace('\n', ' ').replace('\r', ' '));
});
-if (indexOp.equalsIgnoreCase("index")) {
-sb.append("{\"index\": { \"_index\": \"");
-sb.append(index);
-sb.append("\", \"_type\": \"");
-sb.append(docType);
-sb.append("\"");
-if (!StringUtils.isEmpty(id)) {
-sb.append(", \"_id\": \"");
-sb.append(id);
-sb.append("\"");
-}
-sb.append("}}\n");
-sb.append(json);
-sb.append("\n");
-} else if (indexOp.equalsIgnoreCase("upsert") ||
indexOp.equalsIgnoreCase("update")) {
-sb.append("{\"update\": { \"_index\": \"");
-sb.append(index);
-sb.append("\", \"_type\": \"");
-sb.append(docType);
-sb.append("\", \"_id\": \"");
-sb.append(id);
-sb.append("\" }\n");
-sb.append("{\"doc\": ");
-sb.append(json);
-sb.append(", \"doc_as_upsert\": ");
-sb.append(indexOp.equalsIgnoreCase("upsert"));
-sb.append(" }\n");
-} else if (indexOp.equalsIgnoreCase("delete")) {
-sb.append("{\"delete\": { \"_index\": \"");
-sb.append(index);
-sb.append("\", \"_type\": \"");
-sb.append(docType);
-sb.append("\", \"_id\": \"");
-sb.append(id);
-sb.append("\" }\n");
+
+String jsonString = json.toString();
+
+// Ensure the JSON body is well-formed
+try {
--- End diff --
@ottobackwards @mattyb149 As part of NIFI-5261 and NIFI-5271 introduced a
JSON validator. I'm taking this as an opportunity to share it with you guys and
whoever takes a look at this comment. It could be of use here and in your
future contributions.
---
[GitHub] nifi issue #2558: NIFI-4982 - Add a DistributedMapCacheLookupService
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2558 Thanks @ijokarumawak and @mattyb149 for the review. Thanks @pvillard31 Everything looks good. Merged to master. ---
[GitHub] nifi pull request #2558: NIFI-4982 - Add a DistributedMapCacheLookupService
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2558#discussion_r195933469
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-lookup-services-bundle/nifi-lookup-services/src/main/java/org/apache/nifi/lookup/DistributedMapCacheLookupService.java
---
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.lookup;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.distributed.cache.client.Deserializer;
+import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient;
+import org.apache.nifi.distributed.cache.client.Serializer;
+import
org.apache.nifi.distributed.cache.client.exception.DeserializationException;
+import
org.apache.nifi.distributed.cache.client.exception.SerializationException;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+
+@Tags({"lookup", "enrich", "key", "value", "map", "cache", "distributed"})
+@CapabilityDescription("Allows to choose a distributed map cache client to
retrieve the value associated to a key. "
++ "The coordinates that are passed to the lookup must contain the key
'key'.")
+public class DistributedMapCacheLookupService extends
AbstractControllerService implements StringLookupService {
+
+private static final List STANDARD_CHARSETS = Arrays.asList(
+StandardCharsets.UTF_8,
+StandardCharsets.US_ASCII,
+StandardCharsets.ISO_8859_1,
+StandardCharsets.UTF_16,
+StandardCharsets.UTF_16LE,
+StandardCharsets.UTF_16BE);
+
+private static final String KEY = "key";
+private static final Set REQUIRED_KEYS =
Stream.of(KEY).collect(Collectors.toSet());
+
+private volatile DistributedMapCacheClient cache;
+private volatile static Charset charset;
+private final Serializer keySerializer = new
StringSerializer();
+private final Deserializer valueDeserializer = new
StringDeserializer();
+
+public static final PropertyDescriptor PROP_DISTRIBUTED_CACHE_SERVICE
= new PropertyDescriptor.Builder()
+.name("distributed-map-cache-service")
+.displayName("Distributed Cache Service")
+.description("The Controller Service that is used to get the
cached values.")
+.required(true)
+.identifiesControllerService(DistributedMapCacheClient.class)
+.build();
+
+public static final PropertyDescriptor CHARACTER_ENCODING = new
PropertyDescriptor.Builder()
+.name("character-encoding")
+.displayName("Character Encoding")
+.description("Specifies a character encoding to use.")
+.required(true)
+.allowableValues(getStandardCharsetNames())
+.defaultValue(StandardCharsets.UTF_8.displayName())
+.build();
+
+private static Set getStandardCharsetNames() {
+return STANDARD_CHARSETS.
[GitHub] nifi issue #2751: NIFI-5221: Added 'Object Tagging' functionalities to S3 Pr...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2751 Actually after giving it some serious thought, the attribute approach does make more sense. Easy from the flow development perspective, and even when using ExecuteScript or a custom processor to parse and process content to create appropriate tags, the developer can still add them to FlowFile attributes. So Iâll update this one to scrap the JSON approach and instead take a comma separated list of tag prefixes which will be taken and scanned against the incoming FlowFilesâ attributes to generate the tags. Makes sense? ---
[GitHub] nifi pull request #2751: NIFI-5221: Added 'Object Tagging' functionalities t...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2751#discussion_r194756360 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java --- @@ -205,11 +210,21 @@ .defaultValue(NO_SERVER_SIDE_ENCRYPTION) .build(); +public static final PropertyDescriptor OBJECT_TAGS = new PropertyDescriptor.Builder() --- End diff -- The keys not necessarily be the same, right? It can change since the property supports expression language so the key-values can be dynamically generated and substituted using expression language. Having said that, I do understand your point. My only thought is that when the developer is assigning tags to the FlowFiles, he/she would be using either `UpdateAttribute` or more preferably `ExecuteScript` to do some basic processing and assign tags according to some conditions. Correct? In that case, I honestly don't see the regular expression approach adding more value than the JSON approach. They seem more or less the same. I mean, if the developer is using `ExecuteScript` to generate the tags and add it to FlowFile attributes, he/she might as well generate the entire JSON and assign it to a FlowFile attribute and that FlowFile attribute can be given as the property value here and the substitution can be done. Makes sense? ---
[GitHub] nifi pull request #2751: NIFI-5221: Added 'Object Tagging' functionalities t...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2751#discussion_r194751872
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/ListS3.java
---
@@ -307,6 +328,20 @@ private boolean commit(final ProcessContext context,
final ProcessSession sessio
return willCommit;
}
+private Map writeObjectTags(AmazonS3 client,
S3VersionSummary versionSummary) {
+final GetObjectTaggingResult taggingResult =
client.getObjectTagging(new
GetObjectTaggingRequest(versionSummary.getBucketName(),
versionSummary.getKey()));
--- End diff --
I agree. I'll change the default value to `false`.
---
[GitHub] nifi pull request #2747: NIFI-5249 Dockerfile enhancements
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2747#discussion_r194404103
--- Diff: nifi-docker/dockermaven/pom.xml ---
@@ -44,7 +44,6 @@
apache/nifi
${project.version}
-latest
--- End diff --
Understood. Thanks.
---
[GitHub] nifi pull request #2783: NIFI-5293 - add merge.uuid to MergeRecord processor
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2783#discussion_r194399030 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/MergeRecord.java --- @@ -89,6 +89,7 @@ @WritesAttribute(attribute = "merge.count", description = "The number of FlowFiles that were merged into this bundle"), @WritesAttribute(attribute = "merge.bin.age", description = "The age of the bin, in milliseconds, when it was merged and output. Effectively " + "this is the greatest amount of time that any FlowFile in this bundle remained waiting in this processor before it was output"), +@WritesAttribute(attribute = "merge.uuid", description = "UUID of the merged flow file that will be added to the original flow files attributes"), --- End diff -- Pure nitpick: "will be added to the original _FlowFile's attributes_" and I also prefer `FlowFile` instead of `flow file` just to be consistent across NiFi world. Feel free to ignore though :) ---
[GitHub] nifi pull request #2776: NIFI-5284: Added JSON_TYPE support to RunMongoAggre...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2776#discussion_r194246883
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
---
@@ -275,4 +303,13 @@ protected void writeBatch(String payload, FlowFile
parent, ProcessContext contex
session.getProvenanceReporter().receive(flowFile, getURI(context));
session.transfer(flowFile, rel);
}
+
+protected void configureMapper(String setting) {
--- End diff --
@MikeThomsen I again went through the link @joewitt shared. Anyway, we are
configuring `objectMapper` before using it, at least in this case. That
guarantees thread safety. https://stackoverflow.com/a/3909846/2973446. Anyway I
have marked it synchronized. Let me know, if there are any concerns.
@joewitt If you see something odd with this one, please share it over here.
---
[GitHub] nifi pull request #2776: NIFI-5284: Added JSON_TYPE support to RunMongoAggre...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2776 NIFI-5284: Added JSON_TYPE support to RunMongoAggregation Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5284 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2776.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2776 commit b963fb76b96edd383e6fda1568ea21fd49a69990 Author: zenfenan Date: 2018-06-09T12:05:59Z NIFI-5284: Added JSON_TYPE support to RunMongoAggregation ---
[GitHub] nifi issue #2773: NIFI-5155: Add host address info to bulletins
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2773 @mcgilman Are you saying regardless of the `setNodeAddress` API not setting anything, the cluster coordinator takes care of populating it ? ---
[GitHub] nifi pull request #2775: NIFI-5286: Updated FasterXML Jackson libraries to 2...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2775 NIFI-5286: Updated FasterXML Jackson libraries to 2.9.5 Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5286 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2775.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2775 commit bb0b37e68079997de17e74719850a5671ead9841 Author: zenfenan Date: 2018-06-09T06:31:36Z NIFI-5286: Updated FasterXML Jackson libraries to 2.9.5 ---
[GitHub] nifi issue #2056: NIFI-4262 - MergeContent - option to add merged uuid in or...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2056 Pulled it and built it locally. Ran the template attached in [NIFI-4028](https://issues.apache.org/jira/browse/NIFI-4028) and verified it to be working as expected. Thanks @pvillard31 @markap14 @mattyb149 Merged to master. +1 ---
[GitHub] nifi issue #2769: NIFI-5264 - Added attribute for validation error message i...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2769 Built it and tested locally. Tested with few runs and got the expected error messages in the attributes. Thanks @pvillard31 ---
[GitHub] nifi issue #2769: NIFI-5264 - Added attribute for validation error message i...
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2769 Reviewing.. ---
[GitHub] nifi pull request #2751: NIFI-5221: Added 'Object Tagging' functionalities t...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2751#discussion_r194009838 --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java --- @@ -205,11 +210,21 @@ .defaultValue(NO_SERVER_SIDE_ENCRYPTION) .build(); +public static final PropertyDescriptor OBJECT_TAGS = new PropertyDescriptor.Builder() --- End diff -- I don't fully understand this one. Are you suggesting to replace the JSON approach with a different one? ---
[GitHub] nifi pull request #2751: NIFI-5221: Added 'Object Tagging' functionalities t...
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2751#discussion_r194009531
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/ListS3.java
---
@@ -307,6 +328,20 @@ private boolean commit(final ProcessContext context,
final ProcessSession sessio
return willCommit;
}
+private Map writeObjectTags(AmazonS3 client,
S3VersionSummary versionSummary) {
+final GetObjectTaggingResult taggingResult =
client.getObjectTagging(new
GetObjectTaggingRequest(versionSummary.getBucketName(),
versionSummary.getKey()));
--- End diff --
Yep. It'll make additional API call but I don't think it will have any
negative impact on the existing flows? Do you sense any cases where this might
break? This simply calls S3 service to see if this key has any tags associated,
if it has, it will add to the flowfile attributes.
---
[GitHub] nifi issue #2773: NIFI-5155: Add host address info to bulletins
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2773 I'm not entirely sure if this poses an immediate threat, but in the future when other extensions try to use the `getNodeAddress()` API, it might possibly be an issue, especially on the user experience perspective i.e. chances of the user getting confused when all the bulletins print 127.0.0.1 are there, IMO. For the workaround, this feels like a good approach: https://stackoverflow.com/a/28346117 ---
[GitHub] nifi issue #2765: NIFI-5271 Moved JSON_VALIDATOR to its own maven module.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2765 @joewitt @MikeThomsen Thank you so much. I have merged it to the master +1 ---
[GitHub] nifi issue #2765: NIFI-5271 Moved JSON_VALIDATOR to its own maven module.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2765 @MikeThomsen We get compilation failure. Take a look at the travis logs. ---
[GitHub] nifi pull request #2765: NIFI-5271 Moved JSON_VALIDATOR to its own maven mod...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2765#discussion_r193611711 --- Diff: nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-services-api/pom.xml --- @@ -45,6 +45,11 @@ com.github.stephenc.findbugs findbugs-annotations 1.3.9-1 - + --- End diff -- @MikeThomsen Can you please add a `To-do` here explaining that it is an workaround because of a transitive dependency issue. Ideally once the `google-auth-library-oauth2-http` is updated to a version that brings updated jackson artifacts, we can remove this. ---
[GitHub] nifi issue #2765: NIFI-5271 Moved JSON_VALIDATOR to its own maven module.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2765 Link to my Gist that has the full log: https://gist.github.com/zenfenan/2e260004ce3d27ded2312a2f36d8b300 ---
[GitHub] nifi issue #2765: NIFI-5271 Moved JSON_VALIDATOR to its own maven module.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2765 The snippet of the log that throws the error when the service is starting: ``` 2018-06-06 20:43:52,727 INFO [main] org.apache.nifi.web.server.JettyServer Loading WAR: E:\Workspace\ASF\nifi\nifi-assembly\target\nifi-1.7.0-SNAPSHOT-bin\nifi-1.7.0-SNAPSHOT\.\work\nar\framework\nifi-framework-nar-1.7.0-SNAPSHOT.nar-unpacked\META-INF\bundled-dependencies\nifi-web-docs-1.7.0-SNAPSHOT.war with context path set to /nifi-docs 2018-06-06 20:43:52,742 INFO [main] org.apache.nifi.web.server.JettyServer Loading documents web app with context path set to /nifi-docs 2018-06-06 20:43:52,742 INFO [main] org.apache.nifi.web.server.JettyServer Loading WAR: E:\Workspace\ASF\nifi\nifi-assembly\target\nifi-1.7.0-SNAPSHOT-bin\nifi-1.7.0-SNAPSHOT\.\work\nar\framework\nifi-framework-nar-1.7.0-SNAPSHOT.nar-unpacked\META-INF\bundled-dependencies\nifi-web-error-1.7.0-SNAPSHOT.war with context path set to / 2018-06-06 20:43:52,758 INFO [main] org.apache.nifi.web.server.JettyServer Running in HTTP mode; host headers not restricted 2018-06-06 20:43:54,747 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.controller.ControllerService: Provider org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService could not be instantiated java.util.ServiceConfigurationError: org.apache.nifi.controller.ControllerService: Provider org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148) at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:816) at org.apache.nifi.NiFi.(NiFi.java:157) at org.apache.nifi.NiFi.(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:292) Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z at com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:571) at com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:480) at org.apache.nifi.processor.util.JsonValidator.(JsonValidator.java:29) at org.apache.nifi.processors.gcp.credentials.factory.CredentialPropertyDescriptors.(CredentialPropertyDescriptors.java:86) at org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService.(GCPCredentialsControllerService.java:60) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 8 common frames omitted 2018-06-06 20:43:54,747 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2018-06-06 20:43:54,747 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise). ``` ---
[GitHub] nifi issue #2765: NIFI-5271 Moved JSON_VALIDATOR to its own maven module.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2765 Yes. Iâm on windows. I built it with contrib-check. It built successfully but when I execute run-nifi.bat, I got errors thrown during when the GCP bundle gets loaded. Iâm away from my machine now (on Phone). The error stacktrace had messages along the lines NoClassDef : ObjectMapper. on FasterXML. Iâll try to post the full log but extremely sorry for not able to post them now. > ---
[GitHub] nifi issue #2765: NIFI-5271 Moved JSON_VALIDATOR to its own maven module.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2765 @joewitt @MikeThomsen Build is a success but when I run (i.e. execute nifi.sh/run-nifi.bat), it fails to bring up the service. Thatâs to do with the dependency issue in GCP bundle. Can you please confirm you bring up the UI after these changes? ---
[GitHub] nifi issue #2765: NIFI-5271 Moved JSON_VALIDATOR to its own maven module.
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2765 Having it as static method sounds good to me. @MikeThomsen Have you looked into the dependency issue with the GCP bundle that I had raised earlier? On Wed, 6 Jun 2018 at 10:43 PM, Joe Witt wrote: > @MikeThomsen <https://github.com/MikeThomsen> now that objectmapper is > within the call any reason not to have this be a static method invoked > consistently with other validators? I dont know that it matters but just a > heads up. > > As far as L&N i'll send more on that in a sec. It looks now that it will > be very easy > > â > You are receiving this because you were mentioned. > > Reply to this email directly, view it on GitHub > <https://github.com/apache/nifi/pull/2765#issuecomment-395144977>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AWmHLVvaj6GsMtK0fQtC6u8AfZjvtEB_ks5t6A2dgaJpZM4Uce2b> > . > ---
[GitHub] nifi pull request #2765: NIFI-5271 Moved JSON_VALIDATOR to its own maven mod...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2765#discussion_r193427227 --- Diff: nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml --- @@ -80,6 +80,12 @@ json 1.8 + +org.apache.nifi +nifi-json-utils +1.7.0-SNAPSHOT +compile --- End diff -- This is the one what I'm talking about. ---
[GitHub] nifi pull request #2765: NIFI-5271 Moved JSON_VALIDATOR to its own maven mod...
Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2765#discussion_r193425267 --- Diff: nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml --- @@ -80,6 +80,12 @@ json 1.8 + +org.apache.nifi +nifi-json-utils +1.7.0-SNAPSHOT --- End diff -- This causes issue when starting NiFi. Reason seems to be `google-auth-library-oauth2-http` comes with `jackson-core:2.1.3` as its dependency. We can't exclude it because of the version mismatch. Even the latest version of `google-auth-library-oauth2-http` uses `jackson-core:2.1.3` ---
