[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[vivekmuniyandi]

Github user vivekmuniyandi commented on the issue:

https://github.com/apache/nifi/pull/2671
  
Thanks @MikeThomsen ! Have made the changes. 


---

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[vivekmuniyandi]

Github user vivekmuniyandi commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185979858
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/QueryMarkLogic.java
 ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.ExportListener;
+import com.marklogic.client.ext.datamovement.job.SimpleQueryBatcherJob;
+import com.marklogic.client.io.BytesHandle;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@Tags({"MarkLogic"})
+@CapabilityDescription("Creates FlowFiles from batches of documents, 
matching the given criteria," +
+" retrieved from a MarkLogic server using the MarkLogic Data Movement 
SDK (DMSDK)")
+public class QueryMarkLogic extends AbstractMarkLogicProcessor {
--- End diff --

We have added this as a task in our sprint and accommodate in the coming 
weeks. 


---

[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[vivekmuniyandi]

Github user vivekmuniyandi commented on the issue:

https://github.com/apache/nifi/pull/2671
  
I followed this link - 
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-Keepingyourfeaturebranchcurrent

and looks like this pulled other's commits as well.


---

[GitHub] nifi issue #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2671
  
@vivekmuniyandi looks like you pulled a few other folks' commits in with 
your last push. Do this to clear that up:

1. git checkout master
2. git pull  master
3. git checkout nifi-5102
4. git rebase master
5. git push marklogic --force nifi-5102

You probably did a pull on master into nifi-5102. You want to avoid that 
for your own sanity's sake and use a rebase instead.


---

[jira] [Commented] (NIFI-5044) SelectHiveQL accept only one statement

[Ed Berezitsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463091#comment-16463091
 ] 

Ed Berezitsky commented on NIFI-5044:
-

Looks like we are all aligned now. Thanks for your input!

> SelectHiveQL accept only one statement
> --
>
> Key: NIFI-5044
> URL: https://issues.apache.org/jira/browse/NIFI-5044
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Davide Isoardi
>Assignee: Ed Berezitsky
>Priority: Critical
>
> In [this 
> |[https://github.com/apache/nifi/commit/bbc714e73ba245de7bc32fd9958667c847101f7d]
>  ] commit claims to add support to running multiple statements both on 
> SelectHiveQL and PutHiveQL; instead, it adds only the support to PutHiveQL, 
> so SelectHiveQL still lacks this important feature. @Matt Burgess, I saw that 
> you worked on that, is there any reason for this? If not, can we support it?
> If I try to execute this query:
> {quote}set hive.vectorized.execution.enabled = false; SELECT * FROM table_name
> {quote}
> I have this error:
>  
> {quote}2018-04-05 13:35:40,572 ERROR [Timer-Driven Process Thread-146] 
> o.a.nifi.processors.hive.SelectHiveQL 
> SelectHiveQL[id=243d4c17-b1fe-14af--ee8ce15e] Unable to execute 
> HiveQL select query set hive.vectorized.execution.enabled = false; SELECT * 
> FROM table_name for 
> StandardFlowFileRecord[uuid=0e035558-07ce-473b-b0d4-ac00b8b1df93,claim=StandardContentClaim
>  [resourceClaim=StandardResourceClaim[id=1522824912161-2753, 
> container=default, section=705], offset=838441, 
> length=25],offset=0,name=cliente_attributi.csv,size=25] due to 
> org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!; routing to failure: {}
>  org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL$2.process(SelectHiveQL.java:305)
>  at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2529)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:275)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
>  Caused by: java.sql.SQLException: The query did not generate a result set!
>  at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:438)
>  at 
> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
>  at 
> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL$2.process(SelectHiveQL.java:293)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5148) Solr processors failing to authenticate against Kerberized Solr

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463088#comment-16463088
 ] 

ASF GitHub Bot commented on NIFI-5148:
--

GitHub user bbende opened a pull request:

https://github.com/apache/nifi/pull/2674

NIFI-5148 Refactoring Kerberos auth for Solr processors

- Created resuable KeytabUser and KeytabConfiguration in nifi-security-utils
- Refactored Solr processors to use a KeytabControllerService and no longer 
rely on JAAS system property
- Wrapped all calls in SolrProcessor onTrigger in a doAs when kerberos is 
enabled

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbende/nifi solr-kerberos-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2674.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2674


commit d5332ae4e3a958027becb2f41c0da594dab0f7a5
Author: Bryan Bende 
Date:   2018-05-02T20:17:05Z

NIFI-5148 Refactoring Kerberos auth for Solr processors
- Created resuable KeytabUser and KeytabConfiguration in nifi-security-utils
- Refactored Solr processors to use a KeytabControllerService and no longer 
rely on JAAS system property
- Wrapped all calls in SolrProcessor onTrigger in a doAs when kerberos is 
enabled




> Solr processors failing to authenticate against Kerberized Solr
> ---
>
> Key: NIFI-5148
> URL: https://issues.apache.org/jira/browse/NIFI-5148
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.6.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
>
> It appears that with the new default value of "useSubjectCredsOnly=true" in 
> NiFi's bootstrap.conf that this can cause an issue for the Solr processors 
> when talking to a kerberized Solr.
> The SolrJ client code that we are using specifically calls this out as being 
> problematic:
> [https://github.com/apache/lucene-solr/blob/branch_6x/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Krb5HttpClientConfigurer.java#L75-L88]
> We should refactor the kerberos approach in the Solr processors to resolve 
> this issue and make general improvements. We should be performing a JAAS 
> login and wrapping calls in Subject.doAs, and we should try and move away 
> from the system level JAAS property and leverage the new keytab controller 
> service.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5148) Solr processors failing to authenticate against Kerberized Solr

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende updated NIFI-5148:
--
Status: Patch Available  (was: In Progress)

> Solr processors failing to authenticate against Kerberized Solr
> ---
>
> Key: NIFI-5148
> URL: https://issues.apache.org/jira/browse/NIFI-5148
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.6.0
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
>
> It appears that with the new default value of "useSubjectCredsOnly=true" in 
> NiFi's bootstrap.conf that this can cause an issue for the Solr processors 
> when talking to a kerberized Solr.
> The SolrJ client code that we are using specifically calls this out as being 
> problematic:
> [https://github.com/apache/lucene-solr/blob/branch_6x/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Krb5HttpClientConfigurer.java#L75-L88]
> We should refactor the kerberos approach in the Solr processors to resolve 
> this issue and make general improvements. We should be performing a JAAS 
> login and wrapping calls in Subject.doAs, and we should try and move away 
> from the system level JAAS property and leverage the new keytab controller 
> service.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2674: NIFI-5148 Refactoring Kerberos auth for Solr proces...

[bbende]

GitHub user bbende opened a pull request:

https://github.com/apache/nifi/pull/2674

NIFI-5148 Refactoring Kerberos auth for Solr processors

- Created resuable KeytabUser and KeytabConfiguration in nifi-security-utils
- Refactored Solr processors to use a KeytabControllerService and no longer 
rely on JAAS system property
- Wrapped all calls in SolrProcessor onTrigger in a doAs when kerberos is 
enabled

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbende/nifi solr-kerberos-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2674.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2674


commit d5332ae4e3a958027becb2f41c0da594dab0f7a5
Author: Bryan Bende 
Date:   2018-05-02T20:17:05Z

NIFI-5148 Refactoring Kerberos auth for Solr processors
- Created resuable KeytabUser and KeytabConfiguration in nifi-security-utils
- Refactored Solr processors to use a KeytabControllerService and no longer 
rely on JAAS system property
- Wrapped all calls in SolrProcessor onTrigger in a doAs when kerberos is 
enabled




---

[jira] [Commented] (MINIFICPP-368) Implement expression language subjectless functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463083#comment-16463083
 ] 

ASF GitHub Bot commented on MINIFICPP-368:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/316


> Implement expression language subjectless functions
> ---
>
> Key: MINIFICPP-368
> URL: https://issues.apache.org/jira/browse/MINIFICPP-368
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> Subjectless Functions
> ip
> hostname
> UUID
> nextInt
> literal
> getStateValue



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #316: MINIFICPP-368 Implement subjectless EL fu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/316


---

[jira] [Commented] (MINIFICPP-478) Add missing date manipulation heading/intro to EL docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463065#comment-16463065
 ] 

ASF GitHub Bot commented on MINIFICPP-478:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/317


> Add missing date manipulation heading/intro to EL docs
> --
>
> Key: MINIFICPP-478
> URL: https://issues.apache.org/jira/browse/MINIFICPP-478
> Project: NiFi MiNiFi C++
>  Issue Type: Documentation
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> The date manipulation heading/intro is missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-475) Allow preferred image dimensions/mode to be set in GetUSBCamera properties

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463064#comment-16463064
 ] 

ASF GitHub Bot commented on MINIFICPP-475:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/318


> Allow preferred image dimensions/mode to be set in GetUSBCamera properties
> --
>
> Key: MINIFICPP-475
> URL: https://issues.apache.org/jira/browse/MINIFICPP-475
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> GetUSBCamera currently selects the highest-quality image format for a given 
> FPS. This optimizes for image quality, but can be suboptimal for performance 
> on embedded devices where users may need to have low FPS and low/small image 
> quality.
> Add additional optional properties to GetUSBCamera to allow specification of 
> preferred image dimensions/quality, and have this override automatic 
> selection if the properties are set.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #317: MINIFICPP-478 Added missing date manipula...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/317


---

[GitHub] nifi-minifi-cpp pull request #318: MINIFICPP-475 Added support for preferred...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/318


---

[jira] [Updated] (NIFI-5109) AbstractListProcessor stop calling performListing after primary node reelection

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-5109:
--
Fix Version/s: (was: 1.7.0)

> AbstractListProcessor stop calling performListing after primary node 
> reelection
> ---
>
> Key: NIFI-5109
> URL: https://issues.apache.org/jira/browse/NIFI-5109
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Maksym Viazovskyi
>Priority: Major
> Attachments: 
> 0001-NIFI-2109-Reset-justElectedPrimaryNode-flag-right-af.patch, 
> PutAndListSFTP.xml, docker-compose.yml
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> AbstractListProcessor stops performing listing of entities when after 
> reelection the same node becomes Primary.
> STEPS TO REPRODUCE:
> 0. Make sure that you have docker 18.03.0-ce
> 1. Download attached [^docker-compose.yml] and [^PutAndListSFTP.xml]
> 2. Run *_docker-compose up -d_* in the folder where you downloaded 
> docker-compose.yml. This command will run 3 containers - Zookeeper, NiFi 
> 1.6.0 in Cluster Mode with 1 node, SFTPs server 
> 3. After NiFi is up and running (check http://localhost:8080/nifi) upload 
> PutAndListSFTP.xml template and place it on NiFi canvas. It has simple logic 
> - generate flow file each 10 seconds and put it on SFTP, in parallel 
> ListSFTP processor "listens" for the new files and pushes data about them to 
> NiFi log with LogAttribute processor 
> 4. Update PutSFTP and ListSFTP processors with password admin (to check SFTP 
> container use following credentials SFTP://admin:admin@localhost:) 
> 5. Run the flow 
> 6. Check the NiFi log (*_docker logs -f --tail 100 nifi_*) - you should see 
> attributes for the uploaded files on SFTP 
> 7. Check that generated each 10 seconds files appear on SFTP /upload folder 
> 8. Run following command to restart Zookeeper to put NiFi in reelection 
> state: *_docker restart zookeeper_*
> 9. After reelection phase finished check that new files appears on SFTP, but 
> NO new files info appears in NiFi log
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5109) AbstractListProcessor stop calling performListing after primary node reelection

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463062#comment-16463062
 ] 

Joseph Witt commented on NIFI-5109:
---

Please dont set the fix version until we can get review traction.  There are a 
lot of PRs to go through.  Definitely appreciate the contribution

> AbstractListProcessor stop calling performListing after primary node 
> reelection
> ---
>
> Key: NIFI-5109
> URL: https://issues.apache.org/jira/browse/NIFI-5109
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Maksym Viazovskyi
>Priority: Major
> Attachments: 
> 0001-NIFI-2109-Reset-justElectedPrimaryNode-flag-right-af.patch, 
> PutAndListSFTP.xml, docker-compose.yml
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> AbstractListProcessor stops performing listing of entities when after 
> reelection the same node becomes Primary.
> STEPS TO REPRODUCE:
> 0. Make sure that you have docker 18.03.0-ce
> 1. Download attached [^docker-compose.yml] and [^PutAndListSFTP.xml]
> 2. Run *_docker-compose up -d_* in the folder where you downloaded 
> docker-compose.yml. This command will run 3 containers - Zookeeper, NiFi 
> 1.6.0 in Cluster Mode with 1 node, SFTPs server 
> 3. After NiFi is up and running (check http://localhost:8080/nifi) upload 
> PutAndListSFTP.xml template and place it on NiFi canvas. It has simple logic 
> - generate flow file each 10 seconds and put it on SFTP, in parallel 
> ListSFTP processor "listens" for the new files and pushes data about them to 
> NiFi log with LogAttribute processor 
> 4. Update PutSFTP and ListSFTP processors with password admin (to check SFTP 
> container use following credentials SFTP://admin:admin@localhost:) 
> 5. Run the flow 
> 6. Check the NiFi log (*_docker logs -f --tail 100 nifi_*) - you should see 
> attributes for the uploaded files on SFTP 
> 7. Check that generated each 10 seconds files appear on SFTP /upload folder 
> 8. Run following command to restart Zookeeper to put NiFi in reelection 
> state: *_docker restart zookeeper_*
> 9. After reelection phase finished check that new files appears on SFTP, but 
> NO new files info appears in NiFi log
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5148) Solr processors failing to authenticate against Kerberized Solr

[Bryan Bende (JIRA)]

Bryan Bende created NIFI-5148:
-

 Summary: Solr processors failing to authenticate against 
Kerberized Solr
 Key: NIFI-5148
 URL: https://issues.apache.org/jira/browse/NIFI-5148
 Project: Apache NiFi
  Issue Type: Bug
Affects Versions: 1.6.0
Reporter: Bryan Bende
Assignee: Bryan Bende


It appears that with the new default value of "useSubjectCredsOnly=true" in 
NiFi's bootstrap.conf that this can cause an issue for the Solr processors when 
talking to a kerberized Solr.

The SolrJ client code that we are using specifically calls this out as being 
problematic:

[https://github.com/apache/lucene-solr/blob/branch_6x/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Krb5HttpClientConfigurer.java#L75-L88]

We should refactor the kerberos approach in the Solr processors to resolve this 
issue and make general improvements. We should be performing a JAAS login and 
wrapping calls in Subject.doAs, and we should try and move away from the system 
level JAAS property and leverage the new keytab controller service.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-481) Failed repository creation should not result in seg fault

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463057#comment-16463057
 ] 

ASF GitHub Bot commented on MINIFICPP-481:
--

GitHub user phrocker opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/319

MINIFICPP-481: Update agent to exit gracefully with an invalid *repo dir

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-481

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/319.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #319


commit 7b9885178122259d4ee0aa6caec16535d070e28b
Author: Marc Parisi 
Date:   2018-05-03T21:01:18Z

MINIFICPP-481: Update agent to exit gracefully with an invalid *repo dir




> Failed repository creation should not result in seg fault
> -
>
> Key: MINIFICPP-481
> URL: https://issues.apache.org/jira/browse/MINIFICPP-481
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: marco polo
>Assignee: marco polo
>Priority: Major
>
> When a repository isn't created we should exit gracefully and print something 
> to the command line about why we are exiting. A log is created but this isn't 
> sufficient. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #319: MINIFICPP-481: Update agent to exit grace...

[phrocker]

GitHub user phrocker opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/319

MINIFICPP-481: Update agent to exit gracefully with an invalid *repo dir

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-481

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/319.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #319


commit 7b9885178122259d4ee0aa6caec16535d070e28b
Author: Marc Parisi 
Date:   2018-05-03T21:01:18Z

MINIFICPP-481: Update agent to exit gracefully with an invalid *repo dir




---

[jira] [Updated] (NIFI-5109) AbstractListProcessor stop calling performListing after primary node reelection

[Maksym Viazovskyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maksym Viazovskyi updated NIFI-5109:

Fix Version/s: 1.7.0

> AbstractListProcessor stop calling performListing after primary node 
> reelection
> ---
>
> Key: NIFI-5109
> URL: https://issues.apache.org/jira/browse/NIFI-5109
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Maksym Viazovskyi
>Priority: Major
> Fix For: 1.7.0
>
> Attachments: 
> 0001-NIFI-2109-Reset-justElectedPrimaryNode-flag-right-af.patch, 
> PutAndListSFTP.xml, docker-compose.yml
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> AbstractListProcessor stops performing listing of entities when after 
> reelection the same node becomes Primary.
> STEPS TO REPRODUCE:
> 0. Make sure that you have docker 18.03.0-ce
> 1. Download attached [^docker-compose.yml] and [^PutAndListSFTP.xml]
> 2. Run *_docker-compose up -d_* in the folder where you downloaded 
> docker-compose.yml. This command will run 3 containers - Zookeeper, NiFi 
> 1.6.0 in Cluster Mode with 1 node, SFTPs server 
> 3. After NiFi is up and running (check http://localhost:8080/nifi) upload 
> PutAndListSFTP.xml template and place it on NiFi canvas. It has simple logic 
> - generate flow file each 10 seconds and put it on SFTP, in parallel 
> ListSFTP processor "listens" for the new files and pushes data about them to 
> NiFi log with LogAttribute processor 
> 4. Update PutSFTP and ListSFTP processors with password admin (to check SFTP 
> container use following credentials SFTP://admin:admin@localhost:) 
> 5. Run the flow 
> 6. Check the NiFi log (*_docker logs -f --tail 100 nifi_*) - you should see 
> attributes for the uploaded files on SFTP 
> 7. Check that generated each 10 seconds files appear on SFTP /upload folder 
> 8. Run following command to restart Zookeeper to put NiFi in reelection 
> state: *_docker restart zookeeper_*
> 9. After reelection phase finished check that new files appears on SFTP, but 
> NO new files info appears in NiFi log
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (MINIFICPP-481) Failed repository creation should not result in seg fault

[marco polo (JIRA)]

marco polo created MINIFICPP-481:


 Summary: Failed repository creation should not result in seg fault
 Key: MINIFICPP-481
 URL: https://issues.apache.org/jira/browse/MINIFICPP-481
 Project: NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: marco polo
Assignee: marco polo


When a repository isn't created we should exit gracefully and print something 
to the command line about why we are exiting. A log is created but this isn't 
sufficient. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFIREG-170) Upgrade hoek

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462960#comment-16462960
 ] 

ASF GitHub Bot commented on NIFIREG-170:


Github user mcgilman commented on the issue:

https://github.com/apache/nifi-registry/pull/115
  
Will review...


> Upgrade hoek
> 
>
> Key: NIFIREG-170
> URL: https://issues.apache.org/jira/browse/NIFIREG-170
> Project: NiFi Registry
>  Issue Type: Bug
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-registry issue #115: [NIFIREG-170] upgrade hoek.js

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi-registry/pull/115
  
Will review...


---

[jira] [Updated] (NIFI-5135) Support user identity/group name transform

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-5135:
--
Status: Patch Available  (was: In Progress)

> Support user identity/group name transform
> --
>
> Key: NIFI-5135
> URL: https://issues.apache.org/jira/browse/NIFI-5135
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> Need to offer support for applying transforms to user identities and group 
> names. Other tools, like Ranger, offer this support and without the same 
> capabilities, users may be unable to integrate the two.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2673: NIFI-5135: User identities and group name transform...

[mcgilman]

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/2673

NIFI-5135: User identities and group name transforms

NIFI-5135: 
- Adding support for applying transforms to user identities and group names.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-5135

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2673.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2673


commit a87455045120af8b670ace81c321fcf0cafa042b
Author: Matt Gilman 
Date:   2018-05-03T19:26:31Z

NIFI-5135:
- Adding support for applying transforms to user identities and group names.




---

[jira] [Commented] (NIFI-5135) Support user identity/group name transform

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462959#comment-16462959
 ] 

ASF GitHub Bot commented on NIFI-5135:
--

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/2673

NIFI-5135: User identities and group name transforms

NIFI-5135: 
- Adding support for applying transforms to user identities and group names.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-5135

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2673.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2673


commit a87455045120af8b670ace81c321fcf0cafa042b
Author: Matt Gilman 
Date:   2018-05-03T19:26:31Z

NIFI-5135:
- Adding support for applying transforms to user identities and group names.




> Support user identity/group name transform
> --
>
> Key: NIFI-5135
> URL: https://issues.apache.org/jira/browse/NIFI-5135
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> Need to offer support for applying transforms to user identities and group 
> names. Other tools, like Ranger, offer this support and without the same 
> capabilities, users may be unable to integrate the two.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5147) Improve HashAttribute processor

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-5147:

Description: 
The {{HashAttribute}} processor currently has surprising behavior. Barring 
familiarity with the processor, a user would expect {{HashAttribute}} to 
generate a hash value over one or more attributes. Instead, the processor as it 
is implemented "groups" incoming flowfiles into groups based on regular 
expressions which match attribute values, and then generates a 
(non-configurable) MD5 hash over the concatenation of the matching attribute 
keys and values. 

In addition:
* the processor throws an error and routes to failure any incoming flowfile 
which does not have all attributes specified in the processor
* the use of MD5 is vastly deprecated
* no other hash algorithms are available

I am unaware of community use of this processor, but I do not want to break 
backward compatibility. I propose the following steps:

* Implement a new {{CalculateAttributeHash}} processor (awkward name, but this 
processor already has the desired name)
** This processor will perform the "standard" use case -- identify an 
attribute, calculate the specified hash over the value, and write it to an 
output attribute
** This processor will have a required property descriptor allowing a dropdown 
menu of valid hash algorithms
** This processor will accept arbitrary dynamic properties identifying the 
attributes to be hashed as a key, and the resulting attribute name as a value
** Example: I want to generate a SHA-512 hash on the attribute {{username}}, 
and a flowfile enters the processor with {{username}} value {{alopresto}}. I 
configure {{algorithm}} with {{SHA-512}} and add a dynamic property 
{{username}} -- {{username_SHA512}}. The resulting flowfile will have attribute 
{{username_SHA512}} with value 
{{739b4f6722fb5de20125751c7a1a358b2a7eb8f07e530e4bf18561fbff93234908aa9d250c876bca9ede5ba784d5ce6081dbbdfe5ddd446678f223b8d632}}
* Improve the documentation of this processor to explain the goal/expected use 
case (?)
* Link in processor documentation to new processor for standard use cases
* Remove the error alert when an incoming flowfile does not contain all 
expected attributes. I propose changing the severity to INFO and still routing 
to failure

  was:
The {{HashAttribute}} processor currently has surprising behavior. Barring 
familiarity with the processor, a user would expect {{HashAttribute}} to 
generate a hash value over one or more attributes. Instead, the processor as it 
is implemented "groups" incoming flowfiles into groups based on regular 
expressions which match attribute values, and then generates a 
(non-configurable) MD5 hash over the concatenation of the matching attribute 
keys and values. 

In addition:
* the processor throws an error and routes to failure any incoming flowfile 
which does not have all attributes specified in the processor
* the use of MD5 is vastly deprecated
* no other hash algorithms are available

I am unaware of community use of this processor, but I do not want to break 
backward compatibility. I propose the following steps:

* Implement a new {{CalculateAttributeHash}} processor (awkward name, but this 
processor already has the desired name)
* This processor will perform the "standard" use case -- identify an 
attribute, calculate the specified hash over the value, and write it to an 
output attribute
* This processor will have a required property descriptor allowing a 
dropdown menu of valid hash algorithms
* This processor will accept arbitrary dynamic properties identifying the 
attributes to be hashed as a key, and the resulting attribute name as a value
* Example: I want to generate a SHA-512 hash on the attribute {{username}}, 
and a flowfile enters the processor with {{username}} value {{alopresto}}. I 
configure {{algorithm}} with {{SHA-512}} and add a dynamic property 
{{username}} -- {{username_SHA512}}. The resulting flowfile will have attribute 
{{username_SHA512}} with value 
{{739b4f6722fb5de20125751c7a1a358b2a7eb8f07e530e4bf18561fbff93234908aa9d250c876bca9ede5ba784d5ce6081dbbdfe5ddd446678f223b8d632}}
* Improve the documentation of this processor to explain the goal/expected use 
case (?)
* Link in processor documentation to new processor for standard use cases
* Remove the error alert when an incoming flowfile does not contain all 
expected attributes. I propose changing the severity to INFO and still routing 
to failure


> Improve HashAttribute processor
> ---
>
> Key: NIFI-5147
> URL: https://issues.apache.org/jira/browse/NIFI-5147
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Andy LoPresto
>Priority: Major
>  Labels: hash, security
>

[jira] [Created] (NIFI-5147) Improve HashAttribute processor

[Andy LoPresto (JIRA)]

Andy LoPresto created NIFI-5147:
---

 Summary: Improve HashAttribute processor
 Key: NIFI-5147
 URL: https://issues.apache.org/jira/browse/NIFI-5147
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.6.0
Reporter: Andy LoPresto


The {{HashAttribute}} processor currently has surprising behavior. Barring 
familiarity with the processor, a user would expect {{HashAttribute}} to 
generate a hash value over one or more attributes. Instead, the processor as it 
is implemented "groups" incoming flowfiles into groups based on regular 
expressions which match attribute values, and then generates a 
(non-configurable) MD5 hash over the concatenation of the matching attribute 
keys and values. 

In addition:
* the processor throws an error and routes to failure any incoming flowfile 
which does not have all attributes specified in the processor
* the use of MD5 is vastly deprecated
* no other hash algorithms are available

I am unaware of community use of this processor, but I do not want to break 
backward compatibility. I propose the following steps:

* Implement a new {{CalculateAttributeHash}} processor (awkward name, but this 
processor already has the desired name)
* This processor will perform the "standard" use case -- identify an 
attribute, calculate the specified hash over the value, and write it to an 
output attribute
* This processor will have a required property descriptor allowing a 
dropdown menu of valid hash algorithms
* This processor will accept arbitrary dynamic properties identifying the 
attributes to be hashed as a key, and the resulting attribute name as a value
* Example: I want to generate a SHA-512 hash on the attribute {{username}}, 
and a flowfile enters the processor with {{username}} value {{alopresto}}. I 
configure {{algorithm}} with {{SHA-512}} and add a dynamic property 
{{username}} -- {{username_SHA512}}. The resulting flowfile will have attribute 
{{username_SHA512}} with value 
{{739b4f6722fb5de20125751c7a1a358b2a7eb8f07e530e4bf18561fbff93234908aa9d250c876bca9ede5ba784d5ce6081dbbdfe5ddd446678f223b8d632}}
* Improve the documentation of this processor to explain the goal/expected use 
case (?)
* Link in processor documentation to new processor for standard use cases
* Remove the error alert when an incoming flowfile does not contain all 
expected attributes. I propose changing the severity to INFO and still routing 
to failure



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-475) Allow preferred image dimensions/mode to be set in GetUSBCamera properties

[Andrew Christianson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462914#comment-16462914
 ] 

Andrew Christianson commented on MINIFICPP-475:
---

[~iyanmv], I agree about adding the delay/skip options. I put in a PR for this 
ticket, supporting a target frame width/height (closest will be chosen). 
Created MINIFICPP-480 to track adding of skip/delay options.

Unfortunately, I do not yet see a clear way to support color depth as libuvc 
seems to report frame byte size but not color options. There may be a way to do 
it, but it will require further investigation/another ticket.

> Allow preferred image dimensions/mode to be set in GetUSBCamera properties
> --
>
> Key: MINIFICPP-475
> URL: https://issues.apache.org/jira/browse/MINIFICPP-475
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> GetUSBCamera currently selects the highest-quality image format for a given 
> FPS. This optimizes for image quality, but can be suboptimal for performance 
> on embedded devices where users may need to have low FPS and low/small image 
> quality.
> Add additional optional properties to GetUSBCamera to allow specification of 
> preferred image dimensions/quality, and have this override automatic 
> selection if the properties are set.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (MINIFICPP-480) Add skip/delay options to GetUSBCamera

[Andrew Christianson (JIRA)]

Andrew Christianson created MINIFICPP-480:
-

 Summary: Add skip/delay options to GetUSBCamera
 Key: MINIFICPP-480
 URL: https://issues.apache.org/jira/browse/MINIFICPP-480
 Project: NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: Andrew Christianson
Assignee: Andrew Christianson


Add skip/delay options to GetUSBCamera properties to support delayed 
capture/skipping of frames. This supports, e.g., allowing the camera to adjust 
focus & lighting settings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-475) Allow preferred image dimensions/mode to be set in GetUSBCamera properties

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462908#comment-16462908
 ] 

ASF GitHub Bot commented on MINIFICPP-475:
--

GitHub user achristianson opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/318

MINIFICPP-475 Added support for preferred frame width/height to GetUS…

…BCamera properties

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [x] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] If applicable, have you updated the LICENSE file?
- [x] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/achristianson/nifi-minifi-cpp MINIFICPP-475

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/318.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #318


commit f3840dcbe89490de32389c782d39ae8ace333ae9
Author: Andrew I. Christianson 
Date:   2018-05-03T18:37:55Z

MINIFICPP-475 Added support for preferred frame width/height to 
GetUSBCamera properties




> Allow preferred image dimensions/mode to be set in GetUSBCamera properties
> --
>
> Key: MINIFICPP-475
> URL: https://issues.apache.org/jira/browse/MINIFICPP-475
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> GetUSBCamera currently selects the highest-quality image format for a given 
> FPS. This optimizes for image quality, but can be suboptimal for performance 
> on embedded devices where users may need to have low FPS and low/small image 
> quality.
> Add additional optional properties to GetUSBCamera to allow specification of 
> preferred image dimensions/quality, and have this override automatic 
> selection if the properties are set.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #318: MINIFICPP-475 Added support for preferred...

[achristianson]

GitHub user achristianson opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/318

MINIFICPP-475 Added support for preferred frame width/height to GetUS…

…BCamera properties

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [x] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] If applicable, have you updated the LICENSE file?
- [x] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/achristianson/nifi-minifi-cpp MINIFICPP-475

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/318.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #318


commit f3840dcbe89490de32389c782d39ae8ace333ae9
Author: Andrew I. Christianson 
Date:   2018-05-03T18:37:55Z

MINIFICPP-475 Added support for preferred frame width/height to 
GetUSBCamera properties




---

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462861#comment-16462861
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user michalmisiewicz commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@MikeThomsen it supposed to fetch the entire result set of that select 
statement each run


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2666: NIFI-5130 ExecuteInfluxDBQuery processor chunking support

[michalmisiewicz]

Github user michalmisiewicz commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@MikeThomsen it supposed to fetch the entire result set of that select 
statement each run


---

[jira] [Commented] (NIFIREG-170) Upgrade hoek

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462847#comment-16462847
 ] 

ASF GitHub Bot commented on NIFIREG-170:


GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi-registry/pull/115

[NIFIREG-170] upgrade hoek.js



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi-registry NIFIREG-170

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-registry/pull/115.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #115


commit 48aa7103a5688d0036f633f2ee4f5d7218bacdc2
Author: Scott Aslan 
Date:   2018-05-03T17:43:47Z

[NIFIREG-170] upgrade hoek.js




> Upgrade hoek
> 
>
> Key: NIFIREG-170
> URL: https://issues.apache.org/jira/browse/NIFIREG-170
> Project: NiFi Registry
>  Issue Type: Bug
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-registry pull request #115: [NIFIREG-170] upgrade hoek.js

[scottyaslan]

GitHub user scottyaslan opened a pull request:

https://github.com/apache/nifi-registry/pull/115

[NIFIREG-170] upgrade hoek.js



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/scottyaslan/nifi-registry NIFIREG-170

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-registry/pull/115.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #115


commit 48aa7103a5688d0036f633f2ee4f5d7218bacdc2
Author: Scott Aslan 
Date:   2018-05-03T17:43:47Z

[NIFIREG-170] upgrade hoek.js




---

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462836#comment-16462836
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@michaelandrepearce so I pushed about 30k events into InfluxDB using some 
artifacts from @mans2singh's PRs and ran `select * from /*/` on the database. 
Let it run twice and the result sets were different in size. Is each iteration 
of the processor supposed to be a new chunk or is it supposed to chunk the 
entire result set of that select statement each run?


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2666: NIFI-5130 ExecuteInfluxDBQuery processor chunking support

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@michaelandrepearce so I pushed about 30k events into InfluxDB using some 
artifacts from @mans2singh's PRs and ran `select * from /*/` on the database. 
Let it run twice and the result sets were different in size. Is each iteration 
of the processor supposed to be a new chunk or is it supposed to chunk the 
entire result set of that select statement each run?


---

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462818#comment-16462818
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@mans2singh do you have free time to help review this?


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2666: NIFI-5130 ExecuteInfluxDBQuery processor chunking support

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@mans2singh do you have free time to help review this?


---

[jira] [Commented] (NIFI-5146) Ability to configure HTTP and HTTPS simultaneously causes HostHeader issues

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462722#comment-16462722
 ] 

Andy LoPresto commented on NIFI-5146:
-

I agree with Aldrin that simultaneous support of HTTP and HTTPS interfaces does 
not make sense. This was a legacy design decision for an edge case which is no 
longer supported. All current documentation indicates one or the other should 
be selected. 

I will implement a check during Jetty startup which ensures that only one mode 
is configured and prevents startup with both configured. 

> Ability to configure HTTP and HTTPS simultaneously causes HostHeader issues
> ---
>
> Key: NIFI-5146
> URL: https://issues.apache.org/jira/browse/NIFI-5146
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Aldrin Piri
>Assignee: Andy LoPresto
>Priority: Major
>
> The host header whitelisting evaluation is only done when NiFi is configured 
> in secure mode, determined by the setting of an HTTPS port.  (see 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java#L161
>  and 
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/HostHeaderHandler.java#L190).]
> However, in the case where both are enabled, the HTTP port is not enumerated 
> in possible combinations and explicit inclusions of a given socket that would 
> be HTTP is stripped via 
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/HostHeaderHandler.java#L143.]
> It is possible that concurrently running HTTP and HTTPS no longer makes 
> sense, in which case we could evaluate the relevant properties and prevent 
> startup for an unintended configuration.  Alternatively, we would need to 
> adjust the custom hostname interpretation to also include consideration for 
> the HTTP port.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NIFI-5146) Ability to configure HTTP and HTTPS simultaneously causes HostHeader issues

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto reassigned NIFI-5146:
---

Assignee: Andy LoPresto

> Ability to configure HTTP and HTTPS simultaneously causes HostHeader issues
> ---
>
> Key: NIFI-5146
> URL: https://issues.apache.org/jira/browse/NIFI-5146
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Aldrin Piri
>Assignee: Andy LoPresto
>Priority: Major
>
> The host header whitelisting evaluation is only done when NiFi is configured 
> in secure mode, determined by the setting of an HTTPS port.  (see 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java#L161
>  and 
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/HostHeaderHandler.java#L190).]
> However, in the case where both are enabled, the HTTP port is not enumerated 
> in possible combinations and explicit inclusions of a given socket that would 
> be HTTP is stripped via 
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/HostHeaderHandler.java#L143.]
> It is possible that concurrently running HTTP and HTTPS no longer makes 
> sense, in which case we could evaluate the relevant properties and prevent 
> startup for an unintended configuration.  Alternatively, we would need to 
> adjust the custom hostname interpretation to also include consideration for 
> the HTTP port.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (NIFI-1466) Add password strength indicator to password properties

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462710#comment-16462710
 ] 

Andy LoPresto edited comment on NIFI-1466 at 5/3/18 4:23 PM:
-

Troy Hunt's 
[haveibeenpwned.com|https://www.troyhunt.com/ive-just-launched-pwned-passwords-version-2/]
 has introduced a service where suspect passwords can be compared to a list of 
known compromised passwords, helping users choose secure options. 


was (Author: alopresto):
Troy Hunt's 
[https://www.troyhunt.com/ive-just-launched-pwned-passwords-version-2/|haveibeenpwned.com]
 has introduced a service where suspect passwords can be compared to a list of 
known compromised passwords, helping users choose secure options. 

> Add password strength indicator to password properties
> --
>
> Key: NIFI-1466
> URL: https://issues.apache.org/jira/browse/NIFI-1466
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 0.5.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: encryption, security
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> In processor properties which accept a password, enforce minimum entropy 
> limits and provide real-time feedback as to the entropy estimate of the 
> password. This will have to be overridable (either locally or globally) for 
> backward compatibility, but we should require an explicit administrator 
> decision to do so. 
> Password "strength meters" and other such indicators are not perfect, but 
> they do provide an estimate of valuable feedback to users to encourage 
> stronger passwords. 
> Resources:
> * [NIST & CMU Paper on observed password entropy and recommendations for 
> user-friendly 
> restrictions|https://www.cylab.cmu.edu/research/techreports/2011/tr_cylab11008.html]
> * [J. Bonneau - Statistical metrics for individual password strength 
> (PDF)|http://www.jbonneau.com/doc/B12-SPW-statistical_password_strength_metrics.pdf]
> * [Sophos - Why you can't trust password strength 
> meters|https://nakedsecurity.sophos.com/2015/03/02/why-you-cant-trust-password-strength-meters/]
> * [zxcvbn - Dropbox Password Strength 
> Estimator|https://blogs.dropbox.com/tech/2012/04/zxcvbn-realistic-password-strength-estimation/]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-1466) Add password strength indicator to password properties

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462710#comment-16462710
 ] 

Andy LoPresto commented on NIFI-1466:
-

Troy Hunt's 
[https://www.troyhunt.com/ive-just-launched-pwned-passwords-version-2/|haveibeenpwned.com]
 has introduced a service where suspect passwords can be compared to a list of 
known compromised passwords, helping users choose secure options. 

> Add password strength indicator to password properties
> --
>
> Key: NIFI-1466
> URL: https://issues.apache.org/jira/browse/NIFI-1466
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 0.5.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: encryption, security
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> In processor properties which accept a password, enforce minimum entropy 
> limits and provide real-time feedback as to the entropy estimate of the 
> password. This will have to be overridable (either locally or globally) for 
> backward compatibility, but we should require an explicit administrator 
> decision to do so. 
> Password "strength meters" and other such indicators are not perfect, but 
> they do provide an estimate of valuable feedback to users to encourage 
> stronger passwords. 
> Resources:
> * [NIST & CMU Paper on observed password entropy and recommendations for 
> user-friendly 
> restrictions|https://www.cylab.cmu.edu/research/techreports/2011/tr_cylab11008.html]
> * [J. Bonneau - Statistical metrics for individual password strength 
> (PDF)|http://www.jbonneau.com/doc/B12-SPW-statistical_password_strength_metrics.pdf]
> * [Sophos - Why you can't trust password strength 
> meters|https://nakedsecurity.sophos.com/2015/03/02/why-you-cant-trust-password-strength-meters/]
> * [zxcvbn - Dropbox Password Strength 
> Estimator|https://blogs.dropbox.com/tech/2012/04/zxcvbn-realistic-password-strength-estimation/]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5146) Ability to configure HTTP and HTTPS simultaneously causes HostHeader issues

[Aldrin Piri (JIRA)]

Aldrin Piri created NIFI-5146:
-

 Summary: Ability to configure HTTP and HTTPS simultaneously causes 
HostHeader issues
 Key: NIFI-5146
 URL: https://issues.apache.org/jira/browse/NIFI-5146
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Aldrin Piri


The host header whitelisting evaluation is only done when NiFi is configured in 
secure mode, determined by the setting of an HTTPS port.  (see 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java#L161
 and 
[https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/HostHeaderHandler.java#L190).]

However, in the case where both are enabled, the HTTP port is not enumerated in 
possible combinations and explicit inclusions of a given socket that would be 
HTTP is stripped via 
[https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/HostHeaderHandler.java#L143.]

It is possible that concurrently running HTTP and HTTPS no longer makes sense, 
in which case we could evaluate the relevant properties and prevent startup for 
an unintended configuration.  Alternatively, we would need to adjust the custom 
hostname interpretation to also include consideration for the HTTP port.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462541#comment-16462541
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user michalmisiewicz commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@MikeThomsen yes. Chunking is the only way to fetch unlimited number of 
rows in a single query from InfluxDB that has set `max-row-limit` property.

> Many production InfluxDB installation has limited number of rows returned 
in a single query (by default 10k). Chunking query can return an unlimited 
number of rows.


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2666: NIFI-5130 ExecuteInfluxDBQuery processor chunking support

[michalmisiewicz]

Github user michalmisiewicz commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@MikeThomsen yes. Chunking is the only way to fetch unlimited number of 
rows in a single query from InfluxDB that has set `max-row-limit` property.

> Many production InfluxDB installation has limited number of rows returned 
in a single query (by default 10k). Chunking query can return an unlimited 
number of rows.


---

[jira] [Updated] (NIFI-5109) AbstractListProcessor stop calling performListing after primary node reelection

[Maksym Viazovskyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maksym Viazovskyi updated NIFI-5109:

Status: Patch Available  (was: Open)

> AbstractListProcessor stop calling performListing after primary node 
> reelection
> ---
>
> Key: NIFI-5109
> URL: https://issues.apache.org/jira/browse/NIFI-5109
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Maksym Viazovskyi
>Priority: Major
> Attachments: 
> 0001-NIFI-2109-Reset-justElectedPrimaryNode-flag-right-af.patch, 
> PutAndListSFTP.xml, docker-compose.yml
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> AbstractListProcessor stops performing listing of entities when after 
> reelection the same node becomes Primary.
> STEPS TO REPRODUCE:
> 0. Make sure that you have docker 18.03.0-ce
> 1. Download attached [^docker-compose.yml] and [^PutAndListSFTP.xml]
> 2. Run *_docker-compose up -d_* in the folder where you downloaded 
> docker-compose.yml. This command will run 3 containers - Zookeeper, NiFi 
> 1.6.0 in Cluster Mode with 1 node, SFTPs server 
> 3. After NiFi is up and running (check http://localhost:8080/nifi) upload 
> PutAndListSFTP.xml template and place it on NiFi canvas. It has simple logic 
> - generate flow file each 10 seconds and put it on SFTP, in parallel 
> ListSFTP processor "listens" for the new files and pushes data about them to 
> NiFi log with LogAttribute processor 
> 4. Update PutSFTP and ListSFTP processors with password admin (to check SFTP 
> container use following credentials SFTP://admin:admin@localhost:) 
> 5. Run the flow 
> 6. Check the NiFi log (*_docker logs -f --tail 100 nifi_*) - you should see 
> attributes for the uploaded files on SFTP 
> 7. Check that generated each 10 seconds files appear on SFTP /upload folder 
> 8. Run following command to restart Zookeeper to put NiFi in reelection 
> state: *_docker restart zookeeper_*
> 9. After reelection phase finished check that new files appears on SFTP, but 
> NO new files info appears in NiFi log
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2588: NIFI-5022 InvokeAWSGatewayApi processor

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2588
  
Sounds good to me, especially if that project is active, I'd like to keep 
our dependencies as modules rather than code if possible.


---

[jira] [Commented] (NIFI-5022) Create an AWS Gateway Web API version of InvokeHTTP

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462492#comment-16462492
 ] 

ASF GitHub Bot commented on NIFI-5022:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2588
  
Sounds good to me, especially if that project is active, I'd like to keep 
our dependencies as modules rather than code if possible.


> Create an AWS Gateway Web API version of InvokeHTTP
> ---
>
> Key: NIFI-5022
> URL: https://issues.apache.org/jira/browse/NIFI-5022
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Currently the AWS processors are lacking support to call AWS Gateway Web Apis.
> Nifi should provide support for calling this apis, including support for all 
> the same authentication methods available to the other AWS client processors.
> Since these APIs are web services, their expected use would require an 
> interface more like InvokeHTTP however, than the specialized interfaces for 
> the other AWS Services.
> What would be required then would be a new AWS Processor that exposed the 
> same interface as InvokeHTTP, but backed by the AWS client support ( and of 
> course modified to fit the differences between the OK http client and the 
> Amazon client ).
> This new processor should be able to pass all the applicable tests available 
> in the InvokeHttp test suite.
> The processor should also be factored in such a way as to make it possible 
> for the creation of custom processors for specific apis
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5022) Create an AWS Gateway Web API version of InvokeHTTP

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462483#comment-16462483
 ] 

ASF GitHub Bot commented on NIFI-5022:
--

Github user ottobackwards commented on the issue:

https://github.com/apache/nifi/pull/2588
  
It would be better if we would just use RPGreen's maven dep instead off 
bringing in the classes.  I have a pr against his repo to upgrade to the same 
aws package we use.  When that is taken and a release is cut, I'll move over to 
that.  I think we should still review however, in the event that he doesn't 
take that pr and we have to keep the code.





> Create an AWS Gateway Web API version of InvokeHTTP
> ---
>
> Key: NIFI-5022
> URL: https://issues.apache.org/jira/browse/NIFI-5022
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Currently the AWS processors are lacking support to call AWS Gateway Web Apis.
> Nifi should provide support for calling this apis, including support for all 
> the same authentication methods available to the other AWS client processors.
> Since these APIs are web services, their expected use would require an 
> interface more like InvokeHTTP however, than the specialized interfaces for 
> the other AWS Services.
> What would be required then would be a new AWS Processor that exposed the 
> same interface as InvokeHTTP, but backed by the AWS client support ( and of 
> course modified to fit the differences between the OK http client and the 
> Amazon client ).
> This new processor should be able to pass all the applicable tests available 
> in the InvokeHttp test suite.
> The processor should also be factored in such a way as to make it possible 
> for the creation of custom processors for specific apis
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5022) Create an AWS Gateway Web API version of InvokeHTTP

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462484#comment-16462484
 ] 

ASF GitHub Bot commented on NIFI-5022:
--

Github user ottobackwards commented on the issue:

https://github.com/apache/nifi/pull/2588
  
https://github.com/rpgreen/apigateway-generic-java-sdk/pull/8


> Create an AWS Gateway Web API version of InvokeHTTP
> ---
>
> Key: NIFI-5022
> URL: https://issues.apache.org/jira/browse/NIFI-5022
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Currently the AWS processors are lacking support to call AWS Gateway Web Apis.
> Nifi should provide support for calling this apis, including support for all 
> the same authentication methods available to the other AWS client processors.
> Since these APIs are web services, their expected use would require an 
> interface more like InvokeHTTP however, than the specialized interfaces for 
> the other AWS Services.
> What would be required then would be a new AWS Processor that exposed the 
> same interface as InvokeHTTP, but backed by the AWS client support ( and of 
> course modified to fit the differences between the OK http client and the 
> Amazon client ).
> This new processor should be able to pass all the applicable tests available 
> in the InvokeHttp test suite.
> The processor should also be factored in such a way as to make it possible 
> for the creation of custom processors for specific apis
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2588: NIFI-5022 InvokeAWSGatewayApi processor

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/nifi/pull/2588
  
https://github.com/rpgreen/apigateway-generic-java-sdk/pull/8


---

[GitHub] nifi issue #2588: NIFI-5022 InvokeAWSGatewayApi processor

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/nifi/pull/2588
  
It would be better if we would just use RPGreen's maven dep instead off 
bringing in the classes.  I have a pr against his repo to upgrade to the same 
aws package we use.  When that is taken and a release is cut, I'll move over to 
that.  I think we should still review however, in the event that he doesn't 
take that pr and we have to keep the code.





---

[GitHub] nifi issue #2498: [WIP] NIFI-4915 - Creating new nifi-hbase_2-client-service...

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/2498
  
@MikeThomsen thanks! I'll update the PR once things settle down


---

[jira] [Commented] (NIFI-4915) Add support for HBase 2.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462432#comment-16462432
 ] 

ASF GitHub Bot commented on NIFI-4915:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/2498
  
@MikeThomsen thanks! I'll update the PR once things settle down


> Add support for HBase 2.0.0
> ---
>
> Key: NIFI-4915
> URL: https://issues.apache.org/jira/browse/NIFI-4915
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> The HBase community is gearing up for their 2.0.0 release and currently has a 
> 2.0.0-beta-1 release out. We should provide a new HBaseClientService that 
> uses the 2.0.0 client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5134) NiFi can encounter "no TGT" after Hive service outage

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5134:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> NiFi can encounter "no TGT" after Hive service outage
> -
>
> Key: NIFI-5134
> URL: https://issues.apache.org/jira/browse/NIFI-5134
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
> Fix For: 1.7.0
>
>
> NiFi's Hive controller service may encounter a "no TGT" error after an outage 
> in Hive has occurred.  The "no TGT" error can occur after the service has 
> been restarted and the current TGT has expired.  The Hive client/thrift does 
> not seem to handle this case implicitly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5134) NiFi can encounter "no TGT" after Hive service outage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462422#comment-16462422
 ] 

ASF GitHub Bot commented on NIFI-5134:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2667


> NiFi can encounter "no TGT" after Hive service outage
> -
>
> Key: NIFI-5134
> URL: https://issues.apache.org/jira/browse/NIFI-5134
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
> Fix For: 1.7.0
>
>
> NiFi's Hive controller service may encounter a "no TGT" error after an outage 
> in Hive has occurred.  The "no TGT" error can occur after the service has 
> been restarted and the current TGT has expired.  The Hive client/thrift does 
> not seem to handle this case implicitly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2667: NIFI-5134 Explicitly requesting UGI to relogin befo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2667


---

[jira] [Updated] (NIFI-5134) NiFi can encounter "no TGT" after Hive service outage

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5134:
---
Fix Version/s: 1.7.0

> NiFi can encounter "no TGT" after Hive service outage
> -
>
> Key: NIFI-5134
> URL: https://issues.apache.org/jira/browse/NIFI-5134
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
> Fix For: 1.7.0
>
>
> NiFi's Hive controller service may encounter a "no TGT" error after an outage 
> in Hive has occurred.  The "no TGT" error can occur after the service has 
> been restarted and the current TGT has expired.  The Hive client/thrift does 
> not seem to handle this case implicitly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5134) NiFi can encounter "no TGT" after Hive service outage

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462419#comment-16462419
 ] 

ASF subversion and git services commented on NIFI-5134:
---

Commit 3ad324351140d3fb539548fa431463c7decec0c3 in nifi's branch 
refs/heads/master from [~jtstorck]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3ad3243 ]

NIFI-5134 Explicitly requesting UGI to relogin before attempting to get a DB 
connection in HiveConnectionPool

Signed-off-by: Matthew Burgess 

This closes #2667


> NiFi can encounter "no TGT" after Hive service outage
> -
>
> Key: NIFI-5134
> URL: https://issues.apache.org/jira/browse/NIFI-5134
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
> Fix For: 1.7.0
>
>
> NiFi's Hive controller service may encounter a "no TGT" error after an outage 
> in Hive has occurred.  The "no TGT" error can occur after the service has 
> been restarted and the current TGT has expired.  The Hive client/thrift does 
> not seem to handle this case implicitly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5134) NiFi can encounter "no TGT" after Hive service outage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462416#comment-16462416
 ] 

ASF GitHub Bot commented on NIFI-5134:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2667
  
+1 LGTM, looks like it will work like a barrier sync, the first one in will 
relogin if necessary, the rest will no-op, which means there won't be a race 
condition / threading issue when some of the threads call doAs. Thanks for the 
improvement! Merging to master


> NiFi can encounter "no TGT" after Hive service outage
> -
>
> Key: NIFI-5134
> URL: https://issues.apache.org/jira/browse/NIFI-5134
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> NiFi's Hive controller service may encounter a "no TGT" error after an outage 
> in Hive has occurred.  The "no TGT" error can occur after the service has 
> been restarted and the current TGT has expired.  The Hive client/thrift does 
> not seem to handle this case implicitly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462415#comment-16462415
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@michalmisiewicz so if I understand the Jira ticket correctly, the purpose 
of this ticket is to make it possible to stream out everything that matches the 
query, not just one large result payload. Is that right?


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2666: NIFI-5130 ExecuteInfluxDBQuery processor chunking support

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@michalmisiewicz so if I understand the Jira ticket correctly, the purpose 
of this ticket is to make it possible to stream out everything that matches the 
query, not just one large result payload. Is that right?


---

[GitHub] nifi issue #2667: NIFI-5134 Explicitly requesting UGI to relogin before atte...

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2667
  
+1 LGTM, looks like it will work like a barrier sync, the first one in will 
relogin if necessary, the rest will no-op, which means there won't be a race 
condition / threading issue when some of the threads call doAs. Thanks for the 
improvement! Merging to master


---

[jira] [Updated] (NIFI-4393) AbstractDatabaseFetchProcessor handles SQL Server brackets incorrectly

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4393:
---
Fix Version/s: 1.7.0

> AbstractDatabaseFetchProcessor handles SQL Server brackets incorrectly
> --
>
> Key: NIFI-4393
> URL: https://issues.apache.org/jira/browse/NIFI-4393
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Mikołaj Siedlarek
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.7.0
>
> Attachments: 
> 0001-Handle-SQL-Server-square-brackets-in-AbstractDatabas.patch
>
>
> SQL Server allows column names to contain whitespace, in which case they are 
> written in SQL queries inside square brackets. When using processors based on 
> {{AbstractDatabaseFetchProcessor}}, such as {{QueryDatabaseTable}} they have 
> to be specified in  "Maximum-value Columns" in square brackets, because 
> otherwise they would break a SELECT query. However, when such column name is 
> retrieved from ResultSetMetaData, the driver returns it without square 
> brackets. This causes a mismatch between the key used to save last seen 
> maximum value in processor's state and the one used to search for the value 
> later.
> I'm not sure whether the attached patch is very elegant, but it fixes the 
> issue in a simplest way possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4393) AbstractDatabaseFetchProcessor handles SQL Server brackets incorrectly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462410#comment-16462410
 ] 

ASF GitHub Bot commented on NIFI-4393:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2424
  
+1 LGTM merged.


> AbstractDatabaseFetchProcessor handles SQL Server brackets incorrectly
> --
>
> Key: NIFI-4393
> URL: https://issues.apache.org/jira/browse/NIFI-4393
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Mikołaj Siedlarek
>Assignee: Koji Kawamura
>Priority: Major
> Attachments: 
> 0001-Handle-SQL-Server-square-brackets-in-AbstractDatabas.patch
>
>
> SQL Server allows column names to contain whitespace, in which case they are 
> written in SQL queries inside square brackets. When using processors based on 
> {{AbstractDatabaseFetchProcessor}}, such as {{QueryDatabaseTable}} they have 
> to be specified in  "Maximum-value Columns" in square brackets, because 
> otherwise they would break a SELECT query. However, when such column name is 
> retrieved from ResultSetMetaData, the driver returns it without square 
> brackets. This causes a mismatch between the key used to save last seen 
> maximum value in processor's state and the one used to search for the value 
> later.
> I'm not sure whether the attached patch is very elegant, but it fixes the 
> issue in a simplest way possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2424: NIFI-4393: Handle database specific identifier escape char...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2424
  
+1 LGTM merged.


---

[jira] [Updated] (NIFI-4393) AbstractDatabaseFetchProcessor handles SQL Server brackets incorrectly

[Mike Thomsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Thomsen updated NIFI-4393:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

I checked it over, and things LGTM as well. Merged.

> AbstractDatabaseFetchProcessor handles SQL Server brackets incorrectly
> --
>
> Key: NIFI-4393
> URL: https://issues.apache.org/jira/browse/NIFI-4393
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Mikołaj Siedlarek
>Assignee: Koji Kawamura
>Priority: Major
> Attachments: 
> 0001-Handle-SQL-Server-square-brackets-in-AbstractDatabas.patch
>
>
> SQL Server allows column names to contain whitespace, in which case they are 
> written in SQL queries inside square brackets. When using processors based on 
> {{AbstractDatabaseFetchProcessor}}, such as {{QueryDatabaseTable}} they have 
> to be specified in  "Maximum-value Columns" in square brackets, because 
> otherwise they would break a SELECT query. However, when such column name is 
> retrieved from ResultSetMetaData, the driver returns it without square 
> brackets. This causes a mismatch between the key used to save last seen 
> maximum value in processor's state and the one used to search for the value 
> later.
> I'm not sure whether the attached patch is very elegant, but it fixes the 
> issue in a simplest way possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4393) AbstractDatabaseFetchProcessor handles SQL Server brackets incorrectly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462408#comment-16462408
 ] 

ASF GitHub Bot commented on NIFI-4393:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2424


> AbstractDatabaseFetchProcessor handles SQL Server brackets incorrectly
> --
>
> Key: NIFI-4393
> URL: https://issues.apache.org/jira/browse/NIFI-4393
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Mikołaj Siedlarek
>Assignee: Koji Kawamura
>Priority: Major
> Attachments: 
> 0001-Handle-SQL-Server-square-brackets-in-AbstractDatabas.patch
>
>
> SQL Server allows column names to contain whitespace, in which case they are 
> written in SQL queries inside square brackets. When using processors based on 
> {{AbstractDatabaseFetchProcessor}}, such as {{QueryDatabaseTable}} they have 
> to be specified in  "Maximum-value Columns" in square brackets, because 
> otherwise they would break a SELECT query. However, when such column name is 
> retrieved from ResultSetMetaData, the driver returns it without square 
> brackets. This causes a mismatch between the key used to save last seen 
> maximum value in processor's state and the one used to search for the value 
> later.
> I'm not sure whether the attached patch is very elegant, but it fixes the 
> issue in a simplest way possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2424: NIFI-4393: Handle database specific identifier esca...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2424


---

[jira] [Commented] (NIFI-4393) AbstractDatabaseFetchProcessor handles SQL Server brackets incorrectly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462405#comment-16462405
 ] 

ASF subversion and git services commented on NIFI-4393:
---

Commit 2007c207ab1fac0c1d030cf29623796997650b20 in nifi's branch 
refs/heads/master from [~ijokarumawak]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=2007c20 ]

NIFI-4393: Handle database specific identifier escape characters

QueryDatabaseTable and GenerateTableFetch processors were not able to
use max value state as expected, if max value column was wrapped with
escape characters. Due to a mis-match between computed state keys
and actual keys used in the managed state. State keys computed by
getStateKey method included escape characters while actual stored keys
did not. Resulted querying the same dataset again and again.

This commit added unwrapIdentifier method to DatabaseAdapter class to
remove database specific escape characters for identifiers such as table
and column names, so that max value state keys are populated correctly
even if identifiers are wrapped with escape characters.

This commit also added new DatabaseAdapter for MySQL, to handle MySQL
specific identifier escape with back-ticks.

This closes #2424

Signed-off-by: Mike Thomsen 


> AbstractDatabaseFetchProcessor handles SQL Server brackets incorrectly
> --
>
> Key: NIFI-4393
> URL: https://issues.apache.org/jira/browse/NIFI-4393
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
>Reporter: Mikołaj Siedlarek
>Assignee: Koji Kawamura
>Priority: Major
> Attachments: 
> 0001-Handle-SQL-Server-square-brackets-in-AbstractDatabas.patch
>
>
> SQL Server allows column names to contain whitespace, in which case they are 
> written in SQL queries inside square brackets. When using processors based on 
> {{AbstractDatabaseFetchProcessor}}, such as {{QueryDatabaseTable}} they have 
> to be specified in  "Maximum-value Columns" in square brackets, because 
> otherwise they would break a SELECT query. However, when such column name is 
> retrieved from ResultSetMetaData, the driver returns it without square 
> brackets. This causes a mismatch between the key used to save last seen 
> maximum value in processor's state and the one used to search for the value 
> later.
> I'm not sure whether the attached patch is very elegant, but it fixes the 
> issue in a simplest way possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4915) Add support for HBase 2.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462320#comment-16462320
 ] 

ASF GitHub Bot commented on NIFI-4915:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2498
  
@bbende Heads up, [HBase 2.0.0 was 
released](https://www.apache.org/dist/hbase/2.0.0/RELEASENOTES.md) Once the 
review for visibility labels is done, I'll try to find time to help with 
testing 2.0.0.


> Add support for HBase 2.0.0
> ---
>
> Key: NIFI-4915
> URL: https://issues.apache.org/jira/browse/NIFI-4915
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Minor
>
> The HBase community is gearing up for their 2.0.0 release and currently has a 
> 2.0.0-beta-1 release out. We should provide a new HBaseClientService that 
> uses the 2.0.0 client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2498: [WIP] NIFI-4915 - Creating new nifi-hbase_2-client-service...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2498
  
@bbende Heads up, [HBase 2.0.0 was 
released](https://www.apache.org/dist/hbase/2.0.0/RELEASENOTES.md) Once the 
review for visibility labels is done, I'll try to find time to help with 
testing 2.0.0.


---

[jira] [Commented] (NIFI-4980) Typo in ReportAtlasLineage kafka kerberos service name property

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462314#comment-16462314
 ] 

ASF GitHub Bot commented on NIFI-4980:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2550
  
@ijokarumawak I say we just go ahead and merge a breaking change here. 
Atlas is not even at a stable 1.0 release, so anyone using it should be 
prepared for a few bumps as things get ironed out. If you want me to pare down 
the commit, let me know. I have some free time today.


> Typo in ReportAtlasLineage kafka kerberos service name property
> ---
>
> Key: NIFI-4980
> URL: https://issues.apache.org/jira/browse/NIFI-4980
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Trivial
> Attachments: nifi-4980-screenshot1.png, nifi-4980-screenshot2.png
>
>
> "Kafka Kerberos Service Name" property name is 
> "kafka-kerberos-service-name-kafka". 'kafka' is redundant.
> It should be "kafka-kerberos-service-name".
> This is reported by [~nayakmahesh616].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2550: NIFI-4980: Typo in ReportAtlasLineage kafka kerberos servi...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2550
  
@ijokarumawak I say we just go ahead and merge a breaking change here. 
Atlas is not even at a stable 1.0 release, so anyone using it should be 
prepared for a few bumps as things get ironed out. If you want me to pare down 
the commit, let me know. I have some free time today.


---

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462255#comment-16462255
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2672
  
@pvillard31 @joewitt Could one of you do a code review on this? It aims to 
roll back a little of the new testing behavior that breaks 
evaluateExpressionLanguage(FlowFile) when the input is null. The work around to 
get the testing behavior to match the live behavior is [pretty 
hackish](https://issues.apache.org/jira/browse/NIFI-5145).


> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2672: NIFI-5145 Made MockPropertyValue.evaluateExpressionLanguag...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2672
  
@pvillard31 @joewitt Could one of you do a code review on this? It aims to 
roll back a little of the new testing behavior that breaks 
evaluateExpressionLanguage(FlowFile) when the input is null. The work around to 
get the testing behavior to match the live behavior is [pretty 
hackish](https://issues.apache.org/jira/browse/NIFI-5145).


---

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462253#comment-16462253
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2672

NIFI-5145 Made MockPropertyValue.evaluateExpressionLanguage(FlowFile …

…able to accommodate null flowfiles the way live NiFi does.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-5145

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2672.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2672


commit e717a2c68783815767d0b931f4c0d84fd6e6e0f7
Author: Mike Thomsen 
Date:   2018-05-03T10:28:45Z

NIFI-5145 Made MockPropertyValue.evaluateExpressionLanguage(FlowFile able 
to accommodate null flowfiles the way live NiFi does.




> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[Mike Thomsen (JIRA)]

Mike Thomsen created NIFI-5145:
--

 Summary: MockPropertyValue.evaluateExpressionLanguage(FlowFile) 
cannot handle null inputs
 Key: NIFI-5145
 URL: https://issues.apache.org/jira/browse/NIFI-5145
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Mike Thomsen
Assignee: Mike Thomsen


The method mentioned in the title line cannot handle null inputs, even though 
the main NiFi execution classes can handle that scenario. This forces hack to 
pass testing with nulls that looks like this:

String val = flowFile != null ? 
context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
context.getProperty(PROP).evaluateExpressionLanguage(new HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2672: NIFI-5145 Made MockPropertyValue.evaluateExpression...

[MikeThomsen]

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2672

NIFI-5145 Made MockPropertyValue.evaluateExpressionLanguage(FlowFile …

…able to accommodate null flowfiles the way live NiFi does.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-5145

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2672.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2672


commit e717a2c68783815767d0b931f4c0d84fd6e6e0f7
Author: Mike Thomsen 
Date:   2018-05-03T10:28:45Z

NIFI-5145 Made MockPropertyValue.evaluateExpressionLanguage(FlowFile able 
to accommodate null flowfiles the way live NiFi does.




---

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462218#comment-16462218
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r185746716
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java
 ---
@@ -75,6 +83,17 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
 .build();
+protected static final PropertyDescriptor DEFAULT_VISIBILITY_STRING = 
new PropertyDescriptor.Builder()
--- End diff --

Added that.


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r185746716
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java
 ---
@@ -75,6 +83,17 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
 .build();
+protected static final PropertyDescriptor DEFAULT_VISIBILITY_STRING = 
new PropertyDescriptor.Builder()
--- End diff --

Added that.


---

[jira] [Commented] (NIFI-5051) Create a LookupService that uses ElasticSearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462213#comment-16462213
 ] 

ASF GitHub Bot commented on NIFI-5051:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2615
  
@mattyb149 Can you comment on the schema detection strategy issue I raised 
here?


> Create a LookupService that uses ElasticSearch
> --
>
> Key: NIFI-5051
> URL: https://issues.apache.org/jira/browse/NIFI-5051
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2615: NIFI-5051 Created ElasticSearch lookup service.

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2615
  
@mattyb149 Can you comment on the schema detection strategy issue I raised 
here?


---

[jira] [Commented] (NIFI-5052) Create a "delete by query" ElasticSearch processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462206#comment-16462206
 ] 

ASF GitHub Bot commented on NIFI-5052:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2616
  
@mattyb149 ready to merge?


> Create a "delete by query" ElasticSearch processor
> --
>
> Key: NIFI-5052
> URL: https://issues.apache.org/jira/browse/NIFI-5052
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2616: NIFI-5052 Added DeleteByQuery ElasticSearch processor.

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2616
  
@mattyb149 ready to merge?


---

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185738926
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/AbstractMarkLogicProcessor.java
 ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.DatabaseClient;
+import com.marklogic.nifi.controller.DatabaseClientService;
+import com.marklogic.nifi.controller.DatabaseClientService;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Defines common properties for MarkLogic processors.
+ */
+public abstract class AbstractMarkLogicProcessor extends 
AbstractSessionFactoryProcessor {
+
+protected List properties;
+protected Set relationships;
+
+// NiFi requires a validator for every property, even those that don't 
need any validation
+protected static Validator NO_VALIDATION_VALIDATOR = new Validator() {
+@Override
+public ValidationResult validate(String subject, String input, 
ValidationContext context) {
+return new ValidationResult.Builder().valid(true).build();
+}
+};
+
+public static final PropertyDescriptor DATABASE_CLIENT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("DatabaseClient Service")
+.displayName("DatabaseClient Service")
+.required(true)
+.description("The DatabaseClient Controller Service that provides 
the MarkLogic connection")
+.identifiesControllerService(DatabaseClientService.class)
+.build();
+
+public static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("Batch size")
+.displayName("Batch size")
+.required(true)
+.defaultValue("100")
+.description("The number of documents per batch - sets the batch 
size on the Batcher")
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor THREAD_COUNT = new 
PropertyDescriptor.Builder()
+.name("Thread count")
+.displayName("Thread count")
+.required(true)
+.defaultValue("16")
--- End diff --

NiFi uses a lot of threads, so you might want to think about lowering this 
default so it doesn't risk getting greedy.


---

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185739906
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/QueryMarkLogic.java
 ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.ExportListener;
+import com.marklogic.client.ext.datamovement.job.SimpleQueryBatcherJob;
+import com.marklogic.client.io.BytesHandle;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@Tags({"MarkLogic"})
+@CapabilityDescription("Creates FlowFiles from batches of documents, 
matching the given criteria," +
+" retrieved from a MarkLogic server using the MarkLogic Data Movement 
SDK (DMSDK)")
+public class QueryMarkLogic extends AbstractMarkLogicProcessor {
--- End diff --

You might want to think about setting `@InputRequirement` to ALLOWED so 
that you can add an incoming relationship for having the queries provided by 
flowfile properties or content. There are examples of how to do this in 
`GetMongo` and `ExecuteSQL`. When looking there, take note of the top of their 
`onTrigger` methods to see how the presence of a connection is detected.


---

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185742208
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/PutMarkLogic.java
 ---
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.DataMovementManager;
+import com.marklogic.client.datamovement.WriteBatcher;
+import com.marklogic.client.datamovement.WriteEvent;
+import com.marklogic.client.datamovement.impl.WriteEventImpl;
+import com.marklogic.client.document.ServerTransform;
+import com.marklogic.client.io.BytesHandle;
+import com.marklogic.client.io.DocumentMetadataHandle;
+import com.marklogic.client.io.Format;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+/**
+ * The TriggerWhenEmpty annotation is used so that this processor has a 
chance to flush the WriteBatcher when no
+ * flowfiles are ready to be received.
+ */
+@Tags({"MarkLogic"})
+@CapabilityDescription("Write batches of FlowFiles as documents to a 
MarkLogic server using the " +
+"MarkLogic Data Movement SDK (DMSDK)")
+@TriggerWhenEmpty
+public class PutMarkLogic extends AbstractMarkLogicProcessor {
+
+class FlowFileInfo {
+FlowFile flowFile;
+ProcessSession session;
+FlowFileInfo(FlowFile flowFile, ProcessSession session) {
+this.flowFile = flowFile;
+this.session = session;
+}
+}
+private Map URIFlowFileMap = new HashMap<>();
+public static final PropertyDescriptor COLLECTIONS = new 
PropertyDescriptor.Builder()
+.name("Collections")
+.displayName("Collections")
+.description("Comma-delimited sequence of collections to add to 
each document")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor FORMAT = new 
PropertyDescriptor.Builder()
+.name("Format")
+.displayName("Format")
+.description("Format for each document; if not specified, 
MarkLogic will determine the format" +
+" based on the URI")
+.allowableValues(Format.JSON.name(), Format.XML.name(), 
Format.TEXT.name(), Format.BINARY.name(), Format.UNKNOWN.name())
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor JOB_ID = new 
PropertyDescriptor.Builder()
+.name("Job ID")
+.displayName("Job ID")
+.description("ID for the WriteBatcher job")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor JOB_NAME = new 
PropertyDescriptor.Builder()
+.name("Job Name")
+ 

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185743318
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/PutMarkLogic.java
 ---
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.DataMovementManager;
+import com.marklogic.client.datamovement.WriteBatcher;
+import com.marklogic.client.datamovement.WriteEvent;
+import com.marklogic.client.datamovement.impl.WriteEventImpl;
+import com.marklogic.client.document.ServerTransform;
+import com.marklogic.client.io.BytesHandle;
+import com.marklogic.client.io.DocumentMetadataHandle;
+import com.marklogic.client.io.Format;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+/**
+ * The TriggerWhenEmpty annotation is used so that this processor has a 
chance to flush the WriteBatcher when no
+ * flowfiles are ready to be received.
+ */
+@Tags({"MarkLogic"})
+@CapabilityDescription("Write batches of FlowFiles as documents to a 
MarkLogic server using the " +
+"MarkLogic Data Movement SDK (DMSDK)")
+@TriggerWhenEmpty
+public class PutMarkLogic extends AbstractMarkLogicProcessor {
+
+class FlowFileInfo {
+FlowFile flowFile;
+ProcessSession session;
+FlowFileInfo(FlowFile flowFile, ProcessSession session) {
+this.flowFile = flowFile;
+this.session = session;
+}
+}
+private Map URIFlowFileMap = new HashMap<>();
--- End diff --

This variable name violates standard Java conventions.


---

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185740310
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/QueryMarkLogic.java
 ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.ExportListener;
+import com.marklogic.client.ext.datamovement.job.SimpleQueryBatcherJob;
+import com.marklogic.client.io.BytesHandle;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@Tags({"MarkLogic"})
+@CapabilityDescription("Creates FlowFiles from batches of documents, 
matching the given criteria," +
+" retrieved from a MarkLogic server using the MarkLogic Data Movement 
SDK (DMSDK)")
+public class QueryMarkLogic extends AbstractMarkLogicProcessor {
+
+public static final PropertyDescriptor CONSISTENT_SNAPSHOT = new 
PropertyDescriptor.Builder()
+.name("Consistent snapshot")
+.displayName("Consistent snapshot")
+.defaultValue("true")
+.description("Boolean used to indicate that the matching documents 
were retrieved from a " +
+"consistent snapshot")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor COLLECTIONS = new 
PropertyDescriptor.Builder()
+.name("Collections")
+.displayName("Collections")
+.description("Comma-separated list of collections to query from a 
MarkLogic server")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor URIS_QUERY = new 
PropertyDescriptor.Builder()
+.name("URIs query")
+.displayName("URIs query")
+.description("CTS URI Query for retrieving documents from a 
MarkLogic server")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor URI_PATTERN = new 
PropertyDescriptor.Builder()
+.name("URI pattern")
+.displayName("URI pattern")
+.description("URI pattern for retrieving documents from a 
MarkLogic server")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+protected static final Relationship SUCCESS = new 
Relationship.Builder()
+.name("SUCCESS")
+.description("All FlowFiles that are created from documents read 
from MarkLogic are routed to" +
+" this success relationship")
+.build();
+
+@Override
+public void init(ProcessorInitializationContext context) {
+super.init(context);
+
+List list = new ArrayList<>();
+list.addAll(properties);
+list.add(CONSISTENT_SNAPSHOT);
+list.add(COLLECTIONS);
+list.add(URIS_QUERY);
+list.add(URI_PATTERN);
+properties = Collections.unmodifiableList(list);
+Set set = new HashSet<>();
+set.add(SUCCESS);
+relationships = Collections.unmodifiableSet(set);
+}
+
+@Override
+public final void 

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185738577
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/AbstractMarkLogicProcessor.java
 ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.DatabaseClient;
+import com.marklogic.nifi.controller.DatabaseClientService;
+import com.marklogic.nifi.controller.DatabaseClientService;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Defines common properties for MarkLogic processors.
+ */
+public abstract class AbstractMarkLogicProcessor extends 
AbstractSessionFactoryProcessor {
+
+protected List properties;
+protected Set relationships;
+
+// NiFi requires a validator for every property, even those that don't 
need any validation
+protected static Validator NO_VALIDATION_VALIDATOR = new Validator() {
--- End diff --

`Validator.VALID` does the same thing.


---

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185741806
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/PutMarkLogic.java
 ---
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.DataMovementManager;
+import com.marklogic.client.datamovement.WriteBatcher;
+import com.marklogic.client.datamovement.WriteEvent;
+import com.marklogic.client.datamovement.impl.WriteEventImpl;
+import com.marklogic.client.document.ServerTransform;
+import com.marklogic.client.io.BytesHandle;
+import com.marklogic.client.io.DocumentMetadataHandle;
+import com.marklogic.client.io.Format;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+/**
+ * The TriggerWhenEmpty annotation is used so that this processor has a 
chance to flush the WriteBatcher when no
+ * flowfiles are ready to be received.
+ */
+@Tags({"MarkLogic"})
+@CapabilityDescription("Write batches of FlowFiles as documents to a 
MarkLogic server using the " +
+"MarkLogic Data Movement SDK (DMSDK)")
+@TriggerWhenEmpty
+public class PutMarkLogic extends AbstractMarkLogicProcessor {
+
+class FlowFileInfo {
+FlowFile flowFile;
+ProcessSession session;
+FlowFileInfo(FlowFile flowFile, ProcessSession session) {
+this.flowFile = flowFile;
+this.session = session;
+}
+}
+private Map URIFlowFileMap = new HashMap<>();
+public static final PropertyDescriptor COLLECTIONS = new 
PropertyDescriptor.Builder()
--- End diff --

Rule of thumb for all of these:

1. Add an explicit call to `required(boolean)` so it's obvious what your 
intent is there.
2. Switch `NO_VALIDATION_VALIDATOR` to Validator.VALID.


---

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185740483
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/QueryMarkLogic.java
 ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.ExportListener;
+import com.marklogic.client.ext.datamovement.job.SimpleQueryBatcherJob;
+import com.marklogic.client.io.BytesHandle;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@Tags({"MarkLogic"})
+@CapabilityDescription("Creates FlowFiles from batches of documents, 
matching the given criteria," +
+" retrieved from a MarkLogic server using the MarkLogic Data Movement 
SDK (DMSDK)")
+public class QueryMarkLogic extends AbstractMarkLogicProcessor {
+
+public static final PropertyDescriptor CONSISTENT_SNAPSHOT = new 
PropertyDescriptor.Builder()
+.name("Consistent snapshot")
+.displayName("Consistent snapshot")
+.defaultValue("true")
+.description("Boolean used to indicate that the matching documents 
were retrieved from a " +
+"consistent snapshot")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor COLLECTIONS = new 
PropertyDescriptor.Builder()
+.name("Collections")
+.displayName("Collections")
+.description("Comma-separated list of collections to query from a 
MarkLogic server")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor URIS_QUERY = new 
PropertyDescriptor.Builder()
+.name("URIs query")
+.displayName("URIs query")
+.description("CTS URI Query for retrieving documents from a 
MarkLogic server")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor URI_PATTERN = new 
PropertyDescriptor.Builder()
+.name("URI pattern")
+.displayName("URI pattern")
+.description("URI pattern for retrieving documents from a 
MarkLogic server")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+protected static final Relationship SUCCESS = new 
Relationship.Builder()
+.name("SUCCESS")
+.description("All FlowFiles that are created from documents read 
from MarkLogic are routed to" +
+" this success relationship")
+.build();
+
+@Override
+public void init(ProcessorInitializationContext context) {
+super.init(context);
+
+List list = new ArrayList<>();
+list.addAll(properties);
+list.add(CONSISTENT_SNAPSHOT);
+list.add(COLLECTIONS);
+list.add(URIS_QUERY);
+list.add(URI_PATTERN);
+properties = Collections.unmodifiableList(list);
+Set set = new HashSet<>();
+set.add(SUCCESS);
+relationships = Collections.unmodifiableSet(set);
+}
+
+@Override
+public final void 

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185740763
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/PutMarkLogic.java
 ---
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.DataMovementManager;
+import com.marklogic.client.datamovement.WriteBatcher;
+import com.marklogic.client.datamovement.WriteEvent;
+import com.marklogic.client.datamovement.impl.WriteEventImpl;
+import com.marklogic.client.document.ServerTransform;
+import com.marklogic.client.io.BytesHandle;
+import com.marklogic.client.io.DocumentMetadataHandle;
+import com.marklogic.client.io.Format;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+/**
+ * The TriggerWhenEmpty annotation is used so that this processor has a 
chance to flush the WriteBatcher when no
+ * flowfiles are ready to be received.
+ */
+@Tags({"MarkLogic"})
--- End diff --

Should add a few more descriptive tags to help people find it.


---

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185742105
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/PutMarkLogic.java
 ---
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.DataMovementManager;
+import com.marklogic.client.datamovement.WriteBatcher;
+import com.marklogic.client.datamovement.WriteEvent;
+import com.marklogic.client.datamovement.impl.WriteEventImpl;
+import com.marklogic.client.document.ServerTransform;
+import com.marklogic.client.io.BytesHandle;
+import com.marklogic.client.io.DocumentMetadataHandle;
+import com.marklogic.client.io.Format;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+/**
+ * The TriggerWhenEmpty annotation is used so that this processor has a 
chance to flush the WriteBatcher when no
+ * flowfiles are ready to be received.
+ */
+@Tags({"MarkLogic"})
+@CapabilityDescription("Write batches of FlowFiles as documents to a 
MarkLogic server using the " +
+"MarkLogic Data Movement SDK (DMSDK)")
+@TriggerWhenEmpty
+public class PutMarkLogic extends AbstractMarkLogicProcessor {
+
+class FlowFileInfo {
+FlowFile flowFile;
+ProcessSession session;
+FlowFileInfo(FlowFile flowFile, ProcessSession session) {
+this.flowFile = flowFile;
+this.session = session;
+}
+}
+private Map URIFlowFileMap = new HashMap<>();
+public static final PropertyDescriptor COLLECTIONS = new 
PropertyDescriptor.Builder()
+.name("Collections")
+.displayName("Collections")
+.description("Comma-delimited sequence of collections to add to 
each document")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor FORMAT = new 
PropertyDescriptor.Builder()
+.name("Format")
+.displayName("Format")
+.description("Format for each document; if not specified, 
MarkLogic will determine the format" +
+" based on the URI")
+.allowableValues(Format.JSON.name(), Format.XML.name(), 
Format.TEXT.name(), Format.BINARY.name(), Format.UNKNOWN.name())
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor JOB_ID = new 
PropertyDescriptor.Builder()
+.name("Job ID")
+.displayName("Job ID")
+.description("ID for the WriteBatcher job")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor JOB_NAME = new 
PropertyDescriptor.Builder()
+.name("Job Name")
+ 

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185742052
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/PutMarkLogic.java
 ---
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.DataMovementManager;
+import com.marklogic.client.datamovement.WriteBatcher;
+import com.marklogic.client.datamovement.WriteEvent;
+import com.marklogic.client.datamovement.impl.WriteEventImpl;
+import com.marklogic.client.document.ServerTransform;
+import com.marklogic.client.io.BytesHandle;
+import com.marklogic.client.io.DocumentMetadataHandle;
+import com.marklogic.client.io.Format;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+/**
+ * The TriggerWhenEmpty annotation is used so that this processor has a 
chance to flush the WriteBatcher when no
+ * flowfiles are ready to be received.
+ */
+@Tags({"MarkLogic"})
+@CapabilityDescription("Write batches of FlowFiles as documents to a 
MarkLogic server using the " +
+"MarkLogic Data Movement SDK (DMSDK)")
+@TriggerWhenEmpty
+public class PutMarkLogic extends AbstractMarkLogicProcessor {
+
+class FlowFileInfo {
+FlowFile flowFile;
+ProcessSession session;
+FlowFileInfo(FlowFile flowFile, ProcessSession session) {
+this.flowFile = flowFile;
+this.session = session;
+}
+}
+private Map URIFlowFileMap = new HashMap<>();
+public static final PropertyDescriptor COLLECTIONS = new 
PropertyDescriptor.Builder()
+.name("Collections")
+.displayName("Collections")
+.description("Comma-delimited sequence of collections to add to 
each document")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor FORMAT = new 
PropertyDescriptor.Builder()
+.name("Format")
+.displayName("Format")
+.description("Format for each document; if not specified, 
MarkLogic will determine the format" +
+" based on the URI")
+.allowableValues(Format.JSON.name(), Format.XML.name(), 
Format.TEXT.name(), Format.BINARY.name(), Format.UNKNOWN.name())
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor JOB_ID = new 
PropertyDescriptor.Builder()
+.name("Job ID")
+.displayName("Job ID")
+.description("ID for the WriteBatcher job")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor JOB_NAME = new 
PropertyDescriptor.Builder()
+.name("Job Name")
+ 

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185745410
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/PutMarkLogic.java
 ---
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.DataMovementManager;
+import com.marklogic.client.datamovement.WriteBatcher;
+import com.marklogic.client.datamovement.WriteEvent;
+import com.marklogic.client.datamovement.impl.WriteEventImpl;
+import com.marklogic.client.document.ServerTransform;
+import com.marklogic.client.io.BytesHandle;
+import com.marklogic.client.io.DocumentMetadataHandle;
+import com.marklogic.client.io.Format;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+/**
+ * The TriggerWhenEmpty annotation is used so that this processor has a 
chance to flush the WriteBatcher when no
+ * flowfiles are ready to be received.
+ */
+@Tags({"MarkLogic"})
+@CapabilityDescription("Write batches of FlowFiles as documents to a 
MarkLogic server using the " +
+"MarkLogic Data Movement SDK (DMSDK)")
+@TriggerWhenEmpty
+public class PutMarkLogic extends AbstractMarkLogicProcessor {
+
+class FlowFileInfo {
+FlowFile flowFile;
+ProcessSession session;
+FlowFileInfo(FlowFile flowFile, ProcessSession session) {
+this.flowFile = flowFile;
+this.session = session;
+}
+}
+private Map URIFlowFileMap = new HashMap<>();
+public static final PropertyDescriptor COLLECTIONS = new 
PropertyDescriptor.Builder()
+.name("Collections")
+.displayName("Collections")
+.description("Comma-delimited sequence of collections to add to 
each document")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor FORMAT = new 
PropertyDescriptor.Builder()
+.name("Format")
+.displayName("Format")
+.description("Format for each document; if not specified, 
MarkLogic will determine the format" +
+" based on the URI")
+.allowableValues(Format.JSON.name(), Format.XML.name(), 
Format.TEXT.name(), Format.BINARY.name(), Format.UNKNOWN.name())
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor JOB_ID = new 
PropertyDescriptor.Builder()
+.name("Job ID")
+.displayName("Job ID")
+.description("ID for the WriteBatcher job")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor JOB_NAME = new 
PropertyDescriptor.Builder()
+.name("Job Name")
+ 

[GitHub] nifi pull request #2671: NiFi-5102 - Adding Processors for MarkLogic DB

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2671#discussion_r185741393
  
--- Diff: 
nifi-nar-bundles/nifi-marklogic-bundle/nifi-marklogic-processors/src/main/java/com/marklogic/nifi/processor/PutMarkLogic.java
 ---
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.marklogic.nifi.processor;
+
+import com.marklogic.client.datamovement.DataMovementManager;
+import com.marklogic.client.datamovement.WriteBatcher;
+import com.marklogic.client.datamovement.WriteEvent;
+import com.marklogic.client.datamovement.impl.WriteEventImpl;
+import com.marklogic.client.document.ServerTransform;
+import com.marklogic.client.io.BytesHandle;
+import com.marklogic.client.io.DocumentMetadataHandle;
+import com.marklogic.client.io.Format;
+import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessSessionFactory;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+/**
+ * The TriggerWhenEmpty annotation is used so that this processor has a 
chance to flush the WriteBatcher when no
+ * flowfiles are ready to be received.
+ */
+@Tags({"MarkLogic"})
+@CapabilityDescription("Write batches of FlowFiles as documents to a 
MarkLogic server using the " +
+"MarkLogic Data Movement SDK (DMSDK)")
+@TriggerWhenEmpty
+public class PutMarkLogic extends AbstractMarkLogicProcessor {
+
+class FlowFileInfo {
+FlowFile flowFile;
+ProcessSession session;
+FlowFileInfo(FlowFile flowFile, ProcessSession session) {
+this.flowFile = flowFile;
+this.session = session;
+}
+}
+private Map URIFlowFileMap = new HashMap<>();
+public static final PropertyDescriptor COLLECTIONS = new 
PropertyDescriptor.Builder()
+.name("Collections")
+.displayName("Collections")
+.description("Comma-delimited sequence of collections to add to 
each document")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor FORMAT = new 
PropertyDescriptor.Builder()
+.name("Format")
+.displayName("Format")
+.description("Format for each document; if not specified, 
MarkLogic will determine the format" +
+" based on the URI")
+.allowableValues(Format.JSON.name(), Format.XML.name(), 
Format.TEXT.name(), Format.BINARY.name(), Format.UNKNOWN.name())
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor JOB_ID = new 
PropertyDescriptor.Builder()
+.name("Job ID")
+.displayName("Job ID")
+.description("ID for the WriteBatcher job")
+.addValidator(NO_VALIDATION_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor JOB_NAME = new 
PropertyDescriptor.Builder()
+.name("Job Name")
+