[GitHub] nifi-minifi pull request #91: MINIFI-403 - Adjusting included NiFi dependenc...

[apiri]

GitHub user apiri opened a pull request:

https://github.com/apache/nifi-minifi/pull/91

MINIFI-403 - Adjusting included NiFi dependencies

MINIFI-403 - Adjusting included NiFi dependencies and remove those APIJARs 
that were treated as System bundles preventing dependent NARs from finding a 
compatible API NAR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apiri/nifi-minifi MINIFI-403

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi/pull/91.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #91


commit 2e91cbeafd1881a99c4e9b87b841bfb5fe91c0f0
Author: Aldrin Piri 
Date:   2017-10-04T01:42:49Z

MINIFI-403 - Adjusting included NiFi dependencies and remove those API
JARs that were treated as System bundles preventing dependent NARs from
finding a compatible API NAR.




---

[jira] [Commented] (NIFI-4385) Adjust the QueryDatabaseTable processor for handling big tables.

[Peter Wicks (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190648#comment-16190648
 ] 

Peter Wicks commented on NIFI-4385:
---

[~tspaeth] I've thought off and on for a while about adding this feature. Kind 
of like a 'dirty FlowFile' read, but there are issues that need to be tackled.

One of the reason I haven't done it before is because if your connection to the 
database gets interrupted, or NiFi restarts, then the rows you already sent out 
will not have been accounted for in QueryDatabaseTable. The obvious fix for 
this is to update the state of QueryDatabaseTable with the latest max values 
right after you send each file, thus reducing the chance of a duplicate file; 
but...

Right now QueryDatabaseTable does not put an ORDER BY on the SELECT query. 
Which means your max value after any given FlowFile has no bearing on the whole 
ResultSet. This makes sense, why add the performance overhead of an ORDER BY if 
you don't need it. So it's important to understand that there would be some 
additional performance overhead by adding an ORDER BY as part of this change 
too. but...

Then there is the duplicate max value issue to deal with. There will be 
problems if your max value column is not a sequential ID number, but is instead 
a value where duplicates can occur, specifically I'm thinking about time 
stamps. Lets say each FlowFile will be released based on the existing *Max Rows 
Per Flow File* attribute. So we output a FlowFile every 10,000 rows, but two 
rows with the same max value end up split across FlowFile's... and then NiFi 
gets restarted. Now there is no correct max value, some records with the 
current max value have been loaded and some haven't.

This problem can also be handled, but it requires continuing to read rows from 
the ResultSet until the max value column increases. There would be no real way 
to guarantee how many rows would be in a FlowFile, be it 10 or 1M rows. I was 
recently working on some SQL for a NiFi change data capture job and found that 
every few days a batch job would run a single transactions updating 100k rows, 
all with the same update timestamp, into my table...

Thoughts?

> Adjust the QueryDatabaseTable processor for handling big tables.
> 
>
> Key: NIFI-4385
> URL: https://issues.apache.org/jira/browse/NIFI-4385
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Tim Späth
>
> When querying large database tables, the *QueryDatabaseTable* processor does 
> not perform very well.
> The processor will always perform the full query and then transfer all 
> flowfiles as a list instead of 
> transferring them particularly after the *ResultSet* is fetching the next 
> rows(If a fetch size is given). 
> If you want to query a billion rows from a table, 
> the processor will add all flowfiles in an ArrayList in memory 
> before transferring the whole list after the last row is fetched by the 
> ResultSet. 
> I've checked the code in 
> *org.apache.nifi.processors.standard.QueryDatabaseTable.java* 
> and in my opinion, it would be no big deal to move the session.transfer to a 
> proper position in the code (into the while loop where the flowfile is added 
> to the list) to 
> achieve a real _stream support_. There was also a bug report for this problem 
> which resulted in adding the new property *Maximum Number of Fragments*, 
> but this property will just limit the results. 
> Now you have to multiply *Max Rows Per Flow File* with *Maximum Number of 
> Fragments* to get your limit, 
> which is not really a solution for the original problem imho. 
> Also the workaround with GenerateTableFetch and/or ExecuteSQL processors is 
> much slower than using a database cursor or a ResultSet
> and stream the rows in flowfiles directly in the queue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2180: Added GetMongoAggregation to support running Mongo ...

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2180#discussion_r142563421
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/GetMongoAggregation.java
 ---
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.nifi.processors.mongodb;
+
+import com.mongodb.BasicDBObject;
+import com.mongodb.client.AggregateIterable;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.MongoCursor;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.bson.conversions.Bson;
+import org.codehaus.jackson.map.ObjectMapper;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+@Tags({"mongo", "aggregation", "aggregate"})
+@CapabilityDescription("A processor that runs an aggregation query at 
user-defined intervals.")
+public class GetMongoAggregation extends AbstractMongoProcessor {
+
+private final static Set relationships;
+private final static List propertyDescriptors;
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success").description("All files are routed to 
success").build();
--- End diff --

Yes, good catch


---

[jira] [Commented] (NIFI-4431) Support multiple login/SSO configurations concurrently

[Peter Wicks (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190621#comment-16190621
 ] 

Peter Wicks commented on NIFI-4431:
---

[~mcgilman] I don't think I disagree with the spirit of your ticket, but it is 
possible to do both Certificate and Kerberos at the same time already.

I agree that a login selection option would be nice. Right now if I click 
cancel on that certificate selection box it takes me to Kerberos and it's very 
hard to get back if you really wanted a certificate after all.

> Support multiple login/SSO configurations concurrently
> --
>
> Key: NIFI-4431
> URL: https://issues.apache.org/jira/browse/NIFI-4431
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Priority: Minor
>
> Add support for configuring multiple login/SSO providers concurrently. We 
> currently only allow a single option which causes the user to be 
> automatically redirected to the corresponding login page. We should allow the 
> user to initiate which login sequence they would prefer. Additionally, this 
> would give us the opportunity to relay any error messages when an earlier 
> login attempt fails.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4457) "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id"

[Peter Wicks (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190618#comment-16190618
 ] 

Peter Wicks commented on NIFI-4457:
---

Thanks for reporting an issue. What your screenshot appears to show is that the 
maxvalue for tweet_id you provided is already the maximum value in the table 
and that there are not new records with a greater tweet_id.

After replicating the bug can you query the base table and confirm that there 
are new records?

I have not been able to reproduce this. I was working with QueryDatabaseTable 
yesterday, latest NiFi version, and set initial.maxvalue for about 30 tables. 
None of them are being tracked by a column named `id`, and they all are working 
as expected.



> "Maximum-value" not increasing when "initial.maxvalue" is set and 
> "Maximum-value column" name is different from "id" 
> -
>
> Key: NIFI-4457
> URL: https://issues.apache.org/jira/browse/NIFI-4457
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
> Environment: windows 10
>Reporter: meh
> Attachments: Picture1.png, Picture2.png
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> when "Maximum-value column" name is "id" there is no problem, when i add 
> "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works 
> well and maxvalue is increasing by every running.
> !Picture1.png|thumbnail!
> but...
> when the "Maximum-value column" name is different from "id" (such as 
> "tweet_id"), after initial processor working, only given 
> "initial.maxvalue.id" is saves and that repeating just same value for every 
> run.
> !Picture2.png|thumbnail!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-minifi-cpp issue #141: Make libcurl containing classes extensions.

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/141
  
Having new compilation issues in linux. Addressing now. 


---

[GitHub] nifi-registry issue #14: NIFIREG-9: Initial Auth Implementation

[kevdoran]

Github user kevdoran commented on the issue:

https://github.com/apache/nifi-registry/pull/14
  
Thanks a lot for the review @bbende! Great suggestions. Will make those 
changes and update.


---

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190356#comment-16190356
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user kevdoran commented on the issue:

https://github.com/apache/nifi-registry/pull/14
  
Thanks a lot for the review @bbende! Great suggestions. Will make those 
changes and update.


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[kevdoran]

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142518769
  
--- Diff: 
nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/AuthorizationService.java
 ---
@@ -0,0 +1,696 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.service;
+
+import org.apache.nifi.registry.authorization.AccessDeniedException;
+import org.apache.nifi.registry.authorization.AccessPolicyProvider;
+import 
org.apache.nifi.registry.authorization.AccessPolicyProviderInitializationContext;
+import org.apache.nifi.registry.authorization.AuthorizableLookup;
+import org.apache.nifi.registry.authorization.AuthorizationAccessException;
+import org.apache.nifi.registry.authorization.AuthorizeAccess;
+import org.apache.nifi.registry.authorization.Authorizer;
+import 
org.apache.nifi.registry.authorization.AuthorizerCapabilityDetection;
+import 
org.apache.nifi.registry.authorization.AuthorizerConfigurationContext;
+import org.apache.nifi.registry.authorization.AuthorizerCreationException;
+import 
org.apache.nifi.registry.authorization.AuthorizerDestructionException;
+import 
org.apache.nifi.registry.authorization.ConfigurableAccessPolicyProvider;
+import 
org.apache.nifi.registry.authorization.ConfigurableUserGroupProvider;
+import org.apache.nifi.registry.authorization.Group;
+import org.apache.nifi.registry.authorization.ManagedAuthorizer;
+import org.apache.nifi.registry.authorization.RequestAction;
+import org.apache.nifi.registry.authorization.UserAndGroups;
+import org.apache.nifi.registry.authorization.UserGroupProvider;
+import 
org.apache.nifi.registry.authorization.UserGroupProviderInitializationContext;
+import org.apache.nifi.registry.authorization.resource.ResourceFactory;
+import org.apache.nifi.registry.authorization.resource.ResourceType;
+import org.apache.nifi.registry.authorization.user.NiFiUserUtils;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.model.authorization.AccessPolicy;
+import org.apache.nifi.registry.model.authorization.AccessPolicySummary;
+import org.apache.nifi.registry.model.authorization.Resource;
+import org.apache.nifi.registry.model.authorization.Tenant;
+import org.apache.nifi.registry.model.authorization.User;
+import org.apache.nifi.registry.model.authorization.UserGroup;
+import org.apache.nifi.registry.service.params.QueryParameters;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.stream.Collectors;
+
+@Service
+public class AuthorizationService {
+
+public static final String MSG_NON_MANAGED_AUTHORIZER = "This NiFi 
Registry is not configured to internally manage users, groups, or policies. 
Please contact your system administrator.";
+public static final String MSG_NON_CONFIGURABLE_POLICIES = "This NiFi 
Registry is not configured to allow configurable policies. Please contact your 
system administrator.";
+public static final String MSG_NON_CONFIGURABLE_USERS = "This NiFi 
Registry is not configured to allow configurable users and groups. Please 
contact your system administrator.";
+
+private AuthorizableLookup authorizableLookup;
+private Authorizer authorizer;
+private RegistryService registryService;
+private UserGroupProvider userGroupProvider;
+private AccessPolicyProvider accessPolicyProvider;
+
+private final ReentrantReadWriteLock lock = new 
ReentrantReadWriteLock();
+private final Lock readLock = lock.readLock();
+private final Lock writeLock = lock.writeLock();
+
+@Autowired
+public 

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190327#comment-16190327
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142518769
  
--- Diff: 
nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/AuthorizationService.java
 ---
@@ -0,0 +1,696 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.service;
+
+import org.apache.nifi.registry.authorization.AccessDeniedException;
+import org.apache.nifi.registry.authorization.AccessPolicyProvider;
+import 
org.apache.nifi.registry.authorization.AccessPolicyProviderInitializationContext;
+import org.apache.nifi.registry.authorization.AuthorizableLookup;
+import org.apache.nifi.registry.authorization.AuthorizationAccessException;
+import org.apache.nifi.registry.authorization.AuthorizeAccess;
+import org.apache.nifi.registry.authorization.Authorizer;
+import 
org.apache.nifi.registry.authorization.AuthorizerCapabilityDetection;
+import 
org.apache.nifi.registry.authorization.AuthorizerConfigurationContext;
+import org.apache.nifi.registry.authorization.AuthorizerCreationException;
+import 
org.apache.nifi.registry.authorization.AuthorizerDestructionException;
+import 
org.apache.nifi.registry.authorization.ConfigurableAccessPolicyProvider;
+import 
org.apache.nifi.registry.authorization.ConfigurableUserGroupProvider;
+import org.apache.nifi.registry.authorization.Group;
+import org.apache.nifi.registry.authorization.ManagedAuthorizer;
+import org.apache.nifi.registry.authorization.RequestAction;
+import org.apache.nifi.registry.authorization.UserAndGroups;
+import org.apache.nifi.registry.authorization.UserGroupProvider;
+import 
org.apache.nifi.registry.authorization.UserGroupProviderInitializationContext;
+import org.apache.nifi.registry.authorization.resource.ResourceFactory;
+import org.apache.nifi.registry.authorization.resource.ResourceType;
+import org.apache.nifi.registry.authorization.user.NiFiUserUtils;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.model.authorization.AccessPolicy;
+import org.apache.nifi.registry.model.authorization.AccessPolicySummary;
+import org.apache.nifi.registry.model.authorization.Resource;
+import org.apache.nifi.registry.model.authorization.Tenant;
+import org.apache.nifi.registry.model.authorization.User;
+import org.apache.nifi.registry.model.authorization.UserGroup;
+import org.apache.nifi.registry.service.params.QueryParameters;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.stream.Collectors;
+
+@Service
+public class AuthorizationService {
+
+public static final String MSG_NON_MANAGED_AUTHORIZER = "This NiFi 
Registry is not configured to internally manage users, groups, or policies. 
Please contact your system administrator.";
+public static final String MSG_NON_CONFIGURABLE_POLICIES = "This NiFi 
Registry is not configured to allow configurable policies. Please contact your 
system administrator.";
+public static final String MSG_NON_CONFIGURABLE_USERS = "This NiFi 
Registry is not configured to allow configurable users and groups. Please 
contact your system administrator.";
+
+private AuthorizableLookup authorizableLookup;
+private Authorizer authorizer;
+private RegistryService registryService;
+private UserGroupProvider userGroupProvider;
+private AccessPolicyProvider accessPolicyProvider;
+
+

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190322#comment-16190322
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142517215
  
--- Diff: 
nifi-registry-framework/src/main/java/org/apache/nifi/registry/authorization/StandardAuthorizerFactory.java
 ---
@@ -0,0 +1,791 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.authorization;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.registry.authorization.annotation.AuthorizerContext;
+import org.apache.nifi.registry.authorization.generated.Authorizers;
+import org.apache.nifi.registry.authorization.generated.Prop;
+import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import org.apache.nifi.registry.provider.StandardProviderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.xml.sax.SAXException;
+
+import javax.xml.XMLConstants;
+import javax.xml.bind.JAXBContext;
+import javax.xml.bind.JAXBElement;
+import javax.xml.bind.JAXBException;
+import javax.xml.bind.Unmarshaller;
+import javax.xml.transform.stream.StreamSource;
+import javax.xml.validation.Schema;
+import javax.xml.validation.SchemaFactory;
+import java.io.File;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * This implementation of AuthorizerFactory in NiFi Registry is based on a 
combination of
+ * NiFi's AuthorizerFactory and AuthorizerFactoryBean.
+ */
+public class StandardAuthorizerFactory implements AuthorizerFactory, 
UserGroupProviderLookup, AccessPolicyProviderLookup, AuthorizerLookup{
+
+private static final Logger logger = 
LoggerFactory.getLogger(StandardProviderFactory.class);
+
+private static final String AUTHORIZERS_XSD = "/authorizers.xsd";
+private static final String JAXB_GENERATED_PATH = 
"org.apache.nifi.registry.authorization.generated";
+private static final JAXBContext JAXB_CONTEXT = 
initializeJaxbContext();
+
+/**
+ * Load the JAXBContext.
+ */
+private static JAXBContext initializeJaxbContext() {
+try {
+return JAXBContext.newInstance(JAXB_GENERATED_PATH, 
StandardAuthorizerFactory.class.getClassLoader());
+} catch (JAXBException e) {
+throw new RuntimeException("Unable to create JAXBContext.", e);
+}
+}
+
+private final NiFiRegistryProperties properties;
+private Authorizer authorizer;
+private final Map userGroupProviders = new 
HashMap<>();
+private final Map accessPolicyProviders 
= new HashMap<>();
+private final Map authorizers = new HashMap<>();
+
+public StandardAuthorizerFactory(final NiFiRegistryProperties 
properties) {
+this.properties = properties;
+
+if (this.properties == null) {
+throw new IllegalStateException("NiFiRegistryProperties cannot 
be null");
+}
+}
+
+/* UserGroupProviderLookup */
+
+@Override
+public UserGroupProvider getUserGroupProvider(String identifier) {
+return userGroupProviders.get(identifier);
+}
+
+/* AccessPolicyProviderLookup */
+
+@Override
+public AccessPolicyProvider getAccessPolicyProvider(String identifier) 
{
+return accessPolicyProviders.get(identifier);
+}
+
+
+/* AuthorizerLookup */
+
+@Override
+

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[kevdoran]

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142517215
  
--- Diff: 
nifi-registry-framework/src/main/java/org/apache/nifi/registry/authorization/StandardAuthorizerFactory.java
 ---
@@ -0,0 +1,791 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.authorization;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.registry.authorization.annotation.AuthorizerContext;
+import org.apache.nifi.registry.authorization.generated.Authorizers;
+import org.apache.nifi.registry.authorization.generated.Prop;
+import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import org.apache.nifi.registry.provider.StandardProviderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.xml.sax.SAXException;
+
+import javax.xml.XMLConstants;
+import javax.xml.bind.JAXBContext;
+import javax.xml.bind.JAXBElement;
+import javax.xml.bind.JAXBException;
+import javax.xml.bind.Unmarshaller;
+import javax.xml.transform.stream.StreamSource;
+import javax.xml.validation.Schema;
+import javax.xml.validation.SchemaFactory;
+import java.io.File;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * This implementation of AuthorizerFactory in NiFi Registry is based on a 
combination of
+ * NiFi's AuthorizerFactory and AuthorizerFactoryBean.
+ */
+public class StandardAuthorizerFactory implements AuthorizerFactory, 
UserGroupProviderLookup, AccessPolicyProviderLookup, AuthorizerLookup{
+
+private static final Logger logger = 
LoggerFactory.getLogger(StandardProviderFactory.class);
+
+private static final String AUTHORIZERS_XSD = "/authorizers.xsd";
+private static final String JAXB_GENERATED_PATH = 
"org.apache.nifi.registry.authorization.generated";
+private static final JAXBContext JAXB_CONTEXT = 
initializeJaxbContext();
+
+/**
+ * Load the JAXBContext.
+ */
+private static JAXBContext initializeJaxbContext() {
+try {
+return JAXBContext.newInstance(JAXB_GENERATED_PATH, 
StandardAuthorizerFactory.class.getClassLoader());
+} catch (JAXBException e) {
+throw new RuntimeException("Unable to create JAXBContext.", e);
+}
+}
+
+private final NiFiRegistryProperties properties;
+private Authorizer authorizer;
+private final Map userGroupProviders = new 
HashMap<>();
+private final Map accessPolicyProviders 
= new HashMap<>();
+private final Map authorizers = new HashMap<>();
+
+public StandardAuthorizerFactory(final NiFiRegistryProperties 
properties) {
+this.properties = properties;
+
+if (this.properties == null) {
+throw new IllegalStateException("NiFiRegistryProperties cannot 
be null");
+}
+}
+
+/* UserGroupProviderLookup */
+
+@Override
+public UserGroupProvider getUserGroupProvider(String identifier) {
+return userGroupProviders.get(identifier);
+}
+
+/* AccessPolicyProviderLookup */
+
+@Override
+public AccessPolicyProvider getAccessPolicyProvider(String identifier) 
{
+return accessPolicyProviders.get(identifier);
+}
+
+
+/* AuthorizerLookup */
+
+@Override
+public Authorizer getAuthorizer(String identifier) {
+return authorizers.get(identifier);
+}
+
+
+/* AuthorizerFactory */
+
+@Override
+public void initialize() throws 

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190320#comment-16190320
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142516827
  
--- Diff: 
nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/RegistryService.java
 ---
@@ -261,6 +262,29 @@ public Bucket deleteBucket(final String 
bucketIdentifier) {
 }
 }
 
+public List getBucketItems(final QueryParameters 
queryParameters, final Set bucketIdentifiers) {
+if (bucketIdentifiers == null || bucketIdentifiers.isEmpty()) {
+throw new IllegalArgumentException("Bucket Identifiers cannot 
be null or empty");
+}
+
+readLock.lock();
+try {
+final Set filterBuckets = 
bucketIdentifiers.stream()
--- End diff --

Yeah I agree that would be a better approach. Will make that change, thanks.


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190319#comment-16190319
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142516612
  
--- Diff: nifi-registry-security/pom.xml ---
@@ -34,12 +34,28 @@
 jaxb2-maven-plugin
 
 
-xjc
+authorizations
 
 xjc
 
 
-
org.apache.nifi.registry.user.generated
+
--- End diff --

Good find, will update that plugin version.


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[kevdoran]

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142516827
  
--- Diff: 
nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/RegistryService.java
 ---
@@ -261,6 +262,29 @@ public Bucket deleteBucket(final String 
bucketIdentifier) {
 }
 }
 
+public List getBucketItems(final QueryParameters 
queryParameters, final Set bucketIdentifiers) {
+if (bucketIdentifiers == null || bucketIdentifiers.isEmpty()) {
+throw new IllegalArgumentException("Bucket Identifiers cannot 
be null or empty");
+}
+
+readLock.lock();
+try {
+final Set filterBuckets = 
bucketIdentifiers.stream()
--- End diff --

Yeah I agree that would be a better approach. Will make that change, thanks.


---

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[kevdoran]

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142516612
  
--- Diff: nifi-registry-security/pom.xml ---
@@ -34,12 +34,28 @@
 jaxb2-maven-plugin
 
 
-xjc
+authorizations
 
 xjc
 
 
-
org.apache.nifi.registry.user.generated
+
--- End diff --

Good find, will update that plugin version.


---

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190315#comment-16190315
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142516366
  
--- Diff: 
nifi-registry-resources/src/main/resources/conf/nifi-registry.properties ---
@@ -32,6 +32,8 @@ 
nifi.registry.security.truststoreType=${nifi.registry.security.truststoreType}
 
nifi.registry.security.truststorePasswd=${nifi.registry.security.truststorePasswd}
 
nifi.registry.security.needClientAuth=${nifi.registry.security.needClientAuth}
 
nifi.registry.security.authorized.users=${nifi.registry.security.authorized.users}
+nifi.security.user.authorizer=${nifi.security.user.authorizer}
--- End diff --

same as above. removed in the next update


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[kevdoran]

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142516366
  
--- Diff: 
nifi-registry-resources/src/main/resources/conf/nifi-registry.properties ---
@@ -32,6 +32,8 @@ 
nifi.registry.security.truststoreType=${nifi.registry.security.truststoreType}
 
nifi.registry.security.truststorePasswd=${nifi.registry.security.truststorePasswd}
 
nifi.registry.security.needClientAuth=${nifi.registry.security.needClientAuth}
 
nifi.registry.security.authorized.users=${nifi.registry.security.authorized.users}
+nifi.security.user.authorizer=${nifi.security.user.authorizer}
--- End diff --

same as above. removed in the next update


---

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190313#comment-16190313
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142516209
  
--- Diff: nifi-registry-assembly/pom.xml ---
@@ -137,6 +137,8 @@
 
 
 
./conf/authorized-users.xml
+
managed-authorizer
--- End diff --

nope. neither is login.identity.provider. will remove unused properties and 
fix the prefixes to be consistent. good catch!


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[kevdoran]

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142516209
  
--- Diff: nifi-registry-assembly/pom.xml ---
@@ -137,6 +137,8 @@
 
 
 
./conf/authorized-users.xml
+
managed-authorizer
--- End diff --

nope. neither is login.identity.provider. will remove unused properties and 
fix the prefixes to be consistent. good catch!


---

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190310#comment-16190310
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142515622
  
--- Diff: nifi-registry.properties ---
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
--- End diff --

yeah this was a test file. will remove


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[kevdoran]

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142515622
  
--- Diff: nifi-registry.properties ---
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
--- End diff --

yeah this was a test file. will remove


---

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190308#comment-16190308
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142515496
  
--- Diff: 
nifi-registry-security/src/main/java/org/apache/nifi/registry/authorization/AccessPolicy.java
 ---
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.authorization;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.UUID;
+
+/**
+ * Defines a policy for a set of userIdentifiers to perform a set of 
actions on a given resource.
+ */
+public class AccessPolicy {
--- End diff --

It's a good suggestion. I wasn't thrilled with the current organization. 
Your approach both distinguishes interfaces from implementation and adds 
extensibility. Will make that change, thanks!


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190309#comment-16190309
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142515556
  
--- Diff: nifi-registry-web-api/pom.xml ---
@@ -105,6 +105,11 @@
 
 
 org.springframework.boot
+spring-boot-starter-security
--- End diff --

Yep.


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[kevdoran]

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142514956
  
--- Diff: nifi-registry-security/pom.xml ---
@@ -51,11 +67,25 @@
 org.apache.maven.plugins
 maven-checkstyle-plugin
 
-**/user/generated/*.java
+
**/authorization/file/generated/*.java,**/authorization/file/tenants/generated/*.java
 
 
 
 
+
+
+
+
+
+org.springframework
+spring-framework-bom
--- End diff --

Good catch. Yep, will do.


---

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190276#comment-16190276
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142465687
  
--- Diff: nifi-registry-security/pom.xml ---
@@ -51,11 +67,25 @@
 org.apache.maven.plugins
 maven-checkstyle-plugin
 
-**/user/generated/*.java
+
**/authorization/file/generated/*.java,**/authorization/file/tenants/generated/*.java
 
 
 
 
+
+
+
+
+
+org.springframework
+spring-framework-bom
--- End diff --

What do you think about changing this and the spring-security-core 
dependency below to be:
```

org.springframework.boot
spring-boot-starter-security
${spring.boot.version}

```

Just seeing if we can make it easier to keep a consistent version across 
all the places where spring-boot is referenced.


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190278#comment-16190278
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142427496
  
--- Diff: 
nifi-registry-framework/src/main/java/org/apache/nifi/registry/authorization/StandardAuthorizerFactory.java
 ---
@@ -0,0 +1,791 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.authorization;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.registry.authorization.annotation.AuthorizerContext;
+import org.apache.nifi.registry.authorization.generated.Authorizers;
+import org.apache.nifi.registry.authorization.generated.Prop;
+import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import org.apache.nifi.registry.provider.StandardProviderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.xml.sax.SAXException;
+
+import javax.xml.XMLConstants;
+import javax.xml.bind.JAXBContext;
+import javax.xml.bind.JAXBElement;
+import javax.xml.bind.JAXBException;
+import javax.xml.bind.Unmarshaller;
+import javax.xml.transform.stream.StreamSource;
+import javax.xml.validation.Schema;
+import javax.xml.validation.SchemaFactory;
+import java.io.File;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * This implementation of AuthorizerFactory in NiFi Registry is based on a 
combination of
+ * NiFi's AuthorizerFactory and AuthorizerFactoryBean.
+ */
+public class StandardAuthorizerFactory implements AuthorizerFactory, 
UserGroupProviderLookup, AccessPolicyProviderLookup, AuthorizerLookup{
+
+private static final Logger logger = 
LoggerFactory.getLogger(StandardProviderFactory.class);
+
+private static final String AUTHORIZERS_XSD = "/authorizers.xsd";
+private static final String JAXB_GENERATED_PATH = 
"org.apache.nifi.registry.authorization.generated";
+private static final JAXBContext JAXB_CONTEXT = 
initializeJaxbContext();
+
+/**
+ * Load the JAXBContext.
+ */
+private static JAXBContext initializeJaxbContext() {
+try {
+return JAXBContext.newInstance(JAXB_GENERATED_PATH, 
StandardAuthorizerFactory.class.getClassLoader());
+} catch (JAXBException e) {
+throw new RuntimeException("Unable to create JAXBContext.", e);
+}
+}
+
+private final NiFiRegistryProperties properties;
+private Authorizer authorizer;
+private final Map userGroupProviders = new 
HashMap<>();
+private final Map accessPolicyProviders 
= new HashMap<>();
+private final Map authorizers = new HashMap<>();
+
+public StandardAuthorizerFactory(final NiFiRegistryProperties 
properties) {
+this.properties = properties;
+
+if (this.properties == null) {
+throw new IllegalStateException("NiFiRegistryProperties cannot 
be null");
+}
+}
+
+/* UserGroupProviderLookup */
+
+@Override
+public UserGroupProvider getUserGroupProvider(String identifier) {
+return userGroupProviders.get(identifier);
+}
+
+/* AccessPolicyProviderLookup */
+
+@Override
+public AccessPolicyProvider getAccessPolicyProvider(String identifier) 
{
+return accessPolicyProviders.get(identifier);
+}
+
+
+/* AuthorizerLookup */
+
+@Override
+public 

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190272#comment-16190272
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142455640
  
--- Diff: 
nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/RegistryService.java
 ---
@@ -261,6 +262,29 @@ public Bucket deleteBucket(final String 
bucketIdentifier) {
 }
 }
 
+public List getBucketItems(final QueryParameters 
queryParameters, final Set bucketIdentifiers) {
+if (bucketIdentifiers == null || bucketIdentifiers.isEmpty()) {
+throw new IllegalArgumentException("Bucket Identifiers cannot 
be null or empty");
+}
+
+readLock.lock();
+try {
+final Set filterBuckets = 
bucketIdentifiers.stream()
--- End diff --

Should we create a method in metadataService to retrieve all the buckets 
for the set of ids in one call?

I'm just slightly concerned making an individual DB retrieval for each id 
if there are a lot of ids passed in.


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190277#comment-16190277
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142460901
  
--- Diff: nifi-registry-security/pom.xml ---
@@ -34,12 +34,28 @@
 jaxb2-maven-plugin
 
 
-xjc
+authorizations
 
 xjc
 
 
-
org.apache.nifi.registry.user.generated
+
--- End diff --

I noticed this file showed errors in my IDE, even though technically it 
built fine... seems like bumping the jaxb2-maven-plugin to 2.3.1 in the root 
pom pluginManagement cleared this up


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190280#comment-16190280
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142475289
  
--- Diff: 
nifi-registry-security/src/main/java/org/apache/nifi/registry/authorization/AccessPolicy.java
 ---
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.authorization;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.UUID;
+
+/**
+ * Defines a policy for a set of userIdentifiers to perform a set of 
actions on a given resource.
+ */
+public class AccessPolicy {
--- End diff --

I know this would be a tedious change, but what do you think about 
splitting out all of the interfaces/domain objects from nifi-registry-security 
into nifi-registry-security-api?

My thinking is that nifi-registry-security-api would live in the lib 
directory and be something that other people would implement against, where as 
nifi-registry-security would be bundled under the WAR for nifi-registry-web-api.


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190281#comment-16190281
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142481007
  
--- Diff: 
nifi-registry-resources/src/main/resources/conf/nifi-registry.properties ---
@@ -32,6 +32,8 @@ 
nifi.registry.security.truststoreType=${nifi.registry.security.truststoreType}
 
nifi.registry.security.truststorePasswd=${nifi.registry.security.truststorePasswd}
 
nifi.registry.security.needClientAuth=${nifi.registry.security.needClientAuth}
 
nifi.registry.security.authorized.users=${nifi.registry.security.authorized.users}
+nifi.security.user.authorizer=${nifi.security.user.authorizer}
--- End diff --

Is nifi.registry.security.authorized.users still needed?


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190273#comment-16190273
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142457464
  
--- Diff: nifi-registry-provider-api/pom.xml ---
@@ -31,5 +31,10 @@
 4.12
 test
 
+
--- End diff --

I think we can get rid of this, doing a quick test the build still passed 
without it


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190274#comment-16190274
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142479366
  
--- Diff: nifi-registry.properties ---
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
--- End diff --

I think this was maybe left-over from testing and not meant to be checked 
in?


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142427496
  
--- Diff: 
nifi-registry-framework/src/main/java/org/apache/nifi/registry/authorization/StandardAuthorizerFactory.java
 ---
@@ -0,0 +1,791 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.authorization;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.registry.authorization.annotation.AuthorizerContext;
+import org.apache.nifi.registry.authorization.generated.Authorizers;
+import org.apache.nifi.registry.authorization.generated.Prop;
+import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import org.apache.nifi.registry.provider.StandardProviderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.xml.sax.SAXException;
+
+import javax.xml.XMLConstants;
+import javax.xml.bind.JAXBContext;
+import javax.xml.bind.JAXBElement;
+import javax.xml.bind.JAXBException;
+import javax.xml.bind.Unmarshaller;
+import javax.xml.transform.stream.StreamSource;
+import javax.xml.validation.Schema;
+import javax.xml.validation.SchemaFactory;
+import java.io.File;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * This implementation of AuthorizerFactory in NiFi Registry is based on a 
combination of
+ * NiFi's AuthorizerFactory and AuthorizerFactoryBean.
+ */
+public class StandardAuthorizerFactory implements AuthorizerFactory, 
UserGroupProviderLookup, AccessPolicyProviderLookup, AuthorizerLookup{
+
+private static final Logger logger = 
LoggerFactory.getLogger(StandardProviderFactory.class);
+
+private static final String AUTHORIZERS_XSD = "/authorizers.xsd";
+private static final String JAXB_GENERATED_PATH = 
"org.apache.nifi.registry.authorization.generated";
+private static final JAXBContext JAXB_CONTEXT = 
initializeJaxbContext();
+
+/**
+ * Load the JAXBContext.
+ */
+private static JAXBContext initializeJaxbContext() {
+try {
+return JAXBContext.newInstance(JAXB_GENERATED_PATH, 
StandardAuthorizerFactory.class.getClassLoader());
+} catch (JAXBException e) {
+throw new RuntimeException("Unable to create JAXBContext.", e);
+}
+}
+
+private final NiFiRegistryProperties properties;
+private Authorizer authorizer;
+private final Map userGroupProviders = new 
HashMap<>();
+private final Map accessPolicyProviders 
= new HashMap<>();
+private final Map authorizers = new HashMap<>();
+
+public StandardAuthorizerFactory(final NiFiRegistryProperties 
properties) {
+this.properties = properties;
+
+if (this.properties == null) {
+throw new IllegalStateException("NiFiRegistryProperties cannot 
be null");
+}
+}
+
+/* UserGroupProviderLookup */
+
+@Override
+public UserGroupProvider getUserGroupProvider(String identifier) {
+return userGroupProviders.get(identifier);
+}
+
+/* AccessPolicyProviderLookup */
+
+@Override
+public AccessPolicyProvider getAccessPolicyProvider(String identifier) 
{
+return accessPolicyProviders.get(identifier);
+}
+
+
+/* AuthorizerLookup */
+
+@Override
+public Authorizer getAuthorizer(String identifier) {
+return authorizers.get(identifier);
+}
+
+
+/* AuthorizerFactory */
+
+@Override
+public void initialize() throws 

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190279#comment-16190279
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142480736
  
--- Diff: nifi-registry-assembly/pom.xml ---
@@ -137,6 +137,8 @@
 
 
 
./conf/authorized-users.xml
+
managed-authorizer
--- End diff --

Is the nifi.registry.security.authorized.users still needed?


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142453859
  
--- Diff: 
nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/AuthorizationService.java
 ---
@@ -0,0 +1,696 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.service;
+
+import org.apache.nifi.registry.authorization.AccessDeniedException;
+import org.apache.nifi.registry.authorization.AccessPolicyProvider;
+import 
org.apache.nifi.registry.authorization.AccessPolicyProviderInitializationContext;
+import org.apache.nifi.registry.authorization.AuthorizableLookup;
+import org.apache.nifi.registry.authorization.AuthorizationAccessException;
+import org.apache.nifi.registry.authorization.AuthorizeAccess;
+import org.apache.nifi.registry.authorization.Authorizer;
+import 
org.apache.nifi.registry.authorization.AuthorizerCapabilityDetection;
+import 
org.apache.nifi.registry.authorization.AuthorizerConfigurationContext;
+import org.apache.nifi.registry.authorization.AuthorizerCreationException;
+import 
org.apache.nifi.registry.authorization.AuthorizerDestructionException;
+import 
org.apache.nifi.registry.authorization.ConfigurableAccessPolicyProvider;
+import 
org.apache.nifi.registry.authorization.ConfigurableUserGroupProvider;
+import org.apache.nifi.registry.authorization.Group;
+import org.apache.nifi.registry.authorization.ManagedAuthorizer;
+import org.apache.nifi.registry.authorization.RequestAction;
+import org.apache.nifi.registry.authorization.UserAndGroups;
+import org.apache.nifi.registry.authorization.UserGroupProvider;
+import 
org.apache.nifi.registry.authorization.UserGroupProviderInitializationContext;
+import org.apache.nifi.registry.authorization.resource.ResourceFactory;
+import org.apache.nifi.registry.authorization.resource.ResourceType;
+import org.apache.nifi.registry.authorization.user.NiFiUserUtils;
+import org.apache.nifi.registry.bucket.Bucket;
+import org.apache.nifi.registry.model.authorization.AccessPolicy;
+import org.apache.nifi.registry.model.authorization.AccessPolicySummary;
+import org.apache.nifi.registry.model.authorization.Resource;
+import org.apache.nifi.registry.model.authorization.Tenant;
+import org.apache.nifi.registry.model.authorization.User;
+import org.apache.nifi.registry.model.authorization.UserGroup;
+import org.apache.nifi.registry.service.params.QueryParameters;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.stream.Collectors;
+
+@Service
+public class AuthorizationService {
+
+public static final String MSG_NON_MANAGED_AUTHORIZER = "This NiFi 
Registry is not configured to internally manage users, groups, or policies. 
Please contact your system administrator.";
+public static final String MSG_NON_CONFIGURABLE_POLICIES = "This NiFi 
Registry is not configured to allow configurable policies. Please contact your 
system administrator.";
+public static final String MSG_NON_CONFIGURABLE_USERS = "This NiFi 
Registry is not configured to allow configurable users and groups. Please 
contact your system administrator.";
+
+private AuthorizableLookup authorizableLookup;
+private Authorizer authorizer;
+private RegistryService registryService;
+private UserGroupProvider userGroupProvider;
+private AccessPolicyProvider accessPolicyProvider;
+
+private final ReentrantReadWriteLock lock = new 
ReentrantReadWriteLock();
+private final Lock readLock = lock.readLock();
+private final Lock writeLock = lock.writeLock();
+
+@Autowired
+public 

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142481007
  
--- Diff: 
nifi-registry-resources/src/main/resources/conf/nifi-registry.properties ---
@@ -32,6 +32,8 @@ 
nifi.registry.security.truststoreType=${nifi.registry.security.truststoreType}
 
nifi.registry.security.truststorePasswd=${nifi.registry.security.truststorePasswd}
 
nifi.registry.security.needClientAuth=${nifi.registry.security.needClientAuth}
 
nifi.registry.security.authorized.users=${nifi.registry.security.authorized.users}
+nifi.security.user.authorizer=${nifi.security.user.authorizer}
--- End diff --

Is nifi.registry.security.authorized.users still needed?


---

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142465687
  
--- Diff: nifi-registry-security/pom.xml ---
@@ -51,11 +67,25 @@
 org.apache.maven.plugins
 maven-checkstyle-plugin
 
-**/user/generated/*.java
+
**/authorization/file/generated/*.java,**/authorization/file/tenants/generated/*.java
 
 
 
 
+
+
+
+
+
+org.springframework
+spring-framework-bom
--- End diff --

What do you think about changing this and the spring-security-core 
dependency below to be:
```

org.springframework.boot
spring-boot-starter-security
${spring.boot.version}

```

Just seeing if we can make it easier to keep a consistent version across 
all the places where spring-boot is referenced.


---

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142475289
  
--- Diff: 
nifi-registry-security/src/main/java/org/apache/nifi/registry/authorization/AccessPolicy.java
 ---
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.authorization;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Set;
+import java.util.UUID;
+
+/**
+ * Defines a policy for a set of userIdentifiers to perform a set of 
actions on a given resource.
+ */
+public class AccessPolicy {
--- End diff --

I know this would be a tedious change, but what do you think about 
splitting out all of the interfaces/domain objects from nifi-registry-security 
into nifi-registry-security-api?

My thinking is that nifi-registry-security-api would live in the lib 
directory and be something that other people would implement against, where as 
nifi-registry-security would be bundled under the WAR for nifi-registry-web-api.


---

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142479366
  
--- Diff: nifi-registry.properties ---
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
--- End diff --

I think this was maybe left-over from testing and not meant to be checked 
in?


---

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142480736
  
--- Diff: nifi-registry-assembly/pom.xml ---
@@ -137,6 +137,8 @@
 
 
 
./conf/authorized-users.xml
+
managed-authorizer
--- End diff --

Is the nifi.registry.security.authorized.users still needed?


---

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142460901
  
--- Diff: nifi-registry-security/pom.xml ---
@@ -34,12 +34,28 @@
 jaxb2-maven-plugin
 
 
-xjc
+authorizations
 
 xjc
 
 
-
org.apache.nifi.registry.user.generated
+
--- End diff --

I noticed this file showed errors in my IDE, even though technically it 
built fine... seems like bumping the jaxb2-maven-plugin to 2.3.1 in the root 
pom pluginManagement cleared this up


---

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142476593
  
--- Diff: nifi-registry-web-api/pom.xml ---
@@ -105,6 +105,11 @@
 
 
 org.springframework.boot
+spring-boot-starter-security
--- End diff --

If we make the change in one of the previous comments where we use 
spring-boot-starter-security in the nifi-registry-security pom, then we 
probably don't need to directly reference it here because it will be brought in 
transitively


---

[GitHub] nifi-registry pull request #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/14#discussion_r142457464
  
--- Diff: nifi-registry-provider-api/pom.xml ---
@@ -31,5 +31,10 @@
 4.12
 test
 
+
--- End diff --

I think we can get rid of this, doing a quick test the build still passed 
without it


---

[jira] [Commented] (NIFI-4461) DistributedMapCacheClient/Server are inefficient when waiting for data to be received from socket

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190237#comment-16190237
 ] 

ASF GitHub Bot commented on NIFI-4461:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2193

NIFI-4461: When reading from socket channel use blocking mode instead…

… of sleeping; when writing, use a far smaller sleep duration

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4461

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2193.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2193


commit 0cefa7ee8de2abf72008444c5edd19ce96f782d4
Author: Mark Payne 
Date:   2017-10-03T19:55:02Z

NIFI-4461: When reading from socket channel use blocking mode instead of 
sleeping; when writing, use a far smaller sleep duration




> DistributedMapCacheClient/Server are inefficient when waiting for data to be 
> received from socket
> -
>
> Key: NIFI-4461
> URL: https://issues.apache.org/jira/browse/NIFI-4461
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> The DistributedMapCacheClient and DistributedMapCacheServer use the  
> SocketChannelInputStream and SSLSocketChannelInputStream (and output streams) 
> for communicating over a socket in non-blocking mode. This is done to allow a 
> timeout to occur on a socket write. However, when reading from the socket, it 
> ends up calling Thread.sleep(10, TimeUnit.MILLISECONDS) when there is no data 
> available, and that can result in extremely slow performance. Instead, we 
> should using blocking mode when receiving data because it will throw a 
> timeout exception as we desire. When writing, we should continue using 
> non-blocking mode but sleep for far less than 10 milliseconds.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2193: NIFI-4461: When reading from socket channel use blo...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2193

NIFI-4461: When reading from socket channel use blocking mode instead…

… of sleeping; when writing, use a far smaller sleep duration

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4461

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2193.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2193


commit 0cefa7ee8de2abf72008444c5edd19ce96f782d4
Author: Mark Payne 
Date:   2017-10-03T19:55:02Z

NIFI-4461: When reading from socket channel use blocking mode instead of 
sleeping; when writing, use a far smaller sleep duration




---

[jira] [Created] (NIFI-4460) ProcessSession.get(FlowFileFilter) does not search through FlowFiles that are swapped out

[Mark Payne (JIRA)]

Mark Payne created NIFI-4460:


 Summary: ProcessSession.get(FlowFileFilter) does not search 
through FlowFiles that are swapped out
 Key: NIFI-4460
 URL: https://issues.apache.org/jira/browse/NIFI-4460
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Mark Payne
Assignee: Mark Payne


If a Processor calls ProcessSession.get(FlowFileFilter), any FlowFile that is 
swapped out never gets passed to the filter. As a result, those FlowFiles will 
be skipped over. This can be problematic for a Processor like the Wait/Notify 
processors where this could result in a FlowFile not being released at the 
right time and even getting routed to the wrong relationship



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4372) Wait processor - recommend prioritizer in documentation

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190228#comment-16190228
 ] 

Mark Payne commented on NIFI-4372:
--

I was able to replicate the issue that you describe above. Looking into the 
processor it looks like it is calling ProcessSession.get(FlowFileFilter) and 
the StandardProcessSession appears not to look through any FlowFiles that are 
swapped out. I will create a separate JIRA for that issue.

> Wait processor - recommend prioritizer in documentation
> ---
>
> Key: NIFI-4372
> URL: https://issues.apache.org/jira/browse/NIFI-4372
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website, Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Trivial
>  Labels: documentation
>
> When using the Wait processor, the use of a connection prioritizer (FIFO for 
> instance) on the wait relationship should be recommended in the processor 
> documentation. When having tens of thousands of flow files in the wait 
> relationship, not using a prioritizer could lead to flow files not being 
> released when signal is notified and generate expired flow files.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4459) Confluent Schema Registry - not tolerant to error in single Subject

[Dave Torok (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Torok updated NIFI-4459:
-
Description: 
NOTE:  could not select "1.4.0" when creating this Jira.

Getting Could not find Schema with id 12345 from the Confluent Schema Registry.

Root cause:
1.  RestSchemaRegistryClient.java  - gets all subjects from the registry 
{noformat}https://{registry}/subjects{noformat}

2.  Iterates through the subjects to get schema IDs  (getSchema method line 
113-126)
{noformat}https://{registry}/subjects/{subjectname}/ids/latest{noformat}

3.  If  ANY subject in the list of subjects has an error / resource not found, 
lookup FAILS, even if the subject with the matching ID is anywhere in the list 
of subjects.

Proposed Fix: 
Tolerate individual subject calls that return errors (NOT_FOUND status).

  was:
NOTE:  could not select "1.4.0" when creating this Jira.

Getting Could not find Schema with id 12345 from the Confluent Schema Registry.

Root cause:
1.  RestSchemaRegistryClient.java  - gets all subjects from the registry 
https://{registry}/subjects

2.  Iterates through the subjects to get schema IDs  (getSchema method line 
113-126)
{{https://{registry}/subjects/{subjectname}/ids/latest}}

3.  If  ANY subject in the list of subjects has an error / resource not found, 
lookup FAILS, even if the subject with the matching ID is anywhere in the list 
of subjects.

Proposed Fix: 
Tolerate individual subject calls that return errors (NOT_FOUND status).


> Confluent Schema Registry - not tolerant to error in single Subject
> ---
>
> Key: NIFI-4459
> URL: https://issues.apache.org/jira/browse/NIFI-4459
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0, 1.5.0
>Reporter: Dave Torok
>Priority: Minor
>
> NOTE:  could not select "1.4.0" when creating this Jira.
> Getting Could not find Schema with id 12345 from the Confluent Schema 
> Registry.
> Root cause:
> 1.  RestSchemaRegistryClient.java  - gets all subjects from the registry 
> {noformat}https://{registry}/subjects{noformat}
> 2.  Iterates through the subjects to get schema IDs  (getSchema method line 
> 113-126)
> {noformat}https://{registry}/subjects/{subjectname}/ids/latest{noformat}
> 3.  If  ANY subject in the list of subjects has an error / resource not 
> found, lookup FAILS, even if the subject with the matching ID is anywhere in 
> the list of subjects.
> Proposed Fix: 
> Tolerate individual subject calls that return errors (NOT_FOUND status).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4459) Confluent Schema Registry - not tolerant to error in single Subject

[Dave Torok (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Torok updated NIFI-4459:
-
Description: 
NOTE:  could not select "1.4.0" when creating this Jira.

Getting Could not find Schema with id 12345 from the Confluent Schema Registry.

Root cause:
1.  RestSchemaRegistryClient.java  - gets all subjects from the registry 
https://{registry}/subjects

2.  Iterates through the subjects to get schema IDs  (getSchema method line 
113-126)
{{https://{registry}/subjects/{subjectname}/ids/latest}}

3.  If  ANY subject in the list of subjects has an error / resource not found, 
lookup FAILS, even if the subject with the matching ID is anywhere in the list 
of subjects.

Proposed Fix: 
Tolerate individual subject calls that return errors (NOT_FOUND status).

  was:
NOTE:  could not select "1.4.0" when creating this Jira.

Getting Could not find Schema with id 12345 from the Confluent Schema Registry.

Root cause:
1.  RestSchemaRegistryClient.java  - gets all subjects from the registry 
{{https://{registry}/subjects}}

2.  Iterates through the subjects to get schema IDs  (getSchema method line 
113-126)
{{https://{registry}/subjects/{subjectname}/ids/latest}}

3.  If  ANY subject in the list of subjects has an error / resource not found, 
lookup FAILS, even if the subject with the matching ID is anywhere in the list 
of subjects.

Proposed Fix: 
Tolerate individual subject calls that return errors (NOT_FOUND status).


> Confluent Schema Registry - not tolerant to error in single Subject
> ---
>
> Key: NIFI-4459
> URL: https://issues.apache.org/jira/browse/NIFI-4459
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0, 1.5.0
>Reporter: Dave Torok
>Priority: Minor
>
> NOTE:  could not select "1.4.0" when creating this Jira.
> Getting Could not find Schema with id 12345 from the Confluent Schema 
> Registry.
> Root cause:
> 1.  RestSchemaRegistryClient.java  - gets all subjects from the registry 
> https://{registry}/subjects
> 2.  Iterates through the subjects to get schema IDs  (getSchema method line 
> 113-126)
> {{https://{registry}/subjects/{subjectname}/ids/latest}}
> 3.  If  ANY subject in the list of subjects has an error / resource not 
> found, lookup FAILS, even if the subject with the matching ID is anywhere in 
> the list of subjects.
> Proposed Fix: 
> Tolerate individual subject calls that return errors (NOT_FOUND status).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4459) Confluent Schema Registry - not tolerant to error in single Subject

[Dave Torok (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dave Torok updated NIFI-4459:
-
Description: 
NOTE:  could not select "1.4.0" when creating this Jira.

Getting Could not find Schema with id 12345 from the Confluent Schema Registry.

Root cause:
1.  RestSchemaRegistryClient.java  - gets all subjects from the registry 
{quote}https://{registry}/subjects{quote}

2.  Iterates through the subjects to get schema IDs  (getSchema method line 
113-126)
{quote}https://{registry}/subjects/{subjectname}/ids/latest{quote}

3.  If  ANY subject in the list of subjects has an error / resource not found, 
lookup FAILS, even if the subject with the matching ID is anywhere in the list 
of subjects.

Proposed Fix: 
Tolerate individual subject calls that return errors (NOT_FOUND status).

  was:
NOTE:  could not select "1.4.0" when creating this Jira.

Getting Could not find Schema with id 12345 from the Confluent Schema Registry.

Root cause:
1.  RestSchemaRegistryClient.java  - gets all subjects from the registry 
https://{registry}/subjects

2.  Iterates through the subjects to get schema IDs  (getSchema method line 
113-126)
https://{registry}/subjects/{subjectname}/ids/latest

3.  If  ANY subject in the list of subjects has an error / resource not found, 
lookup FAILS, even if the subject with the matching ID is anywhere in the list 
of subjects.

Proposed Fix: 
Tolerate individual subject calls that return errors (NOT_FOUND status).


> Confluent Schema Registry - not tolerant to error in single Subject
> ---
>
> Key: NIFI-4459
> URL: https://issues.apache.org/jira/browse/NIFI-4459
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0, 1.5.0
>Reporter: Dave Torok
>Priority: Minor
>
> NOTE:  could not select "1.4.0" when creating this Jira.
> Getting Could not find Schema with id 12345 from the Confluent Schema 
> Registry.
> Root cause:
> 1.  RestSchemaRegistryClient.java  - gets all subjects from the registry 
> {quote}https://{registry}/subjects{quote}
> 2.  Iterates through the subjects to get schema IDs  (getSchema method line 
> 113-126)
> {quote}https://{registry}/subjects/{subjectname}/ids/latest{quote}
> 3.  If  ANY subject in the list of subjects has an error / resource not 
> found, lookup FAILS, even if the subject with the matching ID is anywhere in 
> the list of subjects.
> Proposed Fix: 
> Tolerate individual subject calls that return errors (NOT_FOUND status).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4459) Confluent Schema Registry - not tolerant to error in single Subject

[Dave Torok (JIRA)]

Dave Torok created NIFI-4459:


 Summary: Confluent Schema Registry - not tolerant to error in 
single Subject
 Key: NIFI-4459
 URL: https://issues.apache.org/jira/browse/NIFI-4459
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.3.0, 1.5.0
Reporter: Dave Torok
Priority: Minor


NOTE:  could not select "1.4.0" when creating this Jira.

Getting Could not find Schema with id 12345 from the Confluent Schema Registry.

Root cause:
1.  RestSchemaRegistryClient.java  - gets all subjects from the registry 
https://{registry}/subjects

2.  Iterates through the subjects to get schema IDs  (getSchema method line 
113-126)
https://{registry}/subjects/{subjectname}/ids/latest

3.  If  ANY subject in the list of subjects has an error / resource not found, 
lookup FAILS, even if the subject with the matching ID is anywhere in the list 
of subjects.

Proposed Fix: 
Tolerate individual subject calls that return errors (NOT_FOUND status).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2180: Added GetMongoAggregation to support running Mongo ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2180#discussion_r142480211
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
 ---
@@ -102,8 +107,26 @@
 .defaultValue(WRITE_CONCERN_ACKNOWLEDGED)
 .build();
 
+static final PropertyDescriptor RESULTS_PER_FLOWFILE = new 
PropertyDescriptor.Builder()
+.name("results-per-flowfile")
+.displayName("Results Per FlowFile")
+.description("How many results to put into a flowfile at once. 
The whole body will be treated as a JSON array of results.")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("1")
+.build();
+
+static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("Batch Size")
--- End diff --

In this case, the guidance for a new `PropertyDescriptor` in an existing 
class (or new class) is to add a `displayName` value and provide a 
"machine-readable" (i.e. no space) value for `name`. When modifying an 
*existing* `PropertyDescriptor`, adding a `displayName` value is encouraged, 
but modifying the `name` will break backward compatibility with existing flows. 


---

[jira] [Updated] (NIFI-4439) WriteAheadProvenanceRepository fails often to cleanup provenance files filling up disks

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4439:
-
 Assignee: Mark Payne
Fix Version/s: 1.5.0
   Status: Patch Available  (was: Open)

> WriteAheadProvenanceRepository fails often to cleanup provenance files 
> filling up disks
> ---
>
> Key: NIFI-4439
> URL: https://issues.apache.org/jira/browse/NIFI-4439
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Extensions
> Environment: Windows 10
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.5.0
>
>
> Lots of warnings in log file which state
> EventFileCompressor Failed to delete .\provenance_repository\1919191914.prov; 
> this file should be cleaned up manually
> Provenance appears to work correctly otherwise but these files fill up disk 
> space since they are the before compression version and they aren't being 
> cleaned up.  Perhaps we're attempting to delete the files before the 
> input/output streams are closed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4458) Adding Expression Language Support for Port Numbers in ListenTCP, ListenUDP, ListenSyslog

[Dhirendra Singh (JIRA)]

Dhirendra Singh created NIFI-4458:
-

 Summary: Adding Expression Language Support for Port Numbers in 
ListenTCP, ListenUDP, ListenSyslog
 Key: NIFI-4458
 URL: https://issues.apache.org/jira/browse/NIFI-4458
 Project: Apache NiFi
  Issue Type: Improvement
 Environment: NiFi component
Reporter: Dhirendra Singh
 Attachments: nifierror_11.PNG

Hi All,

In case of ListenTCP, ListenUDP, ListenSyslog we can have local interface name 
picked from a custom.properties file. However, there is no support for port 
number. Port number is still hardcoded.

Thanks
Dhiren



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4328) Invalid swagger.json generated from ControllerServiceReferencingComponentDTO#referencingCompoents

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190004#comment-16190004
 ] 

ASF GitHub Bot commented on NIFI-4328:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2191
  
@mcgilman thanks for updating this. All looks good, except that i needed to 
make a minor change to the maven pom for the new version of swagger. Then was 
able to build and startup new version and verify that the Rest API docs all 
look good. +1 merged to master. Thanks!


> Invalid swagger.json generated from 
> ControllerServiceReferencingComponentDTO#referencingCompoents
> -
>
> Key: NIFI-4328
> URL: https://issues.apache.org/jira/browse/NIFI-4328
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Michael Werle
>Assignee: Matt Gilman
> Fix For: 1.5.0
>
>
> The referencingComponents field in 
> {{nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ControllerServiceReferencingComponentDTO.java}}
>  field generates the following lines in swagger.json:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "$ref" : "#/definitions/Set"
> }
>   }
> }
> {code}
> Which causes this invalid object definition:
> {code:javascript}
> "Set" : {
>   "properties" : {
> "empty" : {
>   "type" : "boolean",
>   "default" : false
> }
>   }
> }
> {code}
> It is not clear how to fix the annotation, but the generated swagger.json 
> should be:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "type" : "array",
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "uniqueItems" : true,
>   "items" : {
> "$ref" : 
> "#/definitions/ControllerServiceReferencingComponentEntity"
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4328) Invalid swagger.json generated from ControllerServiceReferencingComponentDTO#referencingCompoents

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4328:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Invalid swagger.json generated from 
> ControllerServiceReferencingComponentDTO#referencingCompoents
> -
>
> Key: NIFI-4328
> URL: https://issues.apache.org/jira/browse/NIFI-4328
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Michael Werle
>Assignee: Matt Gilman
> Fix For: 1.5.0
>
>
> The referencingComponents field in 
> {{nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ControllerServiceReferencingComponentDTO.java}}
>  field generates the following lines in swagger.json:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "$ref" : "#/definitions/Set"
> }
>   }
> }
> {code}
> Which causes this invalid object definition:
> {code:javascript}
> "Set" : {
>   "properties" : {
> "empty" : {
>   "type" : "boolean",
>   "default" : false
> }
>   }
> }
> {code}
> It is not clear how to fix the annotation, but the generated swagger.json 
> should be:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "type" : "array",
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "uniqueItems" : true,
>   "items" : {
> "$ref" : 
> "#/definitions/ControllerServiceReferencingComponentEntity"
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4328) Invalid swagger.json generated from ControllerServiceReferencingComponentDTO#referencingCompoents

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190003#comment-16190003
 ] 

ASF GitHub Bot commented on NIFI-4328:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2191


> Invalid swagger.json generated from 
> ControllerServiceReferencingComponentDTO#referencingCompoents
> -
>
> Key: NIFI-4328
> URL: https://issues.apache.org/jira/browse/NIFI-4328
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Michael Werle
>Assignee: Matt Gilman
> Fix For: 1.5.0
>
>
> The referencingComponents field in 
> {{nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ControllerServiceReferencingComponentDTO.java}}
>  field generates the following lines in swagger.json:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "$ref" : "#/definitions/Set"
> }
>   }
> }
> {code}
> Which causes this invalid object definition:
> {code:javascript}
> "Set" : {
>   "properties" : {
> "empty" : {
>   "type" : "boolean",
>   "default" : false
> }
>   }
> }
> {code}
> It is not clear how to fix the annotation, but the generated swagger.json 
> should be:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "type" : "array",
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "uniqueItems" : true,
>   "items" : {
> "$ref" : 
> "#/definitions/ControllerServiceReferencingComponentEntity"
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2191: NIFI-4328: Upgrade Swagger

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2191


---

[GitHub] nifi issue #2191: NIFI-4328: Upgrade Swagger

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2191
  
@mcgilman thanks for updating this. All looks good, except that i needed to 
make a minor change to the maven pom for the new version of swagger. Then was 
able to build and startup new version and verify that the Rest API docs all 
look good. +1 merged to master. Thanks!


---

[jira] [Commented] (NIFI-4328) Invalid swagger.json generated from ControllerServiceReferencingComponentDTO#referencingCompoents

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190002#comment-16190002
 ] 

ASF subversion and git services commented on NIFI-4328:
---

Commit e773fa551362d9adf9c9544b1525fd38e9c07887 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=e773fa5 ]

NIFI-4328: Fix to pom.xml for swagger maven plugin. This closes #2191.


> Invalid swagger.json generated from 
> ControllerServiceReferencingComponentDTO#referencingCompoents
> -
>
> Key: NIFI-4328
> URL: https://issues.apache.org/jira/browse/NIFI-4328
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Michael Werle
>Assignee: Matt Gilman
> Fix For: 1.5.0
>
>
> The referencingComponents field in 
> {{nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ControllerServiceReferencingComponentDTO.java}}
>  field generates the following lines in swagger.json:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "$ref" : "#/definitions/Set"
> }
>   }
> }
> {code}
> Which causes this invalid object definition:
> {code:javascript}
> "Set" : {
>   "properties" : {
> "empty" : {
>   "type" : "boolean",
>   "default" : false
> }
>   }
> }
> {code}
> It is not clear how to fix the annotation, but the generated swagger.json 
> should be:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "type" : "array",
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "uniqueItems" : true,
>   "items" : {
> "$ref" : 
> "#/definitions/ControllerServiceReferencingComponentEntity"
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2180: Added GetMongoAggregation to support running Mongo ...

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2180#discussion_r142458671
  
--- Diff: 
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/AbstractMongoProcessor.java
 ---
@@ -102,8 +107,26 @@
 .defaultValue(WRITE_CONCERN_ACKNOWLEDGED)
 .build();
 
+static final PropertyDescriptor RESULTS_PER_FLOWFILE = new 
PropertyDescriptor.Builder()
+.name("results-per-flowfile")
+.displayName("Results Per FlowFile")
+.description("How many results to put into a flowfile at once. 
The whole body will be treated as a JSON array of results.")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("1")
+.build();
+
+static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("Batch Size")
--- End diff --

I know, but I didn't write the class and it's been around for a while. I 
wouldn't want to risk breaking others' flows over that.


---

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189935#comment-16189935
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2184
  
@frett27 thanks for your contribution! It looks like you're provided some 
test data that is used in a unit test. Any idea what the license would be on 
this test data? I'm guessing from looking at it that it was provided by 
osm.org? If licensing information is not available, it may make sense to just 
create some mock-up data instead?


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4441) Add MapRecord support inside avro union types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189940#comment-16189940
 ] 

ASF GitHub Bot commented on NIFI-4441:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2184#discussion_r142454463
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -238,5 +242,23 @@ public void testComplicatedRecursiveSchema() {
 // Make sure the 'parent' field has a schema reference back to the 
original top level record schema
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
+
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new Schema.Parser().parse(
+
"{\"type\":\"record\",\"name\":\"OSMEntity\",\"namespace\":\"org.osm.avro\",\"fields\":[{\"name\":\"osmtype\",\"type\":{\"type\":\"enum\",\"name\":\"OSMType\",\"symbols\":[\"NODE\",\"WAY\",\"POLYGON\",\"RELATION\"]}},{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"node\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ANode\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"x\",\"type\":\"double\"},{\"name\":\"y\",\"type\":\"double\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"way\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"AComplex\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"geometry\",\"type\":\"bytes\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"polygon\",\"type\":[\"null\",\"AComplex\"]},{\"name\":\"rel\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ARelation\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]},{\"name\":\"related\",\"type\":[\"null\",{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"ARelated\",\"fields\":[{\"name\":\"relatedId\",\"type\":\"long\"},{\"name\":\"type\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"role\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}]}}]}]}]}]}"
--- End diff --

I think we'd be better off to externalize this into a .avsc file under 
src/test/resources


> Add MapRecord support inside avro union types
> -
>
> Key: NIFI-4441
> URL: https://issues.apache.org/jira/browse/NIFI-4441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Patrice Freydiere
>
> Using an avro union type that contain maps in the definition lead to errors 
> in loading avro records.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2184: NIFI-4441 : add maprecord support for avro union ty...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2184#discussion_r142454463
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java
 ---
@@ -238,5 +242,23 @@ public void testComplicatedRecursiveSchema() {
 // Make sure the 'parent' field has a schema reference back to the 
original top level record schema
 Assert.assertEquals(recordASchema, 
((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
 }
+
+@Test
+public void testMapWithNullSchema() throws IOException {
+
+Schema recursiveSchema = new Schema.Parser().parse(
+
"{\"type\":\"record\",\"name\":\"OSMEntity\",\"namespace\":\"org.osm.avro\",\"fields\":[{\"name\":\"osmtype\",\"type\":{\"type\":\"enum\",\"name\":\"OSMType\",\"symbols\":[\"NODE\",\"WAY\",\"POLYGON\",\"RELATION\"]}},{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"node\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ANode\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"x\",\"type\":\"double\"},{\"name\":\"y\",\"type\":\"double\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"way\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"AComplex\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"geometry\",\"type\":\"bytes\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"polygon\",\"type\
 
":[\"null\",\"AComplex\"]},{\"name\":\"rel\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ARelation\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]},{\"name\":\"related\",\"type\":[\"null\",{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"ARelated\",\"fields\":[{\"name\":\"relatedId\",\"type\":\"long\"},{\"name\":\"type\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"role\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}]}}]}]}]}]}"
--- End diff --

I think we'd be better off to externalize this into a .avsc file under 
src/test/resources


---

[GitHub] nifi pull request #2184: NIFI-4441 : add maprecord support for avro union ty...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2184#discussion_r142454362
  
--- Diff: 
nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
 ---
@@ -691,6 +697,10 @@ private static boolean isCompatibleDataType(final 
Object value, final DataType d
 return true;
 }
 break;
+case MAP:
+if (value instanceof MapRecord) {
--- End diff --

We can probably use the more generic Record instead of MapRecord. We should 
probably also consider if value instanceof java.util.Map


---

[GitHub] nifi issue #2184: NIFI-4441 : add maprecord support for avro union types

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2184
  
@frett27 thanks for your contribution! It looks like you're provided some 
test data that is used in a unit test. Any idea what the license would be on 
this test data? I'm guessing from looking at it that it was provided by 
osm.org? If licensing information is not available, it may make sense to just 
create some mock-up data instead?


---

[jira] [Updated] (NIFI-4457) Maximum-value not increasing when initial.maxvalue is set and "Maximum-value column" name is different from "id"

[meh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

meh updated NIFI-4457:
--
Summary: Maximum-value not increasing when initial.maxvalue is set and 
"Maximum-value column" name is different from "id"   (was: Maximum-value not 
increasing when initial.maxvalue is set)

> Maximum-value not increasing when initial.maxvalue is set and "Maximum-value 
> column" name is different from "id" 
> -
>
> Key: NIFI-4457
> URL: https://issues.apache.org/jira/browse/NIFI-4457
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
> Environment: windows 10
>Reporter: meh
> Attachments: Picture1.png, Picture2.png
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> when "Maximum-value column" name is "id" there is no problem, when i add 
> "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works 
> well and maxvalue is increasing by every running.
> !Picture1.png|thumbnail!
> but...
> when the "Maximum-value column" name is different from "id" (such as 
> "tweet_id"), after initial processor working, only given 
> "initial.maxvalue.id" is saves and that repeating just same value for every 
> run.
> !Picture2.png|thumbnail!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4457) "Maximum-value" not increasing when "initial.maxvalue" is set and "Maximum-value column" name is different from "id"

[meh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

meh updated NIFI-4457:
--
Summary: "Maximum-value" not increasing when "initial.maxvalue" is set and 
"Maximum-value column" name is different from "id"   (was: Maximum-value not 
increasing when initial.maxvalue is set and "Maximum-value column" name is 
different from "id" )

> "Maximum-value" not increasing when "initial.maxvalue" is set and 
> "Maximum-value column" name is different from "id" 
> -
>
> Key: NIFI-4457
> URL: https://issues.apache.org/jira/browse/NIFI-4457
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
> Environment: windows 10
>Reporter: meh
> Attachments: Picture1.png, Picture2.png
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> when "Maximum-value column" name is "id" there is no problem, when i add 
> "initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works 
> well and maxvalue is increasing by every running.
> !Picture1.png|thumbnail!
> but...
> when the "Maximum-value column" name is different from "id" (such as 
> "tweet_id"), after initial processor working, only given 
> "initial.maxvalue.id" is saves and that repeating just same value for every 
> run.
> !Picture2.png|thumbnail!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4457) Maximum-value not increasing when initial.maxvalue is set

[meh (JIRA)]

meh created NIFI-4457:
-

 Summary: Maximum-value not increasing when initial.maxvalue is set
 Key: NIFI-4457
 URL: https://issues.apache.org/jira/browse/NIFI-4457
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.3.0
 Environment: windows 10
Reporter: meh
 Attachments: Picture1.png, Picture2.png

when "Maximum-value column" name is "id" there is no problem, when i add 
"initial.maxvalue.id" property in "QueryDatabaseTable" processor, it works well 
and maxvalue is increasing by every running.
!Picture1.png|thumbnail!

but...
when the "Maximum-value column" name is different from "id" (such as 
"tweet_id"), after initial processor working, only given "initial.maxvalue.id" 
is saves and that repeating just same value for every run.
!Picture2.png|thumbnail!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4456) Update JSON Record Reader / Writer to allow for 'json per line' format

[Mark Payne (JIRA)]

Mark Payne created NIFI-4456:


 Summary: Update JSON Record Reader / Writer to allow for 'json per 
line' format
 Key: NIFI-4456
 URL: https://issues.apache.org/jira/browse/NIFI-4456
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: Mark Payne


It is common, especially for archiving purposes, to have many JSON objects 
combined with new-lines in between, in order to delimit the records. It would 
be useful to allow record readers and writers to support this, instead of 
requiring that JSON records being elements in a JSON Array.

For example, the following JSON Is considered two records:
{code}
[
  { "greeting" : "hello", "id" : 1 },
  { "greeting" : "good-bye", "id" : 2 }
]
{code}

It would be beneficial to also support the format:
{code}
{ "greeting" : "hello", "id" : 1 }
{ "greeting" : "good-bye", "id" : 2 }
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-1547) DistributedMapCacheServer: Ambiguous error on misconfiguration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189817#comment-16189817
 ] 

ASF GitHub Bot commented on NIFI-1547:
--

GitHub user jmark99 opened a pull request:

https://github.com/apache/nifi/pull/2192

NIFI-1547: DistributedMapCacheServer: Ambiguous error on misconfiguration

misconfiguration

Updated PersistentMapCache constructor in PersistentMapCache.java to
catch an OverlappingFileLockException and present a more useful error
message before propagating the exception forward. The log message alerts
user to possible duplicated persistencePath in call to
PersistentMapCache.

Created a test method to verify the exception is thrown as expected.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jmark99/nifi NIFI-1547

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2192.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2192


commit cd7a8fc91827688032c57a1955b5443cb588b653
Author: Mark Owens 
Date:   2017-10-03T14:48:42Z

NIFI-1547: DistributedMapCacheServer: Ambiguous error on
misconfiguration

Updated PersistentMapCache constructor in PersistentMapCache.java to
catch an OverlappingFileLockException and present a more useful error
message before propagating the exception forward. The log message alerts
user to possible duplicated persistencePath in call to
PersistentMapCache.

Created a test method to verify the exception is thrown as expected.




> DistributedMapCacheServer: Ambiguous error on misconfiguration
> --
>
> Key: NIFI-1547
> URL: https://issues.apache.org/jira/browse/NIFI-1547
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Brandon DeVries
>Assignee: Mark Owens
>Priority: Minor
>
> If a DistributedMapCacheServer's "Persistence Directory" is set to the same 
> location as another DistributedMapCacheServer (or another controller service 
> with a  "Persistence Directory") only one will enable successfully.  The 
> other(s) will get an OverlappingFileLockException.  Consider catching this 
> exception \[1\] and providing additional information (e.g. check the lock 
> location /  "Persistence Directory").
> \[1\] 
> https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-nar-bundles/nifi-standard-services/nifi-distributed-cache-services-bundle/nifi-distributed-cache-server/src/main/java/org/apache/nifi/distributed/cache/server/map/PersistentMapCache.java#L44



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2192: NIFI-1547: DistributedMapCacheServer: Ambiguous err...

[jmark99]

GitHub user jmark99 opened a pull request:

https://github.com/apache/nifi/pull/2192

NIFI-1547: DistributedMapCacheServer: Ambiguous error on misconfiguration

misconfiguration

Updated PersistentMapCache constructor in PersistentMapCache.java to
catch an OverlappingFileLockException and present a more useful error
message before propagating the exception forward. The log message alerts
user to possible duplicated persistencePath in call to
PersistentMapCache.

Created a test method to verify the exception is thrown as expected.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jmark99/nifi NIFI-1547

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2192.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2192


commit cd7a8fc91827688032c57a1955b5443cb588b653
Author: Mark Owens 
Date:   2017-10-03T14:48:42Z

NIFI-1547: DistributedMapCacheServer: Ambiguous error on
misconfiguration

Updated PersistentMapCache constructor in PersistentMapCache.java to
catch an OverlappingFileLockException and present a more useful error
message before propagating the exception forward. The log message alerts
user to possible duplicated persistencePath in call to
PersistentMapCache.

Created a test method to verify the exception is thrown as expected.




---

[jira] [Commented] (MINIFICPP-197) IOS port for C++

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189800#comment-16189800
 ] 

ASF GitHub Bot commented on MINIFICPP-197:
--

GitHub user phrocker opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/141

Make libCuRL containing classes extensions.

MINIFI-215: Add conditional to HTTP-CURL

MINIFI-215: Update Readme

MINIFICPP-215: Udate tests and ensure extensions doc is up to date

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-215

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/141.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #141


commit 24f052e262414e1e28d373a1081d5f784942a0e3
Author: Marc Parisi 
Date:   2017-09-19T17:09:59Z

Make libCuRL containing classes extensions.

MINIFI-215: Add conditional to HTTP-CURL

MINIFI-215: Update Readme

MINIFICPP-215: Udate tests and ensure extensions doc is up to date




> IOS port for C++
> 
>
> Key: MINIFICPP-197
> URL: https://issues.apache.org/jira/browse/MINIFICPP-197
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: bqiu
>Assignee: bqiu
> Fix For: 0.2.0
>
>
> IOS port for C++



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-minifi-cpp pull request #141: Make libCuRL containing classes extension...

[phrocker]

GitHub user phrocker opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/141

Make libCuRL containing classes extensions.

MINIFI-215: Add conditional to HTTP-CURL

MINIFI-215: Update Readme

MINIFICPP-215: Udate tests and ensure extensions doc is up to date

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-215

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/141.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #141


commit 24f052e262414e1e28d373a1081d5f784942a0e3
Author: Marc Parisi 
Date:   2017-09-19T17:09:59Z

Make libCuRL containing classes extensions.

MINIFI-215: Add conditional to HTTP-CURL

MINIFI-215: Update Readme

MINIFICPP-215: Udate tests and ensure extensions doc is up to date




---

[jira] [Updated] (MINIFICPP-215) Create Conditional build, using libcURL as a first example

[marco polo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

marco polo updated MINIFICPP-215:
-
Summary: Create Conditional build, using libcURL as a first example  (was: 
Create Conditional build)

> Create Conditional build, using libcURL as a first example
> --
>
> Key: MINIFICPP-215
> URL: https://issues.apache.org/jira/browse/MINIFICPP-215
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: marco polo
>Assignee: marco polo
>
> Create a conditional build based on what dependencies exist. The first step 
> that I've taken is to extract CURL capabilities into an extension directory. 
> It will be built IFF libcurl exists. That can be the basis for conditional 
> builds. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4328) Invalid swagger.json generated from ControllerServiceReferencingComponentDTO#referencingCompoents

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-4328:
--
Status: Patch Available  (was: In Progress)

> Invalid swagger.json generated from 
> ControllerServiceReferencingComponentDTO#referencingCompoents
> -
>
> Key: NIFI-4328
> URL: https://issues.apache.org/jira/browse/NIFI-4328
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Michael Werle
>Assignee: Matt Gilman
>
> The referencingComponents field in 
> {{nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ControllerServiceReferencingComponentDTO.java}}
>  field generates the following lines in swagger.json:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "$ref" : "#/definitions/Set"
> }
>   }
> }
> {code}
> Which causes this invalid object definition:
> {code:javascript}
> "Set" : {
>   "properties" : {
> "empty" : {
>   "type" : "boolean",
>   "default" : false
> }
>   }
> }
> {code}
> It is not clear how to fix the annotation, but the generated swagger.json 
> should be:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "type" : "array",
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "uniqueItems" : true,
>   "items" : {
> "$ref" : 
> "#/definitions/ControllerServiceReferencingComponentEntity"
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4328) Invalid swagger.json generated from ControllerServiceReferencingComponentDTO#referencingCompoents

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-4328:
--
Fix Version/s: 1.5.0

> Invalid swagger.json generated from 
> ControllerServiceReferencingComponentDTO#referencingCompoents
> -
>
> Key: NIFI-4328
> URL: https://issues.apache.org/jira/browse/NIFI-4328
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Michael Werle
>Assignee: Matt Gilman
> Fix For: 1.5.0
>
>
> The referencingComponents field in 
> {{nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ControllerServiceReferencingComponentDTO.java}}
>  field generates the following lines in swagger.json:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "$ref" : "#/definitions/Set"
> }
>   }
> }
> {code}
> Which causes this invalid object definition:
> {code:javascript}
> "Set" : {
>   "properties" : {
> "empty" : {
>   "type" : "boolean",
>   "default" : false
> }
>   }
> }
> {code}
> It is not clear how to fix the annotation, but the generated swagger.json 
> should be:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "type" : "array",
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "uniqueItems" : true,
>   "items" : {
> "$ref" : 
> "#/definitions/ControllerServiceReferencingComponentEntity"
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4328) Invalid swagger.json generated from ControllerServiceReferencingComponentDTO#referencingCompoents

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189772#comment-16189772
 ] 

ASF GitHub Bot commented on NIFI-4328:
--

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/2191

NIFI-4328: Upgrade Swagger

NIFI-4328:
- Upgrading swagger and the corresponding maven plugin to address issues 
discovered in recursive domain models.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-4328

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2191.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2191


commit 697c382d9bd4a3aaa05bc28b64a5e74592101c61
Author: Matt Gilman 
Date:   2017-09-28T15:02:27Z

NIFI-4328:
- Upgrading swagger and the corresponding maven plugin to address issues 
discovered in recursive domain models.




> Invalid swagger.json generated from 
> ControllerServiceReferencingComponentDTO#referencingCompoents
> -
>
> Key: NIFI-4328
> URL: https://issues.apache.org/jira/browse/NIFI-4328
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Michael Werle
>Assignee: Matt Gilman
>
> The referencingComponents field in 
> {{nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/ControllerServiceReferencingComponentDTO.java}}
>  field generates the following lines in swagger.json:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "$ref" : "#/definitions/Set"
> }
>   }
> }
> {code}
> Which causes this invalid object definition:
> {code:javascript}
> "Set" : {
>   "properties" : {
> "empty" : {
>   "type" : "boolean",
>   "default" : false
> }
>   }
> }
> {code}
> It is not clear how to fix the annotation, but the generated swagger.json 
> should be:
> {code:javascript}
> "ControllerServiceReferencingComponentDTO" : {
>   "properties" : {
> //... (omitted for brevity)
> "referencingComponents" : {
>   "type" : "array",
>   "description" : "If the referencing component represents a 
> controller service, these are the components that reference it.",
>   "uniqueItems" : true,
>   "items" : {
> "$ref" : 
> "#/definitions/ControllerServiceReferencingComponentEntity"
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2191: NIFI-4328: Upgrade Swagger

[mcgilman]

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/2191

NIFI-4328: Upgrade Swagger

NIFI-4328:
- Upgrading swagger and the corresponding maven plugin to address issues 
discovered in recursive domain models.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-4328

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2191.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2191


commit 697c382d9bd4a3aaa05bc28b64a5e74592101c61
Author: Matt Gilman 
Date:   2017-09-28T15:02:27Z

NIFI-4328:
- Upgrading swagger and the corresponding maven plugin to address issues 
discovered in recursive domain models.




---

[jira] [Created] (NIFI-4455) nifi-nar-maven-plugin - Nar-Dependency-Id not being written to manifest

[Steven Youtsey (JIRA)]

Steven Youtsey created NIFI-4455:


 Summary: nifi-nar-maven-plugin - Nar-Dependency-Id not being 
written to manifest
 Key: NIFI-4455
 URL: https://issues.apache.org/jira/browse/NIFI-4455
 Project: Apache NiFi
  Issue Type: Bug
  Components: Tools and Build
Affects Versions: 1.1.0
Reporter: Steven Youtsey
Priority: Minor


Within a nar pom that extends/depends on a parent nar, the parent nar's id is 
not written to the Nar-Dependency-Id in the Manifest file when the parent nar's 
maven artifact contains a maven classifier.


myGroup
my-service-api-nar
1.2.3
nar
gpu


produces...

Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Built-By: unattributed
Nar-Id: my-service
Created-By: Apache Maven 3.3.3
Build-Jdk: 1.8.0_131

The workaround for this was to override the plugin and add the 
Nar-Dependency-Id via manifestEntries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4008) ConsumeKafkaRecord_0_10 assumes there is always one Record in a message

[Gardella Juan Pablo (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189754#comment-16189754
 ] 

Gardella Juan Pablo commented on NIFI-4008:
---

It breaks an scenario of NIFI-4330, it does not handle null values properly.

> ConsumeKafkaRecord_0_10 assumes there is always one Record in a message
> ---
>
> Key: NIFI-4008
> URL: https://issues.apache.org/jira/browse/NIFI-4008
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>
> ConsumeKafkaRecord_0_10 uses ConsumerLease underneath, and it [assumes there 
> is one Record available in a consumed 
> message|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-10-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumerLease.java#L434]
>  retrieved from a Kafka topic.
> But in fact, a message can contain 0 or more records in it. For example, with 
> a record schema shown below:
> {code}
> {
>   "type": "record",
>   "name": "temp",
>   "fields" : [
> {"name": "value", "type": "string"}
>   ]
> }
> {code}
> Multiple records can be sent within a single message, e.g. using JSON:
> {code}
> [{"value": "a"}, {"value": "b"}, {"value": "c"}]
> {code}
> But ConsumeKafkaRecord only outputs the first record:
> {code}
> [{"value": "a"}]
> {code}
> Also, if a message doesn't contain any record in it, the processor fails with 
> NullPointerException.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4454) When user searches in the nifi Search toolbar, results are not shown for Process Group variables

[Mark Payne (JIRA)]

Mark Payne created NIFI-4454:


 Summary: When user searches in the nifi Search toolbar, results 
are not shown for Process Group variables
 Key: NIFI-4454
 URL: https://issues.apache.org/jira/browse/NIFI-4454
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4439) WriteAheadProvenanceRepository fails often to cleanup provenance files filling up disks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189724#comment-16189724
 ] 

ASF GitHub Bot commented on NIFI-4439:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2190

NIFI-4439: When a Provenance Event File is rolled over, we were faili…

…ng to close the resource before attempting to compress it. Fixed that.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4439

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2190


commit cc7960af57fa67dd0141cbdbb9bbb11121893e3a
Author: Mark Payne 
Date:   2017-10-03T13:44:54Z

NIFI-4439: When a Provenance Event File is rolled over, we were failing to 
close the resource before attempting to compress it. Fixed that.




> WriteAheadProvenanceRepository fails often to cleanup provenance files 
> filling up disks
> ---
>
> Key: NIFI-4439
> URL: https://issues.apache.org/jira/browse/NIFI-4439
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Extensions
>Affects Versions: 1.4.0
> Environment: Windows 10
>Reporter: Joseph Witt
>
> Lots of warnings in log file which state
> EventFileCompressor Failed to delete .\provenance_repository\1919191914.prov; 
> this file should be cleaned up manually
> Provenance appears to work correctly otherwise but these files fill up disk 
> space since they are the before compression version and they aren't being 
> cleaned up.  Perhaps we're attempting to delete the files before the 
> input/output streams are closed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi pull request #2190: NIFI-4439: When a Provenance Event File is rolled o...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2190

NIFI-4439: When a Provenance Event File is rolled over, we were faili…

…ng to close the resource before attempting to compress it. Fixed that.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4439

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2190


commit cc7960af57fa67dd0141cbdbb9bbb11121893e3a
Author: Mark Payne 
Date:   2017-10-03T13:44:54Z

NIFI-4439: When a Provenance Event File is rolled over, we were failing to 
close the resource before attempting to compress it. Fixed that.




---

[jira] [Updated] (NIFI-4008) ConsumeKafkaRecord_0_10 assumes there is always one Record in a message

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4008:
-
Status: Patch Available  (was: Reopened)

> ConsumeKafkaRecord_0_10 assumes there is always one Record in a message
> ---
>
> Key: NIFI-4008
> URL: https://issues.apache.org/jira/browse/NIFI-4008
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>
> ConsumeKafkaRecord_0_10 uses ConsumerLease underneath, and it [assumes there 
> is one Record available in a consumed 
> message|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-10-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumerLease.java#L434]
>  retrieved from a Kafka topic.
> But in fact, a message can contain 0 or more records in it. For example, with 
> a record schema shown below:
> {code}
> {
>   "type": "record",
>   "name": "temp",
>   "fields" : [
> {"name": "value", "type": "string"}
>   ]
> }
> {code}
> Multiple records can be sent within a single message, e.g. using JSON:
> {code}
> [{"value": "a"}, {"value": "b"}, {"value": "c"}]
> {code}
> But ConsumeKafkaRecord only outputs the first record:
> {code}
> [{"value": "a"}]
> {code}
> Also, if a message doesn't contain any record in it, the processor fails with 
> NullPointerException.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-1107) Add multipart upload support to PutS3Object

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189711#comment-16189711
 ] 

ASF GitHub Bot commented on NIFI-1107:
--

Github user coveralls commented on the issue:

https://github.com/apache/nifi/pull/132
  

[![Coverage 
Status](https://coveralls.io/builds/13545015/badge)](https://coveralls.io/builds/13545015)

Changes Unknown when pulling **52fd4180c7e929cbbdd3647d1b7aad68174b2f9c on 
jskora:NIFI-1107** into ** on apache:master**.



> Add multipart upload support to PutS3Object
> ---
>
> Key: NIFI-1107
> URL: https://issues.apache.org/jira/browse/NIFI-1107
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Joe Skora
>Assignee: Joe Skora
>  Labels: s3
> Fix For: 0.5.0
>
> Attachments: NIFI-1107.000.patch
>
>
> A new `PutS3ObjectMultipart` processor using the AWS S3 API to upload files 
> larger than those supported by `PutS3Object` which has a [5GB 
> limit|http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html] 
> limit.
> To support S3 compatible endpoints this will also add an `Endpoint Override 
> URL` property to `AbstractAWSProcessor` to set the service 
> [endpoint|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/AmazonWebServiceClient.html#setEndpoint(java.lang.String)]
>  to override the endpoint URL normally selected based on the the Amazon 
> region.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi issue #132: NIFI-1107 - Integrate Multipart uploads into the PutS3Objec...

[coveralls]

Github user coveralls commented on the issue:

https://github.com/apache/nifi/pull/132
  

[![Coverage 
Status](https://coveralls.io/builds/13545015/badge)](https://coveralls.io/builds/13545015)

Changes Unknown when pulling **52fd4180c7e929cbbdd3647d1b7aad68174b2f9c on 
jskora:NIFI-1107** into ** on apache:master**.



---

[jira] [Commented] (NIFI-4412) Release Management for 1.4.0

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189704#comment-16189704
 ] 

ASF subversion and git services commented on NIFI-4412:
---

Commit 1811008 from [~jtstorck] in branch 'site/trunk'
[ https://svn.apache.org/r1811008 ]

NIFI-4412 Added 1.4.0 documentation

> Release Management for 1.4.0
> 
>
> Key: NIFI-4412
> URL: https://issues.apache.org/jira/browse/NIFI-4412
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4412) Release Management for 1.4.0

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189702#comment-16189702
 ] 

ASF subversion and git services commented on NIFI-4412:
---

Commit 1811006 from [~jtstorck] in branch 'site/trunk'
[ https://svn.apache.org/r1811006 ]

NIFI-4412 removing 1.3.0 docs in preparation for adding 1.4.0 docs

> Release Management for 1.4.0
> 
>
> Key: NIFI-4412
> URL: https://issues.apache.org/jira/browse/NIFI-4412
> Project: Apache NiFi
>  Issue Type: Task
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFIREG-9) Define web REST API endpoints for user, group, and policy management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189690#comment-16189690
 ] 

ASF GitHub Bot commented on NIFIREG-9:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi-registry/pull/14
  
Reviewing...


> Define web REST API endpoints for user, group, and policy management
> 
>
> Key: NIFIREG-9
> URL: https://issues.apache.org/jira/browse/NIFIREG-9
> Project: NiFi Registry
>  Issue Type: New Feature
>Reporter: Kevin Doran
>Assignee: Kevin Doran
> Fix For: 0.0.1
>
>
> NIFIREG-8 defined web REST API endpoints for domain objects such as buckets 
> and flows.
> This ticket scopes work to build on that by adding endpoints for additional 
> resources to facilitate user, group, and policy management.
> The interface for user, group, and policy management will be based on the 
> NiFi API so as to minimize the effort for NiFi API clients to be reused for 
> NiFi Registry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] nifi-registry issue #14: NIFIREG-9: Initial Auth Implementation

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi-registry/pull/14
  
Reviewing...


---