[jira] [Created] (NIFI-2445) PublishKafka sends flowfiles to 'success' if not authorized to write to topic

2016-07-31 Thread Mark Payne (JIRA)
Mark Payne created NIFI-2445:


 Summary: PublishKafka sends flowfiles to 'success' if not 
authorized to write to topic
 Key: NIFI-2445
 URL: https://issues.apache.org/jira/browse/NIFI-2445
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Reporter: Mark Payne
Priority: Blocker
 Fix For: 1.0.0, 0.8.0


I have a PublishKafka processor configured to write to a kafka topic that I am 
not authorized to write to (via PLAINTEXT_SASL configuration). When I attempt 
to send a message to Kafka, I get a stack trace telling me that I am not 
authorized but the FlowFile is still routed to 'success'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (NIFI-2208) Support Custom Properties in Expression Language - 1.x baseline

2016-07-31 Thread Joseph Witt (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401310#comment-15401310
 ] 

Joseph Witt edited comment on NIFI-2208 at 8/1/16 1:19 AM:
---

Above is a PR that reflects the following thoughts.

1)
VariableRegistry should not be editable at least in so far as 
extensions/components are concerned so we must ensure the interface doesn't 
expose a way to alter state.  We'll of course allow it to be editable 
eventually but at that point it will be in a different or extended interface 
not exposed to components themselves as we'll have to manage the lifecycle of 
those edits in a manner that doens't mean a component could have dirty reads 
within an execution cycle.  So slimmed down the interface to a single method.  
Also introduced a new small immutable object class called VariableDescriptor 
which is used as the key to interact with elements in the variable registry.  
It allows us to capture more than just the String key name such as whether the 
variable is sensitive and a description that will help us with logging or user 
display cases. Logging will be helpful when people are confused about variable 
precedence issues.
2) 
Concrete implementations of the VariableRegistry should be avoided in the 
nifi-api as much as possible.  The nifi-api is our most public and most rigid 
definition of extension points so the more we put in there the higher the 
friction and the more challenging it is to evolve.  Slimmed down to a single 
static final instance of a system_environment_variable registry in the 
interface of VariableRegistry and then moved the only needed concrete 
implementation at this time to nifi-framework-core.
3)
Changes for VariableRegistry had an impact beyond what was probably intended 
and beyond what the current lifecycle management really supports.  For example 
the custom UI for UpdateAttribute had it because it was evaluating expressions. 
 However, the expression evaluation being done there was purely to validate the 
structure of the expression to help with user experience.  The actual 
expression evaluation is still in the processor where it belongs.  But the 
VariableRegistry was there seemingly due to constructors requiring it.  That 
had a ripple effect into the NiFiWebContext which caused an API change for that 
and caused spring context properties changes, etc.. Also, the Authorizer API 
was updated for variable registry and this should be avoided unless we have a 
clear need there.  However, in general since we don't have a clear strategy for 
the lifecycle managment of the variable registry, in my opinion, we should take 
a less is more approach for now. It is also now in nifi-bootstrap though it is 
using it for general system/env properties so I kept this in place because it 
isn't exposing the user defined aspects of the variable registry we will have.
4) 
Due to updates to constructors of various mock contexts and property value 
classes it appears the ripple effect was wider than really necessary.  Most 
often the variable registry isn't needed to test the use case at hand.  Added 
constructors that made it optional and a sentinel instance of it in the 
interface to make null handling more explicit and safe.  Was able to remove a 
lot of code due to this.
5) 
The nifi-api AttributeExpression javadocs needed to be updated to reflect the 
intent of the API now that we have the variable registry.
6) 
Introduced a ValueLookup class to the expression language module.  This class 
to help wrap the VariableRegistry with precedence based attributes such as 
processor properties and flowfile attributes rather than having the 
VariableRegistry support these directly.  The VariableRegistry is purely the 
user defined properties, system properties, and environment variables so we 
should avoid extending it to include other transient/optional items which have 
different lifecycles.  The ValueLookup class helps encapsulate the difference 
between the VariableRegistry and the local needs and interface of expression 
language.
7)
To support some of the comments about life cycle concerns we had tests which 
altered system properties then grabbed the registry to verify substitution was 
working.  The problem with this is it goes beyond the intent of system and env 
variables being loaded only once at startup.  We won't have a chance to get 
system properties edited at runtime because if we support that then we run the 
risk of variable registry values changing outside our supported lifecycle. So I 
removed these tests.
8)
Not a change now but commentary for future: Changes to the api could include 
adding support in the PropertyValue class or something else which returns the 
referenced VariableDescriptors in a given expression/property.  This will allow 
us to restrict/manage when users can alter variables in the registry based on 
whether com

[jira] [Created] (NIFI-2444) PublishKafka does not generate bulletins when there are problems

2016-07-31 Thread Mark Payne (JIRA)
Mark Payne created NIFI-2444:


 Summary: PublishKafka does not generate bulletins when there are 
problems
 Key: NIFI-2444
 URL: https://issues.apache.org/jira/browse/NIFI-2444
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Reporter: Mark Payne
Priority: Critical
 Fix For: 1.0.0


I noticed extremely poor performance when trying to send data to Kafka via 
PublishKafka. I saw no bulletins so didn't realize there were problems until I 
looked into the logs and saw a lot of stack traces:

{code}
2016-08-01 00:30:03,749 ERROR [Timer-Driven Process Thread-44] 
o.a.n.p.kafka.pubsub.KafkaPublisher Failed while waiting for acks from Kafka
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
access topics: [nifi-provenance]
at 
org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:686)
 ~[kafka-clients-0.9.0.1.jar:na]
at 
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:449) 
~[kafka-clients-0.9.0.1.jar:na]
at 
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:339) 
~[kafka-clients-0.9.0.1.jar:na]
at 
org.apache.nifi.processors.kafka.pubsub.KafkaPublisher.publish(KafkaPublisher.java:115)
 ~[nifi-kafka-pubsub-processors-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.processors.kafka.pubsub.PublishKafka$1.process(PublishKafka.java:248)
 [nifi-kafka-pubsub-processors-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1880)
 [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
 [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.processors.kafka.pubsub.PublishKafka.doRendezvousWithKafka(PublishKafka.java:244)
 [nifi-kafka-pubsub-processors-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.processors.kafka.pubsub.PublishKafka.rendezvousWithKafka(PublishKafka.java:205)
 [nifi-kafka-pubsub-processors-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.processors.kafka.pubsub.AbstractKafkaProcessor.onTrigger(AbstractKafkaProcessor.java:192)
 [nifi-kafka-pubsub-processors-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
 [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
 [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[na:1.8.0_91]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_91]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.kafka.common.errors.TopicAuthorizationException: Not 
authorized to access topics: [nifi-provenance]
{code}

It appears that the processor creates a KafkaPublisher class (and perhaps 
others?) that have their own loggers. These loggers need to be removed and the 
Processor's logger needs to be passed in so that all log messages generate the 
appropriate bulletins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (NIFI-2443) Search - Unable to find component in different Process Group

2016-07-31 Thread Matt Gilman (JIRA)
Matt Gilman created NIFI-2443:
-

 Summary: Search - Unable to find component in different Process 
Group
 Key: NIFI-2443
 URL: https://issues.apache.org/jira/browse/NIFI-2443
 Project: Apache NiFi
  Issue Type: Sub-task
  Components: Core UI
Reporter: Matt Gilman
 Fix For: 1.0.0


If I click a component that is not in the current process group, it changes to 
that component's process group but then tells me that it can't find the 
component.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (NIFI-2420) Add a section for Multi-tenant Authorization to the Admin Guide

2016-07-31 Thread Andrew Lim (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401341#comment-15401341
 ] 

Andrew Lim commented on NIFI-2420:
--

Thanks [~bende] for the feedback.  Will add a section for the "brand new secure 
flow" scenario to the doc.

> Add a section for Multi-tenant Authorization to the Admin Guide
> ---
>
> Key: NIFI-2420
> URL: https://issues.apache.org/jira/browse/NIFI-2420
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
> Fix For: 1.0.0
>
> Attachments: Multi-tenantAuthManage.pdf, Multi-tenantAuthSetup.pdf
>
>
> Multi-tenant Authorization is a new feature in 1.0.0. A section covering 
> setup and management should be included in the documentation.
> Attaching my first attempt at the doc in case anyone has any thoughts.
> I plan to submit a PR soon depending on feedback and after more 
> editing/review on my part.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (NIFI-2237) REST API Documentation Updates

2016-07-31 Thread Joe Skora (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401301#comment-15401301
 ] 

Joe Skora edited comment on NIFI-2237 at 7/31/16 7:32 PM:
--

[~mcgilman], regarding #2 it seems to me that documenting read-only values only 
applies when a DTO is used for a PUT or POST request.  For instance, 
documenting "This property is read-only" on every property in the 
AccessStatusDTO seems redudant since that is only ever used as a response 
object.

Also, required vs. optional and allowable values seem to track from the 
ApiModelProperty annotation on the DTO to the API docs, but the read-only 
status does not.  

1. Is adding "This property is read-only" the key thing here for the read-only 
fields?

2 Should this be updated even if the DTO is only used for read-only response 
objects?

(edited to clarify)


was (Author: jskora):
[~mcgilman], regarding #2 it seems to me that documenting read-only, required 
vs optional, and allowable values only applies when a DTO is used for a PUT or 
POST request.  For instance, documenting "This property is read-only" on every 
property in the AccessStatusDTO seems redudant since that is only ever used as 
a response object.

Does that make sense or should they be documented anywhere the ApiModelProperty 
annotation tags something as read-only?

> REST API Documentation Updates
> --
>
> Key: NIFI-2237
> URL: https://issues.apache.org/jira/browse/NIFI-2237
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Critical
> Fix For: 1.0.0
>
>
> - Ensure REST APi documentation is up to date.
> - Remove mention of deprecated user roles
> - Mark applicable endpoints with Yetus like descriptions according to 
> intent/stability/openness.
> - Ensure all fields are correctly marked as optional or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (NIFI-2366) After creating a template, it errors out when trying to add it back

2016-07-31 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401319#comment-15401319
 ] 

ASF GitHub Bot commented on NIFI-2366:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/718
  
@olegz looks good! Did a full build and created several process groups, 
processors, input & output ports, funnels, remote process groups, and labels. 
Copied & pasted them all over the place, multiple times. Moved in & out of 
process groups. Created templates and instantiated them. all appear to work as 
expected. Thanks for jumping in and knocking this out - i know it gets tricky!


> After creating a template, it errors out when trying to add it back
> ---
>
> Key: NIFI-2366
> URL: https://issues.apache.org/jira/browse/NIFI-2366
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Assignee: Oleg Zhurakousky
>Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: minifi_test.xml
>
>
> NIFI build on Jul.22nd. 
> Template: GenerateFlowFile -> RPG with a input port
> after the template is created, cannot add it back to the same canvas, due to 
> the following error: "The flow snippet contains a Connection that references 
> a component that is not included."
> after further investigation, realized that the component ID of the input port 
> referenced in the template is not accurate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] nifi issue #718: NIFI-2366 - Fixed ID generation semantics in clustered envi...

2016-07-31 Thread markap14
Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/718
  
@olegz looks good! Did a full build and created several process groups, 
processors, input & output ports, funnels, remote process groups, and labels. 
Copied & pasted them all over the place, multiple times. Moved in & out of 
process groups. Created templates and instantiated them. all appear to work as 
expected. Thanks for jumping in and knocking this out - i know it gets tricky!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (NIFI-2366) After creating a template, it errors out when trying to add it back

2016-07-31 Thread ASF subversion and git services (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401314#comment-15401314
 ] 

ASF subversion and git services commented on NIFI-2366:
---

Commit 1bf10944eaee0e408d870c9e17a6c8224b630fad in nifi's branch 
refs/heads/master from [~ozhurakousky]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1bf1094 ]

NIFI-2366 - Fixed ID generation semantics in clustered environment
- added SnippetUtilsTest
- renamed TypeOneUUIDGenerator to ComponentIdGenerator

- changed lsb part of ComponentIdGenerator back to long
- Fixed 'isCopy' condition for clustered environments

This closes #718.


> After creating a template, it errors out when trying to add it back
> ---
>
> Key: NIFI-2366
> URL: https://issues.apache.org/jira/browse/NIFI-2366
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Assignee: Oleg Zhurakousky
>Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: minifi_test.xml
>
>
> NIFI build on Jul.22nd. 
> Template: GenerateFlowFile -> RPG with a input port
> after the template is created, cannot add it back to the same canvas, due to 
> the following error: "The flow snippet contains a Connection that references 
> a component that is not included."
> after further investigation, realized that the component ID of the input port 
> referenced in the template is not accurate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (NIFI-2366) After creating a template, it errors out when trying to add it back

2016-07-31 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401315#comment-15401315
 ] 

ASF GitHub Bot commented on NIFI-2366:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/718


> After creating a template, it errors out when trying to add it back
> ---
>
> Key: NIFI-2366
> URL: https://issues.apache.org/jira/browse/NIFI-2366
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Assignee: Oleg Zhurakousky
>Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: minifi_test.xml
>
>
> NIFI build on Jul.22nd. 
> Template: GenerateFlowFile -> RPG with a input port
> after the template is created, cannot add it back to the same canvas, due to 
> the following error: "The flow snippet contains a Connection that references 
> a component that is not included."
> after further investigation, realized that the component ID of the input port 
> referenced in the template is not accurate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] nifi pull request #718: NIFI-2366 - Fixed ID generation semantics in cluster...

2016-07-31 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/718


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (NIFI-2208) Support Custom Properties in Expression Language - 1.x baseline

2016-07-31 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401309#comment-15401309
 ] 

ASF GitHub Bot commented on NIFI-2208:
--

GitHub user joewitt opened a pull request:

https://github.com/apache/nifi/pull/754

NIFI-2208 refactored as per comments in JIRA



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joewitt/incubator-nifi NIFI-2208-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/754.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #754


commit 1513b8435d862e9db3fb12228be31e9f5920e38e
Author: joewitt 
Date:   2016-07-31T19:14:22Z

NIFI-2208 refactored as per comments in JIRA




> Support Custom Properties in Expression Language - 1.x baseline
> ---
>
> Key: NIFI-2208
> URL: https://issues.apache.org/jira/browse/NIFI-2208
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Yolanda M. Davis
>Priority: Blocker
> Fix For: 1.0.0
>
>
> NIFI-1974 addressed this for the 0.x baseline but the PR does not apply 
> cleanly to the 1.x baseline. Creating a separate JIRA for 1.x so that we can 
> close out NIFI-1974 since 0.7.0 is ready to be released.
> In addition to the merge this should also include a fix to ensure that 
> variable registry is initialized on startup that variables from the registry 
> are applied during EL compilation based on the following order of precedence:
> 1)  Flow File Attribute
> 2)  Processor provided variables
> 3)  User Defined Variables (via custom properties)
> 4)  JVM System Properties
> 5)  OS Environment Variables
> Finally the following processor properties should be enabled to support 
> expression language:
> Put HDFS/Get HDFS/List HDFS
> - Directory property
> ConsumeJMS/PublishJMS
> - Destination Name property
> MS Connection Factory Provider
> -MQ ConnectionFactory Implementation (fqn classname)
> -MQ client library path
> -Broker URI 
> DBCP Connection Pool: 
> -Database Connection URL
> -Database Driver Class Name
> -DB Driver jar url
> -DB username 
> -DB password
> ConvertCSVToAvro
> -add EL support for the following property
> -csv charset
> -and below...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (NIFI-2208) Support Custom Properties in Expression Language - 1.x baseline

2016-07-31 Thread Joseph Witt (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401310#comment-15401310
 ] 

Joseph Witt commented on NIFI-2208:
---

Above is a PR that reflects the following thoughts.

1)
VariableRegistry should not be editable at least in so far as 
extensions/components are concerned so we must ensure the interface doesn't 
expose a way to alter state.  We'll of course allow it to be editable 
eventually but at that point it will be in a different or extended interface 
not exposed to components themselves as we'll have to manage the lifecycle of 
those edits in a manner that doens't mean a component could have dirty reads 
within an execution cycle.  So slimmed down the interface to a single method.  
Also introduced a new small immutable object class called VariableDescriptor 
which is used as the key to interact with elements in the variable registry.  
It allows us to capture more than just the String key name such as whether the 
variable is sensitive and a description that will help us with logging or user 
display cases. Logging will be helpful when people are confused about variable 
precedence issues.
2) 
Concrete implementations of the VariableRegistry should be avoided in the 
nifi-api as much as possible.  The nifi-api is our most public and most rigid 
definition of extension points so the more we put in there the higher the 
friction and the more challenging it is to evolve.  Slimmed down to a single 
static final instance of a system_environment_variable registry in the 
interface of VariableRegistry and then moved the only needed concrete 
implementation at this time to nifi-framework-core.
3)
Changes for VariableRegistry had an impact beyond what was probably intended 
and beyond what the current lifecycle management really supports.  For example 
the custom UI for UpdateAttribute had it because it was evaluating expressions. 
 However, the expression evaluation being done there was purely to validate the 
structure of the expression to help with user experience.  The actual 
expression evaluation is still in the processor where it belongs.  But the 
VariableRegistry was there seemingly due to constructors requiring it.  That 
had a ripple effect into the NiFiWebContext which caused an API change for that 
and caused spring context properties changes, etc.. Also, the Authorizer API 
was updated for variable registry and this should be avoided unless we have a 
clear need there.  However, in general since we don't have a clear strategy for 
the lifecycle managment of the variable registry, in my opinion, we should take 
a less is more approach for now. It is also now in nifi-bootstrap though it is 
using it for general system/env properties so I kept this in place because it 
isn't exposing the user defined aspects of the variable registry we will have.
4) 
Due to updates to constructors of various mock contexts and property value 
classes it appears the ripple effect was wider than really necessary.  Most 
often the variable registry isn't needed to test the use case at hand.  Added 
constructors that made it optional and a sentinel instance of it in the 
interface to make null handling more explicit and safe.  Was able to remove a 
lot of code due to this.
5) 
The nifi-api AttributeExpression javadocs needed to be updated to reflect the 
intent of the API now that we have the variable registry.
6) 
Introduced a ValueLookup class to the expression language module.  This class 
to help wrap the VariableRegistry with precedence based attributes such as 
processor properties and flowfile attributes rather than having the 
VariableRegistry support these directly.  The VariableRegistry is purely the 
user defined properties, system properties, and environment variables so we 
should avoid extending it to include other transient/optional items which have 
different lifecycles.  The ValueLookup class helps encapsulate the difference 
between the VariableRegistry and the local needs and interface of expression 
language.
7)
To support some of the comments about life cycle concerns we had tests which 
altered system properties then grabbed the registry to verify substitution was 
working.  The problem with this is it goes beyond the intent of system and env 
variables being loaded only once at startup.  We won't have a chance to get 
system properties edited at runtime because if we support that then we run the 
risk of variable registry values changing outside our supported lifecycle. So I 
removed these tests.
8)
Not a change now but commentary for future: Changes to the api could include 
adding support in the PropertyValue class or something else which returns the 
referenced VariableDescriptors in a given expression/property.  This will allow 
us to restrict/manage when users can alter variables in the registry based on 
whether components are running that reference them.  Furth

[GitHub] nifi pull request #754: NIFI-2208 refactored as per comments in JIRA

2016-07-31 Thread joewitt
GitHub user joewitt opened a pull request:

https://github.com/apache/nifi/pull/754

NIFI-2208 refactored as per comments in JIRA



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joewitt/incubator-nifi NIFI-2208-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/754.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #754


commit 1513b8435d862e9db3fb12228be31e9f5920e38e
Author: joewitt 
Date:   2016-07-31T19:14:22Z

NIFI-2208 refactored as per comments in JIRA




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (NIFI-2237) REST API Documentation Updates

2016-07-31 Thread Joe Skora (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401301#comment-15401301
 ] 

Joe Skora commented on NIFI-2237:
-

[~mcgilman], regarding #2 it seems to me that documenting read-only, required 
vs optional, and allowable values only applies when a DTO is used for a PUT or 
POST request.  For instance, documenting "This property is read-only" on every 
property in the AccessStatusDTO seems redudant since that is only ever used as 
a response object.

Does that make sense or should they be documented anywhere the ApiModelProperty 
annotation tags something as read-only?

> REST API Documentation Updates
> --
>
> Key: NIFI-2237
> URL: https://issues.apache.org/jira/browse/NIFI-2237
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Critical
> Fix For: 1.0.0
>
>
> - Ensure REST APi documentation is up to date.
> - Remove mention of deprecated user roles
> - Mark applicable endpoints with Yetus like descriptions according to 
> intent/stability/openness.
> - Ensure all fields are correctly marked as optional or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Reopened] (NIFI-2301) Remove policy when component is deleted

2016-07-31 Thread Matt Gilman (JIRA)

 [ 
https://issues.apache.org/jira/browse/NIFI-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman reopened NIFI-2301:
---

Also need to ensure we're removing any applicable data or data-transfer 
resources.

> Remove policy when component is deleted
> ---
>
> Key: NIFI-2301
> URL: https://issues.apache.org/jira/browse/NIFI-2301
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.0.0
>
>
> When a component is removed, we need to also remove any associated policies. 
> Otherwise, the policies will continue to grow unboundedly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (NIFI-1373) Annotate all classes to indicate their API stability and intended audience

2016-07-31 Thread Joseph Witt (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401204#comment-15401204
 ] 

Joseph Witt commented on NIFI-1373:
---

the yetus audience annotations are now in maven central.

http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22audience-annotations%22

can proceed

> Annotate all classes to indicate their API stability and intended audience
> --
>
> Key: NIFI-1373
> URL: https://issues.apache.org/jira/browse/NIFI-1373
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core Framework, Core UI, Extensions
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> Sean Busbey and Tony Kurc have pointed out the need to document the intended 
> audience and stability of the APIs.  This is very necessary as we need to be 
> able to make changes and be fluid where intended and extremely strict to 
> compatibility where intended.  Presently with things being unmarked it can be 
> ambiguous for those not deeply familiar with the api and codebase.
> By leveraging the annotations from Apache Yetus we can resolve this 
> discrepancy across the codebase:  
> http://yetus.apache.org/documentation/0.1.0/audience-annotations-apidocs/
> To Do:
> 1) Go through entire codebase and tag classes appropriate with Yetus 
> annotations.
> 2) Update our versioning guidance to reflect that our major/minor/incremental 
> model only applies to those classes which are tagged as stable/etc..  We 
> should be able to safely maneuver and update internal APIs without any risks 
> to compatibility.  Through discussions things should become more stable and 
> if we find folks wanting to extend/alter things which we didn't have listed 
> as stable then those are great things to start discussion on.  Right now 
> we're too hamstrung by thinking we have to hold onto everything.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] nifi issue #675: LSB Adherence

2016-07-31 Thread PuspenduBanerjee
Github user PuspenduBanerjee commented on the issue:

https://github.com/apache/nifi/pull/675
  
Agreed. That sounds like a plan!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Comment Edited] (NIFI-2237) REST API Documentation Updates

2016-07-31 Thread Matt Gilman (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401167#comment-15401167
 ] 

Matt Gilman edited comment on NIFI-2237 at 7/31/16 2:41 PM:


Some additional background regarding #3...

We use Swagger to build our Rest Api documentation. This is done using Swagger 
annotations on our Rest Endpoints and DTO model. During the build, those 
annotations are used to build a JSON document according to the Swagger 
specification. We then run that JSON document through a Handlebars Template to 
generate our HTML docs. These templates can be found in 
nifi-web-api/src/main/resources/templates.

Let me know about #3. I have no problem tackling it as well. Thanks for jumping 
in and offering to help!


was (Author: mcgilman):
Some additional background regarding #3...

We use Swagger to build our Rest Api documentation. This is done using Swagger 
annotations on our Rest Endpoints and DTO model. During the build, those 
annotations are used to be a JSON document according to the Swagger 
specification. We then run that JSON document through a Handlebars Template to 
generate our HTML docs. These templates can be found in 
nifi-web-api/src/main/resources/templates.

Let me know about #3. I have no problem tackling it as well. Thanks for jumping 
in and offering to help!

> REST API Documentation Updates
> --
>
> Key: NIFI-2237
> URL: https://issues.apache.org/jira/browse/NIFI-2237
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Critical
> Fix For: 1.0.0
>
>
> - Ensure REST APi documentation is up to date.
> - Remove mention of deprecated user roles
> - Mark applicable endpoints with Yetus like descriptions according to 
> intent/stability/openness.
> - Ensure all fields are correctly marked as optional or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (NIFI-2237) REST API Documentation Updates

2016-07-31 Thread Matt Gilman (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401167#comment-15401167
 ] 

Matt Gilman commented on NIFI-2237:
---

Some additional background regarding #3...

We use Swagger to build our Rest Api documentation. This is done using Swagger 
annotations on our Rest Endpoints and DTO model. During the build, those 
annotations are used to be a JSON document according to the Swagger 
specification. We then run that JSON document through a Handlebars Template to 
generate our HTML docs. These templates can be found in 
nifi-web-api/src/main/resources/templates.

Let me know about #3. I have no problem tackling it as well. Thanks for jumping 
in and offering to help!

> REST API Documentation Updates
> --
>
> Key: NIFI-2237
> URL: https://issues.apache.org/jira/browse/NIFI-2237
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Critical
> Fix For: 1.0.0
>
>
> - Ensure REST APi documentation is up to date.
> - Remove mention of deprecated user roles
> - Mark applicable endpoints with Yetus like descriptions according to 
> intent/stability/openness.
> - Ensure all fields are correctly marked as optional or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

2016-07-31 Thread ijokarumawak
Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72900441
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/pom.xml ---
@@ -35,6 +35,10 @@
 nifi-utils
 
 
+org.apache.nifi
+nifi-expression-language
--- End diff --

We may be able to improve the framework to provide onPropertyModified a 
PropertyValue which is already setup. I think valuable registry can be useful 
with env configs like broker address. Raised an issue 
[NIFI-2364](https://issues.apache.org/jira/browse/NIFI-2364).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

2016-07-31 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401051#comment-15401051
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72900441
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/pom.xml ---
@@ -35,6 +35,10 @@
 nifi-utils
 
 
+org.apache.nifi
+nifi-expression-language
--- End diff --

We may be able to improve the framework to provide onPropertyModified a 
PropertyValue which is already setup. I think valuable registry can be useful 
with env configs like broker address. Raised an issue 
[NIFI-2364](https://issues.apache.org/jira/browse/NIFI-2364).


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (NIFI-2442) Support Expression Language evaluation at onPropertyModified

2016-07-31 Thread Koji Kawamura (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401049#comment-15401049
 ] 

Koji Kawamura commented on NIFI-2442:
-

Without this, NIFI-2078 can't fully support EL.

> Support Expression Language evaluation at onPropertyModified
> 
>
> Key: NIFI-2442
> URL: https://issues.apache.org/jira/browse/NIFI-2442
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>
> Some components have to eagerly get property value change, and do some stuff. 
> Sometimes, the target properties support Expression Language (EL).
> However, since current onPropertyModified method only passes String 
> representation of property value, we can't evaluate EL in that method 
> directly.
> It would be useful if we can use subject-less EL functions (i.e without 
> incoming FlowFile) at onPropertyModified, especially with Variable Registry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (NIFI-2442) Support Expression Language evaluation at onPropertyModified

2016-07-31 Thread Koji Kawamura (JIRA)
Koji Kawamura created NIFI-2442:
---

 Summary: Support Expression Language evaluation at 
onPropertyModified
 Key: NIFI-2442
 URL: https://issues.apache.org/jira/browse/NIFI-2442
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Reporter: Koji Kawamura


Some components have to eagerly get property value change, and do some stuff. 
Sometimes, the target properties support Expression Language (EL).

However, since current onPropertyModified method only passes String 
representation of property value, we can't evaluate EL in that method directly.

It would be useful if we can use subject-less EL functions (i.e without 
incoming FlowFile) at onPropertyModified, especially with Variable Registry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (NIFI-2259) HTTP Site-to-Site doesn't handle TRANSACTION_FINISHED _BUT_DESTINATION_FULL as expected

2016-07-31 Thread Koji Kawamura (JIRA)

 [ 
https://issues.apache.org/jira/browse/NIFI-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-2259:

Priority: Critical  (was: Major)

> HTTP Site-to-Site doesn't handle TRANSACTION_FINISHED _BUT_DESTINATION_FULL 
> as expected
> ---
>
> Key: NIFI-2259
> URL: https://issues.apache.org/jira/browse/NIFI-2259
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Critical
> Fix For: 1.0.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> If a remote NiFi instance input-port's destination becomes full, It's 
> expected that remote NiFi instance returns 
> TRANSACTION_FINISHED_BUT_DESTINATION_FULL response.
> However, an IOException occurred due to DESTINATION_FULL handshake error when 
> org.apache.nifi.remote.Transaction.complete() is called.
> The cause of this issue is SiteToSiteResource.commitInputPortTransaction() 
> checks if all of input-port's destination is full. Which shouldn't be 
> executed at complete(). The transaction is already committed on source NiFi.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

2016-07-31 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401033#comment-15401033
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/563
  
@JPercivall I've tested with broker 0.9.0.1 and 0.10.0. Both worked, 
consumed messages with ConsumeKafka and GetKafka, also view/clear state.


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] nifi issue #563: NIFI-2078, 2363, 2364: External state management. CLUSTER s...

2016-07-31 Thread ijokarumawak
Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/563
  
@JPercivall I've tested with broker 0.9.0.1 and 0.10.0. Both worked, 
consumed messages with ConsumeKafka and GetKafka, also view/clear state.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] nifi pull request #563: NIFI-2078, 2363, 2364: External state management. CL...

2016-07-31 Thread ijokarumawak
Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72899732
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/GetKafka.java
 ---
@@ -481,4 +493,59 @@ private void releaseFlowFile(FlowFile flowFile, 
ProcessSession session, Map partitionOffsets = 
KafkaUtils.retrievePartitionOffsets(zookeeperConnectionString, topic, groupId);
+
+return new StandardStateMap(partitionOffsets, 
System.currentTimeMillis());
+}
+
+private boolean isReadyToAccessState() {
+return !StringUtils.isEmpty(zookeeperConnectionString)
+&& !StringUtils.isEmpty(topic)
+&& !StringUtils.isEmpty(groupId);
+}
+
+@Override
+public void clearExternalState() throws IOException {
+if (!isReadyToAccessState()) {
+return;
+}
+// Block onTrigger starts creating new consumer until clear offset 
finishes.
+synchronized (this.consumerStreamsReady) {
+KafkaUtils.clearPartitionOffsets(zookeeperConnectionString, 
topic, groupId);
+}
+}
+
+/**
+ * GetKafka overrides this method in order to capture processor's 
property values required when it retrieves
+ * its state managed externally at Kafka. Since view/clear state 
operation can be executed before onTrigger() is called,
+ * we need to capture these values as it's modified. This method is 
also called when NiFi restarts and loads configs,
+ * so users can access external states right after restart of NiFi.
+ * @param descriptor of the modified property
+ * @param oldValue non-null property value (previous)
+ * @param newValue the new property value or if null indicates the 
property
+ */
+@Override
+public void onPropertyModified(PropertyDescriptor descriptor, String 
oldValue, String newValue) {
+if (ZOOKEEPER_CONNECTION_STRING.equals(descriptor)) {
+zookeeperConnectionString = newValue;
+} else if (TOPIC.equals(descriptor)) {
+topic = newValue;
+} else if (GROUP_ID.equals(descriptor)) {
+groupId = newValue;
--- End diff --

Thanks for pointing this out. Addressed.
Fixed bootstrap server address for ConsumeKafka, too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

2016-07-31 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401018#comment-15401018
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/563#discussion_r72899732
  
--- Diff: 
nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/GetKafka.java
 ---
@@ -481,4 +493,59 @@ private void releaseFlowFile(FlowFile flowFile, 
ProcessSession session, Map partitionOffsets = 
KafkaUtils.retrievePartitionOffsets(zookeeperConnectionString, topic, groupId);
+
+return new StandardStateMap(partitionOffsets, 
System.currentTimeMillis());
+}
+
+private boolean isReadyToAccessState() {
+return !StringUtils.isEmpty(zookeeperConnectionString)
+&& !StringUtils.isEmpty(topic)
+&& !StringUtils.isEmpty(groupId);
+}
+
+@Override
+public void clearExternalState() throws IOException {
+if (!isReadyToAccessState()) {
+return;
+}
+// Block onTrigger starts creating new consumer until clear offset 
finishes.
+synchronized (this.consumerStreamsReady) {
+KafkaUtils.clearPartitionOffsets(zookeeperConnectionString, 
topic, groupId);
+}
+}
+
+/**
+ * GetKafka overrides this method in order to capture processor's 
property values required when it retrieves
+ * its state managed externally at Kafka. Since view/clear state 
operation can be executed before onTrigger() is called,
+ * we need to capture these values as it's modified. This method is 
also called when NiFi restarts and loads configs,
+ * so users can access external states right after restart of NiFi.
+ * @param descriptor of the modified property
+ * @param oldValue non-null property value (previous)
+ * @param newValue the new property value or if null indicates the 
property
+ */
+@Override
+public void onPropertyModified(PropertyDescriptor descriptor, String 
oldValue, String newValue) {
+if (ZOOKEEPER_CONNECTION_STRING.equals(descriptor)) {
+zookeeperConnectionString = newValue;
+} else if (TOPIC.equals(descriptor)) {
+topic = newValue;
+} else if (GROUP_ID.equals(descriptor)) {
+groupId = newValue;
--- End diff --

Thanks for pointing this out. Addressed.
Fixed bootstrap server address for ConsumeKafka, too.


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] nifi issue #563: NIFI-2078, 2363, 2364: External state management. CLUSTER s...

2016-07-31 Thread ijokarumawak
Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/563
  
@JPercivall Rebased, and added default value handling. Confirmed that I can 
see state with Kafka 0.10.0 broker.  It seems that newer consumer library won't 
work with older broker, so you may need to upgrade Kafka broker (server 
process) to 0.10.0 as well. [1]

[1] https://groups.google.com/forum/#!topic/confluent-platform/jdv10HahRd8


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (NIFI-2078) State management for processors whose states are managed externally

2016-07-31 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/NIFI-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401017#comment-15401017
 ] 

ASF GitHub Bot commented on NIFI-2078:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/563
  
@JPercivall Rebased, and added default value handling. Confirmed that I can 
see state with Kafka 0.10.0 broker.  It seems that newer consumer library won't 
work with older broker, so you may need to upgrade Kafka broker (server 
process) to 0.10.0 as well. [1]

[1] https://groups.google.com/forum/#!topic/confluent-platform/jdv10HahRd8


> State management for processors whose states are managed externally
> ---
>
> Key: NIFI-2078
> URL: https://issues.apache.org/jira/browse/NIFI-2078
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>
> Inherently by the nature of a given processor it may involve state managed by 
> itself (using nifi state management), or can be managed by some external 
> service it interacts with (kafka's offset), and theoretically some might have 
> both going on. With the new state management, we're giving users a way to 
> reset state managed by nifi for a given processor. But it doesnt apply to 
> those processors who have external state.
> we should consider offering a way to reset state that allows a processor to 
> call out to whatever external store it impacts



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (NIFI-2441) View state fails with a JS error: "Cannot read property 'canRead' of undefined"

2016-07-31 Thread Koji Kawamura (JIRA)

 [ 
https://issues.apache.org/jira/browse/NIFI-2441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-2441:

Status: Patch Available  (was: Open)

Removed the permission check code from nf-component-state.js.

https://github.com/apache/nifi/pull/563

> View state fails with a JS error: "Cannot read property 'canRead' of 
> undefined"
> ---
>
> Key: NIFI-2441
> URL: https://issues.apache.org/jira/browse/NIFI-2441
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
> Fix For: 1.0.0
>
>
> NIFI-2355 and its PR #715 added permission check to table sort comparator in 
> nf-component-state.js. And it produces a JS error: "Cannot read property 
> 'canRead' of undefined"
> I think it shouldn't be added to state table, since a given user won't have 
> record level permission difference for component state.
> #715 has the same change for changes nf-controller-services.js and other JS 
> files. I assume adding the permission change to nf-component-state.js was a 
> mistake. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)