Re: Custom Controller Service

[Bryan Bende]

In case it helps, Matt B was nice enough to quickly put up a PR for
the API change discussed yesterday:

https://github.com/apache/nifi/pull/2658

Once this is tested and merged it make it would make it easier to
implement Charlie's approach without needing a new interface.


On Thu, Apr 26, 2018 at 12:36 PM, Charlie Meyer
 wrote:
> Hello,
>
> I'm working to see what we can share, but for a start, here is how we
> handle the delegating controller service:
>
> package com.civitaslearning.nifi.service.impl;
>
>
> import org.apache.nifi.components.PropertyDescriptor;
> import org.apache.nifi.controller.AbstractControllerService;
> import org.apache.nifi.dbcp.DBCPService;
>
> import java.util.Collections;
> import java.util.List;
> import java.util.UUID;
>
> public class DelegatingDBCPServiceImpl extends
> AbstractControllerService implements DelegatingDBCPService {
>
>   private static final List properties =
> Collections.emptyList();
>
>   @Override
>   public DBCPService getDBCPService(UUID uuid) {
> DBCPService dbcpService =
> (DBCPService)
> getControllerServiceLookup().getControllerService(uuid.toString());
>
> if (dbcpService == null) {
>   throw new NullPointerException("Couldn't find DBCP service with
> ID " + uuid);
> }
>
> return dbcpService;
>   }
>
>   @Override
>   public List getSupportedPropertyDescriptors() {
> return properties;
>   }
> }
>
>
> From there, we took existing nifi processor code such as ExecuteSQL and
> modified it to require a reference to a delegating service rather than a
> DBCP service and added another property to specify which flow file
> attribute has the id of the dbcp service that should be used, similar to:
>
> final DelegatingDBCPService delegatingDBCPService =
> context.getProperty(DELEGATING_DBCP_SERVICE).asControllerService(DelegatingDBCPService.class);
> final UUID dbcpServiceId =
> UUID.fromString(context.getProperty(DBCP_SERVICE_ATTRIBUTE).evaluateAttributeExpressions(fileToProcess).getValue());
> final DBCPService dbcpService =
> delegatingDBCPService.getDBCPService(dbcpServiceId);
>
> thanks
>
> On Thu, Apr 26, 2018 at 4:07 AM, RP  wrote:
>
>> Hi Charlie,
>>
>> Thanks for the reply. It seems that you already have a solution to my
>> problem. If possible, can you please share you code for the custom
>> controller service and the custom processor?
>>
>>
>>
>> -
>> RP
>> --
>> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>>

Re: Custom Controller Service

[Charlie Meyer]

Hello,

I'm working to see what we can share, but for a start, here is how we
handle the delegating controller service:

package com.civitaslearning.nifi.service.impl;


import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.controller.AbstractControllerService;
import org.apache.nifi.dbcp.DBCPService;

import java.util.Collections;
import java.util.List;
import java.util.UUID;

public class DelegatingDBCPServiceImpl extends
AbstractControllerService implements DelegatingDBCPService {

  private static final List properties =
Collections.emptyList();

  @Override
  public DBCPService getDBCPService(UUID uuid) {
DBCPService dbcpService =
(DBCPService)
getControllerServiceLookup().getControllerService(uuid.toString());

if (dbcpService == null) {
  throw new NullPointerException("Couldn't find DBCP service with
ID " + uuid);
}

return dbcpService;
  }

  @Override
  public List getSupportedPropertyDescriptors() {
return properties;
  }
}


>From there, we took existing nifi processor code such as ExecuteSQL and
modified it to require a reference to a delegating service rather than a
DBCP service and added another property to specify which flow file
attribute has the id of the dbcp service that should be used, similar to:

final DelegatingDBCPService delegatingDBCPService =
context.getProperty(DELEGATING_DBCP_SERVICE).asControllerService(DelegatingDBCPService.class);
final UUID dbcpServiceId =
UUID.fromString(context.getProperty(DBCP_SERVICE_ATTRIBUTE).evaluateAttributeExpressions(fileToProcess).getValue());
final DBCPService dbcpService =
delegatingDBCPService.getDBCPService(dbcpServiceId);

thanks

On Thu, Apr 26, 2018 at 4:07 AM, RP  wrote:

> Hi Charlie,
>
> Thanks for the reply. It seems that you already have a solution to my
> problem. If possible, can you please share you code for the custom
> controller service and the custom processor?
>
>
>
> -
> RP
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: [DISCUSS] Support for accessing sensitive values safely

[Sivaprasanna]

Andy, That's exactly what I had in mind, but without a separate textbox
popping up. What I had originally though, there is a checkbox instead of a
button named "Use Variables". If it is checked, whatever the user types,
AJAX comes in to play and suggests ( auto-completion ) variables that the
user has access to. If the box is not checked, it is going to be a literal
password. But in this case, the normal textbox that pops up when a property
is edited, because it is more of a "textarea" that a textbox and the
suggestion dropdown would feel awkward there. I haven't worked extensively
on UI so not sure about how difficult it is going to be. Definitely UI
folks would pitch in much better ideas.

Thanks,
Sivaprasanna

On Thu, Apr 26, 2018 at 7:25 PM, Andy LoPresto  wrote:

> Yes, I think a dynamically-populated dropdown is unnecessary, and I think
> there might be a way to do this without requiring complex EL parsing.
>
> For sensitive properties, I imagine a text field as we have now, with a
> button to the right that says “Use Variable”. I do not anticipate having to
> support a combination of variable interpolation and literal text (i.e. no
> one’s password should be “${db.password}12345”), so we only have to support
> the use cases of *literal password entry* OR *variable population*.
>
> If the user clicks “Use Variable”, a dialog would appear which has a text
> field and allows the user to type, and uses an AJAX function to call a
> query API endpoint, which allows them to filter the available variables as
> they type. For example, if the user types “d”, the following list might
> appear:
>
> data.endpoint
> db.url
> db.username
> db.password
>
> As the user continues to type “db.p”, only “db.password” is available. The
> user selects this (thus not allowing true freeform input to the final
> value, only a predetermined list of valid values). Obviously this API would
> need to be protected with proper authorization to ensure that only the
> variables available to that user were exposed, but this would be the case
> on validation as well to ensure the user didn’t manually type in a variable
> reference they were not allowed to use.
>
> I’ve included a couple mockups of how I envision this. I’ve implemented
> the same thing in a Rails app a few years ago, so I know it’s possible, and
> with modern libraries, should not be too difficult. This should also
> demonstrate that people like Rob, Matt, and Scott who actually design the
> UI/UX that goes into NiFi are invaluable.
>
> Current sensitive property entry field
> Proposed “Use Variable” button
>
> “Use Variable” dialog with empty text field
>
> User starts typing, triggering AJAX query
>
> User continues typing, narrowing down result set, and selects it
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Apr 26, 2018, at 9:05 AM, Bryan Bende  wrote:
>
> That is a fair point about the list of variables potentially being
> long, we would probably want to get some UI/UX recommendations from
> the folks that have worked in that are the most.
>
> In practice I wonder if it would really be an issue though...
>
> The variables that would be selectable would only be the sensitive
> variables the user has access to, not all variables. I would imagine
> out of hundreds of variables only a handful are sensitive variables.
> It is similar to processor properties in that there are tons of
> properties across all processors, but only a small batch of sensitive
> properties (I just counted 63 references to sensitive(true) in the
> codebase).
>
> The other factor is that it would only be the sensitive variables that
> are in scope to where you are in the data flow, using the same logic
> of how variables are resolved in a hierarchical order. So you would
> only see sensitive variables that are in the current process group or
> a parent group.
>
> As far as knowing if the enter plain password vs. EL, that was the
> reason I suggested selecting a variable from a list. That is the
> mechanism that tells us that a variable was used. The hope was that
> you could enter free form text or select a variable, any time free
> form text was used it would function the way it does today where the
> value is cleared out when saved to registry. If a variable was
> selected then it could keep the variable name.
>
> If we didn't do a list of variables then we would need a checkbox or
> something for the user to indicate whether the sensitive value is a
> literal or expression language.
>
>
> On Thu, Apr 26, 2018 at 5:24 AM, Sivaprasanna 
> wrote:
>
> Initially when I thought of this, I imagined that we can still leverage
> `PropertyDescriptor` and just add new method(s) to handle the sensitive
> variables. Some thought has to be put on this.
>
> *"It would provide a list of variables that are 

Re: Apache Nifi - How to pass maven contrib-check after adding text file to resources

[M Singh]

Thanks Joe - that's what I was looking for.
 

On Wednesday, April 25, 2018, 7:53:19 PM PDT, Joe Witt  
wrote:  
 
 Mans

See here for an example [1]

The Apache RAT Plugin is what actually would detect the files and
check their licenses/etc..

In the provided example we're excluding a couple test files because
they cannot have headers but they are legit. You'd want to do the same
most likely.

Thanks

[1] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/pom.xml

On Wed, Apr 25, 2018 at 10:51 PM, M Singh  wrote:
> Hi:
>
> I am working on a project and would like to add a text resource file but am 
> not sure how to "register" it so that it  passes maven contrib-check ?
>
> Please let me know where I can find documentation on it.
>
> Thanks
>
> Mans
  

Re: Search for Controller Service UUID

[Matt Gilman]

Mark,

The first doesn't sound like it's been implemented yet. The second sounds
like a bug. Feel free to file JIRAs for these issues. Thanks!

Matt

On Wed, Apr 25, 2018 at 9:22 AM, Mark Bean  wrote:

> When I search for a Controller Service by UUID using the search on the
> toolbar, only processors which reference the service are listed, not the
> service itself. Similarly, when selecting the UUID from the Bulletin Board,
> it reports "Error: Unable to find the specified component".
>
> Is this by design?
>
> Thanks,
> Mark
>

Re: Is there a configuration to limit the size of nifi's flowfile repository

[Mark Payne]

Ben,

There are three things that I've seen cause really massive FlowFile 
Repositories:

1) OutOfMemoryError occurs that causes NiFi to stop working properly.
2) The "nifi.flowfile.repository.checkpoint.interval" property is set really 
long (2 mins is the default).
3) By far, the most common, is that the system runs out of available file 
handles. You can check how many
file handles are available by running "ulimit -Hn" and "ulimit -Sn". We 
recommend at least 50,000 be set,
but the default on most linux-based operating systems is much smaller, like 
4,096. The Admin Guide [1]
will guide you through increasing this value, if this is the problem.

Thanks
-Mark

[1] http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html



On Apr 26, 2018, at 5:26 AM, 尹文才 
> wrote:

hi guys, thanks for all your answers, I actually have seen that the
flowfile repo in one of our openstack centos 7 machine grew up to abour 30
GB, which as a result used up all the disk space allocated for the virtual
machine and the flow inside
NIFI couldn't proceed and many errors started to appear such as fail to
checkpoint, etc.We used NIFI now as a ETL tool to extract some data from
sql server for data analysis.
I actually have no idea why the flowfile repo would grow up like this, in
my idea it is only used to place all flowfile attributes. It would be great
if there're some options to limit the flowfile repo size.

Thanks.
Regard,
Ben

2018-04-26 2:08 GMT+08:00 Brandon DeVries >:

All,

This is something I think we shouldn't dismiss so easily.  While the
FlowFile repo is lighter than the content repo, allowing it to grow too
large can cause major problems.

Specifically, an "overgrown" FlowFile repo may prevent a NiFi instance from
coming back up after a restart due to the way in which records are held in
memory.  If there is more memory available to give to the JVM, this can
sometimes be worked around... but if there isn't you may just be out of
luck.  For that matter, allowing the FlowFile repo to grow so large that it
consumes all the heap isn't going to be good for system health in general
(OOM is probably never where you want to be...).

To Pierre's point "you don't want to limit that repository in size since it
would prevent the workflows to create new flow files"... that's exactly why
I would want to limit the size of the repo.  You do then get into questions
of how exactly to do this.  For example, you may not want to simply block
all transactions that create a FlowFile, because it may remove even more
(e.g. MergeContent).  Additionally, you have to be concerned about
deadlocks (e.g. a "Wait" that hangs forever because its "Notify" is being
starved).  Or, perhaps that's all you can do... freeze everything at some
threshold prior to actual damage being done, and alert operators that
manual intervention is necessary (e.g. bring up the graph with
autoResume=false, and bleed off data in a controlled fashion).

In summary, I believe this is a problem.  Even if it doesn't come up often,
when it does it is significant.  While the solution likely isn't simple,
it's worth putting some thought towards.

Brandon

On Wed, Apr 25, 2018 at 9:43 AM Sivaprasanna 
>
wrote:

No, he actually had mentioned “like content repository”. The answer is,
there aren’t any properties that support this, AFAIK. Pierre’s response
pretty much sums up why there aren’t any properties.

Thanks,
Sivaprasanna

On Wed, 25 Apr 2018 at 7:10 PM, Mike Thomsen 
>
wrote:

I have a feeling that what Ben meant was how to limit the content
repository size.

On Wed, Apr 25, 2018 at 8:26 AM Pierre Villard <
pierre.villard...@gmail.com>
wrote:

Hi Ben,

Since the flow file repository contains the information of the flow
files
currently being processed by NiFi, you don't want to limit that
repository
in size since it would prevent the workflows to create new flow
files.

Besides this repository is very lightweight, I'm not sure it'd need
to
be
limited in size.
Do you have a specific use case in mind?

Pierre


2018-04-25 9:15 GMT+02:00 尹文才 >:

Hi guys, I checked NIFI's system administrator guide trying to
find a
configuration item so that the size of the flowfile repository
could
be
limited similar to the other repositories(e.g. content repository),
but I
didn't find such configuration items, is there currently any
configuration
to limit the flowfile repository's size? thanks.

Regards,
Ben

Re: [DISCUSS] Support for accessing sensitive values safely

[Bryan Bende]

That is a fair point about the list of variables potentially being
long, we would probably want to get some UI/UX recommendations from
the folks that have worked in that are the most.

In practice I wonder if it would really be an issue though...

The variables that would be selectable would only be the sensitive
variables the user has access to, not all variables. I would imagine
out of hundreds of variables only a handful are sensitive variables.
It is similar to processor properties in that there are tons of
properties across all processors, but only a small batch of sensitive
properties (I just counted 63 references to sensitive(true) in the
codebase).

The other factor is that it would only be the sensitive variables that
are in scope to where you are in the data flow, using the same logic
of how variables are resolved in a hierarchical order. So you would
only see sensitive variables that are in the current process group or
a parent group.

As far as knowing if the enter plain password vs. EL, that was the
reason I suggested selecting a variable from a list. That is the
mechanism that tells us that a variable was used. The hope was that
you could enter free form text or select a variable, any time free
form text was used it would function the way it does today where the
value is cleared out when saved to registry. If a variable was
selected then it could keep the variable name.

If we didn't do a list of variables then we would need a checkbox or
something for the user to indicate whether the sensitive value is a
literal or expression language.


On Thu, Apr 26, 2018 at 5:24 AM, Sivaprasanna  wrote:
> Initially when I thought of this, I imagined that we can still leverage
> `PropertyDescriptor` and just add new method(s) to handle the sensitive
> variables. Some thought has to be put on this.
>
> *"It would provide a list of variables that are readable to the current
> user and one can be selected"*
> I think this might lead to bad UX. When the number of variables grow,
> rendering the variables in a drop-down list may not be that good, IMHO.
> Maybe we can still stick with textbox and do the following:
>
>- Once a user enters sensitive variable expression language, the
>framework does a background check if the user is authorized to use this
>variable. If the user doesn't have access, render the same and make the
>component invalid there by preventing it to be enabled/started
>
> Only catch is how are we specifying what the user entered is plain password
> and not EL. Either we can go with the assumption that all sensitive values
> have to be taken from Variable Registry and only the EL for those variables
> will have to be provided as the sensitive property value but this would
> restrict the developers' freedom to choose what approach they want. Or we
> can tweak the UI and present the option better, like have a checkbox which
> when checked, will consider that as an EL and evaluate from the variable
> registry and when not checked, assume that the entered password is plain
> password and no evaluation needs to happen.
>
> -
>
> Sivaprasanna
>
> On Thu, Apr 26, 2018 at 12:19 AM, Bryan Bende  wrote:
>
>> The policy model would need more thought, but the point would be that
>> a user can select variable references they have been given permission
>> to.
>>
>> In order to configure the processor that is referencing the variable,
>> they already need write permissions to that processor, or some parent
>> in the hierarchy if no specific policy exists.
>>
>>
>>
>> On Wed, Apr 25, 2018 at 2:42 PM, Otto Fowler 
>> wrote:
>> >
>> > "It would provide a list of variables that are readable to the current
>> user
>> > and one can be selected, just like allowable values or controller
>> services.”
>> >
>> > A person may have rights to configure nifi without knowing the “value” of
>> > the secure db password ( for example ), but that doesn’t mean they
>> > don’t have there rights to reference it.
>> >
>> >
>> >
>> > On April 25, 2018 at 14:15:16, Bryan Bende (bbe...@gmail.com) wrote:
>> >
>> > There is definitely room for improvement here.
>> >
>> > Keep in mind that often the sensitive information is specific to a
>> > given environment. For example you build a flow in dev with your
>> > db.password. You don't actually want your dev db password to be
>> > propagated to the next environment, but you do want to be able to set
>> > a variable placeholder like ${db.password} and leave that placeholder
>> > so you can just set that variable in the next environment. So to me
>> > the goal here is how to handle secure variables.
>> >
>> > Andy highlighted many of the issues, my proposal would be the
>> following...
>> >
>> > First, we can introduce a concept of a sensitive variable. This would
>> > be something in the UI where a user can indicate a variable is
>> > sensitive, maybe a checkbox, and then the framework can store these
>> 

Re: Custom Controller Service

[RP]

Hi Charlie,

Thanks for the reply. It seems that you already have a solution to my
problem. If possible, can you please share you code for the custom
controller service and the custom processor?



-
RP
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Is there a configuration to limit the size of nifi's flowfile repository

[Pierre Villard]

Hi,

Brandon - agree this shouldn't be dismissed, just saying that's, as you
said, a tough one.
Ben - out of curiosity, how many flow files can you have in the system at
one point? Do you have flow files with a lot of attributes? large
attributes (with a lot of content)?

Pierre

2018-04-26 11:26 GMT+02:00 尹文才 :

> hi guys, thanks for all your answers, I actually have seen that the
> flowfile repo in one of our openstack centos 7 machine grew up to abour 30
> GB, which as a result used up all the disk space allocated for the virtual
> machine and the flow inside
> NIFI couldn't proceed and many errors started to appear such as fail to
> checkpoint, etc.We used NIFI now as a ETL tool to extract some data from
> sql server for data analysis.
> I actually have no idea why the flowfile repo would grow up like this, in
> my idea it is only used to place all flowfile attributes. It would be great
> if there're some options to limit the flowfile repo size.
>
> Thanks.
> Regard,
> Ben
>
> 2018-04-26 2:08 GMT+08:00 Brandon DeVries :
>
> > All,
> >
> > This is something I think we shouldn't dismiss so easily.  While the
> > FlowFile repo is lighter than the content repo, allowing it to grow too
> > large can cause major problems.
> >
> > Specifically, an "overgrown" FlowFile repo may prevent a NiFi instance
> from
> > coming back up after a restart due to the way in which records are held
> in
> > memory.  If there is more memory available to give to the JVM, this can
> > sometimes be worked around... but if there isn't you may just be out of
> > luck.  For that matter, allowing the FlowFile repo to grow so large that
> it
> > consumes all the heap isn't going to be good for system health in general
> > (OOM is probably never where you want to be...).
> >
> > To Pierre's point "you don't want to limit that repository in size since
> it
> > would prevent the workflows to create new flow files"... that's exactly
> why
> > I would want to limit the size of the repo.  You do then get into
> questions
> > of how exactly to do this.  For example, you may not want to simply block
> > all transactions that create a FlowFile, because it may remove even more
> > (e.g. MergeContent).  Additionally, you have to be concerned about
> > deadlocks (e.g. a "Wait" that hangs forever because its "Notify" is being
> > starved).  Or, perhaps that's all you can do... freeze everything at some
> > threshold prior to actual damage being done, and alert operators that
> > manual intervention is necessary (e.g. bring up the graph with
> > autoResume=false, and bleed off data in a controlled fashion).
> >
> > In summary, I believe this is a problem.  Even if it doesn't come up
> often,
> > when it does it is significant.  While the solution likely isn't simple,
> > it's worth putting some thought towards.
> >
> > Brandon
> >
> > On Wed, Apr 25, 2018 at 9:43 AM Sivaprasanna 
> > wrote:
> >
> > > No, he actually had mentioned “like content repository”. The answer is,
> > > there aren’t any properties that support this, AFAIK. Pierre’s response
> > > pretty much sums up why there aren’t any properties.
> > >
> > > Thanks,
> > > Sivaprasanna
> > >
> > > On Wed, 25 Apr 2018 at 7:10 PM, Mike Thomsen 
> > > wrote:
> > >
> > > > I have a feeling that what Ben meant was how to limit the content
> > > > repository size.
> > > >
> > > > On Wed, Apr 25, 2018 at 8:26 AM Pierre Villard <
> > > > pierre.villard...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Ben,
> > > > >
> > > > > Since the flow file repository contains the information of the flow
> > > files
> > > > > currently being processed by NiFi, you don't want to limit that
> > > > repository
> > > > > in size since it would prevent the workflows to create new flow
> > files.
> > > > >
> > > > > Besides this repository is very lightweight, I'm not sure it'd need
> > to
> > > be
> > > > > limited in size.
> > > > > Do you have a specific use case in mind?
> > > > >
> > > > > Pierre
> > > > >
> > > > >
> > > > > 2018-04-25 9:15 GMT+02:00 尹文才 :
> > > > >
> > > > > > Hi guys, I checked NIFI's system administrator guide trying to
> > find a
> > > > > > configuration item so that the size of the flowfile repository
> > could
> > > be
> > > > > > limited similar to the other repositories(e.g. content
> repository),
> > > > but I
> > > > > > didn't find such configuration items, is there currently any
> > > > > configuration
> > > > > > to limit the flowfile repository's size? thanks.
> > > > > >
> > > > > > Regards,
> > > > > > Ben
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Is there a configuration to limit the size of nifi's flowfile repository

[尹文才]

hi guys, thanks for all your answers, I actually have seen that the
flowfile repo in one of our openstack centos 7 machine grew up to abour 30
GB, which as a result used up all the disk space allocated for the virtual
machine and the flow inside
NIFI couldn't proceed and many errors started to appear such as fail to
checkpoint, etc.We used NIFI now as a ETL tool to extract some data from
sql server for data analysis.
I actually have no idea why the flowfile repo would grow up like this, in
my idea it is only used to place all flowfile attributes. It would be great
if there're some options to limit the flowfile repo size.

Thanks.
Regard,
Ben

2018-04-26 2:08 GMT+08:00 Brandon DeVries :

> All,
>
> This is something I think we shouldn't dismiss so easily.  While the
> FlowFile repo is lighter than the content repo, allowing it to grow too
> large can cause major problems.
>
> Specifically, an "overgrown" FlowFile repo may prevent a NiFi instance from
> coming back up after a restart due to the way in which records are held in
> memory.  If there is more memory available to give to the JVM, this can
> sometimes be worked around... but if there isn't you may just be out of
> luck.  For that matter, allowing the FlowFile repo to grow so large that it
> consumes all the heap isn't going to be good for system health in general
> (OOM is probably never where you want to be...).
>
> To Pierre's point "you don't want to limit that repository in size since it
> would prevent the workflows to create new flow files"... that's exactly why
> I would want to limit the size of the repo.  You do then get into questions
> of how exactly to do this.  For example, you may not want to simply block
> all transactions that create a FlowFile, because it may remove even more
> (e.g. MergeContent).  Additionally, you have to be concerned about
> deadlocks (e.g. a "Wait" that hangs forever because its "Notify" is being
> starved).  Or, perhaps that's all you can do... freeze everything at some
> threshold prior to actual damage being done, and alert operators that
> manual intervention is necessary (e.g. bring up the graph with
> autoResume=false, and bleed off data in a controlled fashion).
>
> In summary, I believe this is a problem.  Even if it doesn't come up often,
> when it does it is significant.  While the solution likely isn't simple,
> it's worth putting some thought towards.
>
> Brandon
>
> On Wed, Apr 25, 2018 at 9:43 AM Sivaprasanna 
> wrote:
>
> > No, he actually had mentioned “like content repository”. The answer is,
> > there aren’t any properties that support this, AFAIK. Pierre’s response
> > pretty much sums up why there aren’t any properties.
> >
> > Thanks,
> > Sivaprasanna
> >
> > On Wed, 25 Apr 2018 at 7:10 PM, Mike Thomsen 
> > wrote:
> >
> > > I have a feeling that what Ben meant was how to limit the content
> > > repository size.
> > >
> > > On Wed, Apr 25, 2018 at 8:26 AM Pierre Villard <
> > > pierre.villard...@gmail.com>
> > > wrote:
> > >
> > > > Hi Ben,
> > > >
> > > > Since the flow file repository contains the information of the flow
> > files
> > > > currently being processed by NiFi, you don't want to limit that
> > > repository
> > > > in size since it would prevent the workflows to create new flow
> files.
> > > >
> > > > Besides this repository is very lightweight, I'm not sure it'd need
> to
> > be
> > > > limited in size.
> > > > Do you have a specific use case in mind?
> > > >
> > > > Pierre
> > > >
> > > >
> > > > 2018-04-25 9:15 GMT+02:00 尹文才 :
> > > >
> > > > > Hi guys, I checked NIFI's system administrator guide trying to
> find a
> > > > > configuration item so that the size of the flowfile repository
> could
> > be
> > > > > limited similar to the other repositories(e.g. content repository),
> > > but I
> > > > > didn't find such configuration items, is there currently any
> > > > configuration
> > > > > to limit the flowfile repository's size? thanks.
> > > > >
> > > > > Regards,
> > > > > Ben
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Support for accessing sensitive values safely

[Sivaprasanna]

Initially when I thought of this, I imagined that we can still leverage
`PropertyDescriptor` and just add new method(s) to handle the sensitive
variables. Some thought has to be put on this.

*"It would provide a list of variables that are readable to the current
user and one can be selected"*
I think this might lead to bad UX. When the number of variables grow,
rendering the variables in a drop-down list may not be that good, IMHO.
Maybe we can still stick with textbox and do the following:

   - Once a user enters sensitive variable expression language, the
   framework does a background check if the user is authorized to use this
   variable. If the user doesn't have access, render the same and make the
   component invalid there by preventing it to be enabled/started

Only catch is how are we specifying what the user entered is plain password
and not EL. Either we can go with the assumption that all sensitive values
have to be taken from Variable Registry and only the EL for those variables
will have to be provided as the sensitive property value but this would
restrict the developers' freedom to choose what approach they want. Or we
can tweak the UI and present the option better, like have a checkbox which
when checked, will consider that as an EL and evaluate from the variable
registry and when not checked, assume that the entered password is plain
password and no evaluation needs to happen.

-

Sivaprasanna

On Thu, Apr 26, 2018 at 12:19 AM, Bryan Bende  wrote:

> The policy model would need more thought, but the point would be that
> a user can select variable references they have been given permission
> to.
>
> In order to configure the processor that is referencing the variable,
> they already need write permissions to that processor, or some parent
> in the hierarchy if no specific policy exists.
>
>
>
> On Wed, Apr 25, 2018 at 2:42 PM, Otto Fowler 
> wrote:
> >
> > "It would provide a list of variables that are readable to the current
> user
> > and one can be selected, just like allowable values or controller
> services.”
> >
> > A person may have rights to configure nifi without knowing the “value” of
> > the secure db password ( for example ), but that doesn’t mean they
> > don’t have there rights to reference it.
> >
> >
> >
> > On April 25, 2018 at 14:15:16, Bryan Bende (bbe...@gmail.com) wrote:
> >
> > There is definitely room for improvement here.
> >
> > Keep in mind that often the sensitive information is specific to a
> > given environment. For example you build a flow in dev with your
> > db.password. You don't actually want your dev db password to be
> > propagated to the next environment, but you do want to be able to set
> > a variable placeholder like ${db.password} and leave that placeholder
> > so you can just set that variable in the next environment. So to me
> > the goal here is how to handle secure variables.
> >
> > Andy highlighted many of the issues, my proposal would be the
> following...
> >
> > First, we can introduce a concept of a sensitive variable. This would
> > be something in the UI where a user can indicate a variable is
> > sensitive, maybe a checkbox, and then the framework can store these
> > values encrypted (currently all variable values are stored in plain
> > text because they aren't meant to be sensitive).
> >
> > Second, we can introduce policies on sensitive variables so that we
> > can restrict who can read them elsewhere, just like policies on
> > controller services that determine which controller services show up
> > in the drop down of a processor.
> >
> > Third, we introduce a new kind of PropertyDescriptor that allows
> > selecting a variable from the variable registry rather than free-form
> > expression language. It would provide a list of variables that are
> > readable to the current user and one can be selected, just like
> > allowable values or controller services. Ideally we can have a way to
> > still allow free form values for people who don't want to use
> > variables.
> >
> > Fourth, anytime variables are evaluated from expression language we
> > would prevent evaluating any of these new sensitive variables since we
> > have no way of knowing if a user should have access to it from
> > free-form EL, so they can only be used from the special
> > PropertyDescriptors above.
> >
> > If we put all this in place then when we save flows to the registry,
> > we can leave the variable place-holders in the sensitive properties,
> > and then when you import to the next environment you only need to edit
> > the variables section and not go through individual processors setting
> > sensitive properties.
> >
> > On Wed, Apr 25, 2018 at 1:06 PM, Andy LoPresto 
> wrote:
> >> Hi Sivaprasanna,
> >>
> >> This was a topic that was briefly considered earlier in the lifecycle of
> >> the
> >> project, but was sidelined due to other developments. With the NiFi
> >> Registry
> >> project,