Re: Start Contributing

[James Wing]

Grant,

Welcome to NiFi, and thanks for being willing to contribute!

Wes is right, you can go ahead and submit a pull request (or patch) with
your proposed changes.  You don't need to wait for the JIRA to be
reviewed.

Thanks,

James

On Wed, Jul 12, 2017 at 8:26 PM, Wes Lawrence  wrote:

> Hey Grant.
>
> I'm also new to this, but I recommend writing the solution and providing a
> GitHub PR and/or patch.
>
> I've only fixed some minor bugs I've found, and only today added a very
> minor feature, but I've found that approaching with a solution is the best
> approach.
>
> After all, if you were in charge of something, what do you prefer; I person
> approaching you and saying "This needs fixing", or "This needs fixing, and
> he's a potential fix".
>
> Worst case, if the fix isn't what the commiters/PMC has is mind, providing
> some solution gets the ball rolling on the problem.
>
> --Wes
>
> On Wed, Jul 12, 2017 at 11:10 PM, Grant Langlois <
> grant.r.langl...@gmail.com
> > wrote:
>
> > Hello all,
> >
> > First and foremost, my apologies if this message breaks protocol, this is
> > my first time hitting a dev mailing list.
> >
> > I'm interested in contributing to this project and have a specific
> feature
> > in mind to tackle first. I read through the developer's guide which
> > recommended first submitting a Jira (which I did here
> > .)
> >
> > My question is what happens next? Should I wait for someone to comment on
> > it as to whether or not it's a feasible addition? I'm more than willing
> to
> > write the code, assuming it's an acceptable addition.
> >
> > Thanks for your patience in helping out a newbie.
> >
> > Grant
> >
>

Re: Start Contributing

[Matt Burgess]

Wes,

Very well said :)

Grant,

Welcome!  Thank you for your interest in NiFi, and especially for your
interest in contributing :)  To go along with Wes's last point about
getting the ball rolling; even if someone wasn't a developer or wasn't
sure how to implement something, they can still approach with a
use-case-level solution; a good idea is a good idea even without the
code to implement it :)

For your Jira (thank you for submitting!), IMO that looks legit and
helpful, and a code contribution would be most welcome.

Thanks,
Matt



On Wed, Jul 12, 2017 at 11:26 PM, Wes Lawrence  wrote:
> Hey Grant.
>
> I'm also new to this, but I recommend writing the solution and providing a
> GitHub PR and/or patch.
>
> I've only fixed some minor bugs I've found, and only today added a very
> minor feature, but I've found that approaching with a solution is the best
> approach.
>
> After all, if you were in charge of something, what do you prefer; I person
> approaching you and saying "This needs fixing", or "This needs fixing, and
> he's a potential fix".
>
> Worst case, if the fix isn't what the commiters/PMC has is mind, providing
> some solution gets the ball rolling on the problem.
>
> --Wes
>
> On Wed, Jul 12, 2017 at 11:10 PM, Grant Langlois > wrote:
>
>> Hello all,
>>
>> First and foremost, my apologies if this message breaks protocol, this is
>> my first time hitting a dev mailing list.
>>
>> I'm interested in contributing to this project and have a specific feature
>> in mind to tackle first. I read through the developer's guide which
>> recommended first submitting a Jira (which I did here
>> .)
>>
>> My question is what happens next? Should I wait for someone to comment on
>> it as to whether or not it's a feasible addition? I'm more than willing to
>> write the code, assuming it's an acceptable addition.
>>
>> Thanks for your patience in helping out a newbie.
>>
>> Grant
>>

Re: Start Contributing

[Wes Lawrence]

Hey Grant.

I'm also new to this, but I recommend writing the solution and providing a
GitHub PR and/or patch.

I've only fixed some minor bugs I've found, and only today added a very
minor feature, but I've found that approaching with a solution is the best
approach.

After all, if you were in charge of something, what do you prefer; I person
approaching you and saying "This needs fixing", or "This needs fixing, and
he's a potential fix".

Worst case, if the fix isn't what the commiters/PMC has is mind, providing
some solution gets the ball rolling on the problem.

--Wes

On Wed, Jul 12, 2017 at 11:10 PM, Grant Langlois  wrote:

> Hello all,
>
> First and foremost, my apologies if this message breaks protocol, this is
> my first time hitting a dev mailing list.
>
> I'm interested in contributing to this project and have a specific feature
> in mind to tackle first. I read through the developer's guide which
> recommended first submitting a Jira (which I did here
> .)
>
> My question is what happens next? Should I wait for someone to comment on
> it as to whether or not it's a feasible addition? I'm more than willing to
> write the code, assuming it's an acceptable addition.
>
> Thanks for your patience in helping out a newbie.
>
> Grant
>

Re: NiFi-4181 + patch

[Wes Lawrence]

Ah, an unintended feature! That is useful, and I'll have to keep it in mind.

Glad you're liking it, let me know if anything needs to be fixed up.

--Wes

On Wed, Jul 12, 2017 at 9:39 PM, Matt Burgess  wrote:

> Wes,
>
> I'm reviewing this now, looking good! A cool feature of it, when used
> with "Treat First Line As Header" set to true, is that you can rename
> the columns/fields, so even if you have a header line you don't have
> to use it :)
>
> Regards,
> Matt
>
>
> On Wed, Jul 12, 2017 at 5:37 PM, Wes Lawrence 
> wrote:
> > Hey,
> >
> > I've been enjoying the new *Record processes, but I found
> > CSVReading/Writing a bit much, with having to describe an Avro schema.
> >
> > While we do use Avro, it's a bit much to define a whole schema for the
> > cases where we have 10-15+ columns of CSV data that we are typically
> trying
> > to reduce down to a smaller, say 5 column Avro payload. Just being able
> to
> > list the columns (when they aren't the first line) is helpful, I believe.
> >
> > So, I added a patch for CSVReading/Writing to work given a simple column
> > list, if that helps anyone out!
> >
> > --Wes
>

Re: NiFi-4181 + patch

[Matt Burgess]

Wes,

I'm reviewing this now, looking good! A cool feature of it, when used
with "Treat First Line As Header" set to true, is that you can rename
the columns/fields, so even if you have a header line you don't have
to use it :)

Regards,
Matt


On Wed, Jul 12, 2017 at 5:37 PM, Wes Lawrence  wrote:
> Hey,
>
> I've been enjoying the new *Record processes, but I found
> CSVReading/Writing a bit much, with having to describe an Avro schema.
>
> While we do use Avro, it's a bit much to define a whole schema for the
> cases where we have 10-15+ columns of CSV data that we are typically trying
> to reduce down to a smaller, say 5 column Avro payload. Just being able to
> list the columns (when they aren't the first line) is helpful, I believe.
>
> So, I added a patch for CSVReading/Writing to work given a simple column
> list, if that helps anyone out!
>
> --Wes

NiFi-4181 + patch

[Wes Lawrence]

Hey,

I've been enjoying the new *Record processes, but I found
CSVReading/Writing a bit much, with having to describe an Avro schema.

While we do use Avro, it's a bit much to define a whole schema for the
cases where we have 10-15+ columns of CSV data that we are typically trying
to reduce down to a smaller, say 5 column Avro payload. Just being able to
list the columns (when they aren't the first line) is helpful, I believe.

So, I added a patch for CSVReading/Writing to work given a simple column
list, if that helps anyone out!

--Wes

Re: Question with ExtractText Processor

[Mark Payne]

Atish,

I think there may be a limit on the number of extracted columns, but if you 
exceeded that limit,
then the Processor would be invalid. If you are trying to use a regex that has 
34 .* segments, then
the performance is likely to be awful. Any time you have a .* in a regex it's 
quite expensive. Doing that
34 times can be incredibly expensive.

Is it possible for you to upgrade to a newer version of NiFi? With the newest 
version (1.3) there was the
introduction of a handful of Record-oriented Processors. These should make flow 
design dramatically
easier and should result in far, far better performance.

So instead of using a SplitContext -> ExtractText with regexes -> ReplaceText 
you could just simply use
ConvertRecord (with a CSV Reader and a JSON Writer), and it will keep all of 
the records within a single FlowFile.
No need to fuss with regular expressions or replacing text.

Thanks
-Mark


> On Jul 12, 2017, at 1:24 PM, Atish Ray  wrote:
> 
> Thanks!!! Regex is working for me with smaller number of column. Another
> problem I am facing with ExtractText processor. My pipe delimited file 
> having 34 fields. I need to convert all 34 fields and convert them into
> json. My file size is around 30MB. So I am converting from CSV to JSON using
> "SplitContent">ExtractText>ReplaceText. Queue is stuck before ExtractText.
> Do we have any limitation on number of extracted column? 
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Question-with-ExtractText-Processor-tp16405p16412.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Question with ExtractText Processor

[Lee Laim]

Atish,

ExtractText has a setting called Max Buffer Size that defaults to 1 MB, but
I don't think this is causing your queue to build up before Extract Text.
While I don't know the exact details of your flow, I suggest you try Split
Text instead of Split Content, as this is a more commonly used pattern.

An example template is available on the NiFi dev page:
https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
CsvToJSON.xml

This
flow shows how to convert a CSV entry to a JSON document using ExtractText
and ReplaceText.


If you upgrade to 1.3, you can use the NiFi schema registry to convert
formats.
Here is a great write up:
http://bryanbende.com/development/2017/06/20/apache-nifi-records-and-schema-registries


Thanks,
Lee



On Wed, Jul 12, 2017 at 11:24 AM, Atish Ray  wrote:

> Thanks!!! Regex is working for me with smaller number of column. Another
> problem I am facing with ExtractText processor. My pipe delimited file
> having 34 fields. I need to convert all 34 fields and convert them into
> json. My file size is around 30MB. So I am converting from CSV to JSON
> using
> "SplitContent">ExtractText>ReplaceText. Queue is stuck before ExtractText.
> Do we have any limitation on number of extracted column?
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/Question-with-ExtractText-
> Processor-tp16405p16412.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Question with ExtractText Processor

[Atish Ray]

Thanks!!! Regex is working for me with smaller number of column. Another
problem I am facing with ExtractText processor. My pipe delimited file 
having 34 fields. I need to convert all 34 fields and convert them into
json. My file size is around 30MB. So I am converting from CSV to JSON using
"SplitContent">ExtractText>ReplaceText. Queue is stuck before ExtractText.
Do we have any limitation on number of extracted column? 




--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Question-with-ExtractText-Processor-tp16405p16412.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: APACHE NIFI error message details are not coming

[Pierre Villard]

Hi Vijaya,

I believe the endpoints you're trying are not correct. I'd suggest you to
base your requests on the API documentation:
https://nifi.apache.org/docs/nifi-docs/rest-api/

Thanks!
Pierre

2017-07-12 19:19 GMT+02:00 Vijaya Maddela :

> Hi Team
>
> On NIFI 1.3 version we are not able to access
>
>
>
> http://localhost:8080/nifi-api/controller/status
>
>
>
>
>
> http://localhost:8080/nifi-api/controller/process-groups/
>
>
>
>
>
> http://localhost:8080/nifi-api/controller/process-groups/
> 2cf65be8-015d-1000-b5a5-8be5924a5e4e/status?recursive=true
>
>
>
>
>
> Do we need to configure any parameter? Is it bug in NIFI 1.3 version?
>
>
>
>
>
>
>
>
>
> Thank You
>
> Much Respect
>
> Vijaya Kumar Reddy.Maddela
>
> vijay.re...@eaiesb.com
>
> 91-40-40119993|9704179993 <(970)%20417-9993>
>
> USA:978-306-3751 <(978)%20306-3751>
>
> [image: O_GoldPartner]
>
>
>

Adding properties

[plj]

Howdy,

  I've been away from NiFi for awhile and have forgotten how to do some
things.  I'm using the GetMongo processor (1.2) and I want to use the 'Mongo
Collection Name' property later .  How do I do that?
  I've tried adding a property to the GetMongo processor but it always
complaints that it isn't legal.  I added the property myproperty and set the
value to collectionname.  I get the error:

"'myproperty' validated against 'collectionname' is invalid because
'myproperty' is not a supported property"

Eventually I would like to save the results from the GetMongo to a file.  I
would like the filename to be the collectionname_date_time.  How should I be
doing this.

Thank for your help



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Adding-properties-tp16413.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

APACHE NIFI error message details are not coming

[Vijaya Maddela]

Hi Team

On NIFI 1.3 version we are not able to access 

 

http://localhost:8080/nifi-api/controller/status

 

 

http://localhost:8080/nifi-api/controller/process-groups/

 

 

http://localhost:8080/nifi-api/controller/process-groups/2cf65be8-015d-1000-
b5a5-8be5924a5e4e/status?recursive=true

 

 

Do we need to configure any parameter? Is it bug in NIFI 1.3 version?

 

 

 

 

Thank You

Much Respect

Vijaya Kumar Reddy.Maddela

  vijay.re...@eaiesb.com

91-40-40119993|9704179993

USA:978-306-3751



 

MINIFI-350 minifi-cpp end-to-end integration testing framework

[Andy Christianson]

Hi All,

I am looking at MINIFI-350 and would like to implement some end-to-end 
integration tests for minifi cpp. Essentially, the tests would:


  1.  Stand up a new minifi cpp docker container
  2.  Send test data to HTTP input ports on the container
  3.  Run data through a minifi flow
  4.  Receive output data to a HTTP endpoint
  5.  Verify output data according to some constraints (headers present, hash 
of the content, etc.)

Most of this work, such as setting up a docker container and sending data to 
it, can naturally be done with shell commands. As such, I’ve taken a look at 
the bats [1] testing framework, which seems simple enough and is very 
expressive.

Any thoughts or suggestions on test frameworks to use are appreciated.

[1]: https://github.com/sstephenson/bats

Re: Question with ExtractText Processor

[Mark Payne]

Atish,

I believe you will need to escape the pipe with a backslash:

(.+)\|(.+)\|(.+)\|(.+)\|(.+)

Thanks
-Mark


> On Jul 12, 2017, at 10:13 AM, Atish Ray  wrote:
> 
> Hi,
> I am working with Extract text processor. I have a file which is having Pipe
> Delimited text. I have Spited the file based on every line and now I want to
> extract text from each splited line. I have Difined attribute in ExtractText
> processor as "csv" and given the regex as   (.+)|(.+)|(.+)|(.+)|(.+). 
> 
> Then My expectation is, I will be getting attribute value as csv.1="first
> element of line" ,csv.2="second element of line" ,csv.3="third element of
> line" and csv.4="fourth element of line".
> 
> But it is not working for me. But if I make the file as "," delimited and
> use regex as (.+),(.+),(.+),(.+),(.+), then I am getting the correct
> attribute value.
> 
> Can you please suggest what would be the regex, if I use Pipe delimited
> file?
> 
> 
> Regards
> Atish 
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Question-with-ExtractText-Processor-tp16405.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Adding custom logging Appender to NiFi

[Toivo Adams]

Hi,

I have custom logging appenders (extends ch.qos.logback.core.AppenderBase)
I put custom appenders jar file to /lib directory. 
I added appenders to /conf/ logback.xml and this worked fine using NiFi
1.1.1

Few weeks ago I started to upgrade to NiFi 1.3.0 and along the process
appenders are not found anymore, during NiFi startup I see
java.lang.ClassNotFoundException

What am I missing?
Has classloading changed in 1.3 (or 1.2)?

Thanks
Toivo




--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Adding-custom-logging-Appender-to-NiFi-tp16408.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Question with ExtractText Processor

[Atish Ray]

Hi,
I am working with Extract text processor. I have a file which is having Pipe
Delimited text. I have Spited the file based on every line and now I want to
extract text from each splited line. I have Difined attribute in ExtractText
processor as "csv" and given the regex as   (.+)|(.+)|(.+)|(.+)|(.+). 

Then My expectation is, I will be getting attribute value as csv.1="first
element of line" ,csv.2="second element of line" ,csv.3="third element of
line" and csv.4="fourth element of line".

But it is not working for me. But if I make the file as "," delimited and
use regex as (.+),(.+),(.+),(.+),(.+), then I am getting the correct
attribute value.

Can you please suggest what would be the regex, if I use Pipe delimited
file?


Regards
Atish 



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Question-with-ExtractText-Processor-tp16405.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Question with ExtractText Processor

[Atish Ray]

I am using nifi 1.1



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Question-with-ExtractText-Processor-tp16405p16406.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

NIFI FTP/SFTP modifications

[Vijaya Maddela]

 

 

Hi All,

We are looking to have dynamic behavior for FTP/SFTP processors.

1.  GETFTP processor should be used only as starting point in a flow
(Not supported to use it in middle of the flow)
2.  FetchFTP: This supports using in middle of the flow, but doesn't
support to pass Dynamic properties from previous processor or step

 

When we look into code we identified reasons for not supporting

 

In

GetFTP/GetSFTP:

@InputRequirement(Requirement.INPUT_FORBIDDEN) instead of
@InputRequirement(Requirement.INPUT_REQUIRED)

 

FetchFTP/FetchSFTO:

Doesn't contain @DynamicProperties() in the code for supporting Dynamic
properties

 

 

Can you please some help someone how can we build the code and deploy by our
self. Appreciate a link or tutorial for building code after modification

 

 

Thank You

Much Respect

Vijaya Kumar Reddy.Maddela

  vijay.re...@eaiesb.com

91-40-40119993|9704179993

USA:978-306-3751



 

Trigger timer-driven processor to run from outside NIFI

[尹文才]

Hi guys, is it possible for a Java application outside the NIFI environment
to trigger a timer-driven processor to do its work(I mean its ontrigger
method will be called) when the processor is not yet due to be triggered?
The reason why I'm asking about this is because I have a Java applicatiion
with UI outside NIFI and there're some configuration data that could be
updated into a database, and my processor in NIFI need to get the updated
configuration
data from that database as soon as possible, but my processor is configured
to be timer driven of 5 mins. I hope the processor could be triggered to
run after the configuration is updated by the Java application when it's
not yet reached
the time for it to be triggered. Thanks.

Regards,
Ben