[GitHub] nifi pull request: NIFI-1118 Update SplitText Processor - add supp...

[jskora]

Github user jskora commented on the pull request:

https://github.com/apache/nifi/pull/135#issuecomment-168806914
  
I just pushed changes that address these issues, including tests for these 
and a couple more edge conditions.  It has been squashed (thanks @trkurc) into 
a single commit for readability.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Questions on Apache NiFi

[Aldrin Piri]

Manoj,

1.  In terms of functionality to support this, there are mechanisms that
provide the capability of accomplishing this type of functionality in
components via ControllerServices like the
[DistributedMapCacheServer][1]/Client or
[DistributedSetCacheServer][2]/Client.

There are processors that make use of these controller services in manner
analogous to your desired feature such as [DetectDuplicate][3] or
[ListHDFS][4].

There is work under way for the next release (0.5.0) that brings more
framework functionality to accomplish such tasks and its work is outlined
in our [State Management Feature Proposal][5].

2. ExecuteSQL does not currently perform PreparedStatements for retrieval,
but PutSQL is using them.   Is your desired usage tending toward the
ExecuteSQL processor being a source processor where it is continuously
running the same statement using an updated timestamp value as mentioned in
your first point?

  [1]:
http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.distributed.cache.server.map.DistributedMapCacheServer/index.html
  [2]:
http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.distributed.cache.server.DistributedSetCacheServer/index.html
  [3]:
http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.DetectDuplicate/index.html
  [4]:
http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.ListHDFS/index.html
  [5]: https://cwiki.apache.org/confluence/display/NIFI/State+Management

On Mon, Jan 4, 2016 at 6:49 AM, Kalikodan-Veetil, Manoj K <
manoj.kum...@verizon.com> wrote:

> Hello,
>
> Looking for some guidance.
>
>
> 1.   Is there a way today in Nifi, where I can save some persistent
> data and retrieve later. Use-case I am looking at is to query a table for
> latest updates. I wanted to save the last datetime till which data was
> pulled & use that as start-datetime for next pull.
>
> 2.   Today doe NiFi provide any way to do prepare statements in SQL
> query, to make repetitive queries more efficient?
> Thanks
> Manoj
>

Questions on Apache NiFi

[Kalikodan-Veetil, Manoj K]

Hello,

Looking for some guidance.


1.   Is there a way today in Nifi, where I can save some persistent data 
and retrieve later. Use-case I am looking at is to query a table for latest 
updates. I wanted to save the last datetime till which data was pulled & use 
that as start-datetime for next pull.

2.   Today doe NiFi provide any way to do prepare statements in SQL query, 
to make repetitive queries more efficient?
Thanks
Manoj

Groovy unit tests

[Andy LoPresto]

I am considering writing unit tests in for new development/regression testing 
in Groovy. There are numerous advantages to this [1][2] (such as map coercion, 
relaxed permissions on dependency injection, etc.). Mocking large and complex 
objects, such as NiFiProperties, when only one feature is under test is 
especially easy. I plan to write “Java-style” unit tests, but this would also 
make TDD/BDD frameworks like Spock or Cucumber much easier to use.

I figured before doing this I would poll the community and see if anyone 
strongly objects? In previous situations, I have created a custom Maven profile 
which only runs when triggered (by an environment variable, current username, 
etc.) to avoid polluting the environment of anyone who doesn’t want the Groovy 
test dependencies installed.

Does anyone have thoughts on this?


[1] http://www.ibm.com/developerworks/java/library/j-pg11094/index.html 

[2] 
https://keyholesoftware.com/2015/04/13/short-on-time-switch-to-groovy-for-unit-testing/
 



Andy LoPresto
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Groovy unit tests

[Matt Burgess]

+1, Groovy is a great way to rapidly generate tests, remove boilerplate, and 
enable powerful test frameworks like Spock.

Sent from my iPhone

> On Jan 4, 2016, at 7:29 PM, Andy LoPresto  wrote:
> 
> I am considering writing unit tests in for new development/regression testing 
> in Groovy. There are numerous advantages to this [1][2] (such as map 
> coercion, relaxed permissions on dependency injection, etc.). Mocking large 
> and complex objects, such as NiFiProperties, when only one feature is under 
> test is especially easy. I plan to write “Java-style” unit tests, but this 
> would also make TDD/BDD frameworks like Spock or Cucumber much easier to use. 
> 
> I figured before doing this I would poll the community and see if anyone 
> strongly objects? In previous situations, I have created a custom Maven 
> profile which only runs when triggered (by an environment variable, current 
> username, etc.) to avoid polluting the environment of anyone who doesn’t want 
> the Groovy test dependencies installed. 
> 
> Does anyone have thoughts on this?
> 
> 
> [1] http://www.ibm.com/developerworks/java/library/j-pg11094/index.html
> [2] 
> https://keyholesoftware.com/2015/04/13/short-on-time-switch-to-groovy-for-unit-testing/
> 
> 
> Andy LoPresto
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 

Re: Groovy unit tests

[Joe Witt]

Andy,

I am supportive of this because we clearly have at least a couple
folks in the community who know and are interested in Groovy.  I just
want to make sure we aren't increasing the burden on people to setup
their environments.  As long as this is seamless from a Maven user
perspective (and be mindful of minimum maven versions) then I'm good
with it.  I am slightly concerned about maintainability of these unit
tests relative to Java knowledge we appear to have in the community.

With the above in mind i am a +1.

Thanks
Joe

On Mon, Jan 4, 2016 at 6:42 PM, Matt Burgess  wrote:
> +1, Groovy is a great way to rapidly generate tests, remove boilerplate, and 
> enable powerful test frameworks like Spock.
>
> Sent from my iPhone
>
>> On Jan 4, 2016, at 7:29 PM, Andy LoPresto  wrote:
>>
>> I am considering writing unit tests in for new development/regression 
>> testing in Groovy. There are numerous advantages to this [1][2] (such as map 
>> coercion, relaxed permissions on dependency injection, etc.). Mocking large 
>> and complex objects, such as NiFiProperties, when only one feature is under 
>> test is especially easy. I plan to write “Java-style” unit tests, but this 
>> would also make TDD/BDD frameworks like Spock or Cucumber much easier to use.
>>
>> I figured before doing this I would poll the community and see if anyone 
>> strongly objects? In previous situations, I have created a custom Maven 
>> profile which only runs when triggered (by an environment variable, current 
>> username, etc.) to avoid polluting the environment of anyone who doesn’t 
>> want the Groovy test dependencies installed.
>>
>> Does anyone have thoughts on this?
>>
>>
>> [1] http://www.ibm.com/developerworks/java/library/j-pg11094/index.html
>> [2] 
>> https://keyholesoftware.com/2015/04/13/short-on-time-switch-to-groovy-for-unit-testing/
>>
>>
>> Andy LoPresto
>> alopresto.apa...@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>

Re: Build failure?

[Joe Gresock]

Upgrading to Maven 3.3.9 resolved this failure.  Probably still something
we'll want to either resolve for lower versions of maven (3.2.x) or list a
higher minimum Maven version in the maven-enforcer-plugin.

On Sun, Jan 3, 2016 at 9:55 PM, Joe Witt  wrote:

> And there have been quite a few commits since then so I'm not quite
> sure why you're seeing problems building.  However, in looking through
> the bundle/poms there is room to make it more consistent so will tweak
> and submit a patch for review.
>
> On Sun, Jan 3, 2016 at 9:47 PM, Joe Witt  wrote:
> > Joe
> >
> > I've been building on master throughout the day today and haven't
> > noticed any issues.
> >
> > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > 2015-11-10T11:41:47-05:00)
> > Maven home: /Users/jwitt/Applications/apache-maven-3.3.9
> > Java version: 1.8.0_66, vendor: Oracle Corporation
> > Java home:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "mac os x", version: "10.10.5", arch: "x86_64", family: "mac"
> >
> > Thanks
> > Joe
> >
> > On Sun, Jan 3, 2016 at 9:44 PM, Joe Gresock  wrote:
> >> I just updated master (since a few days ago) and now get this error --
> >> anyone else getting it?  It looks to be related to c45060f, since I
> checked
> >> out the version immediately before it and that one works.
> >>
> >> [INFO] Scanning for projects...
> >> [ERROR] Internal error:
> >> org.apache.maven.artifact.InvalidArtifactRTException: For artifact
> >> {org.apache.nifi:nifi-riemann-processors:null:jar}: The version cannot
> be
> >> empty. -> [Help 1]
> >> org.apache.maven.InternalErrorException: Internal error:
> >> org.apache.maven.artifact.InvalidArtifactRTException: For artifact
> >> {org.apache.nifi:nifi-riemann-processors:null:jar}: The version cannot
> be
> >> empty.
> >> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:166)
> >> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:582)
> >> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
> >> at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
> >>
> >> This is my environment:
> >>
> >> $ mvn -version
> >> Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4;
> >> 2014-08-11T16:58:10-04:00)
> >> Maven home: c:\Program Files\apache-maven-3.2.3
> >> Java version: 1.7.0_71, vendor: Oracle Corporation
> >> Java home: c:\Program Files\Java\jdk1.7.0_71\jre
> >> Default locale: en_US, platform encoding: Cp1252
> >> OS name: "windows 8.1", version: "6.3", arch: "amd64", family: "windows"
> >>
> >> --
> >> I know what it is to be in need, and I know what it is to have plenty.
> I
> >> have learned the secret of being content in any and every situation,
> >> whether well fed or hungry, whether living in plenty or in want.  I can
> do
> >> all this through him who gives me strength.*-Philippians 4:12-13*
>



-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*