[GitHub] nifi pull request: NIFI-1180: Modify PutS3Object to enable encrypt...
Github user alopresto commented on the pull request: https://github.com/apache/nifi/pull/246#issuecomment-206705777 @adamonduty Will do in the morning. Don't want to miss merging into both branches now that `1.0` and `0.x` have split. (Provided docs on that on [the wiki](https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-Stepstomerge/closepullrequestswithtwomainbranches). ) Those commits will also be **GPG-signed** now that [GitHub supports it](https://help.github.com/articles/signing-commits-using-gpg/). Thanks for reminding me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1578: Create PutSlack processor
Github user adamonduty commented on the pull request: https://github.com/apache/nifi/pull/256#issuecomment-206686328 @joewitt I'm only now just realizing that my reply to your comment was in the wrong place. Here's my reply which I originally made on the JIRA: == My use case around PutSlack at this point is pretty simple. Basically the same use cases around PutEmail, but sending to a Slack channel instead of an email distribution list. This allows teams to keep an eye on their dataflows in a natural place, especially since many other Slack integrations exist (Jenkins, github, etc). To configure PutSlack, a Slack user (or maybe admin) has to add the Incoming Webhooks integration to the Slack team, then add a webhook to a specific channel. Once the user has the URL associated with this webhook, they drop the URL into PutSlack along with a webhook text template. Does that help? The use cases would probably also increase with a GetSlack processor, which when paired with PutSlack could allow one to manually trigger flows in NiFi via a chat bot. == Also, just to be clear, the Travis build above failed on JDK8 due to an out of memory exception unrelated to PutSlack. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1180: Modify PutS3Object to enable encrypt...
Github user adamonduty commented on the pull request: https://github.com/apache/nifi/pull/246#issuecomment-206685104 @alopresto What do you think about merging this one with your new committing superpowers? :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1630 Making TestPutUDP select an open port...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/332 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [DRAFT][REPORT] Apache NiFi - April 2016
+1, impressed with community pace, productivity, engagement, and enthusiasm, the report/numbers definitely reflect this. > On Apr 6, 2016, at 11:20 PM, Aldrin Piriwrote: > > Joe, looks great overall. Minor point is that the vote for 0.6.0 wrapped > up Saturday, 26 March. > > Awesome growth for the community. Certainly felt like that was the case, > nice to see actual numbers. > >> On Wed, Apr 6, 2016 at 10:55 PM, Joe Witt wrote: >> >> NiFi Community, >> >> Please review the included April 2016 report for the Apache board. >> If anything seems inaccurate or missing please advise. >> >> Thanks >> Joe >> >> -- >> >> Report from the Apache NiFi committee [Joe Witt] >> >> ## Description: >> - Apache NiFi is an easy to use, powerful, and reliable system to process >> and distribute data. >> >> ## Issues: >> - There are no issues requiring board attention at this time. >> >> ## Activity: >> - Released NiFi 0.5 with considerable new core features and extensions. >> - Released NiFi 0.6 adding features to set stage for 1.0 release. >> - Substantial progress made toward the 1.0 release line which is now >> being built on master. >> - Meetups featuring NiFi talks have occurred in the reporting period in >> Santa Clara, Chicago, WashingtonDC, New York City, Baltimore, >> Toronto, Dublin, Denmark, Montreal >> - Established child project known as 'MiNiFi' to provide an >> alternative command and control and implementation model tailored >> toward the first mile challenges of data acquisition. JIRA and Git >> infrastructure setup and contributions underway. >> >> ## Health report: >> - Activity on mailing lists, JIRA, Git continues trending well. >> - During the last review cycle several new committers have joined. >> - Pipeline for PMC progression is very strong. >> >> ## PMC changes: >> >> - Currently 13 PMC members. >> - No new PMC members in the last three months. >> - Last PMC addition was Sean Busbey on Wed Nov 25 2015 >> >> ## Committer base changes: >> >> - Currently 21 committers. >> - New commmitters: >>- Andy LoPresto was added as a committer on Sat Apr 02 2016 >>- Oleg Zhurakousky was added as a committer on Wed Mar 16 2016 >>- Matt Burgess was added as a committer on Fri Mar 04 2016 >>- Joe Skora was added as a committer on Mon Feb 15 2016 >> >> ## Releases: >> >> - nifi-0.6.0 was released on Fri Mar 25 2016 >> - nifi-0.5.1 was released on Thu Feb 25 2016 >> - nifi-0.5.0 was released on Mon Feb 15 2016 >> >> ## Mailing list activity: >> >> - us...@nifi.apache.org: >>- 262 subscribers (up 56 in the last 3 months): >>- 832 emails sent to list (709 in previous quarter) >> >> - dev@nifi.apache.org: >>- 231 subscribers (up 23 in the last 3 months): >>- 2610 emails sent to list (1389 in previous quarter) >> >> ## JIRA activity: >> >> - 388 JIRA tickets created in the last 3 months >> - 237 JIRA tickets closed/resolved in the last 3 months >>
Re: [DRAFT][REPORT] Apache NiFi - April 2016
Joe, looks great overall. Minor point is that the vote for 0.6.0 wrapped up Saturday, 26 March. Awesome growth for the community. Certainly felt like that was the case, nice to see actual numbers. On Wed, Apr 6, 2016 at 10:55 PM, Joe Wittwrote: > NiFi Community, > > Please review the included April 2016 report for the Apache board. > If anything seems inaccurate or missing please advise. > > Thanks > Joe > > -- > > Report from the Apache NiFi committee [Joe Witt] > > ## Description: > - Apache NiFi is an easy to use, powerful, and reliable system to process > and distribute data. > > ## Issues: > - There are no issues requiring board attention at this time. > > ## Activity: > - Released NiFi 0.5 with considerable new core features and extensions. > - Released NiFi 0.6 adding features to set stage for 1.0 release. > - Substantial progress made toward the 1.0 release line which is now > being built on master. > - Meetups featuring NiFi talks have occurred in the reporting period in > Santa Clara, Chicago, WashingtonDC, New York City, Baltimore, > Toronto, Dublin, Denmark, Montreal > - Established child project known as 'MiNiFi' to provide an > alternative command and control and implementation model tailored > toward the first mile challenges of data acquisition. JIRA and Git > infrastructure setup and contributions underway. > > ## Health report: > - Activity on mailing lists, JIRA, Git continues trending well. > - During the last review cycle several new committers have joined. > - Pipeline for PMC progression is very strong. > > ## PMC changes: > > - Currently 13 PMC members. > - No new PMC members in the last three months. > - Last PMC addition was Sean Busbey on Wed Nov 25 2015 > > ## Committer base changes: > > - Currently 21 committers. > - New commmitters: > - Andy LoPresto was added as a committer on Sat Apr 02 2016 > - Oleg Zhurakousky was added as a committer on Wed Mar 16 2016 > - Matt Burgess was added as a committer on Fri Mar 04 2016 > - Joe Skora was added as a committer on Mon Feb 15 2016 > > ## Releases: > > - nifi-0.6.0 was released on Fri Mar 25 2016 > - nifi-0.5.1 was released on Thu Feb 25 2016 > - nifi-0.5.0 was released on Mon Feb 15 2016 > > ## Mailing list activity: > > - us...@nifi.apache.org: > - 262 subscribers (up 56 in the last 3 months): > - 832 emails sent to list (709 in previous quarter) > > - dev@nifi.apache.org: > - 231 subscribers (up 23 in the last 3 months): > - 2610 emails sent to list (1389 in previous quarter) > > ## JIRA activity: > > - 388 JIRA tickets created in the last 3 months > - 237 JIRA tickets closed/resolved in the last 3 months >
Re: [VOTE] Release Apache NiFi 0.6.1 (RC1)
+1, binding Build and tests worked on OS X and Windows 7. Branch looked good on our Travis build and ran some flow templates from reviews from some of the associated issues with anticipated results. Verified hashes and signatures. On Wed, Apr 6, 2016 at 5:33 PM, Joe Wittwrote: > Hello Apache NiFi Community, > > I am pleased to be calling this vote for the source release of Apache > NiFi 0.6.1. > > The source zip, including signatures, digests, etc. can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-0.6.1/ > > The Git tag is nifi-0.6.1-RC1 > The Git commit hash is d51b24e18356b76ce649c8285af0806acd9071d0 > * > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=d51b24e18356b76ce649c8285af0806acd9071d0 > * > https://github.com/apache/nifi/commit/d51b24e18356b76ce649c8285af0806acd9071d0 > > Checksums of nifi-0.6.1-source-release.zip: > MD5: b82a7fe60e03b679d7c06f57319fa396 > SHA1: 32c45d51d1e1858eaba1df889711c69c17b44781 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 11 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12335496 > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. Then > please vote: > > [ ] +1 Release this package as nifi-0.6.1 > [ ] +0 no opinion > [ ] -1 Do not release this package because... > > Thanks! >
[DRAFT][REPORT] Apache NiFi - April 2016
NiFi Community, Please review the included April 2016 report for the Apache board. If anything seems inaccurate or missing please advise. Thanks Joe -- Report from the Apache NiFi committee [Joe Witt] ## Description: - Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. ## Issues: - There are no issues requiring board attention at this time. ## Activity: - Released NiFi 0.5 with considerable new core features and extensions. - Released NiFi 0.6 adding features to set stage for 1.0 release. - Substantial progress made toward the 1.0 release line which is now being built on master. - Meetups featuring NiFi talks have occurred in the reporting period in Santa Clara, Chicago, WashingtonDC, New York City, Baltimore, Toronto, Dublin, Denmark, Montreal - Established child project known as 'MiNiFi' to provide an alternative command and control and implementation model tailored toward the first mile challenges of data acquisition. JIRA and Git infrastructure setup and contributions underway. ## Health report: - Activity on mailing lists, JIRA, Git continues trending well. - During the last review cycle several new committers have joined. - Pipeline for PMC progression is very strong. ## PMC changes: - Currently 13 PMC members. - No new PMC members in the last three months. - Last PMC addition was Sean Busbey on Wed Nov 25 2015 ## Committer base changes: - Currently 21 committers. - New commmitters: - Andy LoPresto was added as a committer on Sat Apr 02 2016 - Oleg Zhurakousky was added as a committer on Wed Mar 16 2016 - Matt Burgess was added as a committer on Fri Mar 04 2016 - Joe Skora was added as a committer on Mon Feb 15 2016 ## Releases: - nifi-0.6.0 was released on Fri Mar 25 2016 - nifi-0.5.1 was released on Thu Feb 25 2016 - nifi-0.5.0 was released on Mon Feb 15 2016 ## Mailing list activity: - us...@nifi.apache.org: - 262 subscribers (up 56 in the last 3 months): - 832 emails sent to list (709 in previous quarter) - dev@nifi.apache.org: - 231 subscribers (up 23 in the last 3 months): - 2610 emails sent to list (1389 in previous quarter) ## JIRA activity: - 388 JIRA tickets created in the last 3 months - 237 JIRA tickets closed/resolved in the last 3 months
[GitHub] nifi pull request: NIFI-1630 Making TestPutUDP select an open port...
GitHub user bbende opened a pull request: https://github.com/apache/nifi/pull/332 NIFI-1630 Making TestPutUDP select an open port for testing to avoid conflicts You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbende/nifi NIFI-1630 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/332.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #332 commit 13523bd08a07534d130987cbe5cd612a55be7c9a Author: Bryan BendeDate: 2016-04-06T19:09:39Z NIFI-1630 Making TestPutUDP select an open port for testing to avoid conflicts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: Adding requirejs, angularjs and its dependencie...
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/331#issuecomment-206589245 The author email appears tied to a local system name. Probably not what you wanted. There doesn't appear to be updates to the LICENSE/NOTICE of NiFi which will be needed given this is a source dependency. It appears to include a lot of cruft like README files and gitignore files and so on which seem avoidable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Apache NiFi 0.6.1 RC1 Release Helper Guide
Hello Apache NiFi community, Please find the associated guidance to help those interested in validating/verifying the release so they can vote. # Download latest KEYS file: https://dist.apache.org/repos/dist/dev/nifi/KEYS # Import keys file: gpg --import KEYS # [optional] Clear out local maven artifact repository # Pull down nifi-0.6.1 source release artifacts for review: wget https://dist.apache.org/repos/dist/dev/nifi/nifi-0.6.1/nifi-0.6.1-source-release.zip wget https://dist.apache.org/repos/dist/dev/nifi/nifi-0.6.1/nifi-0.6.1-source-release.zip.asc wget https://dist.apache.org/repos/dist/dev/nifi/nifi-0.6.1/nifi-0.6.1-source-release.zip.md5 wget https://dist.apache.org/repos/dist/dev/nifi/nifi-0.6.1/nifi-0.6.1-source-release.zip.sha1 # Verify the signature gpg --verify nifi-0.6.1-source-release.zip.asc # Verify the hashes (md5, sha1) match the source and what was provided in the vote email thread md5sum nifi-0.6.1-source-release.zip sha1sum nifi-0.6.1-source-release.zip # Unzip nifi-0.6.1-source-release.zip # Verify the build works including release audit tool (RAT) checks cd nifi-0.6.1 mvn clean install -Pcontrib-check # Verify the contents contain a good README, NOTICE, and LICENSE. # Verify the git commit ID is correct # Verify the RC was branched off the correct git commit ID # Look at the resulting convenience binary as found in nifi-assembly/target # Make sure the README, NOTICE, and LICENSE are present and correct # Run the resulting convenience binary and make sure it works as expected # Send a response to the vote thread indicating a +1, 0, -1 based on your findings. Thank you for your time and effort to validate the release!
[GitHub] nifi pull request: NIFI-1583 Added ConvertJSONtoCSV processor
Github user joewitt commented on the pull request: https://github.com/apache/nifi/pull/254#issuecomment-206581942 I am not sure we can use org.json:json dependency. Its license as found in the readme of https://github.com/stleary/JSON-java/blob/master/README says "The license includes this restriction: "The software shall be used for good, not evil." If your conscience cannot live with that, then choose a different package."" I'm not sure this is acceptable in that it doesn't claim to be public domain or be part of any approved ASF friendly license. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[VOTE] Release Apache NiFi 0.6.1 (RC1)
Hello Apache NiFi Community, I am pleased to be calling this vote for the source release of Apache NiFi 0.6.1. The source zip, including signatures, digests, etc. can be found at: https://dist.apache.org/repos/dist/dev/nifi/nifi-0.6.1/ The Git tag is nifi-0.6.1-RC1 The Git commit hash is d51b24e18356b76ce649c8285af0806acd9071d0 * https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=d51b24e18356b76ce649c8285af0806acd9071d0 * https://github.com/apache/nifi/commit/d51b24e18356b76ce649c8285af0806acd9071d0 Checksums of nifi-0.6.1-source-release.zip: MD5: b82a7fe60e03b679d7c06f57319fa396 SHA1: 32c45d51d1e1858eaba1df889711c69c17b44781 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/joewitt.asc KEYS file available here: https://dist.apache.org/repos/dist/release/nifi/KEYS 11 issues were closed/resolved for this release: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12335496 Release note highlights can be found here: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [ ] +1 Release this package as nifi-0.6.1 [ ] +0 no opinion [ ] -1 Do not release this package because... Thanks!
[GitHub] nifi pull request: Adding requirejs, angularjs and its dependencie...
GitHub user scottyaslan opened a pull request: https://github.com/apache/nifi/pull/331 Adding requirejs, angularjs and its dependencies, and manually boostr⦠â¦apping an angular app to the body You can merge this pull request into a Git repository by running: $ git pull https://github.com/scottyaslan/nifi responsiveDevBranch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/331.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #331 commit e8499212fc2aeaa735fe26ff8febd84be8663c47 Author: James AslanDate: 2016-04-06T20:37:35Z Adding requirejs, angularjs and its dependencies, and manually boostrapping an angular app to the body --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1678
Github user markap14 commented on the pull request: https://github.com/apache/nifi/pull/323#issuecomment-206513343 I updated the PR with a new commit to address an issue that I ran into. When we create 'container' nodes in ZooKeeper whose paths are not sensitive, we will mark the node as wide-open, rather than applying ACLs. This allows us to not have to worry about if we have state management and heartbeats using different ACLs. Also removed use of username/passwords from ZooKeeper configuration. This was done because ZooKeeper indicates that these should be used only for local testing and not for production. In production, security should be administered via Kerberos. Username and password were always sent over the wire unencrypted. Additionally, we are now using Apache Curator to handle Leader Election, and Curator does not appear to support using digest auth via username/password. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1678
Github user markap14 commented on the pull request: https://github.com/apache/nifi/pull/323#issuecomment-206513196 I updated the PR with a new commit to address an issue that I ran into. When we create 'container' nodes in ZooKeeper whose paths are not sensitive, we will mark the node as wide-open, rather than applying ACLs. This allows us to not have to worry about if we have state management and heartbeats using different ACLs. Also removed use of username/passwords from ZooKeeper configuration. This was done because ZooKeeper indicates that these should be used only for local testing and not for production. In production, security should be administered via Kerberos. Username and password were always sent over the wire unencrypted. Additionally, we are now using Apache Curator to handle Leader Election, and Curator does not appear to support using digest auth via username/password. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request: MINIFI-5: Base MiNiFi executable
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi/pull/5 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1678
Github user markap14 commented on the pull request: https://github.com/apache/nifi/pull/317#issuecomment-206489781 I updated the PR with a new commit to address an issue that I ran into. When we create 'container' nodes in ZooKeeper whose paths are not sensitive, we will mark the node as wide-open, rather than applying ACLs. This allows us to not have to worry about if we have state management and heartbeats using different ACLs. Also removed use of username/passwords from ZooKeeper configuration. This was done because ZooKeeper indicates that these should be used only for local testing and not for production. In production, security should be administered via Kerberos. Username and password were always sent over the wire unencrypted. Additionally, we are now using Apache Curator to handle Leader Election, and Curator does not appear to support using digest auth via username/password. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1582 added state to UpdateAttribute as wel...
Github user JPercivall commented on the pull request: https://github.com/apache/nifi/pull/319#issuecomment-206472004 @mattyb149 I did what you suggested, did a soft reset, re-committed (making a new hash like @apiri pointed out) and did a force push. It's rebuilding now --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request: MINIFI-5: Base MiNiFi executable
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/5#discussion_r58744217 --- Diff: minifi-nar-bundles/minifi-framework-bundle/minifi-framework-nar/pom.xml --- @@ -0,0 +1,67 @@ + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + +minifi-framework-bundle +org.apache.nifi.minifi +0.0.1-SNAPSHOT + +4.0.0 + +org.apache.nifi.minifi +minifi-framework-nar --- End diff -- That works for me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request: MINIFI-5: Base MiNiFi executable
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/5#discussion_r58744060 --- Diff: minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-runtime/src/main/java/org/apache/nifi/minifi/MiNiFi.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.minifi; + +import java.io.File; +import java.io.IOException; +import java.lang.Thread.UncaughtExceptionHandler; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.util.Timer; +import java.util.TimerTask; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; + +import org.apache.nifi.NiFiServer; +import org.apache.nifi.documentation.DocGenerator; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.nar.ExtensionMapping; +import org.apache.nifi.nar.NarClassLoaders; +import org.apache.nifi.nar.NarUnpacker; +import org.apache.nifi.util.FileUtils; +import org.apache.nifi.util.NiFiProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.slf4j.bridge.SLF4JBridgeHandler; + +public class MiNiFi { --- End diff -- Forgot to point out that the BootstrapListener's log and variable names also refer to nifi instead of minifi. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request: MINIFI-5: Base MiNiFi executable
Github user apiri commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/5#discussion_r58742228 --- Diff: minifi-assembly/pom.xml --- @@ -96,160 +85,250 @@ limitations under the License. slf4j-api compile + + -ch.qos.logback -logback-classic -compile +org.apache.nifi.minifi +minifi-resources +resources +runtime +zip +0.0.1-SNAPSHOT + + +org.apache.nifi.minifi +minifi-bootstrap +0.0.1-SNAPSHOT + + +org.apache.nifi.minifi +minifi-runtime +0.0.1-SNAPSHOT + + + + +org.apache.nifi +nifi-processor-utils - + + +org.apache.nifi +nifi-api + + +org.apache.nifi +nifi-runtime + + +org.apache.nifi +nifi-bootstrap --- End diff -- It is not, and I have taken some time to additionally defeat some other dependencies. These were likely remnants of a failed experiment. Good find! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request: MINIFI-5: Base MiNiFi executable
Github user apiri commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/5#discussion_r58742147 --- Diff: minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-runtime/src/main/java/org/apache/nifi/minifi/MiNiFi.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.minifi; + +import java.io.File; +import java.io.IOException; +import java.lang.Thread.UncaughtExceptionHandler; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.util.Timer; +import java.util.TimerTask; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; + +import org.apache.nifi.NiFiServer; +import org.apache.nifi.documentation.DocGenerator; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.nar.ExtensionMapping; +import org.apache.nifi.nar.NarClassLoaders; +import org.apache.nifi.nar.NarUnpacker; +import org.apache.nifi.util.FileUtils; +import org.apache.nifi.util.NiFiProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.slf4j.bridge.SLF4JBridgeHandler; + +public class MiNiFi { --- End diff -- Certainly will adjust. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request: MINIFI-5: Base MiNiFi executable
Github user apiri commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/5#discussion_r58742154 --- Diff: minifi-nar-bundles/minifi-framework-bundle/minifi-framework-nar/pom.xml --- @@ -0,0 +1,67 @@ + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + +minifi-framework-bundle +org.apache.nifi.minifi +0.0.1-SNAPSHOT + +4.0.0 + +org.apache.nifi.minifi +minifi-framework-nar --- End diff -- Was modeling after the nifi-framework-nar that is currently bundled through some of iterations. In the end, given the reuse approach, just a few of these libraries are utilized in lieu of the full NAR. I think that transition will happen later and would like to keep this available moving forward as other folks may wish to contribute to the core framework and things are shifted from NiFi to MiNiFi specific efforts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request: MINIFI-5: Base MiNiFi executable
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/5#discussion_r58732157 --- Diff: minifi-assembly/pom.xml --- @@ -96,160 +85,250 @@ limitations under the License. slf4j-api compile + + -ch.qos.logback -logback-classic -compile +org.apache.nifi.minifi +minifi-resources +resources +runtime +zip +0.0.1-SNAPSHOT + + +org.apache.nifi.minifi +minifi-bootstrap +0.0.1-SNAPSHOT + + +org.apache.nifi.minifi +minifi-runtime +0.0.1-SNAPSHOT + + + + +org.apache.nifi +nifi-processor-utils - + + +org.apache.nifi +nifi-api + + +org.apache.nifi +nifi-runtime + + +org.apache.nifi +nifi-bootstrap --- End diff -- Why is nifi-bootstrap needed? Isn't minifi-bootstrap replacing that? Also nifi-bootstrap is explicitly included in the bootstrap lib in minifi-assembly's dependencies.xml. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request: MINIFI-5: Base MiNiFi executable
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/5#discussion_r58728826 --- Diff: minifi-nar-bundles/minifi-framework-bundle/minifi-framework-nar/pom.xml --- @@ -0,0 +1,67 @@ + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + +minifi-framework-bundle +org.apache.nifi.minifi +0.0.1-SNAPSHOT + +4.0.0 + +org.apache.nifi.minifi +minifi-framework-nar --- End diff -- This package is effectively not used. Running a "find" for "minifi-framework-nar" on the project only turns up two places, here and and in minifi-framework-bundle pom declaring minifi-framwork-nar as a module. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request: MINIFI-5: Base MiNiFi executable
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/5#discussion_r58712548 --- Diff: minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-runtime/src/main/java/org/apache/nifi/minifi/MiNiFi.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.minifi; + +import java.io.File; +import java.io.IOException; +import java.lang.Thread.UncaughtExceptionHandler; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.util.Timer; +import java.util.TimerTask; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; + +import org.apache.nifi.NiFiServer; +import org.apache.nifi.documentation.DocGenerator; +import org.apache.nifi.nar.ExtensionManager; +import org.apache.nifi.nar.ExtensionMapping; +import org.apache.nifi.nar.NarClassLoaders; +import org.apache.nifi.nar.NarUnpacker; +import org.apache.nifi.util.FileUtils; +import org.apache.nifi.util.NiFiProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.slf4j.bridge.SLF4JBridgeHandler; + +public class MiNiFi { --- End diff -- All of the log statements and exception messages refer to "NiFi" instead of "MiNiFi". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Compression of Data in HDFS
Ok one more question... On GetHDFS are you setting the Directory to "\landing\databasename\prodeiw_arc\" and then setting Recurse Sub-Directories to true to have it go into each table's directory? The reason I ask is because the FlowFiles coming out of GetHDFS have an attribute on them called Path, the documentation says: The path is set to the relative path of the file's directory on HDFS. For example, if the Directory property is set to /tmp, then files picked up from /tmp will have the path attribute set to "./". If the Recurse Subdirectories property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to "abc/1/2/3" So theoretically if you were pointing to "\landing\databasename\prodeiw_arc\" and then it recursed into "\landing\databasename\prodeiw_arc\tablename", the path attribute would end up being "tablename". You could then reference this in your PutHDFS processor by setting the Directory to "/landing/teradata/compressed/prodeiw_arc/$[path}" On Wed, Apr 6, 2016 at 8:46 AM, jamesgreenwrote: > Hi Brian, Thanks for the help! > > I have tried two ways > a. > 1. I use GetHDFS to retrieve data from the HDFS , I then use putHDFS > and set > the compression to GZIP. > 2. In the Directory I am putting the complete path i.e > /landing/teradata/compressed/prodeiw_arc > b. > 1. I use GetHDFS to retrieve data from the HDFS, I then use Compress > Content to apply the compression and then use PutHDFS > 2. In the Directory I am putting the complete path i.e > /landing/teradata/compressed/prodeiw_arc > > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Compression-of-Data-in-HDFS-tp8821p8825.html > Sent from the Apache NiFi Developer List mailing list archive at > Nabble.com. >
[GitHub] nifi pull request: NIFI-1582 added state to UpdateAttribute as wel...
Github user apiri commented on the pull request: https://github.com/apache/nifi/pull/319#issuecomment-206384283 There are no facilities to manually start a Travis job. You could likely squash your two commits and push those. The core thing is that a new hash is seen for the Travis webhooks, I believe. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1582 added state to UpdateAttribute as wel...
Github user mattyb149 commented on the pull request: https://github.com/apache/nifi/pull/319#issuecomment-206383566 There was something that could be done, I can't remember. Maybe reset --soft HEAD^ and re-commit and push -f. Will give the same code a new commit SHA and should kick it off again. I thought it was simpler to do but maybe not. Also maybe we can re-run the Travis job manually? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1582 added state to UpdateAttribute as wel...
Github user JPercivall commented on the pull request: https://github.com/apache/nifi/pull/319#issuecomment-206383057 @mattyb149 just tried pushing with no new commits and nothing happened. I believe I could push an empty commit but I don't think it's worth cluttering this PR with unnecessary commits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Compression of Data in HDFS
Hi Brian, Thanks for the help! I have tried two ways a. 1. I use GetHDFS to retrieve data from the HDFS , I then use putHDFS and set the compression to GZIP. 2. In the Directory I am putting the complete path i.e /landing/teradata/compressed/prodeiw_arc b. 1. I use GetHDFS to retrieve data from the HDFS, I then use Compress Content to apply the compression and then use PutHDFS 2. In the Directory I am putting the complete path i.e /landing/teradata/compressed/prodeiw_arc -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Compression-of-Data-in-HDFS-tp8821p8825.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
[GitHub] nifi pull request: NIFI-1690 Changed MonitorMemory to use allowabl...
Github user joewitt commented on a diff in the pull request: https://github.com/apache/nifi/pull/328#discussion_r58706787 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-reporting-tasks/src/main/java/org/apache/nifi/controller/MonitorMemory.java --- @@ -91,12 +92,22 @@ + " that the memory pool is exceeding this threshold.") public class MonitorMemory extends AbstractReportingTask { +private static final AllowableValue[] memPoolAllowableValues; + +static { +List memoryPoolBeans = ManagementFactory.getMemoryPoolMXBeans(); +memPoolAllowableValues = new AllowableValue[memoryPoolBeans.size()]; +for (int i = 0; i < memPoolAllowableValues.length; i++) { +memPoolAllowableValues[i] = new AllowableValue(memoryPoolBeans.get(i).getName()); +} +} + public static final PropertyDescriptor MEMORY_POOL_PROPERTY = new PropertyDescriptor.Builder() .name("Memory Pool") .description("The name of the JVM Memory Pool to monitor") .required(true) -.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) -.defaultValue(null) +.allowableValues(memPoolAllowableValues) +.defaultValue(memPoolAllowableValues.length == 0 ? null : memPoolAllowableValues[0].getValue()) --- End diff -- I think we're better off simply not having a default value. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Compression of Data in HDFS
Hello, Can you describe your flow a bit more? Are you using ListHDFS + FetchHDFS to retrieve the data from HDFS? What value do you have for the Directory property in PutHDFS? Thanks, Bryan On Wed, Apr 6, 2016 at 7:12 AM, jamesgreenwrote: > I am trying to compress a whole lot of files from my HDFS and write to > another folder on the HDFS > My Folder Structure is as follows: > \landing\databasename\prodeiw_arc\tablename\_SUCCESS > \landing\databasename\prodeiw_arc\tablename\part-m-0 > > \landing\databasename\prodeiw_arc\tablename2\_SUCCESS > \landing\databasename\prodeiw_arc\tablename2\part-m-0 > > I am trying to compress to the following > \landing\compressed\prodeiw_arc\tablename\_SUCCESS > \landing\compressed\prodeiw_arc\tablename\part-m-0 > > \landing\compressed\prodeiw_arc\tablename2\_SUCCESS > \landing\compressed\prodeiw_arc\tablename2\part-m-0 > > I have found that it compresses to > \landing\compressed\prodeiw_arc\_SUCCESS > \landing\compressed\prodeiw_arc\tablename\part-m-0 > > it will then continue to overwrite. Is there anyway I can keep the > directory > structure when doing a PutHDFS? > > Thanks and Regards > > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Compression-of-Data-in-HDFS-tp8821.html > Sent from the Apache NiFi Developer List mailing list archive at > Nabble.com. >
Re: How to pass variables to nifi Processor
Hello, For properties that support expression language, they can reference properties passed in through the bootstrap.conf file. So you could define a property in bootstrap.conf like -Dfoo=abc and in a processor property reference that as $[foo}. There are plans for this be improved in the future through the idea of a variable registry: https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry You may also want to take a look at this project that provides a way to deploy templates from one instance to another: https://github.com/aperepel/nifi-api-deploy Thanks, Bryan On Tue, Apr 5, 2016 at 1:32 PM, Rajeswari Raghunathan - Contractor < rajeswari.raghunathan.contrac...@8451.com> wrote: > Hi Team, > > We are using NIFI in our company to migrate all data from SFTP server to > HDFS. > We have different environment like dev, test, prod and want to accomplish > Continuous integration with NIFI. In order to do so, we need to separate > all sensitive data like server hostname,username,password from hardcoding > in Processor (eg:-GETSFTP). > Can anyone help me by providing best solution for this problem? > > Regards, > Rajeswari >
Compression of Data in HDFS
I am trying to compress a whole lot of files from my HDFS and write to another folder on the HDFS My Folder Structure is as follows: \landing\databasename\prodeiw_arc\tablename\_SUCCESS \landing\databasename\prodeiw_arc\tablename\part-m-0 \landing\databasename\prodeiw_arc\tablename2\_SUCCESS \landing\databasename\prodeiw_arc\tablename2\part-m-0 I am trying to compress to the following \landing\compressed\prodeiw_arc\tablename\_SUCCESS \landing\compressed\prodeiw_arc\tablename\part-m-0 \landing\compressed\prodeiw_arc\tablename2\_SUCCESS \landing\compressed\prodeiw_arc\tablename2\part-m-0 I have found that it compresses to \landing\compressed\prodeiw_arc\_SUCCESS \landing\compressed\prodeiw_arc\tablename\part-m-0 it will then continue to overwrite. Is there anyway I can keep the directory structure when doing a PutHDFS? Thanks and Regards -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Compression-of-Data-in-HDFS-tp8821.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
[GitHub] nifi-minifi pull request: MINIFI-5: Base MiNiFi executable
GitHub user apiri opened a pull request: https://github.com/apache/nifi-minifi/pull/5 MINIFI-5: Base MiNiFi executable MINIFI-5: Creating a base MiNiFi project to serve as a basis for further extension and design reusing NiFi libraries You can merge this pull request into a Git repository by running: $ git pull https://github.com/apiri/nifi-minifi minifi-5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/5.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5 commit e94b6dae3dcb18bd60c9234de62543c897e83146 Author: Aldrin PiriDate: 2016-04-06T01:19:57Z MINIFI-5: Creating a base MiNiFi project to serve as a basis for further extension and design reusing NiFi libraries --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---