Re: [DISCUSS] Apache NiFi 0.5.0 RC1
Hi Tony, maybe I missed it in the Repo but did you also provide binaries to test? I'll try building it myself now (as you ask) but wanted to check anyway. Cheers, Lars On Sun, Feb 7, 2016 at 1:34 AM, Tony Kurc wrote: > Hello > We put together the RC1 for nifi-0.5.0. Because we found some early bugs, > as already mentioned on this list, we will not be holding a vote, but > wanted to have some community feedback and testing before putting together > another release candidate. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1071 > > The Git tag is NIFI-1379-RC1 > The Git commit ID is c0c7024ffdcb77267ae60942012c7999085e4b3f > > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=c0c7024ffdcb77267ae60942012c7999085e4b3f > > Checksums of nifi-0.5.0-source-release.zip: > MD5: 6ed99039e0e98e607139c37887d3d35c > SHA1: e64f72a97a5bd484d606af3d12caa41ca531c322 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/tkurc.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/dev/nifi/KEYS > > 103 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334158 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.0 > > Please download the release candidate and evaluate, especially the ability > to build from source, and test. > > Thanks, > > Tony >
Re: [DISCUSS] Apache NiFi 0.5.0 RC1
Okay this turned out easier than expected. Happy to report that building the NiFi tarball as documented in the README worked and was easy and it starts up just fine. On Wed, Feb 10, 2016 at 9:00 AM, Lars Francke wrote: > Hi Tony, > > maybe I missed it in the Repo but did you also provide binaries to test? > I'll try building it myself now (as you ask) but wanted to check anyway. > > Cheers, > Lars > > On Sun, Feb 7, 2016 at 1:34 AM, Tony Kurc wrote: > >> Hello >> We put together the RC1 for nifi-0.5.0. Because we found some early bugs, >> as already mentioned on this list, we will not be holding a vote, but >> wanted to have some community feedback and testing before putting together >> another release candidate. >> >> The source zip, including signatures, digests, etc. can be found at: >> https://repository.apache.org/content/repositories/orgapachenifi-1071 >> >> The Git tag is NIFI-1379-RC1 >> The Git commit ID is c0c7024ffdcb77267ae60942012c7999085e4b3f >> >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=c0c7024ffdcb77267ae60942012c7999085e4b3f >> >> Checksums of nifi-0.5.0-source-release.zip: >> MD5: 6ed99039e0e98e607139c37887d3d35c >> SHA1: e64f72a97a5bd484d606af3d12caa41ca531c322 >> >> Release artifacts are signed with the following key: >> https://people.apache.org/keys/committer/tkurc.asc >> >> KEYS file available here: >> https://dist.apache.org/repos/dist/dev/nifi/KEYS >> >> 103 issues were closed/resolved for this release: >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12334158 >> >> Release note highlights can be found here: >> >> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.0 >> >> Please download the release candidate and evaluate, especially the ability >> to build from source, and test. >> >> Thanks, >> >> Tony >> > >
[GitHub] nifi pull request: [NIFI-1496] State Manager documentation link wr...
GitHub user lfrancke opened a pull request: https://github.com/apache/nifi/pull/214 [NIFI-1496] State Manager documentation link wrong You can merge this pull request into a Git repository by running: $ git pull https://github.com/lfrancke/nifi NIFI-1496 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/214.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #214 commit 1ab06b4db9c2e85a5c2b0445b6ea2b5ff1349b38 Author: Lars Francke Date: 2016-02-10T11:35:15Z [NIFI-1496] State Manager documentation link wrong --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: [NIFI-1496] State Manager documentation link wr...
Github user mcgilman commented on the pull request: https://github.com/apache/nifi/pull/214#issuecomment-182368423 Reviewing now... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: [NIFI-1496] State Manager documentation link wr...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/214 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: [NIFI-1496] State Manager documentation link wr...
Github user mcgilman commented on the pull request: https://github.com/apache/nifi/pull/214#issuecomment-182384398 +1 Thanks for the contribution! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request: NIFI-1484 - ListFile holds unbounded list of fi...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/212 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Nifi - Adding attributes to flow file results in FlowFileHandlingException when flow file is transferred
Hi:
I am processing some flow files and want to add success and failure attributes
to the processed flow file and then transfer it. But this is producing an
exception:
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=432bc163-28a0-4d08-b9e8-1674a649ae8c,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1455121175949-1, container=default,
section=1], offset=0, length=1],offset=0,name=129790440390423,size=1] is not
known in this session (StandardProcessSession[id=70]) at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:2361)
~[nifi-framework-core-0.5.0-SNAPSHOT.jar:0.5.0-SNAPSHOT]
Here is the code segment in the onTrigger method:
Note - If I comment out the lines (as shown below) where I tried to add
attributes to the flow file and it works. If I uncomment the lines (either
adding single attributes or multiple, the exception is produced)
try { List records = new ArrayList<>();
// Prepare batch of records for (int i = 0; i <
flowFiles.size(); i++) { final ByteArrayOutputStream baos = new
ByteArrayOutputStream(); session.exportTo(flowFiles.get(i),
baos); records.add(new
Record().withData(ByteBuffer.wrap(baos.toByteArray(; }
// Send the batch PutRecordBatchRequest
putRecordBatchRequest = new PutRecordBatchRequest();
putRecordBatchRequest.setDeliveryStreamName(streamName);
putRecordBatchRequest.setRecords(records); PutRecordBatchResult
results = client.putRecordBatch(putRecordBatchRequest);
// Separate out the successful and failed flow files
List responseEntries =
results.getRequestResponses(); List failedFlowFiles = new
ArrayList<>(); List successfulFlowFiles = new
ArrayList<>(); for (int i = 0; i < responseEntries.size(); i++ ) {
PutRecordBatchResponseEntry entry = responseEntries.get(i);
FlowFile flowFile = flowFiles.get(i);
Map attributes = new HashMap<>();
attributes.put(RECORD_ID, entry.getRecordId());// NOTE - If I uncomment this
line - or any other which adds attributes to the flowfile - i get the
exception// session.putAttribute(flowFile,RECORD_ID,
entry.getRecordId()); if ( !
StringUtils.isBlank(entry.getErrorCode()) ) {
attributes.put(ERROR_CODE, entry.getErrorCode());
attributes.put(ERROR_MESSAGE, entry.getErrorMessage());//
session.putAllAttributes(flowFile, attributes);
failedFlowFiles.add(flowFile); } else {//
session.putAllAttributes(flowFile, attributes);
successfulFlowFiles.add(flowFile); } }
if ( failedFlowFiles.size() > 0 ) {
session.transfer(failedFlowFiles, REL_FAILURE);
getLogger().error("Failed to send {} records {}", new Object[]{stream,
failedFlowFiles}); }
if ( successfulFlowFiles.size() > 0 ) {// Throws exception when
attributes are added to flow files
session.transfer(successfulFlowFiles, REL_SUCCESS);
getLogger().info("Success sent {} records {}", new Object[]{stream,
successfulFlowFiles}); }
records.clear();
Re: Nifi - Adding attributes to flow file results in FlowFileHandlingException when flow file is transferred
Hello,
The error message is indicating that you are trying to transfer an unknown
FlowFile because it is transferring a reference to the original FlowFile
before you updated the attributes. You would need to assign the result of
putAllAttributes (or putAttribute) and then transfer that:
flowFile = session.putAllAttributes(flowFile, attributes);
Thanks,
Bryan
On Wed, Feb 10, 2016 at 11:41 AM, M Singh
wrote:
> Hi:
> I am processing some flow files and want to add success and failure
> attributes to the processed flow file and then transfer it. But this is
> producing an exception:
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=432bc163-28a0-4d08-b9e8-1674a649ae8c,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1455121175949-1, container=default,
> section=1], offset=0, length=1],offset=0,name=129790440390423,size=1] is
> not known in this session (StandardProcessSession[id=70]) at
> org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:2361)
> ~[nifi-framework-core-0.5.0-SNAPSHOT.jar:0.5.0-SNAPSHOT]
> Here is the code segment in the onTrigger method:
> Note - If I comment out the lines (as shown below) where I tried to add
> attributes to the flow file and it works. If I uncomment the lines (either
> adding single attributes or multiple, the exception is produced)
> try {List records = new ArrayList<>();
> // Prepare batch of recordsfor (int i = 0; i <
> flowFiles.size(); i++) {final ByteArrayOutputStream baos =
> new ByteArrayOutputStream();
> session.exportTo(flowFiles.get(i), baos);records.add(new
> Record().withData(ByteBuffer.wrap(baos.toByteArray(;}
> // Send the batchPutRecordBatchRequest
> putRecordBatchRequest = new PutRecordBatchRequest();
> putRecordBatchRequest.setDeliveryStreamName(streamName);
> putRecordBatchRequest.setRecords(records);PutRecordBatchResult
> results = client.putRecordBatch(putRecordBatchRequest);
> // Separate out the successful and failed flow files
> List responseEntries =
> results.getRequestResponses();List failedFlowFiles =
> new ArrayList<>();List successfulFlowFiles = new
> ArrayList<>();for (int i = 0; i < responseEntries.size(); i++ )
> {PutRecordBatchResponseEntry entry =
> responseEntries.get(i);FlowFile flowFile = flowFiles.get(i);
> Map attributes = new HashMap<>();
> attributes.put(RECORD_ID, entry.getRecordId());// NOTE - If I
> uncomment this line - or any other which adds attributes to the flowfile -
> i get the exception//
> session.putAttribute(flowFile,RECORD_ID, entry.getRecordId());
> if ( ! StringUtils.isBlank(entry.getErrorCode()) ) {
> attributes.put(ERROR_CODE, entry.getErrorCode());
> attributes.put(ERROR_MESSAGE, entry.getErrorMessage());//
> session.putAllAttributes(flowFile, attributes);
> failedFlowFiles.add(flowFile);} else {//
> session.putAllAttributes(flowFile, attributes);
> successfulFlowFiles.add(flowFile);}}
> if ( failedFlowFiles.size() > 0 ) {
> session.transfer(failedFlowFiles, REL_FAILURE);
> getLogger().error("Failed to send {} records {}", new Object[]{stream,
> failedFlowFiles});}
> if ( successfulFlowFiles.size() > 0 ) {// Throws exception
> when attributes are added to flow files
> session.transfer(successfulFlowFiles, REL_SUCCESS);
> getLogger().info("Success sent {} records {}", new Object[]{stream,
> successfulFlowFiles});}
> records.clear();
Re: Nifi - Adding attributes to flow file results in FlowFileHandlingException when flow file is transferred
Bryan :
That was indeed the issue. Thanks for your quick response.
Mans
On Wednesday, February 10, 2016 8:59 AM, Bryan Bende
wrote:
Hello,
The error message is indicating that you are trying to transfer an unknown
FlowFile because it is transferring a reference to the original FlowFile
before you updated the attributes. You would need to assign the result of
putAllAttributes (or putAttribute) and then transfer that:
flowFile = session.putAllAttributes(flowFile, attributes);
Thanks,
Bryan
On Wed, Feb 10, 2016 at 11:41 AM, M Singh
wrote:
> Hi:
> I am processing some flow files and want to add success and failure
> attributes to the processed flow file and then transfer it. But this is
> producing an exception:
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=432bc163-28a0-4d08-b9e8-1674a649ae8c,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1455121175949-1, container=default,
> section=1], offset=0, length=1],offset=0,name=129790440390423,size=1] is
> not known in this session (StandardProcessSession[id=70]) at
> org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:2361)
> ~[nifi-framework-core-0.5.0-SNAPSHOT.jar:0.5.0-SNAPSHOT]
> Here is the code segment in the onTrigger method:
> Note - If I comment out the lines (as shown below) where I tried to add
> attributes to the flow file and it works. If I uncomment the lines (either
> adding single attributes or multiple, the exception is produced)
> try { List records = new ArrayList<>();
> // Prepare batch of records for (int i = 0; i <
> flowFiles.size(); i++) { final ByteArrayOutputStream baos =
> new ByteArrayOutputStream();
> session.exportTo(flowFiles.get(i), baos); records.add(new
> Record().withData(ByteBuffer.wrap(baos.toByteArray(; }
> // Send the batch PutRecordBatchRequest
> putRecordBatchRequest = new PutRecordBatchRequest();
> putRecordBatchRequest.setDeliveryStreamName(streamName);
> putRecordBatchRequest.setRecords(records); PutRecordBatchResult
> results = client.putRecordBatch(putRecordBatchRequest);
> // Separate out the successful and failed flow files
> List responseEntries =
> results.getRequestResponses(); List failedFlowFiles =
> new ArrayList<>(); List successfulFlowFiles = new
> ArrayList<>(); for (int i = 0; i < responseEntries.size(); i++ )
> { PutRecordBatchResponseEntry entry =
> responseEntries.get(i); FlowFile flowFile = flowFiles.get(i);
> Map attributes = new HashMap<>();
> attributes.put(RECORD_ID, entry.getRecordId());// NOTE - If I
> uncomment this line - or any other which adds attributes to the flowfile -
> i get the exception//
> session.putAttribute(flowFile,RECORD_ID, entry.getRecordId());
> if ( ! StringUtils.isBlank(entry.getErrorCode()) ) {
> attributes.put(ERROR_CODE, entry.getErrorCode());
> attributes.put(ERROR_MESSAGE, entry.getErrorMessage());//
> session.putAllAttributes(flowFile, attributes);
> failedFlowFiles.add(flowFile); } else {//
> session.putAllAttributes(flowFile, attributes);
> successfulFlowFiles.add(flowFile); } }
> if ( failedFlowFiles.size() > 0 ) {
> session.transfer(failedFlowFiles, REL_FAILURE);
> getLogger().error("Failed to send {} records {}", new Object[]{stream,
> failedFlowFiles}); }
> if ( successfulFlowFiles.size() > 0 ) {// Throws exception
> when attributes are added to flow files
> session.transfer(successfulFlowFiles, REL_SUCCESS);
> getLogger().info("Success sent {} records {}", new Object[]{stream,
> successfulFlowFiles}); }
> records.clear();
Re: ApacheCon presence?
Sean, I have been tossing a couple of ideas around in my head for a proposal. Hoping to be able to tie a few of them together in a cohesive manner and compose an abstract. On Tue, Feb 9, 2016 at 11:49 AM, Sean Busbey wrote: > Hi folks! > > CFPs for ApacheCon NA Core and ApacheCon NA Big Data close in a few > days, on 12 Feb. > > Anyone already planning to attend? Anyone already submit a proposal? > > I'd really like to see NiFi have a good showing. > > -Sean >
[GitHub] nifi pull request: Fix NiFi-1461 : RPM should create nifi user and...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/204 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: planning for apache nifi 0.5.0
Tony, I believe based on progress with JIRAs today and discussions had that NiFi 0.5.0 is ready for another RC. Thanks Joe On Sat, Feb 6, 2016 at 11:00 AM, Joe Witt wrote: > Hello > > Would like some input/advice on > https://issues.apache.org/jira/browse/NIFI-1484. In my initial > testing of this I would consider it worthy of getting resolved right > away. But perhaps I'm overthinking it. Related, but far less urgent, > is https://issues.apache.org/jira/browse/NIFI-1483. That one is > simply an irritation but could cause pause for every user that sees > it. > > Thanks > Joe > > On Sat, Feb 6, 2016 at 3:41 AM, Tony Kurc wrote: >> Awesome! I will start working on the rc >> On Feb 6, 2016 12:34 AM, "Aldrin Piri" wrote: >> >>> All, >>> >>> Just wanted to follow up that the members of the community had completed >>> their testing against NIFI-259 and I performed the merge of that branch >>> into master. >>> >>> In that time, there were a few other issues that had some added bits of >>> functionality and improvements which were also folded in and incorporated >>> into the codebase. >>> >>> At this time, all issues exclusive of the actual release process for 0.5.0 >>> have been incorporated and resolved. Congrats on all the effort to tie >>> these items up on both the contrib and review efforts. Looking forward to >>> the community's involvement of the upcoming RC during our release >>> evaluation and voting process. >>> >>> Thanks! >>> --aldrin >>> >>> On Thu, Feb 4, 2016 at 10:33 PM, Aldrin Piri wrote: >>> >>> > All, >>> > >>> > Just as an update on some of the remaining items. I have been doing some >>> > review work to help close out some of these final issues. Finished >>> review >>> > and merged 1257 and 1259 which brings a lot of great security and >>> > encryption functions that will help a long way for providing a nice >>> > foundation for additional support across the entirety of the project. >>> > >>> > Along with several of the members of the community, I have been reviewing >>> > and providing feedback 259 along with several other members of the >>> > community and its incorporated tickets of 1223, 1379. We are doing some >>> > final testing particularly surrounding its interactions and utilization >>> of >>> > Kerberos. Once that wraps up, we will be able to merge that in, closing >>> > the last of the items scheduled for this release. >>> > >>> > Thanks! >>> > Aldrin >>> > >>> > On Thu, Feb 4, 2016 at 9:53 AM, Joe Witt wrote: >>> > >>> >> Tony >>> >> >>> >> Agreed. I will work through these as well and try to ensure the >>> >> wording accurately reflect what happened. The release notes are a >>> >> really important piece of communication we need to get right. >>> >> >>> >> Thanks >>> >> Joe >>> >> >>> >> On Thu, Feb 4, 2016 at 6:04 AM, Tony Kurc wrote: >>> >> > I am prepping for the release, going through the jiras that have been >>> >> > closed, I think several have a description that does not match. Good >>> >> > example is: >>> >> > >>> >> > https://issues.apache.org/jira/browse/NIFI-1325 >>> >> > >>> >> > I believe the final scope of the patch surpassed the original >>> >> description. >>> >> > Should this be adjusted for people perusing the JIRA report we link to >>> >> in >>> >> > our release notes? I'd say yes. I made a couple small edits to some >>> >> tickets >>> >> > already for increased readability. One change I wanted to make was >>> >> change >>> >> > NIFI-259 to "New Feature" rather than subtask, but I was concerned >>> about >>> >> > this disrupting folks. >>> >> > >>> >> > I think a quick consistency and spell check would be good - I've had >>> >> > several people mention out of band that prior release notes had a >>> couple >>> >> > errors. Apparently people love the release notes page on the JIRA >>> >> > NIFI-1107 is done, just needs a merge, which I'll do tonight. >>> >> > >>> >> > On Wed, Feb 3, 2016 at 4:32 PM, Joe Witt wrote: >>> >> > >>> >> >> Latest update for all >>> >> >> >>> >> >> You can see the current status here [1]. >>> >> >> >>> >> >> Looks like several tickets but really it is three efforts: >>> >> >> 1) Improve encryption options in EncryptContent NIFI-1257,1259 >>> >> >> 2) Provide state management as a framework function >>> NIFI-259,1223,1339 >>> >> >> 3) Support multi-part S3 uploads NIFI-1107 >>> >> >> >>> >> >> Based on comments on each they are just in very late stage/detailed >>> >> >> review. So we're close. >>> >> >> >>> >> >> Thanks >>> >> >> Joe >>> >> >> >>> >> >> [1] >>> >> >> >>> >> >>> https://issues.apache.org/jira/browse/NIFI-1339?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.5.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC >>> >> >> >>> >> >> On Thu, Jan 21, 2016 at 12:55 AM, Joe Witt >>> wrote: >>> >> >> > Team >>> >> >> > >>> >> >> > Did a full round of 050 JIRA cleanup this evening. Great progress >>> is >>> >> >> > being made. >>> >> >> > >>> >> >>
