RE: Java exception when attempting to update version from registry

[Russ Weiher]

Kevin -

Glad to hear there is a root cause for this.   Am I correct in assuming that I 
won't be able to just rename the offending input port to get this current 
change pushed out?  Basically the "bad thing" is already embedded in the change 
history of the flow?

Russ Weiher
BI CI Solutions Architect
Progenity, Inc.


-Original Message-
From: Kevin Doran [mailto:kdo...@apache.org]
Sent: Thursday, January 10, 2019 4:41 PM
To: users@nifi.apache.org; Russ Weiher 
Subject: RE: Java exception when attempting to update version from registry

Russ,

The diff was perfect. It showed the problem was two input ports with the same 
name but different IDs, which I was able to reproduce. Your stack trace and 
flow diff helped us uncover a bug in our update flow logic -- thanks!

I verified it still affects the latest version of NiFi and I opened a Jira 
issue [1] that captures this. Feel free to sign up to ASF Jira to "watch" that 
issue if you want to see when this gets patched. In the meantime, you can avoid 
this by avoiding deleting/adding input ports with the same name (during a 
single version change) within a versioned process group. It's a pretty rare 
case that should not come up often, so I don't think this should impact your 
workflow for moving flows from dev to higher envs in general. If you do run 
into other failures in your workflow, please let us know.

[1] 
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-5950data=01%7C01%7CRuss.Weiher%40progenity.com%7Cb1c40b5ddbbb45dd4ec208d677445b2a%7C0666349c4979449297792a34e68103b3%7C0sdata=kN4WhTPPOGlO%2FlogPv5BB7716DnzEPyrfkMjSeedQ2E%3Dreserved=0

Thanks!
Kevin

On January 10, 2019 at 15:45:43, Russ Weiher (russ.wei...@progenity.com) wrote:
> Kevin -
>
> Hopefully the diff will provide some insight. As you have suggested, I
> don't think I could actually provide the whole flow outside of an established 
> customer support situation.
>
> Here is the diff:
>
> {
> "bucketId": "251301de-a141-46f5-b4ac-106e751f0dc9",
> "componentDifferenceGroups": [
> {
> "componentId": "99d38ec9-4c72-3395-9f14-10d6d115f7a8",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "99d38ec9-4c72-3395-9f14-10d6d115f7a8"
> }
> ],
> "processGroupId": "eebc95fc-7381-34a3-84a7-79861d88b3d5"
> },
> {
> "componentId": "4f281a67-5d32-38b1-bb49-071a3005f559",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "4f281a67-5d32-38b1-bb49-071a3005f559"
> }
> ],
> "processGroupId": "eebc95fc-7381-34a3-84a7-79861d88b3d5"
> },
> {
> "componentId": "00088125-f087-3f98-a58a-f4e102f2bc0e",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "00088125-f087-3f98-a58a-f4e102f2bc0e"
> }
> ],
> "processGroupId": "1785e36e-6b5d-3451-b3bc-465ab0f21e81"
> },
> {
> "componentId": "7a1db67e-5b9f-38df-9fb8-2b4fdf113d45",
> "componentName": "InvokeHTTP",
> "componentType": "Processor",
> "differences": [
> {
> "changeDescription": "Processor was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "7a1db67e-5b9f-38df-9fb8-2b4fdf113d45"
> }
> ],
> "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
> },
> {
> "componentId": "6add78e1-41b4-3b17-b588-040d01f38ec0",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "6add78e1-41b4-3b17-b588-040d01f38ec0"
> }
> ],
> "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
> },
> {
> "componentId": "53fbf44b-0c16-3492-9b70-c6fdc7e3c8e7",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "53fbf44b-0c16-3492-9b70-c6fdc7e3c8e7"
> }
> ],
> "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
> },
> {
> "componentId": "38390485-ec6a-306b-a9d3-9bbba3bbcbf2",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "38390485-ec6a-306b-a9d3-9bbba3bbcbf2"
> }
> ],
> "processGroupId": "1116ed97-c7f7-3776-a2c4-82c5e7fd298d"
> },
> {
> "componentId": "a36d8b99-48d1-3a87-8aaf-2c59c6affd2d",
> "componentName": 

RE: Java exception when attempting to update version from registry

[Kevin Doran]

Russ,

The diff was perfect. It showed the problem was two input ports with the same 
name but different IDs, which I was able to reproduce. Your stack trace and 
flow diff helped us uncover a bug in our update flow logic -- thanks! 

I verified it still affects the latest version of NiFi and I opened a Jira 
issue [1] that captures this. Feel free to sign up to ASF Jira to "watch" that 
issue if you want to see when this gets patched. In the meantime, you can avoid 
this by avoiding deleting/adding input ports with the same name (during a 
single version change) within a versioned process group. It's a pretty rare 
case that should not come up often, so I don't think this should impact your 
workflow for moving flows from dev to higher envs in general. If you do run 
into other failures in your workflow, please let us know.

[1] https://issues.apache.org/jira/browse/NIFI-5950

Thanks!
Kevin

On January 10, 2019 at 15:45:43, Russ Weiher (russ.wei...@progenity.com) wrote:
> Kevin -
>  
> Hopefully the diff will provide some insight. As you have suggested, I don't 
> think I could  
> actually provide the whole flow outside of an established customer support 
> situation.  
>  
> Here is the diff:
>  
> {
> "bucketId": "251301de-a141-46f5-b4ac-106e751f0dc9",
> "componentDifferenceGroups": [
> {
> "componentId": "99d38ec9-4c72-3395-9f14-10d6d115f7a8",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "99d38ec9-4c72-3395-9f14-10d6d115f7a8"
> }
> ],
> "processGroupId": "eebc95fc-7381-34a3-84a7-79861d88b3d5"
> },
> {
> "componentId": "4f281a67-5d32-38b1-bb49-071a3005f559",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "4f281a67-5d32-38b1-bb49-071a3005f559"
> }
> ],
> "processGroupId": "eebc95fc-7381-34a3-84a7-79861d88b3d5"
> },
> {
> "componentId": "00088125-f087-3f98-a58a-f4e102f2bc0e",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "00088125-f087-3f98-a58a-f4e102f2bc0e"
> }
> ],
> "processGroupId": "1785e36e-6b5d-3451-b3bc-465ab0f21e81"
> },
> {
> "componentId": "7a1db67e-5b9f-38df-9fb8-2b4fdf113d45",
> "componentName": "InvokeHTTP",
> "componentType": "Processor",
> "differences": [
> {
> "changeDescription": "Processor was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "7a1db67e-5b9f-38df-9fb8-2b4fdf113d45"
> }
> ],
> "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
> },
> {
> "componentId": "6add78e1-41b4-3b17-b588-040d01f38ec0",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "6add78e1-41b4-3b17-b588-040d01f38ec0"
> }
> ],
> "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
> },
> {
> "componentId": "53fbf44b-0c16-3492-9b70-c6fdc7e3c8e7",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "53fbf44b-0c16-3492-9b70-c6fdc7e3c8e7"
> }
> ],
> "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
> },
> {
> "componentId": "38390485-ec6a-306b-a9d3-9bbba3bbcbf2",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "38390485-ec6a-306b-a9d3-9bbba3bbcbf2"
> }
> ],
> "processGroupId": "1116ed97-c7f7-3776-a2c4-82c5e7fd298d"
> },
> {
> "componentId": "a36d8b99-48d1-3a87-8aaf-2c59c6affd2d",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was added",
> "differenceType": "COMPONENT_ADDED",
> "differenceTypeDescription": "Component Added",
> "valueB": "a36d8b99-48d1-3a87-8aaf-2c59c6affd2d"
> }
> ],
> "processGroupId": "1116ed97-c7f7-3776-a2c4-82c5e7fd298d"
> },
> {
> "componentId": "fae582e7-b988-3160-b0c2-29af4ba788e3",
> "componentName": "",
> "componentType": "Connection",
> "differences": [
> {
> "changeDescription": "Connection was removed",
> "differenceType": "COMPONENT_REMOVED",
> "differenceTypeDescription": "Component Removed",
> "valueA": "fae582e7-b988-3160-b0c2-29af4ba788e3"
> }
> ],
> "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
> },
> {
> "componentId": 

Re: merging with default values?

[l vic]

Yes i do know them ahead of time, UpdateAttribute rules might work.
Thank you

On Thu, Jan 10, 2019 at 2:46 PM Russ Weiher 
wrote:

> Do you know the name of the attributes you are looking for ahead of time?
> If so, the UpdateAttribute processor has an “Advanced” section that lets
> you write conditional rules based on flowfile attributes and properties.
> If you haven’t already checked that out, it might be a good place to
> start.  There are some good examples in the NiFi documentation for that
> processor.
>
>
>
> *Russ Weiher*
>
> BI CI Solutions Architect
> Progenity, Inc.
>
>
>
> *From:* l vic [mailto:lvic4...@gmail.com]
> *Sent:* Thursday, January 10, 2019 2:18 PM
> *To:* users@nifi.apache.org
> *Subject:* merging with default values?
>
>
>
> I have to merge results of 3 different queries into one flowfile
>
> The problem happens if some of the queries result in empty resultset. For
> example, suppose i have 3 incoming connections for MergeContent:
>
> query1->flowfile1/attribute1, query2->flowfile2/attribute2,
> query3->flowfile3/attribute3
>
> And query2 returns empty resultset. In this case attribute2 never shows up
> in the merged flowfile.
>
> Can someone recommend merge solution where i would always have attribute2
> in the result but with some default value ( empty string assuming
> attribute2 is text)?
>
> Thank you...
> IMPORTANT: This email (and any attachments) is intended for the use of
> only the person or entity to which it is addressed, and may contain
> information that is privileged and confidential. You, the recipient, are
> obligated to maintain it in a safe, secure and confidential manner.
> Unauthorized redisclosure or failure to maintain confidentiality may
> subject you to federal and state penalties. If you are not the intended
> recipient, please immediately notify us by return email, and delete this
> message from your computer.
>

RE: Java exception when attempting to update version from registry

[Russ Weiher]

Kevin -

Hopefully the diff will provide some insight.  As you have suggested, I don't 
think I could actually provide the whole flow outside of an established 
customer support situation.

Here is the diff:

{
  "bucketId": "251301de-a141-46f5-b4ac-106e751f0dc9",
  "componentDifferenceGroups": [
{
  "componentId": "99d38ec9-4c72-3395-9f14-10d6d115f7a8",
  "componentName": "",
  "componentType": "Connection",
  "differences": [
{
  "changeDescription": "Connection was removed",
  "differenceType": "COMPONENT_REMOVED",
  "differenceTypeDescription": "Component Removed",
  "valueA": "99d38ec9-4c72-3395-9f14-10d6d115f7a8"
}
  ],
  "processGroupId": "eebc95fc-7381-34a3-84a7-79861d88b3d5"
},
{
  "componentId": "4f281a67-5d32-38b1-bb49-071a3005f559",
  "componentName": "",
  "componentType": "Connection",
  "differences": [
{
  "changeDescription": "Connection was added",
  "differenceType": "COMPONENT_ADDED",
  "differenceTypeDescription": "Component Added",
  "valueB": "4f281a67-5d32-38b1-bb49-071a3005f559"
}
  ],
  "processGroupId": "eebc95fc-7381-34a3-84a7-79861d88b3d5"
},
{
  "componentId": "00088125-f087-3f98-a58a-f4e102f2bc0e",
  "componentName": "",
  "componentType": "Connection",
  "differences": [
{
  "changeDescription": "Connection was added",
  "differenceType": "COMPONENT_ADDED",
  "differenceTypeDescription": "Component Added",
  "valueB": "00088125-f087-3f98-a58a-f4e102f2bc0e"
}
  ],
  "processGroupId": "1785e36e-6b5d-3451-b3bc-465ab0f21e81"
},
{
  "componentId": "7a1db67e-5b9f-38df-9fb8-2b4fdf113d45",
  "componentName": "InvokeHTTP",
  "componentType": "Processor",
  "differences": [
{
  "changeDescription": "Processor was removed",
  "differenceType": "COMPONENT_REMOVED",
  "differenceTypeDescription": "Component Removed",
  "valueA": "7a1db67e-5b9f-38df-9fb8-2b4fdf113d45"
}
  ],
  "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
},
{
  "componentId": "6add78e1-41b4-3b17-b588-040d01f38ec0",
  "componentName": "",
  "componentType": "Connection",
  "differences": [
{
  "changeDescription": "Connection was removed",
  "differenceType": "COMPONENT_REMOVED",
  "differenceTypeDescription": "Component Removed",
  "valueA": "6add78e1-41b4-3b17-b588-040d01f38ec0"
}
  ],
  "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
},
{
  "componentId": "53fbf44b-0c16-3492-9b70-c6fdc7e3c8e7",
  "componentName": "",
  "componentType": "Connection",
  "differences": [
{
  "changeDescription": "Connection was added",
  "differenceType": "COMPONENT_ADDED",
  "differenceTypeDescription": "Component Added",
  "valueB": "53fbf44b-0c16-3492-9b70-c6fdc7e3c8e7"
}
  ],
  "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
},
{
  "componentId": "38390485-ec6a-306b-a9d3-9bbba3bbcbf2",
  "componentName": "",
  "componentType": "Connection",
  "differences": [
{
  "changeDescription": "Connection was added",
  "differenceType": "COMPONENT_ADDED",
  "differenceTypeDescription": "Component Added",
  "valueB": "38390485-ec6a-306b-a9d3-9bbba3bbcbf2"
}
  ],
  "processGroupId": "1116ed97-c7f7-3776-a2c4-82c5e7fd298d"
},
{
  "componentId": "a36d8b99-48d1-3a87-8aaf-2c59c6affd2d",
  "componentName": "",
  "componentType": "Connection",
  "differences": [
{
  "changeDescription": "Connection was added",
  "differenceType": "COMPONENT_ADDED",
  "differenceTypeDescription": "Component Added",
  "valueB": "a36d8b99-48d1-3a87-8aaf-2c59c6affd2d"
}
  ],
  "processGroupId": "1116ed97-c7f7-3776-a2c4-82c5e7fd298d"
},
{
  "componentId": "fae582e7-b988-3160-b0c2-29af4ba788e3",
  "componentName": "",
  "componentType": "Connection",
  "differences": [
{
  "changeDescription": "Connection was removed",
  "differenceType": "COMPONENT_REMOVED",
  "differenceTypeDescription": "Component Removed",
  "valueA": "fae582e7-b988-3160-b0c2-29af4ba788e3"
}
  ],
  "processGroupId": "b1eea430-4d05-381d-99b1-4ea4d1d567bf"
},
{
  "componentId": "af62e20b-ecea-3594-97a9-22b0654462c7",
  "componentName": "",
  "componentType": "Connection",
  "differences": [
{
  "changeDescription": "Connection was added",
  "differenceType": "COMPONENT_ADDED",
  "differenceTypeDescription": "Component Added",
  "valueB": "af62e20b-ecea-3594-97a9-22b0654462c7"
}
  ],
  

RE: Java exception when attempting to update version from registry

[Kevin Doran]

Hi Russ,

Thanks for the stacktrace and details. I'm going to try to recreate
the issue. To do this I will need some specifics about your flow.

I expect you can't share the flow with me (if you can that would be
great...), but would it be possible to share the diff between versions
-- i.e., the diff between the version that was in QA and the version
that you tried to update to

You can get this by going to the following URL for your NiFi Registry server:

http://:/nifi-registry-api/buckets/{bucketId}/flows/{flowId}/diff/{versionA}/{versionB}

(If Registry is secured, you will need to do this with an HTTP client
that lets you authenticate as a user with access to the specified
Bucket, using either a TLS client cert or LDAP credentials passed to
the http://:/nifi-registry-api/access/token/login endpoint
to obtain a JWT)

Short of that, perhaps you could describe the steps to create a flow
diff that will recreate this error.

Thanks,
Kevin


On January 10, 2019 at 13:34:41, Russ Weiher (russ.wei...@progenity.com) wrote:
> Kevin -
>
> Here is the full stack trace from the log:
>
> 2019-01-10 10:10:55,085 ERROR [Version Control Update Thread-1] 
> org.apache.nifi.web.api.VersionsResource
> Failed to update flow to new version
> java.lang.IllegalStateException: The input port name or identifier is not 
> available
> to be added.
> at 
> org.apache.nifi.groups.StandardProcessGroup.addInputPort(StandardProcessGroup.java:505)
> at 
> org.apache.nifi.groups.StandardProcessGroup.addInputPort(StandardProcessGroup.java:4051)
> at 
> org.apache.nifi.groups.StandardProcessGroup.updateProcessGroup(StandardProcessGroup.java:3565)
> at 
> org.apache.nifi.groups.StandardProcessGroup.updateProcessGroup(StandardProcessGroup.java:3528)
> at 
> org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3335)
> at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.updateProcessGroupFlow(StandardProcessGroupDAO.java:273)
> at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at 
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
> at 
> org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:258)
> at sun.reflect.GeneratedMethodAccessor862.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
> at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
> at 
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
> at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$EnhancerBySpringCGLIB$$cc629156.updateProcessGroupFlow()
> at 
> org.apache.nifi.web.StandardNiFiServiceFacade$13.update(StandardNiFiServiceFacade.java:4382)
> at 
> org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:120)
> at 
> org.apache.nifi.web.StandardNiFiServiceFacade.updateProcessGroupContents(StandardNiFiServiceFacade.java:4378)
> at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at 
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
> at 
> org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithWriteLock(NiFiServiceFacadeLock.java:173)
> at 
> org.apache.nifi.web.NiFiServiceFacadeLock.updateLock(NiFiServiceFacadeLock.java:66)
> at sun.reflect.GeneratedMethodAccessor557.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at 

RE: merging with default values?

[Russ Weiher]

Do you know the name of the attributes you are looking for ahead of time?  If 
so, the UpdateAttribute processor has an “Advanced” section that lets you write 
conditional rules based on flowfile attributes and properties.  If you haven’t 
already checked that out, it might be a good place to start.  There are some 
good examples in the NiFi documentation for that processor.

Russ Weiher
BI CI Solutions Architect
Progenity, Inc.

From: l vic [mailto:lvic4...@gmail.com]
Sent: Thursday, January 10, 2019 2:18 PM
To: users@nifi.apache.org
Subject: merging with default values?

I have to merge results of 3 different queries into one flowfile
The problem happens if some of the queries result in empty resultset. For 
example, suppose i have 3 incoming connections for MergeContent:
query1->flowfile1/attribute1, query2->flowfile2/attribute2, 
query3->flowfile3/attribute3
And query2 returns empty resultset. In this case attribute2 never shows up in 
the merged flowfile.
Can someone recommend merge solution where i would always have attribute2 in 
the result but with some default value ( empty string assuming attribute2 is 
text)?
Thank you...
IMPORTANT: This email (and any attachments) is intended for the use of only the 
person or entity to which it is addressed, and may contain information that is 
privileged and confidential. You, the recipient, are obligated to maintain it 
in a safe, secure and confidential manner. Unauthorized redisclosure or failure 
to maintain confidentiality may subject you to federal and state penalties. If 
you are not the intended recipient, please immediately notify us by return 
email, and delete this message from your computer.

merging with default values?

[l vic]

I have to merge results of 3 different queries into one flowfile
The problem happens if some of the queries result in empty resultset. For
example, suppose i have 3 incoming connections for MergeContent:
query1->flowfile1/attribute1, query2->flowfile2/attribute2,
query3->flowfile3/attribute3
And query2 returns empty resultset. In this case attribute2 never shows up
in the merged flowfile.
Can someone recommend merge solution where i would always have attribute2
in the result but with some default value ( empty string assuming
attribute2 is text)?
Thank you...

RE: Java exception when attempting to update version from registry

[Russ Weiher]

Kevin -

Here is the full stack trace from the log:

2019-01-10 10:10:55,085 ERROR [Version Control Update Thread-1] 
org.apache.nifi.web.api.VersionsResource Failed to update flow to new version
java.lang.IllegalStateException: The input port name or identifier is not 
available to be added.
at 
org.apache.nifi.groups.StandardProcessGroup.addInputPort(StandardProcessGroup.java:505)
at 
org.apache.nifi.groups.StandardProcessGroup.addInputPort(StandardProcessGroup.java:4051)
at 
org.apache.nifi.groups.StandardProcessGroup.updateProcessGroup(StandardProcessGroup.java:3565)
at 
org.apache.nifi.groups.StandardProcessGroup.updateProcessGroup(StandardProcessGroup.java:3528)
at 
org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3335)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.updateProcessGroupFlow(StandardProcessGroupDAO.java:273)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at 
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at 
org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:258)
at sun.reflect.GeneratedMethodAccessor862.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at 
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at 
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$EnhancerBySpringCGLIB$$cc629156.updateProcessGroupFlow()
at 
org.apache.nifi.web.StandardNiFiServiceFacade$13.update(StandardNiFiServiceFacade.java:4382)
at 
org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:120)
at 
org.apache.nifi.web.StandardNiFiServiceFacade.updateProcessGroupContents(StandardNiFiServiceFacade.java:4378)
at 
org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at 
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at 
org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithWriteLock(NiFiServiceFacadeLock.java:173)
at 
org.apache.nifi.web.NiFiServiceFacadeLock.updateLock(NiFiServiceFacadeLock.java:66)
at sun.reflect.GeneratedMethodAccessor557.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at 
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at 
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at 
org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$f3a45649.updateProcessGroupContents()
at 
org.apache.nifi.web.api.VersionsResource.updateFlowVersion(VersionsResource.java:1473)
at 
org.apache.nifi.web.api.VersionsResource.lambda$null$19(VersionsResource.java:1139)
at 
org.apache.nifi.web.api.concurrent.AsyncRequestManager$2.run(AsyncRequestManager.java:109)
at 

RE: Java exception when attempting to update version from registry

[Russ Weiher]

Kevin -

(sorry for the delay, still waiting for someone from the DevOps team to pull 
that log file)

We are currently using Registry version 0.2.0.

In our setup we have DEV, QA, and PROD Nifi instances all pointed to the same 
instance of the registry.

The sequence of steps I used was:

1. Finish making changes to the Process Group in DEV.
2. Right click -> Version - > Commit local changes
3. Wait for check mark to turn green in the bread crumbs at the bottom of the 
screen.
4. Open QA Nifi interface
5. Navigate to the Process Group in question and open it
6. Right click -> Version - > Change Version
7. Select the new version I saved from dev

This particular change version is number 32, and I've only had one other issue 
when moving updates.

Getting a new instance of Nifi running 1.8.0 up and running to test with might 
be possible, but it's a request that has to go through infrastructure to get a 
VM spun up, and then devops to get the software installed, so it's 
unfortunately not something that I can do quickly.


Thanks,
Russ Weiher
Data Engineer
Progenity, Inc.


-Original Message-
From: Kevin Doran [mailto:kdoran.apa...@gmail.com]
Sent: Thursday, January 10, 2019 11:16 AM
To: users@nifi.apache.org; Russ Weiher 
Subject: Re: Java exception when attempting to update version from registry

Sorry, forgot to add -- can you include the full stack trace for the 
IllegalStateException as well? It should be in the nifi-app.log file.

Thanks,
Kevin


On January 10, 2019 at 11:13:33, Kevin Doran (kdo...@apache.org) wrote:
> Hi Russ,
>
> You mentioned you're using NiFi 1.6. What version of NiFi Registry are
> you running? Is the error message consistent?
>
> What is the sequence of steps you are using to move the flow snapshot
> from DEV to QA? Is it all through the NiFi UI (i.e., both environments
> are pointed to the same NiFi Registry instance, and you save a version
> from DEV and then change version in QA), or is there more to it than that?
>
> I find it interesting that change version is failing but delete and import is 
> working...
> That seems to indicate that the flow snapshot is being saved to
> registry correctly and completly, but there is some error in the logic
> of change version that is applying the changes in the wrong order.
>
> I know this may not be super helpful, but you may be running into a
> bug in the Change Version logic that has alreay been fixed in the
> latest version of NiFi. (To others on the mailing list, does that ring
> any bells?) If you're not able to test this on the NiFi 1.8 and Registry 0.3, 
> can you provide the steps to reproduce it so that I can try?
>
> Thanks,
> Kevin
>
>
> On January 10, 2019 at 10:57:19, Russ Weiher (russ.wei...@progenity.com) 
> wrote:
> > Hello -
> >
> >
> >
> > I'm trying to move changes from my DEV instance to the QA instance
> > (via the= Registry), but it's giving me the following error:
> >
> >
> >
> > "Failed to update flow to new version due to
> > java.lang.IllegalStateException: The input port name or identifier is not 
> > available to be added."
> >
> >
> >
> > I had a similar issue in the past where it wouldn't update to a new
> > version= , but due to
> deadlines
> > I ended up just deleting the QA version and re-impor= ting the flow from 
> > the registry.
> > However, that approach means that dozens o= f variables have to be
> > double checked and/or updated for the new environmen= t (QA and then again 
> > for PROD which gives the same error).
> >
> >
> >
> > If this is going to keep happening, I'd rather try and get to the
> > bottom of= it, but haven't been able to find other examples of
> > people having issues w= hen updating flow versions using the registry.
> >
> >
> >
> > Currently running Nifi version 1.6.
> >
> >
> > Russ Weiher
> > BI CI Solutions Architect
> > Progenity, Inc.
> >
> > 5230 S. State Road
> > Ann Arbor, MI 48108
> > https://na01.safelinks.protection.outlook.com/?url=www.progenity.com
> > data=01%7C01%7Cruss.weiher%40progenity.com%7Cee429294932c46e746
> > aa08d67716d9eb%7C0666349c4979449297792a34e68103b3%7C0sdata=gLZU
> > %2BH4ahYOuNSCcCKrkF4nOL39gTXhOXLl3KdQ9beM%3Dreserved=0
> >
> > (734) 794-9488 Office
> >
> > IMPORTANT: This email (and any attachments) is intended for the use
> > of only the person or entity to which it is addressed, and may
> > contain information that is privileged and confidential. You, the
> > recipient, are obligated to maintain it in a safe, secure and
> > confidential manner. Unauthorized redisclosure or failure to
> > maintain confidentiality may subject you to federal and state penalties. If 
> > you are not the intended recipient, please immediately notify us by return 
> > email, and delete this message from your computer.
> >
>
IMPORTANT: This email (and any attachments) is intended for the use of only the 
person or entity to which it is addressed, and may contain information that is 
privileged and confidential. You, the recipient, are obligated to maintain 

Re: Java exception when attempting to update version from registry

[Kevin Doran]

Sorry, forgot to add -- can you include the full stack trace for the
IllegalStateException as well? It should be in the nifi-app.log file.

Thanks,
Kevin


On January 10, 2019 at 11:13:33, Kevin Doran (kdo...@apache.org) wrote:
> Hi Russ,
>
> You mentioned you're using NiFi 1.6. What version of NiFi Registry are you 
> running? Is
> the error message consistent?
>
> What is the sequence of steps you are using to move the flow snapshot from 
> DEV to QA? Is it
> all through the NiFi UI (i.e., both environments are pointed to the same NiFi 
> Registry
> instance, and you save a version from DEV and then change version in QA), or 
> is there more
> to it than that?
>
> I find it interesting that change version is failing but delete and import is 
> working...
> That seems to indicate that the flow snapshot is being saved to registry 
> correctly and
> completly, but there is some error in the logic of change version that is 
> applying the
> changes in the wrong order.
>
> I know this may not be super helpful, but you may be running into a bug in 
> the Change Version
> logic that has alreay been fixed in the latest version of NiFi. (To others on 
> the mailing
> list, does that ring any bells?) If you're not able to test this on the NiFi 
> 1.8 and Registry
> 0.3, can you provide the steps to reproduce it so that I can try?
>
> Thanks,
> Kevin
>
>
> On January 10, 2019 at 10:57:19, Russ Weiher (russ.wei...@progenity.com) 
> wrote:
> > Hello -
> >
> >
> >
> > I'm trying to move changes from my DEV instance to the QA instance (via 
> > the= Registry),
> > but it's giving me the following error:
> >
> >
> >
> > "Failed to update flow to new version due to 
> > java.lang.IllegalStateException: The
> > input port name or identifier is not available to be added."
> >
> >
> >
> > I had a similar issue in the past where it wouldn't update to a new 
> > version= , but due to
> deadlines
> > I ended up just deleting the QA version and re-impor= ting the flow from 
> > the registry.
> > However, that approach means that dozens o= f variables have to be double 
> > checked and/or
> > updated for the new environmen= t (QA and then again for PROD which gives 
> > the same error).
> >
> >
> >
> > If this is going to keep happening, I'd rather try and get to the bottom 
> > of= it, but haven't
> > been able to find other examples of people having issues w= hen updating 
> > flow versions
> > using the registry.
> >
> >
> >
> > Currently running Nifi version 1.6.
> >
> >
> > Russ Weiher
> > BI CI Solutions Architect
> > Progenity, Inc.
> >
> > 5230 S. State Road
> > Ann Arbor, MI 48108
> > www.progenity.com
> >
> > (734) 794-9488 Office
> >
> > IMPORTANT: This email (and any attachments) is intended for the use of only 
> > the person
> > or entity to which it is addressed, and may contain information that is 
> > privileged and
> > confidential. You, the recipient, are obligated to maintain it in a safe, 
> > secure and
> > confidential manner. Unauthorized redisclosure or failure to maintain 
> > confidentiality
> > may subject you to federal and state penalties. If you are not the intended 
> > recipient,
> > please immediately notify us by return email, and delete this message from 
> > your computer.
> >
>

Re: Java exception when attempting to update version from registry

[Kevin Doran]

Hi Russ,

You mentioned you're using NiFi 1.6. What version of NiFi Registry are you 
running? Is the error message consistent?

What is the sequence of steps you are using to move the flow snapshot from DEV 
to QA? Is it all through the NiFi UI (i.e., both environments are pointed to 
the same NiFi Registry instance, and you save a version from DEV and then 
change version in QA), or is there more to it than that?

I find it interesting that change version is failing but delete and import is 
working... That seems to indicate that the flow snapshot is being saved to 
registry correctly and completly, but there is some error in the logic of 
change version that is applying the changes in the wrong order.

I know this may not be super helpful, but you may be running into a bug in the 
Change Version logic that has alreay been fixed in the latest version of NiFi. 
(To others on the mailing list, does that ring any bells?) If you're not able 
to test this on the NiFi 1.8 and Registry 0.3, can you provide the steps to 
reproduce it so that I can try?

Thanks,
Kevin


On January 10, 2019 at 10:57:19, Russ Weiher (russ.wei...@progenity.com) wrote:
> Hello -
>  
>  
>  
> I'm trying to move changes from my DEV instance to the QA instance (via the= 
> Registry),  
> but it's giving me the following error:
>  
>  
>  
> "Failed to update flow to new version due to java.lang.IllegalStateException: 
> The  
> input port name or identifier is not available to be added."
>  
>  
>  
> I had a similar issue in the past where it wouldn't update to a new version= 
> , but due to deadlines  
> I ended up just deleting the QA version and re-impor= ting the flow from the 
> registry.  
> However, that approach means that dozens o= f variables have to be double 
> checked and/or  
> updated for the new environmen= t (QA and then again for PROD which gives the 
> same error).  
>  
>  
>  
> If this is going to keep happening, I'd rather try and get to the bottom of= 
> it, but haven't  
> been able to find other examples of people having issues w= hen updating flow 
> versions  
> using the registry.
>  
>  
>  
> Currently running Nifi version 1.6.
>  
>  
> Russ Weiher
> BI CI Solutions Architect
> Progenity, Inc.
>  
> 5230 S. State Road
> Ann Arbor, MI 48108
> www.progenity.com
>  
> (734) 794-9488 Office
>  
> IMPORTANT: This email (and any attachments) is intended for the use of only 
> the person  
> or entity to which it is addressed, and may contain information that is 
> privileged and  
> confidential. You, the recipient, are obligated to maintain it in a safe, 
> secure and  
> confidential manner. Unauthorized redisclosure or failure to maintain 
> confidentiality  
> may subject you to federal and state penalties. If you are not the intended 
> recipient,  
> please immediately notify us by return email, and delete this message from 
> your computer.  
>  

Java exception when attempting to update version from registry

[Russ Weiher]

Hello -



I'm trying to move changes from my DEV instance to the QA instance (via the=  
Registry), but it's giving me the following error:



"Failed to update flow to new version due to java.lang.IllegalStateException: 
The input port name or identifier is not available to be added."



I had a similar issue in the past where it wouldn't update to a new version= , 
but due to deadlines I ended up just deleting the QA version and re-impor= ting 
the flow from the registry. However, that approach means that dozens o= f 
variables have to be double checked and/or updated for the new environmen= t 
(QA and then again for PROD which gives the same error).



If this is going to keep happening, I'd rather try and get to the bottom of=  
it, but haven't been able to find other examples of people having issues w= hen 
updating flow versions using the registry.



Currently running Nifi version 1.6.


Russ Weiher
BI CI Solutions Architect
Progenity, Inc.

5230 S. State Road
Ann Arbor, MI 48108
www.progenity.com

(734) 794-9488  Office

IMPORTANT: This email (and any attachments) is intended for the use of only the 
person or entity to which it is addressed, and may contain information that is 
privileged and confidential. You, the recipient, are obligated to maintain it 
in a safe, secure and confidential manner. Unauthorized redisclosure or failure 
to maintain confidentiality may subject you to federal and state penalties. If 
you are not the intended recipient, please immediately notify us by return 
email, and delete this message from your computer.