Joe,
Responses to your bullets: 1) I wasn't aware of the details of how provenance
works. After reading about it, it seems quite powerful and it may help for some
cases. I especially like the ability to replay files and track timing of
events. However, there are some problems I think I've experienced in regards to
provenance: we have thousands of flowfiles going through daily. We like to keep
the flowfiles from certain processors for several days, but if we keep them
from all processors we will quickly run out of disk space. When I first
upgraded to 0.3.0, it had provenance on by default (I think that was a bug,
since in 0.2.0 it was off by default) and I quickly and unexpectedly ran out of
disk space on my production server. Furthermore, there are many cases where we
store a collection of flowfile content in a directory and we grep looking for
certain strings to debug something - can you do something similar to a grep on
content through provenance? Can you select which processors you want to store
flow files for so not to fill up the disk with flowfile content from processors
we don't care as much about? If the answer to these questions could be yes,
then I think provenance would work for the kind of tracking I've done with
PutFile and LogAttributes.
2) For emphasizing the main parts of a flow, I think the ability to align
selected processors would help. I would select the main processors, line them
up horizontally, and maybe it would automatically move other processors above
or below. Also, maybe I didn't fully explain my processor minimization idea - I
would like an option on a processor to "show icon only" so instead of a full
processor box with all the stats and details, it would look something like a
Processor Group port looks - no stats, just text and some icons indicating
started/stopped/disabled. (It should still look differently than a port does so
no one gets confused).
Of course, I look forward to hearing what others have to say about these ideas!
-Elli
On Wednesday, January 20, 2016 4:49 PM, Joe Witt <joe.w...@gmail.com> wrote:
Elli,
Ok great feedback. I'll categorize these in the following ways (if
you disagree please share)
1) How best to handle log/debug sort of flows
One thing to consider is that with our provenance and content archival
it eliminated the need for many uses of
LogAttribute and for PutFile for these cases. Something to consider.
2) How best to de-emphasize 'special case' handling of flows or parts
of a flow which are necessary but not the primary logic thread
I 'see' the idea but not sure what a good user experience for it would
be. Anyone have visuals/UX concept in mind for this?
3) Modes of edit / Lock-out
Makes sense. This has been asked for before. Basically allow the
user to express that they want to go into an edit mode. How do others
feel?
Thanks
Joe
On Wed, Jan 20, 2016 at 4:00 PM, Elli Schwarz
<eliezer_schw...@yahoo.com.invalid> wrote:
> Joe,
> What I'm referring to as far as emphasizing "important" processors is that
> there are many places in my workflow where I do a PutFile for logging
> purposes, and I have one for success and a separate one for failure. So for
> many of my main processors, there are two connectors to PutFiles. This makes
> the workflow look very cluttered, and when doing a demo it is difficult to
> see the main path that flowfiles are taking without explanation, when I'd
> like it to be intuitive. In fact, there are some cases that I really want to
> add PutFiles for logging but since it will make the workflow look so
> cluttered I don't do it. Maybe this problem can be solved by an option to
> route all failures or even successes from some or all processors to do a
> PutFile to a specific folder? Maybe we can minimize some processor such as
> these PutFiles to a small icon instead of the whole box to save screen space?
> I often don't care about processor stats for these processors anyway, but
> maybe they'd be displayed on mouse hover.
>
> Furthermore, in some of my workflows I have my main tasks that deal with the
> actual processing of the incoming data, and other "side tasks" that I do with
> the data like collect special metrics, storing some metadata in a special
> database, sending status emails, etc. I handle this now with Processor
> Groups, and that helps, but I find it a bit unwieldy to create many processor
> groups that only have two or three processors in them (and then the processor
> group needs the input/output ports, further complicated an otherwise simple
> workflow). There are also cases where for some failures, I have a ControlRate
> processor and then retry the flowfile after a certain period of time - this
> is not a "main" part of the workflow, but it clutters it and it's not
> intuitive to see what's happening. I think I'd like to solve this problem by
> just being able to align selected processors that I consider more important,
> and having the others off to a side.
>
> As far as accidentally dragging processors is concerned, sometimes I intend
> to the pan screen and end up moving a processor or moving a label that I used
> to highlight a section of my flow. For demos, it would be nice to lock the
> entire page layout. Maybe this can be accomplished with a button on the top
> right corner to enable/disable layout changes or disable adding new
> processors, and only allow viewing processor properties - a sort of "read
> only" mode.
> Thanks for taking my feedback into consideration. I still find Nifi
> incredibly useful for handling my complicated workflows and appreciate your
> work in developing it!
> -Elli
>
>
>
>
>
> On Tuesday, January 19, 2016 5:03 PM, Joe Witt <joe.w...@gmail.com> wrote:
>
>
>
> Elli
>
> "it is sometimes too easy to mis-align processors by dragging them
> accidentally"
> Great point I must admit I too do that. Often in really important
> demos. I've gotten good at making jokes about it. Probably should
> have gotten good at submitting a JIRA :-)
>
> I'd like to understand more about your other idea for emphasizing
> processors which are more important. I can understand the idea I
> think but I'm worried about how we could make the user experience
> worth the effort for the person signaling the emphasis to be of use
> for the people consuming that detail.
>
> Thanks
> Joe
>
> On Tue, Jan 19, 2016 at 4:46 PM, Matt Burgess <mattyb...@gmail.com> wrote:
>> +1 for "snap to grid" feature
>>
>> Sent from my iPhone
>>
>>> On Jan 19, 2016, at 4:20 PM, dan bress <danbr...@gmail.com> wrote:
>>>
>>> Maybe not exactly "auto-layout" but I would back a notion of having the
>>> components snap to a coarser grain grid than what we currently have.
>>> Sometimes I care a lot about having everything line up in the graph
>>> horizontally and vertically, and it always takes a long time to achieve
>>> this.
>>>
>>> I could see this being achieved by snapping the component to the same spot
>>> horizontally as the component above it when you move it underneath another
>>> component. Or some magical "auto snap" button that does its best to align
>>> everything with its nearest neighbors.
>>>
>>>> On Tue, Jan 19, 2016 at 12:37 PM Ryan H <rhendrickson.w...@gmail.com>
>>>> wrote:
>>>>
>>>> I like your idea Rob, that would help with lining up relationships too
>>>> (straight lines).
>>>>
>>>> On Matt's note, I don't think there should be a "standard" either, although
>>>> best practices are always out there.
>>>>
>>>> On Matt's note of putting failures up above processes, we do that too.
>>>> Totally depends on who made the flow first. Sometimes, people don't even
>>>> follow a convention in the same flow.xml file.
>>>>
>>>> For these reasons, I'd recommend alternate views to the flow.
>>>>
>>>> We have a couple projects that just allow you to rearrange a node-based
>>>> graph, based on your preference, hierarchy, circular, pyramid, etc.
>>>>
>>>> Applying this to NiFi, having a couple different default auto-layout
>>>> options that you can swap your current view to, but NOT change the original
>>>> flow, would be nice.
>>>>
>>>> It would let you walk into someone else's, potentially large, dataflow and
>>>> have a familiar way to view the flow.
>>>>
>>>> Ryan
>>>>
>>>>
>>>>> On Tue, Jan 19, 2016 at 2:03 PM, Rob Moran <rmo...@gmail.com> wrote:
>>>>>
>>>>> I agree with Matt's points. I was just replying with something similar
>>>>> basically saying I think trying to set a standard would not be
>>>>> well-received.
>>>>>
>>>>> I believe what could be more useful are layout tools that would help
>>>> users
>>>>> place components to help achieve their preferred layouts. For example,
>>>> the
>>>>> ability to align (left, right, center) components
>>>>> or horizontally/vertically distribute components evenly. Other features
>>>>> such as snap-to and/or smart-guides could make it easier for users to
>>>>> follow their organization's best practices when designing a flow.
>>>>>
>>>>> Rob
>>>>>
>>>>> On Tue, Jan 19, 2016 at 1:49 PM, Matthew Clarke <
>>>> matt.clarke....@gmail.com
>>>>> wrote:
>>>>>
>>>>>> Ryan,
>>>>>>
>>>>>> Setting a standard is a difficult thing to do. The
>>>> complexity
>>>>>> that can exist in many flows would make enforcing a standard difficult.
>>>>> The
>>>>>> first example you provide of success to points right while failures
>>>> point
>>>>>> up is not recommended. It would be better to have failures point down
>>>>> since
>>>>>> it is common to put labels over processor(s). Any relationships
>>>> pointing
>>>>> up
>>>>>> would pass through these labels making both the relationship box and
>>>> the
>>>>>> label hard to read. It is often coomon to see flows designed with a
>>>>>> combination of left to right and top to bottom design.
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> On Tue, Jan 19, 2016 at 12:07 PM, Ryan H <rhendrickson.w...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Rob,
>>>>>>> Yea we did, it was at the end of the meeting.
>>>>>>>
>>>>>>> I think it would be useful to have a couple default type views
>>>> that
>>>>>>> help standardize flow layout across the community.
>>>>>>>
>>>>>>> For example, when we organize processors left-to-right, failure
>>>>>>> relationships always point up, and success always point right.
>>>>>>> Alternatively, when we organize processors up-and-down, failure
>>>>>>> relationships always point left, and successes always point down.
>>>>>>>
>>>>>>> Of course, in some of these scenarios there are processors that
>>>>> have
>>>>>>> more than 1 success relationship, but that's when a good layout
>>>> library
>>>>>>> would come into play.
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>> On Tue, Jan 19, 2016 at 11:10 AM, Rob Moran <rmo...@gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>>> Ryan - I think we spoke briefly (at a very high level) about this
>>>> at
>>>>> a
>>>>>>>> prior meetup. What alternate views did you have in mind, and in
>>>> what
>>>>>> ways
>>>>>>>> do you think they'd be useful?
>>>>>>>>
>>>>>>>> Rob
>>>>>>>>
>>>>>>>> On Tue, Jan 19, 2016 at 10:56 AM, Ryan H <
>>>>> rhendrickson.w...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> It'd be pretty awesome if NiFi provided the ability to
>>>>> auto-organize
>>>>>> a
>>>>>>>>> layout.
>>>>>>>>>
>>>>>>>>> Maybe even just a auto-organized layout that doesn't change the
>>>>>>> flow.xml,
>>>>>>>>> just an alternate view.
>>>>>>>>>
>>>>>>>>> Looking at these demos here: http://js.cytoscape.org/#demos
>>>>>>>>>
>>>>>>>>> Ryan
>>>>
>
>
>
>
