Hi Philipp, thanks for the feedback. An alternative would be a continuous web socket connection from the core, but this has the disadvantage that the browser could get overloaded with events from high-frequency stream. We could extend the long-polling interval so that the chance that events get lost is reduced.
I absolutely agree that the preview feature has not much value if any events get lost. Dominik On 2021/05/23 20:27:26, Philipp Zehnder <[email protected]> wrote: > Hi Dominik, > > This is a very cool feature and I already tried it. > I really like how fast it is and it definitely improves the development > experience and reduces the amount of (debug) dashboard sinks in a pipeline ;) > > While trying, I have encountered a problem. I had a pipeline with a filter, > which should only let an event through from time to time. > Unfortunately this was not displayed in the preview. When I started this with > a data lake sink, the correct values arrived. > I suspect the reason for this behavior is because we poll the values every > two seconds in the preview component. > Do you have any ideas how to fix this? Because as a user you then think that > the pipeline does not work as expected. > > Philipp > > > On 21. May 2021, at 12:24, Dominik Riemer <[email protected]> wrote: > > > > Hi all, > > > > > > > > after talking to some first-time users I discovered that our live dashboard > > is often used as some kind of “debugging feature” to better see if the > > created pipeline performs as expected. This leads to quite a confusing and > > time-consuming pipeline development approach (creating a pipeline, > > appending dashboard sinks everywhere, going to the dashboard, checking the > > output, going back to the pipeline editor to change the pipeline and so on). > > > > So I quickly implemented a “preview feature” that can be enabled in the > > pipeline editor and then shows the output for each pipeline element in form > > of a table below the element (data streams & data processors and limited to > > 1 event/sec to avoid web browser performance issues). > > > > In the background, a pipeline is started and stopped and we reuse the > > pipeline element live data fetcher that was already implemented for data > > streams. > > > > > > > > It would be great if you could check if the feature is working as expected > > or if any changes are needed or critical bugs occur, as the implementation > > of the feature seemed to be a bit too easy and I’ve expected it to be much > > more complicated 😉 > > > > > > > > Dominik > > > > > > > > > > > >
