Hello everyone, I'm Caideyipi, a contributor to IoTDB, currently working on Streaming Engine. Here I wish to share a new feature I introduced to IoTDB "Show Pipes" SQL. Previously, the "Show Pipes" SQL contains 7 columns, like "ID|CreationTime|State|PipeSource|PipeProcessor|PipeSink|ExceptionMessage". By that, we can easily see the pipes' definitions, statuses, and exceptions if exist. However, the pipe's current progress is left unclear, and users must query the receiver or browse the dashboard to acquire the progress. In some typical scenarios, not only the dashboard is absent due to resource limitation, but the receiver is inaccessible for secrecy concerns. In that circumstance, the progress may not be easily known by users. Hence, two new columns, "RemainingEventCount" and "EstimatedRemainingSeconds", are introduced in show pipes result. They are also introduced to the dashboard. Detailed illustrations are provided below. - "RemainingEventCount" is that how many events are left unprocessed/untransferred in the pipe framework, consisting of: 1. The tsfile, insertion, raw tablet, heartbeat and schema events. 2. Events in the config region and all the schema regions and data regions. - "EstimatedRemainingSeconds" is the estimated elapsed seconds when the Streaming Engine has processed/transferred all the existing data. It's calculated by the "RemainingEventCount" of each region, and uses each region's average event commit rate in one minute in calculation. The commit rate will apply the "single exponential smoothing" method when updated to get more precise result, and the largest displayed time is 1 year. To avoid affecting the cluster's performance, the two values are updated in the pipe's heartbeat and may have some delay. As an improvement, in a cluster having only one dataNode, if the pipe comprising only data region and schema region, it's calculation is done at the dataNode and is refreshed on query. I heartily hope that you are interested in this feature. Any participation in testing, and any comments or suggestions in this thread are welcome.
Thanks for your reading and support. Sincerely, Caideyipi
