Hi Julian, well as I told you, I got on board of the project shortly before IBM shifted their priorities. Unfortunately I didn't have the chance to dig into all the details before that happened and other dropped out. I wouldn't be expecting much input from the rest. So I guess we would have to dig through stuff like this on our own.
But I am really willing to do this as I don't know of a similar project and I really want to help revive Edgent again. Chris Am 22.09.18, 08:13 schrieb "Julian Feinauer" <j.feina...@pragmaticminds.de>: Hey all, just a short appendum. I profiled through the code some more and found out the following: * Edgent (ETIAO Runtime) creates (and deletes) one Thread for each scheduled invocation of a Polling Source * This thread is not only used to poll but also to do all subsequent processing of the next pipeline steps (Oplets) * This leads to creations of hundres of Threads / min which imposes significant pressure on GC (I looked at the SensorAnalyticsApplication where GC was triggered on my laptop each 30s and about 300MB in Thread Objets were collected (about 1000 instances) Furthermore this has the following consequences: * No backpressure can be used with the current setup slow sinks have to throw away messages(?) * No guarantee of order as sheduleAtFixedRate is used and the runnable that’s scheduled processes the complete pipeline for the fetched element, so if, e.g., some intense operations is only performed on some objects (or simple filtering) this could lead to races where elements that were pulled later enter the sink earlier I will try to provide a concrete code example where these situations are demonstrated. It would be interesting for me to get an idea why this implementation was chosen or what you think are the advantages of this approach. I think Edgent has some very nice features and can solve some of the common problems for "edge" analytics in an elegant way but I'm not that sure with the ETIAO Runtime for the reasons stated above. But perhaps I am also wrong with some of my conclusions as they are solely from profiling and debugging through the code so please correct me if I am. Best Julian Am 22.09.18, 10:07 schrieb "Julian Feinauer" <j.feina...@pragmaticminds.de>: Hi Edgenteers, after chris does not stop to convince me that Edgent is THE THING to use for IoT I started once again a deeper look into its internals. I do not yet fully understand the core of Edgent and its runtime or how it physically runs the users operation at the end. But what I observed is that it uses a pretty high number of threads and what’s way worse it consumes many threads, i.e., it starts and stops threads all the time (about 1 per second in several Demo Applications (Water / Sensor Analytics Application). And this is also not dependent on the Provider (Direct vs Development). Can anyone here give me some explanations on how the Threading Model of Edgents runtime is and why it is done that way as for me, it seems a bit wasteful with regards to resources and we want to be as low footprint in our applications as possible. Thank you! Julian