Hello all, I think this is a good compromise to get started and then publish it with a bit of fanfare (marketing) as an Apache project to the world once it's ready.
I am also interested in working with you on this project. I think PLC4X and IoTDB need something like this to become popular in the automation world. Regards Niklas On November 28, 2022, Julian Feinauer <[email protected]> wrote: > I fully agree with that. > If we get something to work in private we could easily go TLP or > through the Incubator whatever is decided then to do. > > Julian > > PS.: Do you setup the repo and invite myself? > > Von: Christofer Dutz <[email protected]> > Datum: Montag, 28. November 2022 um 13:47 > An: [email protected] <[email protected]> > Betreff: Re: Building a PLC4X and IoTDB Historian? > And if we already agree on a name (I think “Apache Historian” would be > a good name), > we could even already work on code with that package-name and maven > coordinates, so we don’t have to change much when coming to Apache? > > Chris > > From: Christofer Dutz <[email protected]> > Date: Monday, 28. November 2022 at 13:18 > To: [email protected] <[email protected]> > Subject: Re: Building a PLC4X and IoTDB Historian? > Hi Julian, > > Indeed, your concerns are valid regarding creating an already > abandoned TLP. > > In that case, I would opt for a private repo on GitHub which we invite > everyone willing to help work on it and as soon as we have something > workable, we make it public as part of a TLP? > > This avoids at least the fear of creating a new TLP that’s product > doesn’t even see the light of day. And as soon as we have something we > want to show the world, we join Apache? > > Would that be a compromise? > > > Chris > > From: Julian Feinauer <[email protected]> > Date: Monday, 28. November 2022 at 13:14 > To: [email protected] <[email protected]> > Subject: AW: Building a PLC4X and IoTDB Historian? > Hey, > > as I said.. we could just do it as part of one of the TLPs like PLC4X > or IoTDB easily or in a private repo as well, for sure. > I just want to avoid to create a TLP which is soon abandoned, that’s > my only concern. > Regarding all your comments I totally agree and am with you. > > Julian > > Von: Christofer Dutz <[email protected]> > Datum: Montag, 28. November 2022 um 13:07 > An: [email protected] <[email protected]> > Betreff: Re: Building a PLC4X and IoTDB Historian? > Hi Julian, > > well … I would be hesitant to try something like this without the > legal shield of the ASF … from what I’ve learned in the early days, > Siemens and Co. continuously threatened to sue me for my work on > PLC4X. > The only thing that helped me then, was to say: I am doing everything > legally, so you’d have trouble succeeding with that and you also can’t > sue me but would have to sue the ASF and good luck on that not > backfiring badly on you publicly. > > I know that in the sector of drivers in the past there were several > occasions where open-source developers were annoyed so much by > nuisance law-suites, that they gave up. > > I also know Historians are silly expensive systems. I haven’t seen one > under 100k€ and the annual fees for certified solutions tend to start > at 100k€/year/production-line … so there’s a lot of money for them to > fear losing. So yes: I’m too scared to try something like this > publicly without a legal shield. > > We could however work on something like this in a protected repo at > GitHub etc. and go to TLP as soon as we have something. > > Would that be an option? > > Chris > > > From: Julian Feinauer <[email protected]> > Date: Monday, 28. November 2022 at 12:57 > To: [email protected] <[email protected]> > Subject: AW: Building a PLC4X and IoTDB Historian? > Hey Chris, > > wrt testing I meant how „sustainable” the community is, that we intend > to building. > You know the effect pretty well if you ask “How wants to start this” > everybody is like “me, me, me” and after 2 weeks you stand there > alone. > And one important aspect of a TLP for me is that the community is also > “resilient” and will last “long” (whatever that means). > And this is something I think is hard to guarantee in the current > state. > > Technically I have no worries, this is something we would get to work > rather easily, I see no “magics” hidden there. > > Julian > > Von: Christofer Dutz <[email protected]> > Datum: Montag, 28. November 2022 um 12:48 > An: [email protected] <[email protected]> > Betreff: Re: Building a PLC4X and IoTDB Historian? > Hi Julian, > > Well, the problem with starting at one Project as a subproject … We > then need to make everyone involved or wanting to be involved in the > other project to become committers in the other. > I wouldn’t have problems with doing it under the hood of IoTDB, > personally. > > A separate Project would have the benefit of giving the industry an > impression that this is something built just for them instead of being > a side-product of something that’s already there (I have given up > expecting the Automation industry to use any form of common sense ;-) > … guess that will be my definition of Industry 5.0 … “connected > everything + common sense”) > > The incubator is intended for teaching the initial community how > things work at Apache, I guess as we’d be starting with mainly people > that have already brought projects from outside into the incubator and > into TLPs, this teaching effort will not exist in this case. Even > Justin mentioned that this would probably be a case for straight TLP. > > Regarding the testing … not sure I understand what you want to test? > The idea of an open-source historian? > I mean the need exists … I’ve seen it in multiple occasions and our > friends from IoTDB have confirmed they have built or at least seen > IoTDB serve as such a system. > Not 100% sure what you want to try out. > > But I agree … we should probably start with an (online) workshop, as > this time we’d probably be spread out a bit wider geographically, than > with our PLC4X workshops in the past. > > Chris > > > > From: Julian Feinauer <[email protected]> > Date: Monday, 28. November 2022 at 12:37 > To: [email protected] <[email protected]> > Subject: AW: Building a PLC4X and IoTDB Historian? > Hey, > > I understand your point and also agree generally. > But I personally dislike the straight to TLP route as this is > something I’d like to try out first. > And my point with the separate repo (or whatever) was just to make the > community ready for the Incubator (where a codebase and community > normally should exist). > > But the easiest way in any regard would be to start of as a subproject > in an existing PMC. This would only require to get another repo and we > could start (formally no need to even register it as a subproject). > And the repo thing could be done by you alone I think as Chair. > > Next steps would then be more of brainstorming and workshoping and > building something (whatever?). > I think the timing is quite good before Christmas because if its > intereting, I see myself contributing over the Holidays (better don’t > show this mail to my wife…). > > Best > Julian > > Von: Christofer Dutz <[email protected]> > Datum: Montag, 28. November 2022 um 09:27 > An: [email protected] <[email protected]> > Cc: Tim Mitsch <[email protected]>, [email protected] > <[email protected]> > Betreff: Re: Building a PLC4X and IoTDB Historian? > Hi Julian, > > happy you like the idea ;-) > > We would probably outclass the established ones in point of price ;-) > Right now, I think we should probably set up something that bundles > IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, > that brings the two together and then provides an API that is > equivalent to existing Historian products. > I think the only way we can score points here, is if switching to it > is as easy as possible and that only works if it’s sort of a drop-in > replacement. > > Currently also discussing with other board members and the IPMC chair > … he at least would not see us in the incubator. If there are enough > PMC members of other projects and Apache Members on board, we could > take the straight to TLP route. > > And I would really like to keep it at Apache and not set up something > outside. This for multiple reasons: > > * Protection (The market for the established products is huge and the > companies behind them are also huge with huge legal departments, I > would fear the same as I did with PLC4X, when I started it … don’t > want to be stuck in nuisance lawsuits, just filed for keeping us > occupied) > * The satement: we want Open-Source to be accepted by the industry > and for me “Apache” is the form of Open-Source that I admire most. If > it’s built up out of Apache stuff, it should be an Apache project (If > we establish this in a way, that the industry notices Apache as a > “vendor”, this will benefit many other projects) > > Chris > > > > From: Julian Feinauer <[email protected]> > Date: Monday, 28. November 2022 at 09:17 > To: [email protected] <[email protected]> > Cc: Tim Mitsch <[email protected]>, [email protected] > <[email protected]> > Subject: AW: Building a PLC4X and IoTDB Historian? > Hi all, > > first of all.. sorry for coming in so late. > I think this is a great idea and there really is a product market fit > for such a solution. > I think technology wise its not "that complicated" to build such a > system, I think many of the details that are necessary for a good > adoption of such a software is in the areas around like documentation, > marketing, more marketing, even more and even bolder marketing and > sales (how to sell an open source project??). > > But technology wise I really like the idea and I think with the > technology one has today it is very easy to "outclass" these > established systems e.g. in performance, efficiency and als user > friendliness. > The only thing which might be challenging for the future are things > like plugins or an extension system to allow users to customize their > installation. > > Regarding the "separate project" approach I am not 100% certain. > Personally, I would consider starting EITHER as a subproject in an > existing PMC (PLC4X?) or as a separate undertaking on GitHub or > somewhere else and not directly go to the Incubator or something. > Because I think we should really find out if there is enough developer > interest to build and sustain such a system. > > But I'm totally in for such a system either way! > > Julian > > PS.: Also forwarding the email to Björn and Tim who work(ed) a lot > with historians > > Von: Christofer Dutz <[email protected]> > Datum: Montag, 28. November 2022 um 08:56 > An: [email protected] <[email protected]> > Betreff: Re: Building a PLC4X and IoTDB Historian? > Hi Xiangdong, > > that’s perfect that some of you folks would be on board. I should > probably check, if there are enough PLC4X folks on board too … would > be a shame if it was just me ;-) > > And regarding the name … yeah … you are absolutely right: Historian > might really be the ideal choice … as it instantly explains what it is > … A Historian from the Apache Software Foundation and probably also no > bias towards any regional culture. > > So (Speaking to my fellow PLC4X folks … who would be on board with > this?) > > > Chris > > > > From: Xiangdong Huang <[email protected]> > Date: Monday, 28. November 2022 at 04:24 > To: [email protected] <[email protected]> > Subject: Re: Building a PLC4X and IoTDB Historian? > > I guess I would sort of also vote for a separate Project. Would > however only make sense if some people from both of our projects would > join in. > > Agree, I think the iotdb developers in Timecho can join. > > > And name-wise … one of the names of Historic Historians (Ideally a > Greek one) would make it into my top 10 ;-) … sort of like Hudson or > Jenkins are “famous” Buttler names ;-) > > If we consider famous names (well, there will be culture bias), I'd > like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as > "Records of the Grand Historian" is tooo famous (at least in China). > > BTW. naming "Apache Historian" directly may be another option. > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > > Christofer Dutz <[email protected]> 于2022年11月27日周日 22:56写道： > > > > How about Apache Ephorous? ;-) > > > > https://en.wikipedia.org/wiki/Ephorus > > Ephorus of Cyme > (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; > Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, > Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient > Greek<https://en.wikipedia.org/wiki/Ancient_Greece> > historian<https://en.wikipedia.org/wiki/Historian> known for his > universal history<https://en.wikipedia.org/wiki/Universal_history>. > > > > Chris > > > > From: Christofer Dutz <[email protected]> > > Date: Sunday, 27. November 2022 at 15:28 > > To: [email protected] <[email protected]> > > Subject: Re: Building a PLC4X and IoTDB Historian? > > Hi all, > > > > I guess I would sort of also vote for a separate Project. Would > however only make sense if some people from both of our projects would > join in. > > > > And name-wise … one of the names of Historic Historians (Ideally a > Greek one) would make it into my top 10 ;-) … sort of like Hudson or > Jenkins are “famous” Buttler names ;-) > > https://en.wikipedia.org/wiki/List_of_Greek_historiographers > > https://en.wikipedia.org/wiki/List_of_historians > > > > I think anything but a turnkey-ready solution will not be accepted > by anyone in the Automation industry. > > > > Chris > > > > > > > > > > From: Xiangdong Huang <[email protected]> > > Date: Sunday, 27. November 2022 at 14:05 > > To: [email protected] <[email protected]> > > Subject: Re: Building a PLC4X and IoTDB Historian? > > If we just provide a solution, or demonstration, then either is ok. > > If we want to provide an "one-box thing" (even without GUI), +1 for > a > > new project. > > > > > I don’t even think it would be bad if an entity like Timecho would > add enterprise offerings, because I know, that the industry won’t > “buy” something, if there’s no commercial support or anyone, they can > throw money at, even if it’s free (That might even make them more > skeptical). > > Yes, indeed. :D > > > > Best, > > ----------------------------------- > > Xiangdong Huang > > > > Ben Hutcheson <[email protected]> 于2022年11月27日周日 18:38写道： > > > > > > Hi, > > > > > > I think it's a great idea. > > > > > > I'd vote for having a separate project for something like this, > just so > > > it's clear that it is a historian and people can build it > separately. I > > > wouldn't want it hidden away in an obscure folder in the PLC4X > repo. > > > > > > Cake does go with Cafe. > > > > > > Ben > > > > > > > > > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz > <[email protected]> > > > wrote: > > > > > > > Hi all, > > > > > > > > Trying to respond to all (except the naming question ;-) ): > > > > > > > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we > could be > > > > collecting data on one node, writing that to TSFiles using the > IoTDB-Client > > > > (As Xaingdong mentioned). > > > > These could either be online and transfer data to the server > right away, > > > > or operate in an offline-like mode and periodically fetch data > in bursts > > > > from the client (Hope that’s correct, please correct me if I’m > wrong) > > > > > > > > @Xaiangdong: In general, PLC4X operates in an active mode, but > we also > > > > have a proposed and prototypically proven “passive-mode”. Here > the software > > > > doesn’t actively participate in the data acquisition directly, > but simply > > > > listens to the data stream and makes data available. We also > have something > > > > we call a “Data-Diode”, which technically makes interference > impossible as > > > > it lets ethernet packets flow in one direction, but nothing in > the other > > > > (Firewalls usually have the problem of letting packets pass in > both > > > > directions for established connections). We invented this, > because we knew > > > > we will never have validated and audited open-source software, > to a level > > > > that it would be certified for some of these use cases. > > > > > > > > So, either we can live with everything the SCADA system is > already > > > > requesting, or we add an active PLC4X node in the secure > network, that > > > > requests data, and simply ignores it, and a second – passive > mode – node > > > > sits outside the secure network to capture the information. > > > > > > > > I agree that this sort of thing needs to be a “product”. The > Automation > > > > Industry just doesn’t know how to work with frameworks. Ideally > a one-box > > > > thing. > > > > Right now, most tools I have seen don’t even need the > calculation or the > > > > visualization. This is usually done on another level. Important > would be > > > > that we could be somewhat API compatible with existing products. > Them > > > > usually having SQL or REST APIs, should make it relatively easy > to sort of > > > > produce frontends to our Apache Historian, that are API > compatible with > > > > some existing industry products, so they are replaceable. > > > > > > > > If we wanted to add a visualization layer or a data-curation > layer later > > > > on, we should have a chat with the NiFi or StreamPipes folks as > that’s what > > > > they already have. > > > > > > > > For now, I would be proposing to build something that uses PLC4X > for the > > > > data-acquisition, IoTDB for the storage and build a REST > frontend for this, > > > > that’s somewhat API compatible with one of the major established > products > > > > and to add more on a step-by-step basis. > > > > > > > > I don’t even think it would be bad if an entity like Timecho > would add > > > > enterprise offerings, because I know, that the industry won’t > “buy” > > > > something, if there’s no commercial support or anyone, they can > throw money > > > > at, even if it’s free (That might even make them more > skeptical). > > > > > > > > > > > > What do you all think? Does this make sense? If yes, where would > it make > > > > sense to start working on something like this? In the IoTDB > project? In the > > > > PLC4X project, in a separate (new) project? > > > > > > > > > > > > Chris > > > > > > > > > > > > From: Xiangdong Huang <[email protected]> > > > > Date: Sunday, 27. November 2022 at 06:02 > > > > To: [email protected] <[email protected]> > > > > Subject: Re: Building a PLC4X and IoTDB Historian? > > > > Hi all, > > > > > > > > It makes sense to use PLC4X+IoTDB to build a historian software. > > > > Actually, there are several users using IoTDB as their historian > > > > solution... > > > > > > > > According to my knowledge, a historian software contains several > features: > > > > > > > > - get data from OPC/modubs server and write to IoTDB > > > > * use PLC4x + IoTDB-client. (we call this data collector) > > > > > > > > - (optional) there is a single-way network gateway for security. > > > > * If the gateway is deployed between device and PLC4x program, > then > > > > we need to check whether plc4x supports that. > > > > * If the gateway is deployed between IoTDB-client and IoTDB, > then we > > > > need to check wheter IoTDB-client supports that. > > > > > > > > - Calculation Engine, e.g., we want to write C to IoTDB when we > > > > collect A and B from a device (for example, C=A+B/2 ) > > > > * IoTDB's trigger supports that but we need a Drag-and-drop > > > > programming GUI. (maybe integrating some open source projects, > like > > > > Red-node is a good idea) > > > > > > > > - get the latest data from IoTDB for visualization (usually > users need > > > > to draw a process flow diagram, and put the latest data on each > device > > > > icon on the diagram) > > > > * IoTDB supports getting latest data. but we need a new software > for > > > > drawing the diagram (I do not know if there is any open source > project > > > > for this. In our real applications, we ususally buy and > integrate some > > > > other commerical software) > > > > > > > > - send alert message if the real time data meets some rules > > > > * IoTDB trigger supports the rule. but currently we have no a > GUI. > > > > (IoTDB also supports Prometheus Alert Manager) > > > > > > > > - get the historical data > > > > * IoTDB supports that and grafana is good at visualization > > > > > > > > - last but not the least, if we provide all the features to > industrial > > > > users, all the GUIs should be integrated into one entrance. > > > > > > > > > > > > Best, > > > > ----------------------------------- > > > > Xiangdong Huang > > > > School of Software, Tsinghua University > > > > > > > > 黄向东 > > > > 清华大学 软件学院 > > > > > > > > Ryan Truran <[email protected]> 于2022年11月27日周日 08:18写道： > > > > > > > > > > Hi folks, long time lurker… > > > > > > > > > > A historian would be great. Does PLC4X queue data on the > machine it’s > > > > > running on already? > > > > > > > > > > A common pattern in historians is to queue data on the runtime > server > > > > until > > > > > data can be extracted to a sql server, preventing data loss, > and reducing > > > > > the need for redundancy. > > > > > > > > > > There are a ton of edge cases to watch out for which I can > elaborate on > > > > > further. > > > > > > > > > > I’m a former Industrial Controls Engineer turned software dev > and can > > > > help > > > > > out where needed. > > > > > > > > > > -Ryan > > > > > > > > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler > <[email protected]> > > > > > wrote: > > > > > > > > > > > Would you name it “Cake”? > > > > > > > > > > > > From: Christofer Dutz <[email protected]> > > > > > > <[email protected]> > > > > > > Reply: [email protected] <[email protected]> < > > > > [email protected]> > > > > > > Date: November 26, 2022 at 10:36:23 > > > > > > To: [email protected] <[email protected]> > <[email protected] > > > > >, > > > > > > [email protected] <[email protected]> > <[email protected]> > > > > > > Subject: Building a PLC4X and IoTDB Historian? > > > > > > > > > > > > Hi all, > > > > > > > > > > > > I am currently thinking, if it wouldn’t be a good idea to > build > > > > something > > > > > > like an open-source Historian based on PLC4X and IoTDB. > > > > > > > > > > > > For those of you, who don’t know what a Historian is. In the > > > > manufacturing > > > > > > industry they have these extremely expensive servers > (usually you buy > > > > them > > > > > > as a bundle of hard- and software). > > > > > > These servers are nothing else than a really crappy and > brutally > > > > expensive > > > > > > Database for storing Time-Series data. > > > > > > > > > > > > The thing is most commercial products are currently really > having > > > > trouble > > > > > > to keep up with the increasing amount of Data being sent. > > > > > > > > > > > > So instead of suggesting building an Historian at each > customer’s > > > > site, I > > > > > > thought: Perhaps a ready-to-use solution based on open- > source would be > > > > a > > > > > > good idea. > > > > > > Not 100% sure where I’d locate such an initiative, but I > would tend to > > > > see > > > > > > it more on the Database side. > > > > > > > > > > > > I think IoTDB would be the perfect storage system, all we > seem to need > > > > is > > > > > > some sort of rest-interface that matches the industry > standards for > > > > > > querying the information and on the other side something > like PLC4X to > > > > fill > > > > > > the database. > > > > > > > > > > > > Here some examples: > > > > > > > > > > > > > > > > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html > > > > > > > > > > https://cdn.logic- > control.com/docs/aveva/historian/HistorianRetrieval.pdf > > > > > > > > > > > > What do you folks think? > > > > > > > > > > > > > > > > > > Chris > > > > > > > > > >
