Hi Florian, thanks for sorting the adapters. Maybe we can extend the category for each type (e.g. for all message brokers I would add ‘Generic’).
The list looks good so far, I guess the mentioned adapters were removed. Let’s use the adapters in the git repository as our source of truth, I guess the website can be outdated. Philipp Von: Florian Micklich <[email protected]> Datum: Mittwoch, 14. September 2022 um 23:12 An: [email protected] <[email protected]> Betreff: Re: [DISCUSS] Documentation - Description for Adapters and Processing Elements Hi all, I sorted the Adapters a little bit, to start the documentation The following adapters are mentioned on the website, but are not included in my installation. Are they outdated or somehow lost? MySQL Set Adapter MySql Stream Adapter NSW Traffic Cameras The other adapters are divided into 2 groups. Is something missing or does it need to be corrected in some way? Some adapters do not belong to any category. connect-adapters-iiot NAME | TYPE | CATEGORIES Apache Kafka | Data Stream Apache Pulsar | Data Stream File Set | Data Set File Stream | Data Stream HTTP Server | Data Stream HTTP Set | Data Set HTTP Stream | Data Stream InfluxDB Set Adapter | Data Set InfluxDB Stream Adapter | Data Stream Machine Data Simulator | Data Stream | Testing & Debugging MQTT | Data Stream OPC UA | Data Stream | Generic Adapters, Production & Manufactoring PLC4X MODBUS | Data Stream | Production & Manufactoring PLC4X S7 | Data Stream | Production & Manufactoring ROS Bridge | Data Stream | Production & Manufactoring connect-adapters NAME | TYPE | CATEGORIES Coindesk Bitcoin Price | Data Stream | Finance Flic MQTT | Data Stream | Energy GDELT | Data Stream IEX Cloud News | Data Stream | Finance, News IEX Cloud Stock Quotes | Data Stream | Finance Image Upload (Set) | Data Set Image Upload (Stream) | Data Stream ISS Location | Data Stream | Open Data NETIO http JSON | Data Stream | Energy NETIO MQTT M2M | Data Stream | Energy Random Data Simulator (Set) | Data Set | Testing & Debugging Random Data Simulator (Stream) | Data Stream | Testing & Debugging Slack | Data Stream | Social Media TI Sensor Tag | Data Stream | Open Data Wikipedia Edits | Data Stream | Social Media, Open Data Wikipedia New Articles | Data Stream | Social Media, Open Data Greetings Florian Am Mittwoch, dem 14.09.2022 um 22:52 +0200 schrieb Florian Micklich: > Hi, > just recognized that the format of my list was totally unreadable > inthe emailI added my list in a propper way here [1] > Florian > [1] https://issues.apache.org/jira/browse/STREAMPIPES-581 > > Am Mittwoch, dem 14.09.2022 um 16:52 +0200 schrieb Florian Micklich: > > Hi all,I made a rouge structure about a possible > > documentation. • Introduction: ◦ Overview ◦ Basic > > Elements (formerStreamPipes Concept) ◦ Time in StreamPipes > > (Why timestamp isimportant in the data?) ◦ Semantic in > > StreamPipes ◦OtherTopics? ◦ Installation > > ▪ BasicInstallation ▪ Docker Deployment > > ▪ K8s/K3s ▪ Use SSL ▪ > > Security ◦GettingHelp • Overview (Basic GUI Explanation > > and Introduction for eachGUIpoint and why it is implemented (short > > description text) for exampleDashboard why Dashboard, what you can > > do. Basically little more deepdiver than Basic Elements section. > > Section with Most of theScreenshots, So easy to maintain due > > changes) ◦ Login Page andBasics ◦ Welcome > > Page ◦ Pipelines ◦Connect ◦ Dashboard ◦ > > Data Explorer ◦ Apps(deprecated?) ◦ Install Pipeline > > Elements ◦ AssetManagement ◦ File Management ◦ > > Configuration & UserPrefences ◦ Notification • Data > > Source Elements(Adapter) ◦ Why Adapter. Connect to the Real > > World ◦WhatKind of Sources (Data Set vs Data Stream / > > Categories) ◦ BasicAdapter Setup in 4 Steps (at the > > moment) ▪ Step 1Settings ▪ Step 2 > > Select Format •JSONwith Description for Array > > Field / GeoJSON / Single Object / ArraywithShort Data > > Examples • XML •CSV > > • Image ▪ Step 3 > > configurefields • Different > > between Measurement,DimensionHeader • Add > > Nested / Add static Value / AddTimestamp • Edit > > fields (Big Thinks.Also) • Preview > > ▪ Step4 startadapters • Settings / > > Description (why Importantand where can I find this info later > > on?) •Remove Duplicates / Reduce event rates > > /Persist events ◦ConnectAdapters Overview > > ▪ Apache Kafka with ExampleSources ▪ Apache > > Pulsar ▪ FileSet ▪ File > > Stream ▪ ... ◦ConnectAdapters IIOT > > Overview ▪ ... ◦ How ToAdapter ▪ > > Setup Adapter Example 1 ▪SetupHTTPS Adapter ◦ > > Technical Concept of Adapters (DevPart???) •Data Processors • > > Data Sinks • Create Pipeline •Dashboard • AssetsThe > > working example will be in the section HowTo and some > > exampleresources in each adapter description.What do you > > think?First step would be to check if it possible to move the files > > so thatthe structure is not destroyed and the simply start with the > > adapters:)I made already a rebase between dev and STREAMPIPES-581 > > branch, duesome changes in the dev branch.FlorianAm Montag, dem > > 29.08.2022 um 01:07 +0000 schrieb Philipp Zehnder: > > > Hi Florian,I am not sure if we will find many available public > > > data sources,especially for PLCs. But maybe we find some docker > > > containers thatcan be used for the tutorial. If we do not find > > > some, maybe we canprovide them in our registry.We could also use > > > them for theautomatedtests. That would be great, because > > > currently those tests are notintegrated in the nightly run.I > > > think for most users it is already helpful to have a > > > workingexample for the different adapters, so they do not > > > necessarily haveto try out the examples.As you said we should > > > collect all the information somewhere. Shouldwe use thwiki for > > > that?Cheers,Philipp________________________________Von: Florian > > > Micklich<[email protected] > > > > Gesendet: Samstag, August 27, 2022 1:20 AMAn: > > > [email protected] > > > <[email protected] > > > > Betreff: Re: > > > [DISCUSS] Documentation - Description for Adapters and > > > ProcessingElementsHi,perfekt, I will create this branch. I will > > > think about apossiblestructure over the weekend. I let you > > > know.How are youtesting and using the adapters and wich one are > > > you usingmost? Doyouhave some URL sources you always for > > > developing processor testingand what is the typical workflow to > > > add them?Maybe we can start tocollect them in some way so I can > > > add them later inthedocumentation.GreetingsFlorianAm Donnerstag, > > > dem 25.08.2022 um 19:38 +0000 schrieb DominikRiemer: > > > > Sounds good!We usually use only the issue name as > > > > branchname(STREAMPIPES-581) without the "feature" prefix, but > > > > both istotallyfine!Dominik-----Original Message-----From: > > > > Florian Micklich <[email protected] > > > > > > > > > Sent: Wednesday, August 24, 2022 9:44 PMTo: > > > > > > > > [email protected] > > > > > > > > Subject: Re: [DISCUSS] Documentation - Description for > > > > AdaptersandProcessing ElementsHi Dominik,thanks for the answer > > > > and beforeIask more questions,I'll simply start :-)Therefore I > > > > created aticket > > > > https://issues.apache.org/jira/browse/STREAMPIPES-581 > > > > > > > > I will also create a feature branch feature/STREAMPIPES-581 > > > > orwhatis here best practise naming convention?GreetingFlorian > > > > Am Mittwoch, dem 24.08.2022 um 19:07 +0000 schrieb > > > > DominikRiemer: > > > > > Hi Florian,happy to help! 1. There is a header in > > > > > eachmarkdownfilewith an ID (e.g. in the 01_try- > > > > > installation.md).Thisidcorrespondsto the id in the > > > > > sidebar.json2. Yes, all imagesareinthe staticfolder 3. Well, > > > > > they are not really auto-autogenerated😉 There is amaven > > > > > plugin which extracts the markdownfiles ande.g., > > > > > createsentries for the sidebar.json and modifiesthe > > > > > imagelinks so thatthey work on the website, but it's > > > > > notsomethingautomated in Githubactions or so, although it > > > > > would beawesome tosupport this4. Youdon't need to manually > > > > > add theversion, this ismanaged byDocusaurus. The markdown > > > > > files in thedocumentationfolder alwaysrefer to the next > > > > > unreleased version.When you startthe website, youcan click on > > > > > the version (0.69.0should be thedefault) and thenselect > > > > > "master" which points youtothe unreleasedversion. Aftereach > > > > > release, we run the "ds-version" command andthen > > > > > Docusauruschecks all files in the documentationfolder > > > > > againstthe files fromthe previous release and creates anew > > > > > docs version.E.g., see olderdocumentation files at [1]. > > > > > Incase you want tobackport changes tothe documentation pages > > > > > toprevious versions,these files can also bemodified, but > > > > > that'srather rare.5. Goodpoint, I _think_ this shouldbe > > > > > possiblewithout any problems, atleast it's worth a try 😉Hope > > > > > thishelps!CheersDominik[1] > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/versioned_docs > > > > > > > > > > > > > > > > > > > > -----Original Message-----From: Florian Micklich < > > > > > [email protected] > > > > > > > > > > > Sent: Wednesday, August 24, 2022 7:59 PMTo: > > > > > > > > > > [email protected] > > > > > > > > > > Subject: Re: [DISCUSS] Documentation - Description > > > > > forAdaptersandProcessing ElementsHi,ok, I started docusaurus > > > > > fromsourceDominik has previouslymentioned and did a build & > > > > > runwebsite likementioned in theReadme.Next time I am > > > > > even faster,because I willstart the "BuildDocumentation" > > > > > from thedocumentation/websitefolder, as it itclearly > > > > > mentioned in theReadme :-D Before I start,I am trying > > > > > tounderstand thestructureof this documentation and Ihave > > > > > somequestions.1.So everythingisorganised by thesidebar.jsonSo > > > > > the try-installation [1] islinkedto the 01_try- > > > > > installation.md file[2]. How is this working ifthe names > > > > > doesnot match?2. I see that all images are stored inthe > > > > > static websitepath here[3]. Is this correct?3. Do Iunderstand > > > > > it correctly thatall filesin the PE folder [4] areauto > > > > > generated and will beautogenerated next time again? SoifI > > > > > change something here,will thisbe overwritten the > > > > > nexttime?4.Is it OK if I add the 0.70version inthe version > > > > > filehere[5]?5. Is it OK to structure thedocuments insome more > > > > > sup-folders? We could discuss the sub-foldernames here > > > > > ofcourse butIthink that would make the documentationlittle > > > > > biteasier tonavigate for the authors. This includes > > > > > alsopictures instep2.Greetings Florian[1] > > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json#L5[2] > > > > > > > > > > > > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/docs/01_try-installation.md[3]<https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/docs/01_try-installation.md%5b3%5d> > > > > > > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/static/img[4]<https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/static/img%5b4%5d> > > > > > > > > > > https://github.com > > > > > > > > > > /apache/incubator-streampipes- > > > > > website/tree/dev/documentation/docs/pe[5] > > > > > https://github.com/apache/inc > > > > > ubator-streampipes- > > > > > website/blob/dev/documentation/website/versions.jsonAm > > > > > Mittwoch, dem 24.08.2022 um 12:32 +0000 schrieb > > > > > PhilippZehnder: > > > > > > Hi together,it would be great to have more > > > > > > contentinthedocumentation.I am alsoin favor of using > > > > > > docusaurus.Itusesmarkdown, which leaves us thepossibility > > > > > > to move toanothertoolingat some point in the future ifwe > > > > > > want.@Florianpleasewrite if yousee something that should be > > > > > > updatedorchanged > > > > > > inthecurrentversion.Cheers,Philipp_________________________ > > > > > > _______Von:DominikRiemer <[email protected] > > > > > > > Gesendet: Mittwoch,August > > > > > > 24,202211:47 AMAn: [email protected] > > > > > > > > > > > > <[email protected] > > > > > > > Betreff: RE: [DISCUSS] > > > > > > Documentation-Description for Adapters > > > > > > andProcessingElementsHiFlorian,Cool!Thedocumentation on the > > > > > > website is based onDocusauruswhichsupportsthings like docs > > > > > > versioning.Docusaurususesmarkdown files,so werely on > > > > > > markdown for thedocumentation.Thereis a 'website' folderat > > > > > > [1] where you canfind the npm scriptstostart docusaurus and > > > > > > a'docs' folderwhichincludesthedocumentation markdown files > > > > > > forthe next releaseversion(selectg"next" in the docs > > > > > > website). Aftera newrelease,we run a"npm ds-version" which > > > > > > then detects changesin themarkdown filesand createsa new > > > > > > release docsversion.Thedocuments in the "docs"folder can be > > > > > > freelymodifiedandadded.That's a good place wherewe can add > > > > > > new tutorials.The"pe"foldercontains thedocumentation of > > > > > > pipeline elements.Thesearecurrentlygeneratedfrom the > > > > > > StreamPipes source code usingthestreampipes-maven-plugin > > > > > > (as some rewriting is needed,e.g.,to fixthe pathsofimages). > > > > > > This process is far from perfectandneedssome manualwork.It > > > > > > would be nice to have some > > > > > > Githubactionsscriptwhichautomatically updates the > > > > > > Documentation,butthat's abitmoreadvanced task I'd say 😉If > > > > > > you want to makechanges tothestructure, this can be done > > > > > > inthe"sidebars.json"file [3],whichis also auto-versioned by > > > > > > Docusaurus.[1] > > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website > > > > > > > > > > > > [2] > > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/docs > > > > > > > > > > > > [3] > > > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json > > > > > > > > > > > > > > > > > > -----Original Message-----From: Florian Micklich < > > > > > > [email protected] > > > > > > > > > > > > > Sent: Wednesday, August 24, 2022 10:13 AMTo: > > > > > > > > > > > > [email protected] > > > > > > > > > > > > Subject: Re: [DISCUSS] Documentation - > > > > > > DescriptionforAdaptersandProcessing ElementsHello,I would > > > > > > also prefer ashortIn-Appdocumentation and > > > > > > largerdocumentation "outside".Personally, Iprefer to read > > > > > > pdf documentsso I can takeadditionalnotes. ButHTML is also > > > > > > good for a quickonlinesearch.We must keepin mindwhat kind > > > > > > of documentation wewanttocreate and should notmixthem. > > > > > > Also, cross-references betweentheindividual"chapters"should > > > > > > be avoided. This makes > > > > > > iteasierifsomethingshouldchange. 1. Quick > > > > > > starterdocumentation. 2.SetupDocumentation 3.TechnicalD > > > > > > ocumentation 4. Usermanualdocumentation with > > > > > > bestpractiseIthink we should hit theroad andtake a look > > > > > > along the way :)Iknow the in appdocumentation is inmarkdown > > > > > > so far?Iwouldrecommend to write thedocumentation > > > > > > inasciidoc [1] [2]This couldbe set up withasciidoctorj [3]. > > > > > > Thismakes it easytoincludeeverything in Gitand also keep > > > > > > thedocumentationalongside thesource code. So ifsomething > > > > > > isrewritten, thedeveloper can alsochange thedocumentation. > > > > > > Makesnewscreenshotsand deletes oldones.3] Thecreation of a > > > > > > PDF orHTML file canbedone withGithubAction. With every > > > > > > commitor with everyrelease. Theresults can bepushed to the > > > > > > StreamPipeswebsite, Iguess?What doyouthink?Florian[1] > > > > > > https://docs.asciidoctor.org/asciidoc/latest/[2]<https://docs.asciidoctor.org/asciidoc/latest/%5b2%5d> > > > > > > > > > > > > German Source > > > > > > https://blog.ordix.de/docs-as-code-dokumentation-mit-asciidoctor[3]<https://blog.ordix.de/docs-as-code-dokumentation-mit-asciidoctor%5b3%5d> > > > > > > > > > > > > https://github.com/asciidoctor/asciidoctorjAmMittwoch,dem24.08.2022 > > > > > > um 07:07 +0000 schrieb Dominik > > > > > > Riemer: > > > > > > > Hi,yes, we can do both: In-app > > > > > > > documentationwhichonlycoverstheconfiguration and a short > > > > > > > description oftheadapterandpipelineelement and additional > > > > > > > step-by-stepguides/tutorialswithnicescreenshots that are > > > > > > > availableintheonlinedocumentation. Wecanhave such step- > > > > > > > by-step guidesfor themostfrequently used > > > > > > > adapters.@Florian, do you wanttowork onsuch aguide on > > > > > > > thewebsite? I knowyou had some > > > > > > > reallygooddocumentationideas somewhile ago > > > > > > > 😉CheersDominik-----OriginalMessage-----From: Philipp > > > > > > > Zehnder <[email protected] > > > > > > > > Sent:Sunday, August > > > > > > > 21,20221:07PMTo: [email protected] > > > > > > > > > > > > > > Subject: [DISCUSS] Documentation > > > > > > > -DescriptionforAdaptersandProcessing Elements Hi all, > > > > > > > hereisa new threadtodiscusstheadapter documentation.Here > > > > > > > is alinkto thepreviousdiscussionin therelease thread > > > > > > > [1].@Dominik Iknow thattheregistration isdifferent, but > > > > > > > since alladaptersalready havethethree > > > > > > > files(documentation.md, > > > > > > > icon.png,andstring.en),shouldn’tit bepossible to also > > > > > > > display thedescriptionin theUI.How are wedoingcurrently > > > > > > > for theicons?Because they areloaded dynamicallyaswell. I > > > > > > > think themaindifference is thatthe icons > > > > > > > areloadeddirectly from theworkercontainer and notfrom > > > > > > > thebackend.Couldn’t we do itsimilarly forthedocumentation > > > > > > > file?(Pleasecorrect me if Iamwrong)Regardingthe > > > > > > > description on theWebsite.I think itwouldbegreat if we > > > > > > > couldgenerate this basedon themarkdown filesforeach > > > > > > > adapter.The documentation.md > > > > > > > filesmostlydescribetheconfigurationparameters of adapters > > > > > > > andprocessors.Ithinkitwould be great tohave additional > > > > > > > exampleshow to usethem.(e.g.describe how a CSVfile is > > > > > > > uploaded or how aPLCisconnected).Should we include > > > > > > > suchexamplesintothedocumentation.md file orshould there > > > > > > > beanotherlocation forthis?Any thoughts onthat?@Florian > > > > > > > youwrote that you would liketo workon thedocumentation.If > > > > > > > youneed anything to start > > > > > > > with,pleaseletusknow.Cheers,Philipp[1] > > > > > > > https://lists.apache.org/thread/29w586db1btqo0ps1rgo3fscvt3tvrzg > > > > > > > The other adapters are divided into 2 groups.
