Hi, I have already reported the issue on Jira (https://issues.apache.org/jira/projects/STREAMPIPES/issues/STREAMPIPES-321?filter=allopenissues). Please give it a read before continuing.
I am currently implementing a service management layer on-top of StreamPipes and Flink for better controllability over the individual pieces of the pipelines. This issue is a blocking point for me right now. So I am trying to figure out a way around. As the code is inside the streampipes scope, so we need to change it there to fix the issue. Right now, I see two approaches: 1. Put the line 141 (https://github.com/apache/incubator-streampipes/blob/2f10a17b77a93eee48527e4e88baf[…]streampipes/container/api/InvocablePipelineElementResource.java<https://github.com/apache/incubator-streampipes/blob/2f10a17b77a93eee48527e4e88baf7e4b144d6fe/streampipes-container/src/main/java/org/apache/streampipes/container/api/InvocablePipelineElementResource.java#L141>) inside a try/catch block, and execute line 144 in any case. 1. Or, make a change in FlinkRuntime.java (https://github.com/apache/incubator-streampipes/blob/2f10a17b77a93eee48527e4e88baf[…]ain/java/org/apache/streampipes/wrapper/flink/FlinkRuntime.java<https://github.com/apache/incubator-streampipes/blob/2f10a17b77a93eee48527e4e88baf%5b…%5dain/java/org/apache/streampipes/wrapper/flink/FlinkRuntime.java>) in method discardRuntime() to also check if its present and its state is running then cancel the job, else check if its status is CANCELLED or FAILED, then return and don't throw exception. Otherwise throw exception. Let me know what do you think? Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu i.A. Muhammad Faizan Student / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH Phone: +49 8165 944 201 / NiceNet: 5033-201 / [email protected]<mailto:[email protected]> / www.denso.com<http://www.denso.com> From: Patrick Wiener <[email protected]> Sent: Monday, March 22, 2021 2:13 PM To: Muhammad Faizan <[email protected]> Cc: Dominik Riemer <[email protected]>; Christian Prehofer <[email protected]>; Philipp Zehnder <[email protected]> Subject: Re: Streampipes für Automotive btw: the JobStatusMessage class of the Flink runtime contains a status enum with the following options: CREATED(JobStatus.TerminalState.NON_TERMINAL), RUNNING(JobStatus.TerminalState.NON_TERMINAL), FAILING(JobStatus.TerminalState.NON_TERMINAL), FAILED(JobStatus.TerminalState.GLOBALLY), CANCELLING(JobStatus.TerminalState.NON_TERMINAL), CANCELED(JobStatus.TerminalState.GLOBALLY), FINISHED(JobStatus.TerminalState.GLOBALLY), RESTARTING(JobStatus.TerminalState.NON_TERMINAL), SUSPENDED(JobStatus.TerminalState.LOCALLY), RECONCILING(JobStatus.TerminalState.NON_TERMINAL); So as mentioned in the previous mail you could check for any of the required states and execute the necessary logic accordingly. Am 22.03.2021 um 14:10 schrieb Patrick Wiener <[email protected]<mailto:[email protected]>>: Hi Faizan, at best you’d be given a list of all jobs by the flink rest api including the failed/stopped one (see Line 298 in FlinkRuntime.java): RestClusterClient<String> restClient = getRestClient(); CompletableFuture<Collection<JobStatusMessage>> jobs = restClient.listJobs(); Here, we filter based on the condition „RUNNING“. My hope would be that you could similarly check for „FAILED/CANCELLED“ or whatever state the flink jobmanager API returns. Did you already debug this part of the code? Patrick Am 22.03.2021 um 13:54 schrieb Muhammad Faizan <[email protected]<mailto:[email protected]>>: Hi Patrick, Yes, I am on SNAPSHOT. So, as a workaround I am trying to delete the PE by sending a DELETE Rest request as in line 136 (https://github.com/apache/incubator-streampipes/blob/2f10a17b77a93eee48527e4e88baf7e4b144d6fe/streampipes-container/src/main/java/org/apache/streampipes/container/api/InvocablePipelineElementResource.java#L136). It works fine in normal case when the PE is running, I can use this delete call to stop the PE and start again using my last mentioned approach. The problem is when I manually cancel the Flink job, then this delete request doesn’t find the Flink job and neither deletes the running instance Id from its in-memory data. <image002.jpg> This is very similar to another problem, for which I have opened this Jira ticket: https://issues.apache.org/jira/projects/STREAMPIPES/issues/STREAMPIPES-321?filter=allopenissues Is it possible to resolve this issue? Also, if it seems small fix, then you can also guide me to provide you a PR 😊 Thanks again for your support! Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu i.A. Muhammad Faizan Student / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH Phone: +49 8165 944 201 / NiceNet: 5033-201 / [email protected]<mailto:[email protected]> / www.denso.com<http://www.denso.com/> From: Patrick Wiener <[email protected]<mailto:[email protected]>> Sent: Friday, March 19, 2021 4:12 PM To: Muhammad Faizan <[email protected]<mailto:[email protected]>> Cc: Dominik Riemer <[email protected]<mailto:[email protected]>>; Christian Prehofer <[email protected]<mailto:[email protected]>>; Philipp Zehnder <[email protected]<mailto:[email protected]>> Subject: Re: Streampipes für Automotive I assume you use the SNAPSHOT version? We store a running instance id in-memory for every processor/sink that has been deployed - if you cancel the job and re-sent the graph via the GraphSubmitter it is checked upon receiving HTTP request at the element service itself wether that instance id is still there, because, well it never got terminated (see line 71 onwards) [1]. Maybe you can find a workaround here. To date, we don’t have such a check. Patrick [1] https://github.com/apache/incubator-streampipes/blob/dev/streampipes-container/src/main/java/org/apache/streampipes/container/api/InvocablePipelineElementResource.java Am 19.03.2021 um 15:53 schrieb Muhammad Faizan <[email protected]<mailto:[email protected]>>: Thanks Patrick and Dominik. Now, I have tried this approach, and the invocation request is received by “pipeline-elements-all-flink”, but its somehow detecting that the job is still running and skipping the invocation. See the following log: _______________ 14:39:11.137 SP [XNIO-1 task-1] INFO o.a.s.c.a.InvocablePipelineElementResource - Pipeline element Aggregation with id 309829d0-3cec-4038-88a7-05bbe43962e8-org.apache.streampipes.examples.waterlevel-aggregation-0 seems to be already running, skipping invocation request. ______________ Any suggestions, How or from where I need to reset the job status? Although the job status on flink is CANCELLED. Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu i.A. Muhammad Faizan Student / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH Phone: +49 8165 944 201 / NiceNet: 5033-201 / [email protected]<mailto:[email protected]> / www.denso.com<http://www.denso.com/> From: Dominik Riemer <[email protected]<mailto:[email protected]>> Sent: Friday, March 19, 2021 12:38 PM To: Patrick Wiener <[email protected]<mailto:[email protected]>>; Muhammad Faizan <[email protected]<mailto:[email protected]>> Cc: Christian Prehofer <[email protected]<mailto:[email protected]>>; Philipp Zehnder <[email protected]<mailto:[email protected]>> Subject: RE: Streampipes für Automotive Hi, that’s correct – I think the updated groupId could also be removed right now – we have an open issue to let the backend decide on Kafka settings [1], the idea here is to allow also for “pausing” of pipelines so that consumers can also start at an earlier offset. But so far, the groupId is newly created so that the pipeline always picks up the latest offset. This doesn’t have any effect on topic creation, the topic remains the same. Hope this clarifies the current approach! Dominik [1] https://issues.apache.org/jira/browse/STREAMPIPES-15 From: Patrick Wiener Sent: Friday, March 19, 2021 10:20 AM To: Muhammad Faizan <[email protected]<mailto:[email protected]>> Cc: Dominik Riemer <[email protected]<mailto:[email protected]>>; Christian Prehofer <[email protected]<mailto:[email protected]>>; Philipp Zehnder <[email protected]<mailto:[email protected]>> Subject: Re: Streampipes für Automotive Topics are auto-created only once at initial start and reused - consumer and producer threads are created/stopped with pipeline start/stop. ......................................................... M.Sc. Patrick Wiener Wissenschaftlicher Mitarbeiter | Research Scientist Information Process Engineering (IPE) FZI Forschungszentrum Informatik Haid-und-Neu-Str. 10–14 76131 Karlsruhe, Germany Tel.: +49 721 9654-822 [email protected]<mailto:[email protected]> www.fzi.de/mitarbeiter/wiener<http://www.fzi.de/mitarbeiter/wiener> ......................................................... FZI Forschungszentrum Informatik Stiftung des bürgerlichen Rechts Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. Marius Zöllner Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus ......................................................... Am 19.03.2021 um 09:57 schrieb Muhammad Faizan <[email protected]<mailto:[email protected]>>: Thanks Patrick for the information. @Dominik Riemer<mailto:[email protected]> Please let me know about the updatedGroupIds? Also, does it mean that every time we stop and start a pipeline, all the communication channels (i.e. kafka topics) are created newly again? Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu i.A. Muhammad Faizan Student / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH Phone: +49 8165 944 201 / NiceNet: 5033-201 / [email protected]<mailto:[email protected]> / www.denso.com<http://www.denso.com/> From: Patrick Wiener <[email protected]<mailto:[email protected]>> Sent: Thursday, March 18, 2021 7:41 PM To: Muhammad Faizan <[email protected]<mailto:[email protected]>> Cc: Dominik Riemer <[email protected]<mailto:[email protected]>>; Christian Prehofer <[email protected]<mailto:[email protected]>>; Philipp Zehnder <[email protected]<mailto:[email protected]>> Subject: Re: Streampipes für Automotive Hi Faizan, your case sounds interesting and you’re already in the right module/package. As you correctly pointed out, the PipelineExecutor handles the pipeline life cycle, i.e., start and stop individual pipeline elements. Regarding the notation, we refer to all elements that are used in a pipeline as pipeline elements. Thus data streams originating from adapters are logically also denoted as pipeline elements. While streams are already existing at modeling time, neither of the so called InvocableStreamPipesEntity types are. That are: * Data Processors * Data Sinks We refer to the notation of Opher Etzion: All data processors are considered event processing agents (EPA), sometimes semantic EPA (S-EPA). All data sinks are considered event consumers (EC) or semantic EC (S-EC). You will often find this in the code base. Thus each pipeline may carry a List<DataProcessorInvokation>, List<DataSinkInvokation>. These entities are often referred to „graphs“, short for invokation graphs, as they carry user-defined static properties selected in the UI. These graphs are build in the InvokationGraphBuilder class. These very graph descriptions are handed over to the GraphSubmitter class, that actually sends the descriptions to the corresponding pipeline element service endpoint, in your case the Flink service PE that you defined. After all its basically sending a post request to the corresponding API endpoint of the PE service. Hope this helps to get going :) Maybe @Dominik can tell you about the updatedGroupIds for Kafka because I’m currently not quite sure about this Cheers Patrick Am 18.03.2021 um 18:55 schrieb Muhammad Faizan <[email protected]<mailto:[email protected]>>: Hi Patrick & Dominik, I have been taking a look into DataProcessorInvocation and StreamPipes source code. I have few questions. My use case is: [Steps] 1. Start a pipeline (with two flink PE’s) 2. Go to Flink dashboard, and cancel a single PE from this pipeline. 3. All other PE’s are running except the cancelled PE. 4. Now, I am trying to start this single PE separately. To do this I found PipelineExecuter<https://github.com/apache/incubator-streampipes/blob/2f10a17b77a93eee48527e4e88baf7e4b144d6fe/streampipes-pipeline-management/src/main/java/org/apache/streampipes/manager/execution/http/PipelineExecutor.java#L57> class which has the logic to start a pipeline. What I am trying to do is to create a pipeline object with only single PE (i.e. sepa) and no actions, no streams. And then start this pipeline to start the single PE. Do you think it’s a correct approach? Please take a look at below code: <image002.jpg> Also, what is the purpose of this line “pipeline.getSepas().forEach(this::updateGroupIds);<https://github.com/apache/incubator-streampipes/blob/2f10a17b77a93eee48527e4e88baf7e4b144d6fe/streampipes-pipeline-management/src/main/java/org/apache/streampipes/manager/execution/http/PipelineExecutor.java#L59>”? Does updateGroupIds method changes the kafka topics and stuff? Will it effect in my use case? Thank you for your time & help! Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu i.A. Muhammad Faizan Student / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH Phone: +49 8165 944 201 / NiceNet: 5033-201 / [email protected]<mailto:[email protected]> / www.denso.com<http://www.denso.com/> From: Muhammad Faizan Sent: Saturday, March 13, 2021 4:25 PM To: Dominik Riemer <[email protected]<mailto:[email protected]>>; Patrick Wiener <[email protected]<mailto:[email protected]>> Cc: Christian Prehofer <[email protected]<mailto:[email protected]>>; Philipp Zehnder <[email protected]<mailto:[email protected]>> Subject: RE: Streampipes für Automotive Hi Dominik, Thanks very much for your detailed information. I have now upgraded my extensions and deployment to 0.68.0-SNAPSHOT. I will now try out the StreamPipes-client, swagger API’s and especially theDataProcessorInvocation. I will let you know if any questions. Regarding the health checks of PE’s. I am wondering if I get some kind of status of the each PE of a running pipeline like (#flink) whether it is Running properly, or is it crashed and yet to be re-scheduled by jobmanager, or is it failed. In these cases I can then add some logic in my service management layer to fix the issue either manually (by user input) or auto. Also, in case of failure on flink, how to handle the checkpoints/savepoints through the management layer. As we can have different runtime for PE (Flink / JVM), it would be good if StreamPipes is providing an abstraction over getting the health status of PE’s. Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu i.A. Muhammad Faizan Student / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH Phone: +49 8165 944 201 / NiceNet: 5033-201 / [email protected]<mailto:[email protected]> / www.denso.com<http://www.denso.com/> From: Dominik Riemer <[email protected]<mailto:[email protected]>> Sent: Wednesday, March 10, 2021 9:14 PM To: Muhammad Faizan <[email protected]<mailto:[email protected]>>; Patrick Wiener <[email protected]<mailto:[email protected]>> Cc: Christian Prehofer <[email protected]<mailto:[email protected]>>; Philipp Zehnder <[email protected]<mailto:[email protected]>> Subject: RE: Streampipes für Automotive Hi Muhammad, sounds cool, having a management layer API would be great! The client and REST API are not yet officially released, but if you are running the latest snapshot version from dev, you can already play around with it. Although there is not yet a documentation, you can find some examples here [1] and there is a (yet incomplete) swagger documentation that can be opened from the login page. In order to use the API, you need to create an API key from the UI by clicking the user icon in the top-right corner and then navigate to “Profile->API keys”. In general, all endpoints (also the undocumented) are accessible with a valid API key. Some of your requirements are already supported by the API, so you can get all pipelines which also includes the PEs and their configuration. Also, pipeline element templates can be fetched and there is a feature in the client that instantiates a DataProcessorInvocation or a DataSinkInvocation with a provided pipeline element template config. As there is not yet a convenient way to create a pipeline element template, you can create one in the UI and inspect the config in the database (the DB is called pipelineelementtemplate and you can access the CouchDB instance on port 5984, there is a UI at path /_utils. There might be some things that do not yet fully work, so if you find any bugs, we’ll be glad to fix them. Regarding the health status, there is not yet anything in the UI, but it’s some of the next things on my roadmap, so it would be great to discuss your ideas and requirements. What information would you expect from the health status message? I guess information on the general node health, but probably also on errors and exception that occur at runtime. The conceptual question behind I’m currently thinking about is whether the Pes should provide an endpoint that holds health information and is called by the core if health status is requested from anywhere, or, as an alternative, if there should be some push communication from the PEs to the core that provides health information. I tend to prefer option 1, but what is your opinion? Best, Dominik [1] https://github.com/apache/incubator-streampipes-examples/blob/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/client/example/StreamPipesClientExample.java From: Muhammad Faizan <[email protected]<mailto:[email protected]>> Sent: Wednesday, March 10, 2021 10:55 AM To: Patrick Wiener <[email protected]<mailto:[email protected]>>; Dominik Riemer <[email protected]<mailto:[email protected]>> Cc: Christian Prehofer <[email protected]<mailto:[email protected]>> Subject: RE: Streampipes für Automotive Hi Patrick & Dominik, Hope you are you doing good! I was thinking about implementing a small management layer API’s for better application specific controllability. I have the following list of operations in mind which I am thinking of achieving: * Get list of running pipelines. * Get list of pipeline elements (PE) of a certain pipeline. * Get health status of each PE. * Option to restart an individual PE of a running pipeline. * See configurations of individual PE. * Export a pipeline template. * Create new pipeline from template. Do you think, is it possible right now to access these controllability through StreamPipes using either StreamPipes Client or REST API’s. If yes, can you share any documentation or info? Thank you very much for your time! Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu i.A. Muhammad Faizan Student / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH Phone: +49 8165 944 201 / NiceNet: 5033-201 / [email protected]<mailto:[email protected]> / www.denso.com<http://www.denso.com/> From: Christian Prehofer <[email protected]<mailto:[email protected]>> Sent: Monday, February 1, 2021 3:17 PM To: Patrick Wiener <[email protected]<mailto:[email protected]>> Cc: Dominik Riemer <[email protected]<mailto:[email protected]>>; Muhammad Faizan <[email protected]<mailto:[email protected]>> Subject: RE: Streampipes für Automotive Hi All, here you go with the paper. And thanks a lot for this open discussion – was really helpful for us! Christian Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu i.V. Dr. Christian Prehofer Director / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH Phone: +49 8165 944 776 / Mobile: +49 152 04962246 / NiceNet: 5033-776 / [email protected]<mailto:[email protected]> / www.denso.com<http://www.denso.com/> From: Patrick Wiener <[email protected]<mailto:[email protected]>> Sent: Monday, 1 February 2021 15:14 To: Christian Prehofer <[email protected]<mailto:[email protected]>> Cc: Dominik Riemer <[email protected]<mailto:[email protected]>>; Muhammad Faizan <[email protected]<mailto:[email protected]>> Subject: Re: Streampipes für Automotive Hi all, @Christian: Unfortunately I was not able to download the paper in Teams :/ could you send it to me via mail? @Faizan: Please see the latest changes to the k8s installation here: https://github.com/apache/incubator-streampipes-installer/tree/dev/k8s Here we mount ${HOME}/streampipes-k8s:/streampipes-k8s as the host volume to persist the configuration + db’s. Best Patrick ......................................................... M.Sc. Patrick Wiener Wissenschaftlicher Mitarbeiter | Research Scientist Information Process Engineering (IPE) FZI Forschungszentrum Informatik Haid-und-Neu-Str. 10–14 76131 Karlsruhe, Germany Tel.: +49 721 9654-822 [email protected]<mailto:[email protected]> www.fzi.de/mitarbeiter/wiener<http://www.fzi.de/mitarbeiter/wiener> ......................................................... FZI Forschungszentrum Informatik Stiftung des bürgerlichen Rechts Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. Marius Zöllner Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus ......................................................... Am 28.01.2021 um 11:31 schrieb Christian Prehofer <[email protected]<mailto:[email protected]>>: Dear Dominik, thanks for quick reply, and sorry for delay. From our side, Monday 14:00 or Tuesday at 15:00 would work. I will just send an invite for Monday Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu i.V. Dr. Christian Prehofer Director / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH Phone: +49 8165 944 776 / Mobile: +49 152 04962246 / NiceNet: 5033-776 / [email protected]<mailto:[email protected]> / www.denso.com<http://www.denso.com/> From: Dominik Riemer <[email protected]<mailto:[email protected]>> Sent: Wednesday, 27 January 2021 06:58 To: Christian Prehofer <[email protected]<mailto:[email protected]>>; Patrick Wiener <[email protected]<mailto:[email protected]>> Cc: Muhammad Faizan <[email protected]<mailto:[email protected]>> Subject: RE: Streampipes für Automotive Dear Christian, sure we can have a talk and discuss your questions! What about early next week, e.g., Monday 14:00, Tuesday 14:00 (or later) or Wednesday morning? Best, Dominik ......................................................... Dr.-Ing. Dominik Riemer Bereichsleiter | Division Manager Information Process Engineering (IPE) FZI Forschungszentrum Informatik Haid-und-Neu-Str. 10–14 76131 Karlsruhe, Germany Tel.: +49 721 9654-724 [email protected]<mailto:[email protected]> www.fzi.de<http://www.fzi.de/> ......................................................... FZI Forschungszentrum Informatik Stiftung des bürgerlichen Rechts Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. Marius Zöllner Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus ......................................................... From: Christian Prehofer <[email protected]<mailto:[email protected]>> Sent: Monday, January 25, 2021 3:17 PM To: Patrick Wiener <[email protected]<mailto:[email protected]>>; Dominik Riemer <[email protected]<mailto:[email protected]>> Cc: Muhammad Faizan <[email protected]<mailto:[email protected]>> Subject: RE: Streampipes für Automotive Dear Patricl, Dominik, Nice to talk to you in October. Just heard that you were in contact with Muhammad (see CC) discussing some issues with Streampipes. He is one of the MS thesis students currently working with me, and has already setup a nice case study in Streampipes. We are currently discussing how to proceed – mainly to evaluate and possibly extend Streampipes to more automotive use cases. We have some questions for some points on your agenda. See slide below. A main point of interest would be the management of pipeline elements and also fault tolerance – the second and third last items in your list. He had also some issues with Kubernetes setup. Would you have a minute to talk about these items? We would like to find a topic and plan such that he can complete a nice thesis in the remaining months. Best regards, Christian Prehofer <image001.jpg> Mit freundlichen Grüßen / Best regards / Yoroshiku Onegai Shimasu i.V. Dr. Christian Prehofer Director / Corporate R&D / DENSO AUTOMOTIVE Deutschland GmbH Phone: +49 8165 944 776 / Mobile: +49 152 04962246 / NiceNet: 5033-776 / [email protected]<mailto:[email protected]> / www.denso.com<http://www.denso.com/> From: Patrick Wiener <[email protected]<mailto:[email protected]>> Sent: Thursday, 22 October 2020 10:49 To: Christian Prehofer <[email protected]<mailto:[email protected]>> Cc: Dominik Riemer <[email protected]<mailto:[email protected]>> Subject: Re: Streampipes für Automotive Hallo Herr Prehofer, Danke für den netten Austausch. Vllt für sie auch interessant, falls noch nicht bekannt: „Musketeer“ http://people.csail.mit.edu/malte/pub/papers/2015-eurosys-musketeer.pdf Viele Grüße Patrick Wiener ......................................................... M.Sc. Patrick Wiener Wissenschaftlicher Mitarbeiter | Research Scientist Information Process Engineering (IPE) FZI Forschungszentrum Informatik Haid-und-Neu-Str. 10–14 76131 Karlsruhe, Germany Tel.: +49 721 9654-822 [email protected]<mailto:[email protected]> www.fzi.de/mitarbeiter/wiener<http://www.fzi.de/mitarbeiter/wiener> ......................................................... FZI Forschungszentrum Informatik Stiftung des bürgerlichen Rechts Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. Marius Zöllner Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus ......................................................... Am 16.10.2020 um 11:19 schrieb Patrick Wiener <[email protected]<mailto:[email protected]>>: Hallo Herr Prehofer, das ist kein Problem. Viele Grüße Patrick Wiener ......................................................... M.Sc. Patrick Wiener Wissenschaftlicher Mitarbeiter | Research Scientist Information Process Engineering (IPE) FZI Forschungszentrum Informatik Haid-und-Neu-Str. 10–14 76131 Karlsruhe, Germany Tel.: +49 721 9654-822 [email protected]<mailto:[email protected]> www.fzi.de/mitarbeiter/wiener<http://www.fzi.de/mitarbeiter/wiener> ......................................................... FZI Forschungszentrum Informatik Stiftung des bürgerlichen Rechts Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. Marius Zöllner Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus ......................................................... Am 16.10.2020 um 11:18 schrieb Christian Prehofer <[email protected]<mailto:[email protected]>>: Hallo, Termin wie besprochen, hoffe MS Teams ist ok. Viele Grüsse, Christian Prehofer ________________________________________________________________________________ Join Microsoft Teams Meeting<https://teams.microsoft.com/l/meetup-join/19%3ameeting_NGUyNGNkZmEtNTZhOS00OWJhLWI3MGYtYTJmZjhmNmUwMWNh%40thread.v2/0?context=%7b%22Tid%22%3a%2269405920-b673-4f7c-8845-e124e9d08af2%22%2c%22Oid%22%3a%222b1355a3-3eac-4fac-a83a-b2569192f6d4%22%7d> +49 69 365057908<tel:+49%2069%20365057908,,535068127> Germany, Frankfurt am Main (Toll) Conference ID: 535 068 127# Local numbers<https://dialin.teams.microsoft.com/86a44213-e985-4164-9704-d9cc945f5413?id=535068127> | Reset PIN<https://mysettings.lync.com/pstnconferencing> | Learn more about Teams<https://aka.ms/JoinTeamsMeeting> | Meeting options<https://teams.microsoft.com/meetingOptions/?organizerId=2b1355a3-3eac-4fac-a83a-b2569192f6d4&tenantId=69405920-b673-4f7c-8845-e124e9d08af2&threadId=19_meeting_NGUyNGNkZmEtNTZhOS00OWJhLWI3MGYtYTJmZjhmNmUwMWNh@thread.v2&messageId=0&language=en-US> ________________________________________________________________________________
