style95 commented on a change in pull request #4446: Actionloop docs URL: https://github.com/apache/incubator-openwhisk/pull/4446#discussion_r277702852
########## File path: docs/actions-actionloop.md ########## @@ -0,0 +1,776 @@ + +<!-- +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--> +# Developing a new Runtime with the ActionLoop proxy + +The [runtime specification](actions-new.md) defines the expected behavior of a runtime. You can implement a runtime from scratch just following the spec. + +However, the fastest way to develop a new runtime is reusing the *ActionLoop* proxy, that already implements the specification and provides just a few hooks to get a fully functional (and *fast*) runtime in a few hours. + +## What is the ActionLoop proxy + +ActionLoop proxy is a runtime "engine", written in Go, originally developed precisely to support the Go language. However it was written in a pretty generic way, and it has been then adopted also to implement runtimes for Swift, PHP, Python, Rust, Java, Ruby and Crystal. It was developed with compiled languages in minds but works well also with scripting languages. + +Using it, you can develop a new runtime in a fraction of the time needed for a full-fledged runtime, since you have only to write a command line protocol and not a fully featured web server, with an amount of corner case to take care. + +Also, you will likely get a pretty fast runtime, since it is currently the most rapid. It was also adopted to improve performances of existing runtimes, that gained from factor 2x to a factor 20x for languages like Python, Ruby, PHP, and Java. + +ActionLoop also supports "precompilation". You can take a raw image and use the docker image to perform the transformation in action. You will get a zip file that you can use as an action that is very fast to start because it contains only the binaries and not the sources. + +So it is likely are using ActionLoop a better bet than implementing the specification from scratch. If you are convinced and want to use it, read on: this page is a tutorial on how to write an ActionLoop runtime, using Ruby as an example. + +## How to write a new runtime with ActionLoop + +The development procedure for ActionLoop requires the following steps: + +* building a docker image containing your target language compiler and the ActionLoop runtime +* writing a simple line-oriented protocol in your target language (converting a python example) +* write (or just adapt the existing) a compilation script for your target language +* write some mandatory tests for your language + +To facilitate the process, there is an `actionloop-starter-kit` in the devtools repository, that implements a fully working runtime for Python. It is a stripped down version of the real Python runtime (removing some advanced details of the real one). + +So you can implement your runtime translating some Python code in your target language. This tutorial shows step by step how to do it writing the Ruby runtime. This code is also used in the real Ruby runtime. + +Using the starter kit, the process becomes: + +- checking out the `actionloop-starter-kit` from the `incubator-openwhisk-devtools` repository +- editing the `Dockerfile` to create the target environment for your language +- rewrite the `launcher.py` in your language +- edit the `compile` script to compile your action in your target language +- write the mandatory tests for your language, adapting the `ActionLoopPythonBasicTests.scala` + +Since we need to show the code you have to translate in some language, we picked Python as it is one of the more readable languages, the closer to be real-world `pseudo-code`. + +You need to know a bit of Python to understand the sample `launcher.py`, just enough to rewrite it in your target language. + +You may need to write some real Python coding to edit the `compile` script, but basic knowledge is enough. + +Finally, you do not need to know Scala, even if the tests are embedded in a Scala test, as all you need is to embed your tests in the code. +## Notation + +In this tutorial we have either terminal transcripts to show what you need to do at the terminal, or "diffs" to show changes to existing files. + +In terminal transcripts, the prefix `$` means commands you have to type at the terminal; the rest are comments (prefixed with `#`) or sample output you should check to verify everything is ok. Generally in a transcript I do not put verbatim output of the terminal as it is generally irrelevant. + +When I show changes to existing files, lines without a prefix should be left as is, lines with `-` should be removed and lines with `+` should be added. + +## Setup the development directory + +So let's start to create our own `actionloop-demo-ruby-2.6`. First, check out the `devtools` repository to access the starter kit, then move it in your home directory to work on it. + +``` +$ git clone https://github.com/apache/incubator-openwhisk-devtools +$ mv incubator-openwhisk-devtools/actionloop-starter-kit ~/actionloop-demo-ruby-v2.6 +``` + +Now we take the directory `python3.7` and rename it to `ruby2.6`; we also fix a couple of references, in order to give a name to our new runtime. + +``` +$ cd actionloop-demo-ruby-v2.6 +$ mv python3.7 ruby2.6 +$ sed -i.bak -e 's/python3.7/ruby2.6/' settings.gradle +$ sed -i.bak -e 's/actionloop-demo-python-v3.7/actionloop-demo-ruby-v2.6/' ruby2.6/build.gradle +``` + +Let's check everything is fine building the image. + +``` +# building the image +$ ./gradlew distDocker +... omissis ... +BUILD SUCCESSFUL in 1s +2 actionable tasks: 2 executed +# checking the image is available +$ docker images actionloop-demo-ruby-v2.6 +REPOSITORY TAG IMAGE ID CREATED SIZE +actionloop-demo-ruby-v2.6 latest df3e77c9cd8f 8 days ago 94MB +``` + +So we have built a new image `actionloop-demo-ruby-v2.6`. However, aside from the renaming, internally is still the old Python. We will change it to support Ruby in the rest of the tutorial. + +## Preparing the Docker environment + +The `Dockerfile` has the task of preparing an environment for executing our actions, so we have to find (or build and deploy on Docker Hub) an image suitable to run our target programming language. We use multistage Docker build to "extract" the *ActionLoop* proxy from the Docker image. + +For the purposes of this tutorial, you should use the `/bin/proxy` binary you can find in the `openwhisk/actionlooop-v2` image on Docker Hub. + +In your runtime image, you have then copied the ActionLoop proxy, the `compile` and the file `launcher.rb` we are going to write. + +Let's rename the launcher and fix the `Dockerfile` to create the environment for running Ruby. + +``` +$ mv ruby2.6/lib/launcher.py ruby2.6/lib/launcher.rb +``` + +Now let's edit the `ruby2.6/Dockerfile` to use, instead of the python image, the official ruby image on Docker Hub, and add out files: + +``` + FROM openwhisk/actionloop-v2:latest as builder +-FROM python:3.7-alpine ++FROM ruby:2.6.2-alpine3.9 + RUN mkdir -p /proxy/bin /proxy/lib /proxy/action + WORKDIR /proxy + COPY --from=builder /bin/proxy /bin/proxy +-ADD lib/launcher.py /proxy/lib/launcher.py ++ADD lib/launcher.rb /proxy/lib/launcher.rb + ADD bin/compile /proxy/bin/compile ++RUN apk update && apk add python3 + ENV OW_COMPILER=/proxy/bin/compile + ENTRYPOINT ["/bin/proxy"] +``` + +Note that: + +1. You changed the base action to use a Ruby image +1. You included the ruby launcher instead of the python one +1. Since the Docker image we picked is a Ruby one, and the `compile` script is still a python script, we had to add it too + +Of course, you can avoid having to add python inside, but you may need to rewrite the entire `compile` in Ruby. You may decide to translate the entire `compile` in your target language, but this is not the focus of this tutorial. + +## Implementing the ActionLoop protocol + +Now you have to convert the `launcher.py` in your programming language. Let's recap the ActionLoop protocol. + +### What the launcher should do + +The launcher must imports your function first. It is the job of the `compile` script to make the function available to the launcher, as we will see in the next paragraph. + +Once the function is imported, it opens the file descriptor 3 for output then reads the standard input line by line. + +For each line, it parses the input in JSON and expects it to be a JSON object (not an array nor a scalar). + +In this object, the key `value` is the payload to be passed to your functions. All the other keys will be passed as environment variables, uppercases and with prefix `__OW_`. + +Finally, your function is invoked with the payload. Once the function returns the result, standard out and standard error is flushed. The result is encoded in JSON, ensuring it is only one line and it is terminated with one newline and it is written in file descriptor 3. + +Then the loop starts again. That's it. + +### Converting `launcher.py` in `launcher.rb` + +Now, let's see the protocol in code, converting the Python launcher in Ruby. + +The compilation script as we will see later will ensure the sources are ready for the launcher. + +You are free to decide where your source action is. I generally ensure that the starting point is a file named like `main__.rb`, with the two underscore final, as those names are pretty unusual to ensure uniqueness. + +Let's skip the imports as they are not interesting. So in Python, the first (significant) line is: + +``` +# now import the action as process input/output +from main__ import main as main +``` + +In Ruby, this translates in: + +``` +# requiring user's action code +require "./main__" +``` + +Now, we open the file descriptor 3, as the proxy will invoke the action with this descriptor attached to a pipe where it can read the results. In Python: + +``` +out = fdopen(3, "wb") +``` + +becomes: + +``` +out = IO.new(3) +``` + +Let's read in Python line by line: + +``` +while True: + line = stdin.readline() + if not line: break + # ...continue... +``` + +becomes: + +``` +while true + # JSON arguments get passed via STDIN + line = STDIN.gets() + break unless line + # ...continue... +end +``` + +Now, you have to read and parse in JSON one line, then extract the payload and set the other values as environment variables: + +``` + # ... continuing ... + args = json.loads(line) + payload = {} + for key in args: + if key == "value": + payload = args["value"] + else: + os.environ["__OW_%s" % key.upper()]= args[key] + # ... continue ... +``` + +translated: + +``` + # ... continuing ... + args = JSON.parse(line) + payload = {} + args.each do |key, value| + if key == "value" + payload = value + else + # set environment variables for other keys + ENV["__OW_#{key.upcase}"] = value + end + end + # ... continue ... +``` + +We are at the point of invoking our functions. You should capture exceptions and produce an `{"error": <result> }` if something goes wrong. In Python: + +``` + # ... continuing ... + res = {} + try: + res = main(payload) + except Exception as ex: + print(traceback.format_exc(), file=stderr) + res = {"error": str(ex)} + # ... continue ... +``` + +Translated in Ruby: + +``` + # ... continuing ... + res = {} + begin + res = main(payload) + rescue Exception => e + puts "exception: #{e}" + res ["error"] = "#{e}" + end + # ... continue ... +``` + +Finally, you flush standard out and standard error and write the result back in file descriptor 3. In Python: + +``` + out.write(json.dumps(res, ensure_ascii=False).encode('utf-8')) + out.write(b'\n') + stdout.flush() + stderr.flush() + out.flush() +``` + +That becomes in Ruby: + +``` + STDOUT.flush() + STDERR.flush() + out.puts(res.to_json) + out.flush() +``` + +Congratulations! You wrote your ActionLoop handler. + +## Writing the compilation script + +Now, you need to write the compilation script. It is basically a script that will prepare the uploaded sources for execution, adding the launcher code and generating the final executable. + +For interpreted languages, the compilation script will only "prepare" the sources for execution. The executable is simply a shell script to invoke the interpreter. + +For compiled languages, like Go it will actually invoke a compiler in order to produce the final executable. There are also cases like Java where you still need to execute the compilation step that produces intermediate code, but the executable is just a shell script that will launch the Java runtime. + +So let's go first examine how ActionLoop handles file upload, then what the provided compilation script does for Python, and finally we will see how to modify the existing `compile` for Python to work for Ruby. Review comment: There are two blanks between `finally we` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
