Mousius commented on a change in pull request #8: URL: https://github.com/apache/tvm-rfcs/pull/8#discussion_r663824915
########## File path: rfcs/0008-microtvm-project-api.md ########## @@ -0,0 +1,509 @@ +- Feature Name: microtvm_project_api +- Start Date: 2020-06-09 +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) + +# Summary +[summary]: #summary + +This RFC describes how TVM integrates with build systems for unconventional platforms, such as those +for microcontrollers and for other bare-metal scenarios. + +# Motivation +[motivation]: #motivation + +Though TVM's primary goal is generating code to implement models from a high-level description, +there are several reasons why a user might want to interact with a platform's build system through +TVM: + +1. To perform autotuning. TVM's internal operator implementations are merely templates and rely on + an automatic search process to arrive at a fast configuration for the template on a given + platform. This search process requires that TVM iteratively build and time code on the platform. +2. To perform remote model execution. A user may wish to try several different models or schedules + on a different platform without rewriting the platform-specific code. Users can do this by + building generated model code against a generic implementation of the microTVM RPC server. +3. To debug model execution remotely. Some aspects of model execution are easy to debug using a + platform-specific debugger; however, some things, such as analyzing intermediate tensor values, + are more easily accomplished with TVM-specific tooling. By leveraging the generic microTVM + RPC server used in (2), TVM can provide such tooling in a platform-agnostic way. + +TVM currently supports these use cases through a set of interfaces: +1. `tvm.micro.Compiler`: used to produce binary and library artifacts. +2. `tvm.micro.Flasher`: used to program attached hardware +3. `tvm.micro.Transport`: used to communicate with on-device microTVM RPC server + +Thus far, implementations of these interfaces have been made for Zephyr, mBED OS, and for +simulated hardware using a POSIX subprocess. The latter two interfaces have proven to be a +relatively good fit; however, `tvm.micro.Compiler` is difficult to implement because it attempts +to replicate a platform's build system in TVM. TVM does not want to incorporate platform-specific +build logic into its codebase. + +This proposal unifies these three interfaces to form a "Project-level" interface, recognizing that +it's typical to interact with unconventional platforms and their build systems at this level. It +simplifies the `Compiler` interaction into a project-level Build, and adds an explicit +`generate_project` method to the interface. These changes remove the need to build components +and drive the link process from TVM. + +As a goal, this proposal aims to allow for the same use cases as are currently supported with these +improvements: + +1. Integrating more naturally with build systems typical of embedded platforms. +2. Allowing TVM to automatically generate projects platforms to define automated scripts to build projects Review comment: `Allowing TVM to automatically generate projects for platforms and define automated scripts to build such projects` ? ########## File path: rfcs/0008-microtvm-project-api.md ########## @@ -0,0 +1,509 @@ +- Feature Name: microtvm_project_api +- Start Date: 2020-06-09 +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) + +# Summary +[summary]: #summary + +This RFC describes how TVM integrates with build systems for unconventional platforms, such as those +for microcontrollers and for other bare-metal scenarios. + +# Motivation +[motivation]: #motivation + +Though TVM's primary goal is generating code to implement models from a high-level description, +there are several reasons why a user might want to interact with a platform's build system through +TVM: + +1. To perform autotuning. TVM's internal operator implementations are merely templates and rely on + an automatic search process to arrive at a fast configuration for the template on a given + platform. This search process requires that TVM iteratively build and time code on the platform. +2. To perform remote model execution. A user may wish to try several different models or schedules + on a different platform without rewriting the platform-specific code. Users can do this by + building generated model code against a generic implementation of the microTVM RPC server. +3. To debug model execution remotely. Some aspects of model execution are easy to debug using a + platform-specific debugger; however, some things, such as analyzing intermediate tensor values, + are more easily accomplished with TVM-specific tooling. By leveraging the generic microTVM + RPC server used in (2), TVM can provide such tooling in a platform-agnostic way. + +TVM currently supports these use cases through a set of interfaces: +1. `tvm.micro.Compiler`: used to produce binary and library artifacts. +2. `tvm.micro.Flasher`: used to program attached hardware +3. `tvm.micro.Transport`: used to communicate with on-device microTVM RPC server + +Thus far, implementations of these interfaces have been made for Zephyr, mBED OS, and for +simulated hardware using a POSIX subprocess. The latter two interfaces have proven to be a +relatively good fit; however, `tvm.micro.Compiler` is difficult to implement because it attempts +to replicate a platform's build system in TVM. TVM does not want to incorporate platform-specific +build logic into its codebase. + +This proposal unifies these three interfaces to form a "Project-level" interface, recognizing that +it's typical to interact with unconventional platforms and their build systems at this level. It +simplifies the `Compiler` interaction into a project-level Build, and adds an explicit +`generate_project` method to the interface. These changes remove the need to build components +and drive the link process from TVM. + +As a goal, this proposal aims to allow for the same use cases as are currently supported with these +improvements: + +1. Integrating more naturally with build systems typical of embedded platforms. +2. Allowing TVM to automatically generate projects platforms to define automated scripts to build projects + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +TVM can interact with platform SDKs via its **Project API**. Such SDKs are common when working with +non-traditional OS platforms, such as those commonly used in embedded systems (e.g. Arduino, Zephyr, +iOS). Given a platform-specific implementation of this Project API, TVM can: +1. Generate projects that integrate implemented TVM models with generic platform runtime componeents +2. Build those generated projects +3. Program attached hardware +4. Drive remote model execution via the TVM RPC Server interface. + +This last capability means that TVM can drive autotuning, remotely perform model inference, and debug +models on non-traditional OS such as Arduino, Zephyr, and mobile platforms such as iOS and Android. + +To provide support for a platform, a **template project** is first defined. Template projects are +expected to exist entirely inside a directory and are identified to TVM by the path to the directory +locally. Template projects may live either inside the TVM repository (when they can be included in the +TVM CI) or in other version control repositories. The template project contains at minium an Review comment: Typo: `minimum` ########## File path: rfcs/0008-microtvm-project-api.md ########## @@ -0,0 +1,509 @@ +- Feature Name: microtvm_project_api +- Start Date: 2020-06-09 +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) + +# Summary +[summary]: #summary + +This RFC describes how TVM integrates with build systems for unconventional platforms, such as those +for microcontrollers and for other bare-metal scenarios. + +# Motivation +[motivation]: #motivation + +Though TVM's primary goal is generating code to implement models from a high-level description, +there are several reasons why a user might want to interact with a platform's build system through +TVM: + +1. To perform autotuning. TVM's internal operator implementations are merely templates and rely on + an automatic search process to arrive at a fast configuration for the template on a given + platform. This search process requires that TVM iteratively build and time code on the platform. +2. To perform remote model execution. A user may wish to try several different models or schedules + on a different platform without rewriting the platform-specific code. Users can do this by + building generated model code against a generic implementation of the microTVM RPC server. +3. To debug model execution remotely. Some aspects of model execution are easy to debug using a + platform-specific debugger; however, some things, such as analyzing intermediate tensor values, + are more easily accomplished with TVM-specific tooling. By leveraging the generic microTVM + RPC server used in (2), TVM can provide such tooling in a platform-agnostic way. + +TVM currently supports these use cases through a set of interfaces: +1. `tvm.micro.Compiler`: used to produce binary and library artifacts. +2. `tvm.micro.Flasher`: used to program attached hardware +3. `tvm.micro.Transport`: used to communicate with on-device microTVM RPC server + +Thus far, implementations of these interfaces have been made for Zephyr, mBED OS, and for +simulated hardware using a POSIX subprocess. The latter two interfaces have proven to be a +relatively good fit; however, `tvm.micro.Compiler` is difficult to implement because it attempts +to replicate a platform's build system in TVM. TVM does not want to incorporate platform-specific +build logic into its codebase. + +This proposal unifies these three interfaces to form a "Project-level" interface, recognizing that +it's typical to interact with unconventional platforms and their build systems at this level. It +simplifies the `Compiler` interaction into a project-level Build, and adds an explicit +`generate_project` method to the interface. These changes remove the need to build components +and drive the link process from TVM. + +As a goal, this proposal aims to allow for the same use cases as are currently supported with these +improvements: + +1. Integrating more naturally with build systems typical of embedded platforms. +2. Allowing TVM to automatically generate projects platforms to define automated scripts to build projects + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +TVM can interact with platform SDKs via its **Project API**. Such SDKs are common when working with +non-traditional OS platforms, such as those commonly used in embedded systems (e.g. Arduino, Zephyr, +iOS). Given a platform-specific implementation of this Project API, TVM can: +1. Generate projects that integrate implemented TVM models with generic platform runtime componeents +2. Build those generated projects +3. Program attached hardware +4. Drive remote model execution via the TVM RPC Server interface. + +This last capability means that TVM can drive autotuning, remotely perform model inference, and debug +models on non-traditional OS such as Arduino, Zephyr, and mobile platforms such as iOS and Android. + +To provide support for a platform, a **template project** is first defined. Template projects are +expected to exist entirely inside a directory and are identified to TVM by the path to the directory +locally. Template projects may live either inside the TVM repository (when they can be included in the +TVM CI) or in other version control repositories. The template project contains at minium an +implementation of the **Project API** inside an executable program known as the +**TVM Project API Server**. + +To begin working with a particular platform's Project API implementation, the user supplies TVM with +the path to the top-level directory. TVM launches an instance the Project API Server (found at a +standard location in that directory). TVM communicates with the Project API Server using JSON-RPC +over standard OS pipes. + +TVM supplies generated code to the Project API Server using [Model Library Format](0001-model-library-format.md). + +Below is a survey of example workflows used with the Project API Server: + +## Generating a project + +1. The user imports a model into TVM and builds it using `tvm.relay.build`. Review comment: Do you have ideas for the `tvmc` user flow for this rather than directly scripting it? ########## File path: rfcs/0008-microtvm-project-api.md ########## @@ -0,0 +1,509 @@ +- Feature Name: microtvm_project_api +- Start Date: 2020-06-09 +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) + +# Summary +[summary]: #summary + +This RFC describes how TVM integrates with build systems for unconventional platforms, such as those +for microcontrollers and for other bare-metal scenarios. + +# Motivation +[motivation]: #motivation + +Though TVM's primary goal is generating code to implement models from a high-level description, +there are several reasons why a user might want to interact with a platform's build system through +TVM: + +1. To perform autotuning. TVM's internal operator implementations are merely templates and rely on + an automatic search process to arrive at a fast configuration for the template on a given + platform. This search process requires that TVM iteratively build and time code on the platform. +2. To perform remote model execution. A user may wish to try several different models or schedules + on a different platform without rewriting the platform-specific code. Users can do this by + building generated model code against a generic implementation of the microTVM RPC server. +3. To debug model execution remotely. Some aspects of model execution are easy to debug using a + platform-specific debugger; however, some things, such as analyzing intermediate tensor values, + are more easily accomplished with TVM-specific tooling. By leveraging the generic microTVM + RPC server used in (2), TVM can provide such tooling in a platform-agnostic way. + +TVM currently supports these use cases through a set of interfaces: +1. `tvm.micro.Compiler`: used to produce binary and library artifacts. +2. `tvm.micro.Flasher`: used to program attached hardware +3. `tvm.micro.Transport`: used to communicate with on-device microTVM RPC server + +Thus far, implementations of these interfaces have been made for Zephyr, mBED OS, and for +simulated hardware using a POSIX subprocess. The latter two interfaces have proven to be a +relatively good fit; however, `tvm.micro.Compiler` is difficult to implement because it attempts +to replicate a platform's build system in TVM. TVM does not want to incorporate platform-specific +build logic into its codebase. + +This proposal unifies these three interfaces to form a "Project-level" interface, recognizing that +it's typical to interact with unconventional platforms and their build systems at this level. It +simplifies the `Compiler` interaction into a project-level Build, and adds an explicit +`generate_project` method to the interface. These changes remove the need to build components +and drive the link process from TVM. + +As a goal, this proposal aims to allow for the same use cases as are currently supported with these +improvements: + +1. Integrating more naturally with build systems typical of embedded platforms. +2. Allowing TVM to automatically generate projects platforms to define automated scripts to build projects + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +TVM can interact with platform SDKs via its **Project API**. Such SDKs are common when working with +non-traditional OS platforms, such as those commonly used in embedded systems (e.g. Arduino, Zephyr, +iOS). Given a platform-specific implementation of this Project API, TVM can: +1. Generate projects that integrate implemented TVM models with generic platform runtime componeents +2. Build those generated projects +3. Program attached hardware +4. Drive remote model execution via the TVM RPC Server interface. + +This last capability means that TVM can drive autotuning, remotely perform model inference, and debug +models on non-traditional OS such as Arduino, Zephyr, and mobile platforms such as iOS and Android. + +To provide support for a platform, a **template project** is first defined. Template projects are +expected to exist entirely inside a directory and are identified to TVM by the path to the directory +locally. Template projects may live either inside the TVM repository (when they can be included in the +TVM CI) or in other version control repositories. The template project contains at minium an +implementation of the **Project API** inside an executable program known as the +**TVM Project API Server**. + +To begin working with a particular platform's Project API implementation, the user supplies TVM with +the path to the top-level directory. TVM launches an instance the Project API Server (found at a +standard location in that directory). TVM communicates with the Project API Server using JSON-RPC +over standard OS pipes. + +TVM supplies generated code to the Project API Server using [Model Library Format](0001-model-library-format.md). + +Below is a survey of example workflows used with the Project API Server: + +## Generating a project + +1. The user imports a model into TVM and builds it using `tvm.relay.build`. +2. The user supplies TVM with the path to the template project and a path to a non-existent + directory where the generated project should live. +3. TVM launches a Project API server in the template project. +4. TVM verifies that the template project is indeed a template by invoking the Project API server + `server_info_query` method. +5. TVM invokes the Project API server `generate_project` method to generate the new project. + +## Building and Flashing + +1. The user follows the steps under [Generating a project](#generating-a-project). +2. TVM expects the Project API server to copy itself to the generated project. It launches a + Project API server in the generated project directory. +3. TVM verifies that the generated project is not a template by invoking the Project API server + `server_info_query` method. This method also returns `options` that can be used to customize + the build. +4. TVM invokes the Project API server `build` method to build the project. +5. TVM invokes the Project API server `flash` method to program the attached device. The + `options` can be used to specify a device serial number. + +## Host-driven model inference + +1. The user follows the steps under [Generating a project](#generating-a-project). +2. TVM invokes the Project API server `connect_transport` method to connect to the remote on-device + microTVM RPC server. +3. The microTVM RPC server is attached to a traditional TVM RPC session on the host device. +4. TVM drives inference on-device using the traditional TVM RPC methods. The Project API server + methods `read_transport` and `write_transport` are used to receive and send data. +5. When the inference session is over, TVM invokes the Project API server method `close_transport` + to release any underlying I/O resources, and terminates the Project API server. + +## AutoTVM + +1. The user supplies a kernel to the AutoTVM tuner for search-based optimization. +2. AutoTVM generates a set of task configurations, instantiates each task, and then invokes + `tvm.build` to produce a `Module` for each instantiated task. +3. AutoTVM produces a Model Library Format archive from the `Module` for each instantiated task. +4. AutoTVM passes the Model Library Format archive to the AutoTVM `runner`. Project API overrides the + traditional AutoTVM runner by providing a [`module_loader`](#module-loader). The microTVM + `module_loader` connects to a _supervisor_ TVM RPC server which carries out the microTVM project build + as part of the TVM RPC `session_constructor`. The following steps occur in the + `session_constructor`: + + 1. The Model Library Format tar is uploaded to the supervisor. + 2. The user supplies a path, on the supervisor, to a template project. + 3. The supervisor `session_constructor` performs the steps under + [Building and Flashing](#building-and-flashing). + 4. The supervisor `session_constructor` invokes the Project API server `connect_transport` method + to connect to the remote device. The session constructor registers a traditional TVM RPC + session on the supervisor, and this session is also used by the AutoTVM runner due to the + `session_constructor` mechanism. + +5. The AutoTVM runner measures runtime as normal. +6. The AutoTVM runner disconnects the session, closing the Project API server on the supervisor. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Project API implementation + +The Project API is a Remote Procedure Call (RPC)-type mechanism implemented using +[JSON-RPC](https://www.jsonrpc.org/specification). The client and server are implemented in +`python/tvm/micro/project_api`. Tests are implemented in +`tests/python/unittest/test_micro_project_api.py`. + +## Project API interface + +The functions that need to be implemented as part of a Project API server are defined on the +`ProjectAPIHandler` class in `python/tvm/micro/project_api/server.py`: + +``` +class ProjectAPIHandler(metaclass=abc.ABCMeta): + + @abc.abstractmethod + def server_info_query(self) -> ServerInfo: + raise NotImplementedError() + + @abc.abstractmethod + def generate_project(self, model_library_format_path : pathlib.Path, standalone_crt_dir : pathlib.Path, project_dir : pathlib.Path, options : dict): + """Generate a project from the given artifacts, copying ourselves to that project. + + Parameters + ---------- + model_library_format_path : pathlib.Path + Path to the Model Library Format tar archive. + standalone_crt_dir : pathlib.Path + Path to the root directory of the "standalone_crt" TVM build artifact. This contains the + TVM C runtime. + project_dir : pathlib.Path + Path to a nonexistent directory which should be created and filled with the generated + project. + options : dict + Dict mapping option name to ProjectOption. + """ + raise NotImplementedError() + + @abc.abstractmethod + def build(self, options : dict): + """Build the project, enabling the flash() call to made. + + Parameters + ---------- + options : Dict[str, ProjectOption] + ProjectOption which may influence the build, keyed by option name. + """ + raise NotImplementedError() + + @abc.abstractmethod + def flash(self, options : dict): + """Program the project onto the device. + + Parameters + ---------- + options : Dict[str, ProjectOption] + ProjectOption which may influence the programming process, keyed by option name. + """ + raise NotImplementedError() + + @abc.abstractmethod + def open_transport(self, options : dict) -> TransportTimeouts: + """Open resources needed for the transport layer. + + This function might e.g. open files or serial ports needed in write_transport or read_transport. + + Calling this function enables the write_transport and read_transport calls. If the + transport is not open, this method is a no-op. + + Parameters + ---------- + options : Dict[str, ProjectOption] + ProjectOption which may influence the programming process, keyed by option name. + """ + raise NotImplementedError() + + @abc.abstractmethod + def close_transport(self): + """Close resources needed to operate the transport layer. + This function might e.g. close files or serial ports needed in write_transport or read_transport. + + Calling this function disables the write_transport and read_transport calls. If the + transport is not open, this method is a no-op. + """ + raise NotImplementedError() + + @abc.abstractmethod + def read_transport(self, n : int, timeout_sec : typing.Union[float, type(None)]) -> bytes: + """Read data from the transport. + + Parameters + ---------- + n : int + The exact number of bytes to read from the transport. + timeout_sec : Union[float, None] + Number of seconds to wait for at least one byte to be written before timing out. If + timeout_sec is 0, write should attempt to service the request in a non-blocking fashion. + If timeout_sec is None, write should block until all `n` bytes of data can be returned. + + Returns + ------- + bytes : + Data read from the channel. Should be exactly `n` bytes long. + + Raises + ------ + TransportClosedError : + When the transport layer determines that the transport can no longer send or receive + data due to an underlying I/O problem (i.e. file descriptor closed, cable removed, etc). + IoTimeoutError : + When `timeout_sec` elapses without receiving any data. + """ + raise NotImplementedError() + + @abc.abstractmethod + def write_transport(self, data : bytes, timeout_sec : float): + """Write data to the transport. + This function should either write all bytes in `data` or raise an exception. + + Parameters + ---------- + data : bytes + The data to write over the channel. + timeout_sec : Union[float, None] + Number of seconds to wait for all bytes to be written before timing out. If timeout_sec + is 0, write should attempt to service the request in a non-blocking fashion. If + timeout_sec is None, write should block until it has written all data. + + Raises + ------ + TransportClosedError : + When the transport layer determines that the transport can no longer send or receive + data due to an underlying I/O problem (i.e. file descriptor closed, cable removed, etc). + IoTimeoutError : + When `timeout_sec` elapses without receiving any data. + """ + raise NotImplementedError() +``` + +## Project Options + +Each Project API server can return `project_options` as part of the `server_info_query` response. +These can be specified by the user to allow them to give platform SDK-specific options to each API +method. + +``` +ProjectOption = collections.namedtuple('ProjectOption', ('name', 'help')) Review comment: Should we represent the server response in JSON rather than the client data types? ########## File path: rfcs/0008-microtvm-project-api.md ########## @@ -0,0 +1,509 @@ +- Feature Name: microtvm_project_api +- Start Date: 2020-06-09 +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) + +# Summary +[summary]: #summary + +This RFC describes how TVM integrates with build systems for unconventional platforms, such as those +for microcontrollers and for other bare-metal scenarios. + +# Motivation +[motivation]: #motivation + +Though TVM's primary goal is generating code to implement models from a high-level description, +there are several reasons why a user might want to interact with a platform's build system through +TVM: + +1. To perform autotuning. TVM's internal operator implementations are merely templates and rely on + an automatic search process to arrive at a fast configuration for the template on a given + platform. This search process requires that TVM iteratively build and time code on the platform. +2. To perform remote model execution. A user may wish to try several different models or schedules + on a different platform without rewriting the platform-specific code. Users can do this by + building generated model code against a generic implementation of the microTVM RPC server. +3. To debug model execution remotely. Some aspects of model execution are easy to debug using a + platform-specific debugger; however, some things, such as analyzing intermediate tensor values, + are more easily accomplished with TVM-specific tooling. By leveraging the generic microTVM + RPC server used in (2), TVM can provide such tooling in a platform-agnostic way. + +TVM currently supports these use cases through a set of interfaces: +1. `tvm.micro.Compiler`: used to produce binary and library artifacts. +2. `tvm.micro.Flasher`: used to program attached hardware +3. `tvm.micro.Transport`: used to communicate with on-device microTVM RPC server + +Thus far, implementations of these interfaces have been made for Zephyr, mBED OS, and for +simulated hardware using a POSIX subprocess. The latter two interfaces have proven to be a +relatively good fit; however, `tvm.micro.Compiler` is difficult to implement because it attempts +to replicate a platform's build system in TVM. TVM does not want to incorporate platform-specific +build logic into its codebase. + +This proposal unifies these three interfaces to form a "Project-level" interface, recognizing that +it's typical to interact with unconventional platforms and their build systems at this level. It +simplifies the `Compiler` interaction into a project-level Build, and adds an explicit +`generate_project` method to the interface. These changes remove the need to build components +and drive the link process from TVM. + +As a goal, this proposal aims to allow for the same use cases as are currently supported with these +improvements: + +1. Integrating more naturally with build systems typical of embedded platforms. +2. Allowing TVM to automatically generate projects platforms to define automated scripts to build projects + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +TVM can interact with platform SDKs via its **Project API**. Such SDKs are common when working with +non-traditional OS platforms, such as those commonly used in embedded systems (e.g. Arduino, Zephyr, +iOS). Given a platform-specific implementation of this Project API, TVM can: +1. Generate projects that integrate implemented TVM models with generic platform runtime componeents +2. Build those generated projects +3. Program attached hardware +4. Drive remote model execution via the TVM RPC Server interface. + +This last capability means that TVM can drive autotuning, remotely perform model inference, and debug +models on non-traditional OS such as Arduino, Zephyr, and mobile platforms such as iOS and Android. + +To provide support for a platform, a **template project** is first defined. Template projects are +expected to exist entirely inside a directory and are identified to TVM by the path to the directory +locally. Template projects may live either inside the TVM repository (when they can be included in the +TVM CI) or in other version control repositories. The template project contains at minium an +implementation of the **Project API** inside an executable program known as the +**TVM Project API Server**. + +To begin working with a particular platform's Project API implementation, the user supplies TVM with +the path to the top-level directory. TVM launches an instance the Project API Server (found at a +standard location in that directory). TVM communicates with the Project API Server using JSON-RPC +over standard OS pipes. + +TVM supplies generated code to the Project API Server using [Model Library Format](0001-model-library-format.md). + +Below is a survey of example workflows used with the Project API Server: + +## Generating a project + +1. The user imports a model into TVM and builds it using `tvm.relay.build`. +2. The user supplies TVM with the path to the template project and a path to a non-existent + directory where the generated project should live. +3. TVM launches a Project API server in the template project. +4. TVM verifies that the template project is indeed a template by invoking the Project API server + `server_info_query` method. +5. TVM invokes the Project API server `generate_project` method to generate the new project. + +## Building and Flashing + +1. The user follows the steps under [Generating a project](#generating-a-project). +2. TVM expects the Project API server to copy itself to the generated project. It launches a + Project API server in the generated project directory. +3. TVM verifies that the generated project is not a template by invoking the Project API server + `server_info_query` method. This method also returns `options` that can be used to customize + the build. +4. TVM invokes the Project API server `build` method to build the project. +5. TVM invokes the Project API server `flash` method to program the attached device. The + `options` can be used to specify a device serial number. + +## Host-driven model inference + +1. The user follows the steps under [Generating a project](#generating-a-project). +2. TVM invokes the Project API server `connect_transport` method to connect to the remote on-device + microTVM RPC server. +3. The microTVM RPC server is attached to a traditional TVM RPC session on the host device. +4. TVM drives inference on-device using the traditional TVM RPC methods. The Project API server + methods `read_transport` and `write_transport` are used to receive and send data. +5. When the inference session is over, TVM invokes the Project API server method `close_transport` + to release any underlying I/O resources, and terminates the Project API server. + +## AutoTVM + +1. The user supplies a kernel to the AutoTVM tuner for search-based optimization. +2. AutoTVM generates a set of task configurations, instantiates each task, and then invokes + `tvm.build` to produce a `Module` for each instantiated task. +3. AutoTVM produces a Model Library Format archive from the `Module` for each instantiated task. Review comment: Just to clarify, this is one combined model library format with all `Module`s combined so it only flashes once for many task permutations? ########## File path: rfcs/0008-microtvm-project-api.md ########## @@ -0,0 +1,509 @@ +- Feature Name: microtvm_project_api +- Start Date: 2020-06-09 +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) + +# Summary +[summary]: #summary + +This RFC describes how TVM integrates with build systems for unconventional platforms, such as those +for microcontrollers and for other bare-metal scenarios. + +# Motivation +[motivation]: #motivation + +Though TVM's primary goal is generating code to implement models from a high-level description, +there are several reasons why a user might want to interact with a platform's build system through +TVM: + +1. To perform autotuning. TVM's internal operator implementations are merely templates and rely on + an automatic search process to arrive at a fast configuration for the template on a given + platform. This search process requires that TVM iteratively build and time code on the platform. +2. To perform remote model execution. A user may wish to try several different models or schedules + on a different platform without rewriting the platform-specific code. Users can do this by + building generated model code against a generic implementation of the microTVM RPC server. +3. To debug model execution remotely. Some aspects of model execution are easy to debug using a + platform-specific debugger; however, some things, such as analyzing intermediate tensor values, + are more easily accomplished with TVM-specific tooling. By leveraging the generic microTVM + RPC server used in (2), TVM can provide such tooling in a platform-agnostic way. + +TVM currently supports these use cases through a set of interfaces: +1. `tvm.micro.Compiler`: used to produce binary and library artifacts. +2. `tvm.micro.Flasher`: used to program attached hardware +3. `tvm.micro.Transport`: used to communicate with on-device microTVM RPC server + +Thus far, implementations of these interfaces have been made for Zephyr, mBED OS, and for +simulated hardware using a POSIX subprocess. The latter two interfaces have proven to be a +relatively good fit; however, `tvm.micro.Compiler` is difficult to implement because it attempts +to replicate a platform's build system in TVM. TVM does not want to incorporate platform-specific +build logic into its codebase. + +This proposal unifies these three interfaces to form a "Project-level" interface, recognizing that +it's typical to interact with unconventional platforms and their build systems at this level. It +simplifies the `Compiler` interaction into a project-level Build, and adds an explicit +`generate_project` method to the interface. These changes remove the need to build components +and drive the link process from TVM. + +As a goal, this proposal aims to allow for the same use cases as are currently supported with these +improvements: + +1. Integrating more naturally with build systems typical of embedded platforms. +2. Allowing TVM to automatically generate projects platforms to define automated scripts to build projects + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +TVM can interact with platform SDKs via its **Project API**. Such SDKs are common when working with +non-traditional OS platforms, such as those commonly used in embedded systems (e.g. Arduino, Zephyr, +iOS). Given a platform-specific implementation of this Project API, TVM can: +1. Generate projects that integrate implemented TVM models with generic platform runtime componeents +2. Build those generated projects +3. Program attached hardware +4. Drive remote model execution via the TVM RPC Server interface. + +This last capability means that TVM can drive autotuning, remotely perform model inference, and debug +models on non-traditional OS such as Arduino, Zephyr, and mobile platforms such as iOS and Android. + +To provide support for a platform, a **template project** is first defined. Template projects are +expected to exist entirely inside a directory and are identified to TVM by the path to the directory +locally. Template projects may live either inside the TVM repository (when they can be included in the +TVM CI) or in other version control repositories. The template project contains at minium an +implementation of the **Project API** inside an executable program known as the +**TVM Project API Server**. + +To begin working with a particular platform's Project API implementation, the user supplies TVM with +the path to the top-level directory. TVM launches an instance the Project API Server (found at a +standard location in that directory). TVM communicates with the Project API Server using JSON-RPC +over standard OS pipes. + +TVM supplies generated code to the Project API Server using [Model Library Format](0001-model-library-format.md). + +Below is a survey of example workflows used with the Project API Server: + +## Generating a project + +1. The user imports a model into TVM and builds it using `tvm.relay.build`. +2. The user supplies TVM with the path to the template project and a path to a non-existent + directory where the generated project should live. +3. TVM launches a Project API server in the template project. +4. TVM verifies that the template project is indeed a template by invoking the Project API server + `server_info_query` method. +5. TVM invokes the Project API server `generate_project` method to generate the new project. + +## Building and Flashing + +1. The user follows the steps under [Generating a project](#generating-a-project). +2. TVM expects the Project API server to copy itself to the generated project. It launches a + Project API server in the generated project directory. +3. TVM verifies that the generated project is not a template by invoking the Project API server + `server_info_query` method. This method also returns `options` that can be used to customize + the build. +4. TVM invokes the Project API server `build` method to build the project. +5. TVM invokes the Project API server `flash` method to program the attached device. The + `options` can be used to specify a device serial number. + +## Host-driven model inference + +1. The user follows the steps under [Generating a project](#generating-a-project). +2. TVM invokes the Project API server `connect_transport` method to connect to the remote on-device + microTVM RPC server. +3. The microTVM RPC server is attached to a traditional TVM RPC session on the host device. +4. TVM drives inference on-device using the traditional TVM RPC methods. The Project API server + methods `read_transport` and `write_transport` are used to receive and send data. +5. When the inference session is over, TVM invokes the Project API server method `close_transport` + to release any underlying I/O resources, and terminates the Project API server. + +## AutoTVM + +1. The user supplies a kernel to the AutoTVM tuner for search-based optimization. +2. AutoTVM generates a set of task configurations, instantiates each task, and then invokes + `tvm.build` to produce a `Module` for each instantiated task. +3. AutoTVM produces a Model Library Format archive from the `Module` for each instantiated task. +4. AutoTVM passes the Model Library Format archive to the AutoTVM `runner`. Project API overrides the + traditional AutoTVM runner by providing a [`module_loader`](#module-loader). The microTVM + `module_loader` connects to a _supervisor_ TVM RPC server which carries out the microTVM project build + as part of the TVM RPC `session_constructor`. The following steps occur in the + `session_constructor`: + + 1. The Model Library Format tar is uploaded to the supervisor. + 2. The user supplies a path, on the supervisor, to a template project. + 3. The supervisor `session_constructor` performs the steps under + [Building and Flashing](#building-and-flashing). + 4. The supervisor `session_constructor` invokes the Project API server `connect_transport` method + to connect to the remote device. The session constructor registers a traditional TVM RPC + session on the supervisor, and this session is also used by the AutoTVM runner due to the + `session_constructor` mechanism. + +5. The AutoTVM runner measures runtime as normal. +6. The AutoTVM runner disconnects the session, closing the Project API server on the supervisor. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Project API implementation + +The Project API is a Remote Procedure Call (RPC)-type mechanism implemented using +[JSON-RPC](https://www.jsonrpc.org/specification). The client and server are implemented in +`python/tvm/micro/project_api`. Tests are implemented in +`tests/python/unittest/test_micro_project_api.py`. + +## Project API interface + +The functions that need to be implemented as part of a Project API server are defined on the +`ProjectAPIHandler` class in `python/tvm/micro/project_api/server.py`: + +``` +class ProjectAPIHandler(metaclass=abc.ABCMeta): + + @abc.abstractmethod + def server_info_query(self) -> ServerInfo: + raise NotImplementedError() + + @abc.abstractmethod + def generate_project(self, model_library_format_path : pathlib.Path, standalone_crt_dir : pathlib.Path, project_dir : pathlib.Path, options : dict): + """Generate a project from the given artifacts, copying ourselves to that project. + + Parameters + ---------- + model_library_format_path : pathlib.Path + Path to the Model Library Format tar archive. + standalone_crt_dir : pathlib.Path + Path to the root directory of the "standalone_crt" TVM build artifact. This contains the + TVM C runtime. + project_dir : pathlib.Path + Path to a nonexistent directory which should be created and filled with the generated + project. + options : dict + Dict mapping option name to ProjectOption. + """ + raise NotImplementedError() + + @abc.abstractmethod + def build(self, options : dict): + """Build the project, enabling the flash() call to made. + + Parameters + ---------- + options : Dict[str, ProjectOption] + ProjectOption which may influence the build, keyed by option name. + """ + raise NotImplementedError() + + @abc.abstractmethod + def flash(self, options : dict): + """Program the project onto the device. + + Parameters + ---------- + options : Dict[str, ProjectOption] + ProjectOption which may influence the programming process, keyed by option name. + """ + raise NotImplementedError() + + @abc.abstractmethod + def open_transport(self, options : dict) -> TransportTimeouts: + """Open resources needed for the transport layer. + + This function might e.g. open files or serial ports needed in write_transport or read_transport. + + Calling this function enables the write_transport and read_transport calls. If the + transport is not open, this method is a no-op. + + Parameters + ---------- + options : Dict[str, ProjectOption] + ProjectOption which may influence the programming process, keyed by option name. + """ + raise NotImplementedError() + + @abc.abstractmethod + def close_transport(self): + """Close resources needed to operate the transport layer. + This function might e.g. close files or serial ports needed in write_transport or read_transport. + + Calling this function disables the write_transport and read_transport calls. If the + transport is not open, this method is a no-op. + """ + raise NotImplementedError() + + @abc.abstractmethod + def read_transport(self, n : int, timeout_sec : typing.Union[float, type(None)]) -> bytes: + """Read data from the transport. + + Parameters + ---------- + n : int + The exact number of bytes to read from the transport. + timeout_sec : Union[float, None] + Number of seconds to wait for at least one byte to be written before timing out. If + timeout_sec is 0, write should attempt to service the request in a non-blocking fashion. + If timeout_sec is None, write should block until all `n` bytes of data can be returned. + + Returns + ------- + bytes : + Data read from the channel. Should be exactly `n` bytes long. + + Raises + ------ + TransportClosedError : + When the transport layer determines that the transport can no longer send or receive + data due to an underlying I/O problem (i.e. file descriptor closed, cable removed, etc). + IoTimeoutError : + When `timeout_sec` elapses without receiving any data. + """ + raise NotImplementedError() + + @abc.abstractmethod + def write_transport(self, data : bytes, timeout_sec : float): + """Write data to the transport. + This function should either write all bytes in `data` or raise an exception. + + Parameters + ---------- + data : bytes + The data to write over the channel. + timeout_sec : Union[float, None] + Number of seconds to wait for all bytes to be written before timing out. If timeout_sec + is 0, write should attempt to service the request in a non-blocking fashion. If + timeout_sec is None, write should block until it has written all data. + + Raises + ------ + TransportClosedError : + When the transport layer determines that the transport can no longer send or receive + data due to an underlying I/O problem (i.e. file descriptor closed, cable removed, etc). + IoTimeoutError : + When `timeout_sec` elapses without receiving any data. + """ + raise NotImplementedError() +``` + +## Project Options + +Each Project API server can return `project_options` as part of the `server_info_query` response. +These can be specified by the user to allow them to give platform SDK-specific options to each API +method. + +``` +ProjectOption = collections.namedtuple('ProjectOption', ('name', 'help')) +``` + +It's expected that user-facing clients of the Project API could expose these either as command-line +flags or e.g. accepting them via a JSON or YAML file. + +## ServerInfo + +In response to a `server_info_query`, an API server should return this structure: + +``` +ServerInfo = collections.namedtuple('ServerInfo', ('platform_name', 'is_template', 'model_library_format_path', 'project_options')) +``` + +Its members are documented below: +- `platform_name`: A unique slug identifying this API server. +- `is_template`: True when this server lives in a template project. When True, `generate_project` can be called. +- `model_library_format_path`: None when `is_template` is True; otherwise, the path, relative to the API server, + of the Model Library Format archive used to create this project. +- `project_options`: list of `ProjectOption`, defined above. + + +## Changes to AutoTVM + +There are two changes to AutoTVM needed to interwork with Project API. They are documented in the sections below. + +### Build Model Library Format artifacts + +At present, the AutoTVM `Builder` creates shared libraries. To interoperate with Project API servers, it needs to +create Model Library Format archives. Currently, only `fcompile` may be given to customize the output format. +`Builder` will accept an additional keyword argument `output_format` which defaults to `so`. When `mlf` is given, +Model Library Format `.tar` will be produced. + +### Introducing `module_loader` to the runner + +Before TVM measures inference time for a given artifact, it needs to connect to a TVM RPC server and load the +generated code. This process will be abstracted behind `module_loader`. The default implementation is as follows: + +``` +def default_module_loader(pre_load_function=None): + """Returns a default function that can be passed as module_loader to run_through_rpc. + Parameters + ---------- + pre_load_function : Optional[Function[tvm.rpc.Session, tvm.runtime.Module]] + Invoked after a session is established and before the default code-loading RPC calls are + issued. Allows performing pre-upload actions, e.g. resetting the remote runtime environment. + Returns + ------- + ModuleLoader : + A function that can be passed as module_loader to run_through_rpc. + """ + + @contextlib.contextmanager + def default_module_loader_mgr(remote_kwargs, build_result): + remote = request_remote(**remote_kwargs) + if pre_load_function is not None: + pre_load_function(remote, build_result) + + remote.upload(build_result.filename) + try: + yield remote, remote.load_module(os.path.split(build_result.filename)[1]) + + finally: + # clean up remote files + remote.remove(build_result.filename) + remote.remove(os.path.splitext(build_result.filename)[0] + ".so") + remote.remove("") + + return default_module_loader_mgr +``` + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +## Choice of `JSON-RPC` + +There were a couple of RPC options considered for this: + +1. JSON-RPC. + Pros: + - Human-readable encoding + - Very simple to implement (could be in bash with `jq`) + - Concise specification + - Packages in several popular languages + Cons: + - Heavyweight encoding + - No streaming facility + - Implementations aren't as cohesively authored as gRPC + - Makes for two RPC implementations checked-in to TVM. + +2. gRPC + Pros: + - Widely supported, compact encoding + - Clearly-documented API and good support forums + - Supports streaming, the most natural way to forward TVM RPC traffic. + Cons: + - Requires the use of another Python package + - Requires the use of an IDL compiler + - Intended use case (datacenter-scale RPC) is overkill. + - Makes for two RPC implementations checked-in to TVM. + +3. TVM RPC + Pros: + - Already exists in TVM + - Some prior art for session forwarding + - Binary encoding + Cons: + - Binary encoding + - Impossible to use today without compiling TVM + - Implementation is designed around TVM's remote inference use cases, and will + likely change as new demands arise there. + +TVM RPC was decided against given the requirement that TVM must be compiled. gRPC was considered, +but ultimately rejected because JSON-RPC can be implemented in a single Python file without adding +the complexities of an IDL compiler. + +## Transport functions Review comment: I'd suggest this tightly coupling autotuning and prototyping to specific frameworks called directly from TVM is the main alternative to using the Project API, inclusive of RPC server and support frameworks? Tightly coupling would be less code and potentially easier user journeys without the consideration of the additional server. Given we can mark dependencies to include as extras, are they a real concern for someone doing `pip install tlcpack` ? ########## File path: rfcs/0008-microtvm-project-api.md ########## @@ -0,0 +1,509 @@ +- Feature Name: microtvm_project_api +- Start Date: 2020-06-09 +- RFC PR: [apache/tvm-rfcs#0000](https://github.com/apache/tvm-rfcs/pull/0000) +- GitHub Issue: [apache/tvm#0000](https://github.com/apache/tvm/issues/0000) + +# Summary +[summary]: #summary + +This RFC describes how TVM integrates with build systems for unconventional platforms, such as those +for microcontrollers and for other bare-metal scenarios. + +# Motivation +[motivation]: #motivation + +Though TVM's primary goal is generating code to implement models from a high-level description, +there are several reasons why a user might want to interact with a platform's build system through +TVM: + +1. To perform autotuning. TVM's internal operator implementations are merely templates and rely on + an automatic search process to arrive at a fast configuration for the template on a given + platform. This search process requires that TVM iteratively build and time code on the platform. +2. To perform remote model execution. A user may wish to try several different models or schedules + on a different platform without rewriting the platform-specific code. Users can do this by + building generated model code against a generic implementation of the microTVM RPC server. +3. To debug model execution remotely. Some aspects of model execution are easy to debug using a + platform-specific debugger; however, some things, such as analyzing intermediate tensor values, + are more easily accomplished with TVM-specific tooling. By leveraging the generic microTVM + RPC server used in (2), TVM can provide such tooling in a platform-agnostic way. + +TVM currently supports these use cases through a set of interfaces: +1. `tvm.micro.Compiler`: used to produce binary and library artifacts. +2. `tvm.micro.Flasher`: used to program attached hardware +3. `tvm.micro.Transport`: used to communicate with on-device microTVM RPC server + +Thus far, implementations of these interfaces have been made for Zephyr, mBED OS, and for +simulated hardware using a POSIX subprocess. The latter two interfaces have proven to be a +relatively good fit; however, `tvm.micro.Compiler` is difficult to implement because it attempts +to replicate a platform's build system in TVM. TVM does not want to incorporate platform-specific +build logic into its codebase. + +This proposal unifies these three interfaces to form a "Project-level" interface, recognizing that +it's typical to interact with unconventional platforms and their build systems at this level. It +simplifies the `Compiler` interaction into a project-level Build, and adds an explicit +`generate_project` method to the interface. These changes remove the need to build components +and drive the link process from TVM. + +As a goal, this proposal aims to allow for the same use cases as are currently supported with these +improvements: + +1. Integrating more naturally with build systems typical of embedded platforms. +2. Allowing TVM to automatically generate projects platforms to define automated scripts to build projects + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +TVM can interact with platform SDKs via its **Project API**. Such SDKs are common when working with +non-traditional OS platforms, such as those commonly used in embedded systems (e.g. Arduino, Zephyr, +iOS). Given a platform-specific implementation of this Project API, TVM can: +1. Generate projects that integrate implemented TVM models with generic platform runtime componeents +2. Build those generated projects +3. Program attached hardware +4. Drive remote model execution via the TVM RPC Server interface. + +This last capability means that TVM can drive autotuning, remotely perform model inference, and debug +models on non-traditional OS such as Arduino, Zephyr, and mobile platforms such as iOS and Android. + +To provide support for a platform, a **template project** is first defined. Template projects are +expected to exist entirely inside a directory and are identified to TVM by the path to the directory +locally. Template projects may live either inside the TVM repository (when they can be included in the +TVM CI) or in other version control repositories. The template project contains at minium an +implementation of the **Project API** inside an executable program known as the +**TVM Project API Server**. + +To begin working with a particular platform's Project API implementation, the user supplies TVM with +the path to the top-level directory. TVM launches an instance the Project API Server (found at a +standard location in that directory). TVM communicates with the Project API Server using JSON-RPC +over standard OS pipes. + +TVM supplies generated code to the Project API Server using [Model Library Format](0001-model-library-format.md). + +Below is a survey of example workflows used with the Project API Server: + +## Generating a project + +1. The user imports a model into TVM and builds it using `tvm.relay.build`. +2. The user supplies TVM with the path to the template project and a path to a non-existent + directory where the generated project should live. +3. TVM launches a Project API server in the template project. +4. TVM verifies that the template project is indeed a template by invoking the Project API server + `server_info_query` method. +5. TVM invokes the Project API server `generate_project` method to generate the new project. + +## Building and Flashing + +1. The user follows the steps under [Generating a project](#generating-a-project). +2. TVM expects the Project API server to copy itself to the generated project. It launches a + Project API server in the generated project directory. +3. TVM verifies that the generated project is not a template by invoking the Project API server + `server_info_query` method. This method also returns `options` that can be used to customize + the build. +4. TVM invokes the Project API server `build` method to build the project. +5. TVM invokes the Project API server `flash` method to program the attached device. The + `options` can be used to specify a device serial number. + +## Host-driven model inference + +1. The user follows the steps under [Generating a project](#generating-a-project). +2. TVM invokes the Project API server `connect_transport` method to connect to the remote on-device + microTVM RPC server. +3. The microTVM RPC server is attached to a traditional TVM RPC session on the host device. +4. TVM drives inference on-device using the traditional TVM RPC methods. The Project API server + methods `read_transport` and `write_transport` are used to receive and send data. +5. When the inference session is over, TVM invokes the Project API server method `close_transport` + to release any underlying I/O resources, and terminates the Project API server. + +## AutoTVM + +1. The user supplies a kernel to the AutoTVM tuner for search-based optimization. +2. AutoTVM generates a set of task configurations, instantiates each task, and then invokes + `tvm.build` to produce a `Module` for each instantiated task. +3. AutoTVM produces a Model Library Format archive from the `Module` for each instantiated task. +4. AutoTVM passes the Model Library Format archive to the AutoTVM `runner`. Project API overrides the + traditional AutoTVM runner by providing a [`module_loader`](#module-loader). The microTVM + `module_loader` connects to a _supervisor_ TVM RPC server which carries out the microTVM project build + as part of the TVM RPC `session_constructor`. The following steps occur in the + `session_constructor`: + + 1. The Model Library Format tar is uploaded to the supervisor. + 2. The user supplies a path, on the supervisor, to a template project. + 3. The supervisor `session_constructor` performs the steps under + [Building and Flashing](#building-and-flashing). + 4. The supervisor `session_constructor` invokes the Project API server `connect_transport` method + to connect to the remote device. The session constructor registers a traditional TVM RPC + session on the supervisor, and this session is also used by the AutoTVM runner due to the + `session_constructor` mechanism. + +5. The AutoTVM runner measures runtime as normal. +6. The AutoTVM runner disconnects the session, closing the Project API server on the supervisor. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Project API implementation + +The Project API is a Remote Procedure Call (RPC)-type mechanism implemented using +[JSON-RPC](https://www.jsonrpc.org/specification). The client and server are implemented in +`python/tvm/micro/project_api`. Tests are implemented in +`tests/python/unittest/test_micro_project_api.py`. + +## Project API interface + +The functions that need to be implemented as part of a Project API server are defined on the +`ProjectAPIHandler` class in `python/tvm/micro/project_api/server.py`: + +``` +class ProjectAPIHandler(metaclass=abc.ABCMeta): + + @abc.abstractmethod + def server_info_query(self) -> ServerInfo: + raise NotImplementedError() + + @abc.abstractmethod + def generate_project(self, model_library_format_path : pathlib.Path, standalone_crt_dir : pathlib.Path, project_dir : pathlib.Path, options : dict): + """Generate a project from the given artifacts, copying ourselves to that project. + + Parameters + ---------- + model_library_format_path : pathlib.Path + Path to the Model Library Format tar archive. + standalone_crt_dir : pathlib.Path + Path to the root directory of the "standalone_crt" TVM build artifact. This contains the + TVM C runtime. + project_dir : pathlib.Path + Path to a nonexistent directory which should be created and filled with the generated + project. + options : dict + Dict mapping option name to ProjectOption. + """ + raise NotImplementedError() + + @abc.abstractmethod + def build(self, options : dict): + """Build the project, enabling the flash() call to made. + + Parameters + ---------- + options : Dict[str, ProjectOption] + ProjectOption which may influence the build, keyed by option name. + """ + raise NotImplementedError() + + @abc.abstractmethod + def flash(self, options : dict): + """Program the project onto the device. + + Parameters + ---------- + options : Dict[str, ProjectOption] + ProjectOption which may influence the programming process, keyed by option name. + """ + raise NotImplementedError() + + @abc.abstractmethod + def open_transport(self, options : dict) -> TransportTimeouts: + """Open resources needed for the transport layer. + + This function might e.g. open files or serial ports needed in write_transport or read_transport. + + Calling this function enables the write_transport and read_transport calls. If the + transport is not open, this method is a no-op. + + Parameters + ---------- + options : Dict[str, ProjectOption] + ProjectOption which may influence the programming process, keyed by option name. + """ + raise NotImplementedError() + + @abc.abstractmethod + def close_transport(self): + """Close resources needed to operate the transport layer. + This function might e.g. close files or serial ports needed in write_transport or read_transport. + + Calling this function disables the write_transport and read_transport calls. If the + transport is not open, this method is a no-op. + """ + raise NotImplementedError() + + @abc.abstractmethod + def read_transport(self, n : int, timeout_sec : typing.Union[float, type(None)]) -> bytes: + """Read data from the transport. + + Parameters + ---------- + n : int + The exact number of bytes to read from the transport. + timeout_sec : Union[float, None] + Number of seconds to wait for at least one byte to be written before timing out. If + timeout_sec is 0, write should attempt to service the request in a non-blocking fashion. + If timeout_sec is None, write should block until all `n` bytes of data can be returned. + + Returns + ------- + bytes : + Data read from the channel. Should be exactly `n` bytes long. + + Raises + ------ + TransportClosedError : + When the transport layer determines that the transport can no longer send or receive + data due to an underlying I/O problem (i.e. file descriptor closed, cable removed, etc). + IoTimeoutError : + When `timeout_sec` elapses without receiving any data. + """ + raise NotImplementedError() + + @abc.abstractmethod + def write_transport(self, data : bytes, timeout_sec : float): + """Write data to the transport. + This function should either write all bytes in `data` or raise an exception. + + Parameters + ---------- + data : bytes + The data to write over the channel. + timeout_sec : Union[float, None] + Number of seconds to wait for all bytes to be written before timing out. If timeout_sec + is 0, write should attempt to service the request in a non-blocking fashion. If + timeout_sec is None, write should block until it has written all data. + + Raises + ------ + TransportClosedError : + When the transport layer determines that the transport can no longer send or receive + data due to an underlying I/O problem (i.e. file descriptor closed, cable removed, etc). + IoTimeoutError : + When `timeout_sec` elapses without receiving any data. + """ + raise NotImplementedError() +``` + +## Project Options + +Each Project API server can return `project_options` as part of the `server_info_query` response. +These can be specified by the user to allow them to give platform SDK-specific options to each API +method. + +``` +ProjectOption = collections.namedtuple('ProjectOption', ('name', 'help')) +``` + +It's expected that user-facing clients of the Project API could expose these either as command-line +flags or e.g. accepting them via a JSON or YAML file. + +## ServerInfo + +In response to a `server_info_query`, an API server should return this structure: + +``` +ServerInfo = collections.namedtuple('ServerInfo', ('platform_name', 'is_template', 'model_library_format_path', 'project_options')) +``` + +Its members are documented below: +- `platform_name`: A unique slug identifying this API server. +- `is_template`: True when this server lives in a template project. When True, `generate_project` can be called. +- `model_library_format_path`: None when `is_template` is True; otherwise, the path, relative to the API server, + of the Model Library Format archive used to create this project. +- `project_options`: list of `ProjectOption`, defined above. + + +## Changes to AutoTVM + +There are two changes to AutoTVM needed to interwork with Project API. They are documented in the sections below. + +### Build Model Library Format artifacts + +At present, the AutoTVM `Builder` creates shared libraries. To interoperate with Project API servers, it needs to +create Model Library Format archives. Currently, only `fcompile` may be given to customize the output format. +`Builder` will accept an additional keyword argument `output_format` which defaults to `so`. When `mlf` is given, +Model Library Format `.tar` will be produced. + +### Introducing `module_loader` to the runner + +Before TVM measures inference time for a given artifact, it needs to connect to a TVM RPC server and load the +generated code. This process will be abstracted behind `module_loader`. The default implementation is as follows: + +``` +def default_module_loader(pre_load_function=None): + """Returns a default function that can be passed as module_loader to run_through_rpc. + Parameters + ---------- + pre_load_function : Optional[Function[tvm.rpc.Session, tvm.runtime.Module]] + Invoked after a session is established and before the default code-loading RPC calls are + issued. Allows performing pre-upload actions, e.g. resetting the remote runtime environment. + Returns + ------- + ModuleLoader : + A function that can be passed as module_loader to run_through_rpc. + """ + + @contextlib.contextmanager + def default_module_loader_mgr(remote_kwargs, build_result): + remote = request_remote(**remote_kwargs) + if pre_load_function is not None: + pre_load_function(remote, build_result) + + remote.upload(build_result.filename) + try: + yield remote, remote.load_module(os.path.split(build_result.filename)[1]) + + finally: + # clean up remote files + remote.remove(build_result.filename) + remote.remove(os.path.splitext(build_result.filename)[0] + ".so") + remote.remove("") + + return default_module_loader_mgr +``` + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? Review comment: I don't think this extra work for those coming directly from an embedded background as they'll likely produce model files directly into their own embedded projects. For example, I wouldn't expect a current Zephyr user to use this flow over using `tvmc` and `west` directly due to their existing familiarity. Though I agree, with the variety of embedded OSes available, this will likely mean TVM as a project has to support a number of OSes - whether directly in checked in project generators or indirectly by features required to support different OSes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
