Changeset: 47410ebe674d for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=47410ebe674d Added Files: clients/iotapi/documentation/api_server_arguments.rst clients/iotapi/documentation/conclusion.rst clients/iotapi/documentation/conf.py clients/iotapi/documentation/index.rst clients/iotapi/documentation/introduction.rst clients/iotapi/documentation/websockets_api.rst Removed Files: clients/iotclient/README Modified Files: clients/iotapi/src/Settings/filesystem.py clients/iotapi/src/Streams/datatypes.py clients/iotapi/src/Streams/streampolling.py clients/iotapi/src/Streams/streams.py clients/iotapi/src/Streams/streamscontext.py clients/iotapi/src/WebSockets/jsonschemas.py clients/iotapi/src/WebSockets/websockets.py clients/iotapi/src/main.py clients/iotclient/documentation/conclusion.rst clients/iotclient/documentation/introduction.rst clients/iotclient/documentation/iot_server_arguments.rst clients/iotclient/documentation/restful_resources.rst clients/iotclient/src/Settings/filesystem.py clients/iotclient/src/Streams/streams.py clients/iotclient/src/Streams/streamscontext.py Branch: iot Log Message:
Web api working with iot server. Started adding documentation for web api, rectified the websocket protocol to be used. Fixed an error on binary conversion. diffs (truncated from 987 to 300 lines): diff --git a/clients/iotclient/documentation/iot_server_arguments.rst b/clients/iotapi/documentation/api_server_arguments.rst copy from clients/iotclient/documentation/iot_server_arguments.rst copy to clients/iotapi/documentation/api_server_arguments.rst --- a/clients/iotclient/documentation/iot_server_arguments.rst +++ b/clients/iotapi/documentation/api_server_arguments.rst @@ -1,12 +1,12 @@ -.. _starting_webserver: +.. _starting_webapi: -*********************** -Starting the web server -*********************** +******************** +Starting the web api +******************** -The packages listened on requirements.txt must be installed before running the server. To avoid conflicts with other versions of the packages existing on the host machine, is recommended to create a Python virtual environment for the server. `About Python virtual environments <http://docs.python-guide.org/en/latest/dev/virtualenvs/>`_. +The packages listened on :code:`requirements.txt` file must be installed before running the server. To avoid conflicts with other versions of the packages existing on the host machine, is recommended to create a Python virtual environment for the server. `About Python virtual environments <http://docs.python-guide.org/en/latest/dev/virtualenvs/>`_. -The server starts as a regular Python program invoking the :code:`main.py` file, however there are several arguments that can be passed to change its behaviour. +The server starts as a regular Python program invoking the :code:`main.py` file, however there are several arguments that can be passed to change its behavior. :: @@ -26,45 +26,29 @@ Paths **-f - -filesystem=** -Set the filesystem directory where the baskets will be created. By default in UNIX systems is on :code:`/etc/iotcollector` directory, while on Windows is on the directory where the :code:`main.py` script was invoked. While running the web server creates a :code:`baskets` directory. +Set the filesystem directory where the baskets will be created. By default in UNIX systems is on :code:`/etc/iotapi` directory, while on Windows is on the directory where the :code:`main.py` script was invoked. **-l - -log=** -Location of logfile. On the logfile is reported when streams are created or removed, when tuples are inserted and when the baskets are flushed. By default in UNIX systems is :code:`/var/log/iot/iot.log`, while on Windows is the :code:`iot.log` on the directory where the :code:`main.py` script was called. +Location of logfile. On the logfile is reported when streams are created or removed, when tuples are inserted and when the baskets are flushed. By default in UNIX systems is :code:`/var/log/iot/iotapi.log`, while on Windows is the :code:`iotapi.log` on the directory where the :code:`main.py` script was called. -Host Identifier ---------------- -If the *useidentifier* parameter is provided, an extra column on streams will be added with a custom name of the host for later identification. +Web API Listening and Behavior +------------------------------ -**-ui - -useidentifier** +Customize the Web API server parameters and behavior. -Use a host identifier for every new stream. - -**-in - -name=** - -Host identifier name. By default is the host's MAC address. - -Web Server Listening --------------------- - -For security purposes, two web servers are created by the application. On the administration server it's present operations to create and delete streams, while on application server the insert operations are present instead. The administration server should be listening on the host only, while the application server should be listening to all interfaces. - -**-ih - -ihost=** +**-sh - -shost=** Listening host of the application (IOT) server. By default is on :code:`0.0.0.0`. -**-ip - -iport=** +**-sp - -sport=** -Listening port of the application (IOT) server. By default is on port :code:`8000`. +Listening port of the application (IOT) server. By default is on port :code:`8002`. -**-ah - -ahost=** +**-pi - -polling=** -Listening host of the administration server. By default is on :code:`127.0.0.1`. - -**-ap - -aport=** - -Listening port of the administration server. By default is on port :code:`8001`. +Set the polling interval to MonetDB database for updates. By default is :code:`60` seconds. Database Connection ------------------- diff --git a/clients/iotclient/documentation/conclusion.rst b/clients/iotapi/documentation/conclusion.rst copy from clients/iotclient/documentation/conclusion.rst copy to clients/iotapi/documentation/conclusion.rst --- a/clients/iotclient/documentation/conclusion.rst +++ b/clients/iotapi/documentation/conclusion.rst @@ -7,9 +7,7 @@ Final Acknowledgements Future Work =========== -Currently the project is on a testing phase. More data types will also be added. - -Soon another web server will be built to implement an API for the baskets outputs from DataCell. +Currently the project is on a testing phase. .. _about-the-project: diff --git a/clients/iotclient/documentation/conf.py b/clients/iotapi/documentation/conf.py copy from clients/iotclient/documentation/conf.py copy to clients/iotapi/documentation/conf.py --- a/clients/iotclient/documentation/conf.py +++ b/clients/iotapi/documentation/conf.py @@ -1,6 +1,6 @@ # -*- coding: utf-8 -*- # -# MonetDB IOT WebServer documentation build configuration file, created by +# MonetDB IOT WebApi documentation build configuration file, created by # sphinx-quickstart on Wed Apr 20 09:44:47 2016. # # This file is execfile()d with the current directory set to its @@ -55,7 +55,7 @@ source_suffix = '.rst' master_doc = 'index' # General information about the project. -project = u'MonetDB IOT WebServer' +project = u'MonetDB IOT WebApi' copyright = u'2016, MonetDB Solutions' author = u'Pedro Ferreira' @@ -130,7 +130,7 @@ html_theme = 'sphinx_rtd_theme' # The name for this set of Sphinx documents. # "<project> v<release> documentation" by default. -#html_title = u'MonetDB IOT WebServer v1' +#html_title = u'MonetDB IOT WebApi v1' # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None @@ -212,7 +212,7 @@ html_static_path = ['_static'] #html_search_scorer = 'scorer.js' # Output file base name for HTML help builder. -htmlhelp_basename = 'MonetDBIOTWebServerdoc' +htmlhelp_basename = 'MonetDBIOTWebApidoc' # -- Options for LaTeX output --------------------------------------------- @@ -234,7 +234,7 @@ latex_elements = { # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ - (master_doc, 'MonetDBIOTWebServer.tex', u'MonetDB IOT WebServer Documentation', + (master_doc, 'MonetDBIOTWebApi.tex', u'MonetDB IOT WebApi Documentation', u'Pedro Ferreira', 'manual'), ] @@ -264,7 +264,7 @@ latex_documents = [ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ - (master_doc, 'monetdbiotwebserver', u'MonetDB IOT WebServer Documentation', + (master_doc, 'monetdbiotwebapi', u'MonetDB IOT WebApi Documentation', [author], 1) ] @@ -278,8 +278,8 @@ man_pages = [ # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ - (master_doc, 'MonetDBIOTWebServer', u'MonetDB IOT WebServer Documentation', - author, 'MonetDBIOTWebServer', 'One line description of project.', + (master_doc, 'MonetDBIOTWebApi', u'MonetDB IOT WebApi Documentation', + author, 'MonetDBIOTWebApi', 'One line description of project.', 'Miscellaneous'), ] diff --git a/clients/iotclient/documentation/index.rst b/clients/iotapi/documentation/index.rst copy from clients/iotclient/documentation/index.rst copy to clients/iotapi/documentation/index.rst --- a/clients/iotclient/documentation/index.rst +++ b/clients/iotapi/documentation/index.rst @@ -1,10 +1,10 @@ -.. MonetDB IOT WebServer documentation master file, created by +.. MonetDB IOT WebApi documentation master file, created by sphinx-quickstart on Wed Apr 20 09:44:47 2016. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Welcome to MonetDB IOT WebServer's documentation! -================================================= +Welcome to MonetDB IOT WebAPI's documentation! +============================================== Contents: @@ -14,7 +14,6 @@ Contents: :caption: Table of Contents introduction.rst - iot_server_arguments.rst - restful_resources.rst - streams_data_types.rst + api_server_arguments.rst + websockets_api.rst conclusion.rst diff --git a/clients/iotclient/documentation/introduction.rst b/clients/iotapi/documentation/introduction.rst copy from clients/iotclient/documentation/introduction.rst copy to clients/iotapi/documentation/introduction.rst --- a/clients/iotclient/documentation/introduction.rst +++ b/clients/iotapi/documentation/introduction.rst @@ -4,10 +4,10 @@ Introduction ************ -This application is a Python web server destined to extend MonetDB RDBMS to the Internet-of-Things while using a streaming engine. This project is derived from Pedro Ferreira master thesis, a junior researcher at CWI. The objective of the master thesis to re-evaluate the DataCell extension of MonetDB to the IOT world, which was postponed for several years since its release. To accomplish this, an IOT topology around MonetDB is being built, where this server aims to analyse the input of streams. Later it will also be included another web server to analyze the output of streams. +This application is a Python web server destined to extend MonetDB RDBMS to the Internet-of-Things while using a streaming engine. This project is derived from Pedro Ferreira master thesis, a junior researcher at CWI. The objective of the master thesis to build a streaming extension of MonetDB to the IOT world, which is obtaining much demand nowadays. To accomplish this, an IOT topology around MonetDB is being built, where this server aims to analyse the input of streams. At the same time, another web server is being built to analyze the streams' output. -This web server is built using Python programming language, version 2.7, using `Flask-RESTful <https://pypi.python.org/pypi/Flask-RESTful>`_ framework. The required packages on requirements.txt should be installed before running the server. +This web server is built using Python programming language, version 2.7, using a `Simple Websockets server <https://github.com/dpallot/simple-websocket-server>`_ . The required packages on requirements.txt should be installed before running the server. -The web server is capable of creating and deleting streams for the renewed DataCell extension. After a stream is created, it's possible to make batch inserts on it. The requests are made using RESTful requests with JSON content. Both stream creation and stream insertion are validated using `JSON Schema Draft 4 <http://json-schema.org/documentation.html>`_. On a batch insert if a tuple is invalid, then no tuples are inserted. The stream engine adds an implicit timestamp column to recognize when the tuple was inserted. The inserted tuples are later evaluated using MonetDB's relational engine with continuous queries. +The server creates a polling connection with MonetDB, listening for every new output queries. The Streaming context will then be constantly updated during the polling connection. A small publisher/subscriber pattern was added to the server where the clients can to subscribe to notified right away when an output basket is created. It is also possible to perform small queries on the created data for pagination while including an offset or a limit of the number of tuples to retrieve. All the communication process is assured using a full-duplex WebSockets connection. -The following chapters explain the server parameters, RESTful resources and how to manage the streams. +The following chapters explain the server parameters and the Websockets protocol used. diff --git a/clients/iotapi/documentation/websockets_api.rst b/clients/iotapi/documentation/websockets_api.rst new file mode 100644 --- /dev/null +++ b/clients/iotapi/documentation/websockets_api.rst @@ -0,0 +1,68 @@ +.. _websockets_api: + +************** +WebSockets API +************** + +Bellow is listed the available request and response messages for the WebSockets API. + +Requests +======== + +The client must always provide a JSON string in a request with a request field indicating the intended action to perform in the server, followed by the other specific fields depending on the request. + +.. important:: The :code:`request` field on the JSON request must always be lowercase! + +The following sections explain the available :code:`request` fields. + +sub +--- + +Subscribes for new basket creations from a specific stream. Whenever a basket is created, the server sends a notification message indicating the number of inserted tuples in the new basket. The user has to specify the stream's name and schema. + +.. code-block:: json + + { + "request": "sub", + "schema": "measures", + "stream": "temperature" + } + +unsub +----- + +Unsubscribes a previous subscribed stream for a client. The user has to specify the stream's name and schema. The example is the same as above, just changing the request keyword. + +read +---- + +Reads output result from baskets generated by a stream. + +.. important:: The user has not to be subscribed to the stream in order to read data from it! + +It's possible to provide an offset, a limit and a basket number where the read should start. The request will always provide a result, even if the query provides no tuples to read. The user has to specify the stream's name and schema. + +.. code-block:: json + + { + "request": "read", + "basket": 2, + "offset": 10, + "limit": 100 + } + +info +---- + +Retrieves information about a giving stream if a stream's name and schema are provided, or all the existing streams in the system otherwise. + +.. code-block:: json + + { + "request": "info", + "schema": "measures", + "stream": "temperature" + } + +Responses +========= diff --git a/clients/iotapi/src/Settings/filesystem.py b/clients/iotapi/src/Settings/filesystem.py --- a/clients/iotapi/src/Settings/filesystem.py +++ b/clients/iotapi/src/Settings/filesystem.py @@ -11,7 +11,7 @@ def init_file_system(new_location=None): if new_location is None: if sys.platform in ("linux", "linux2", "darwin"): - new_location = '/etc/iotcollector' + new_location = '/etc/iotapi' elif sys.platform == "win32": new_location = os.path.join(os.path.dirname(__file__), os.pardir) else: @@ -22,7 +22,7 @@ def init_file_system(new_location=None): _______________________________________________ checkin-list mailing list checkin-list@monetdb.org https://www.monetdb.org/mailman/listinfo/checkin-list
