CoFlows Tutorial 1 - Simple WebAPI in only 4 lines (C#, F#, Java, Scala, JS or Python)
This tutorial shows how to create and host a WebAPI through 4 simple terminal commands using CoFlows CE:
docker run -v $(pwd):/app/mnt coflows/ce add workflow pyflowcd pyflow/bin/sh add.sh query py pyapish server.shLet's dive into a bit more detail to explain the statement above and create our first WebAPI that prints a JSON object to the console and returns the same object with an extra property.
We will start with a description of CoFlows and then continue with how to install the pre-requisites and finish off with creating an API, managing permissions and hosting.
CoFlows CE (Community Edition) is a Containerized Polyglot Runtime that simplifies the development, hosting and deployment of powerful data-centric workflows. CoFlows enables developers to create rich Web-APIs with almost zero boiler plate and scheduled / reactive processes through a range of languages including CoreCLR (C#, F# and VB), JVM (Java and Scala), Python and Javascript. Furthermore, functions written in any of these languages can call each other within the same process with full interop.
Please make sure that you have installed docker with support for Linux containers. If not, you can download it here
https://www.docker.com/products/docker-desktop
Windows users must make sure that docker has support for Linux containers.
To be able to follow this tutorial step-by-step, please install Microsoft VSCode unless you have a favourite IDE. VSCode can be found here
I wrote this tutorial using MacOS meaning the syntax is identical if you are using Linux and very similar if you are using Windows.
Open VSCode and a terminal from within VSCode to see the following:
For those that are not used to working with VSCode, once you open VSCode, choose "Open Folder" and select a folder that you want to work from. Then open a terminal from the top menu. If this is confusing perhaps you should follow a quick VSCode tutorial here
https://code.visualstudio.com/docs
Create a new workflow (CoFlows Project) called pyflow by running the following command in the terminal:
docker run -v $(pwd):/app/mnt coflows/ce add workflow pyflow A folder is then created with the necessary file structure for CoFlows CE to work:
This file structure contains the necessary configuration files and scripts that make interacting with CoFlows CE very easy. Enter the bin folder which is where the helper scripts are located in order to start adding functionality to the new pyflow workflow
cd pyflow/bin/The bin folder contains a variety of scripts to build, deploy, host and run the CoFlows functionality. To add a new WebAPI, we must create a Query which in the context of CoFlows is a collection of functions that are automatically exposed as WebAPIs. Queries also define permissions and generate OpenAPI specs. A Query can be thought of as a Controller in an MVC context but Queries have much richer features.
sh add.sh query py pyapiif you are using windows you the following command instead
cd bat #placing you in pyflow/bin/batadd.bat query py pyapiAlternatively, we could have created a Query in any of the supported languages using
sh add.sh query (cs, fs, py, java, scala, js, vb) {name of query}The result is a new sub folder called Queries with a new python file called pyapi.py
Here is where the magic happens. The pyapi.py file defines the CoFlows Query which contains all the functions and meta code to manage permissions and generate the OpenAPI spec.
The initial section defines the description used to generate the OpenAPI spec. These parameters should be changed to fit your custom code.
### <info version="1.0.100">
### <title>Python Query Test API</title>
### <description>Python Query API with samples for permissions, documentation and function definitions</description>
### <termsOfService url="https://www.coflo.ws"/>
### <contact name="Arturo Rodriguez" url="https://www.coflo.ws" email="arturo@coflo.ws"/>
### <license name="Apache 2.0" url="https://www.apache.org/licenses/LICENSE-2.0.html"/>
### </info>Each function also has meta code used to generate the OpenAPI spec and define permissions. One of the functions in this example Adds two numbers:
### <api name="Add">
### <description>Function that adds two numbers</description>
### <returns>returns an integer</returns>
### <param name="x" type="integer">First number</param>
### <param name="y" type="integer">Second number</param>
### <permissions>
### <group id="$WID$" permission="read"/>
### </permissions>
### </api>
def Add(x, y):
return int(x) + int(y)When creating this code on your end, you will notice that $WID$ is exchanged for an actual GUID which is a random ID that can look something like this "9a7adf48-183f-4d44-8ab2-c0afd1610c71".
The declaration of the permissions is done in the snippet below:
### <permissions>
### <group id="$WID$" permission="read"/>
### </permissions>This section declares that a user must be a member of group $WID$ and at least have a permission of type Read. It is possible to add multiple groups and assign different permissions to each group. User's are deemd to have access if they are part of a group and at least have the specified permission in this group.
There are four different permissions:
Denied (-1)
View (0)
Read (1)
Write (2)
The numbers linked to each permission defines the permission's value. In the example above, the minimum permission a user must have in the $WID$ group is Read. If a user is not part of the group, the user has no access. If the user has the View permission, the user has no access because View < Read. If the user has the Read or Write permission, then the user has access and is allowed to call the API.
More on permissions and how to add users to permission groups can be read in the following tutorial.
On a side note, perhaps you noticed that the parameters are being transformed to int in the example above:
return int(x) + int(y)this is because CoFlows passes all parameter values from the web call as strings and need to be managed by the developer accordingly.
Execute
To execute this workflow simply run
sh server.shor if you are using windows
server.batThis will generate the following output
CoFlows CE - NetCoreApp 3.1... Python starting... QuantApp Server 11/01/2020 12:17:17DB Connected
Local deployment
Workflow-Clean started
SSL encryption is not used....
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
No XML encryptor configured. Key {5ce74f79-603d-4145-bd52-f3f3b2489778} may be persisted to storage in unencrypted form.Once this happens, please enter the http://localhost through your browser to see
Login with
Username: root
Password: 123
leading you to
where you will click on pyflow to enter the workflow page which shows you all of the Queries in this workflow:
In this example you will see the only Query that was created. Clicking on the pyapi.py will show you and IDE where you can both view and edit the source code of the Query.
Please not that changing the code in this UI WILL change the code in your local file and vice versa!
Finally you will be able to run the query to see the values of all apis / functions that take NO paramteres. In the this case it is only the Permission API since Add requires two parameters:
The end-points to these APIs are now visible and are for the Permission API:
and for the Add API:
where X and Y are any numbers. hUsing curl you are able to test this API is h. For the sake of security, it is preferable to include the
curl "http://localhost/flow/query/9a7adf48-183f-4d44-8ab2-c0afd1610c71/pyapi.py/Add?p[0]=1&p[1]=2&_cokey=26499e5e555e9957725f51cc4d400384"The cokey is a secret key used to authenticate a user without the need to login. These keys can also be added in the Header which is a more secure way of using them. More information about the cokey is available here.
New WebAPI
The new WebAPI we are adding a Print function that
takes a serialised JSON object as a parameter
deserialises the parameter and prints it in the console
adds a property to the object and then prints it again
returns the new object to the caller of the WebAPI
In order to achieve this, start by adding the following function
import json
### <api name="Print">
### <description>Function that prints a json</description>
### <returns>returns an integer</returns>
### <param name="x" type="string">JSON object</param>
### <permissions>
### <group id="$WID$" permission="read"/>
### </permissions>
### </api>
def Print(x):
js = json.loads(x)
print('------ Initial Object')
print(js)
js['NewProperty'] = 'HELLO'
print('------ New Object')
print(js)
return jsStop the server by pressing Control-C and then start it again. This will reload the new function and make it available as a WebAPI.
We can call this WebAPI through Curl to see the result:
curl -d '{"key":"value"}' -H "Content-Type: application/json" -H "_cokey: 26499e5e555e9957725f51cc4d400384" -X POST -g "http://localhost/flow/query/9a7adf48-183f-4d44-8ab2-c0afd1610c71/pyapi.py/Print"Please note that the workflow ID 9a7adf48-183f-4d44-8ab2-c0afd1610c71 in the example above will need to be changed for the ID automatically generated on your end. This ID can be found in the package.json file or in the permissions section of the previous functions.
This curl command also shows you how to add the cokey (secret key) in the header instead of the URL for the sake of increased security.
The output on the terminal that is executing the CoFlows server will be:
------ Initial Object
{'key': 'value'}
------ New Object
{'key': 'value', 'NewProperty': 'HELLO'}and the output of the curl command is:
{"key": "value","NewProperty": "HELLO"}Conclusion
We have now gone through the steps required to create and configure a new WebAPI through CoFlows CE. For further information please go to the GitHub page.