Service compilation & deploy

The goal of this document is to explain the evolution of the deploy api and the introduction of the compilation of service.

The idea is to have a single way to deploy / publish service to the Engine but also to the Marketplace.

The Marketplace reveals that in order to deploy a service to another Engine, not only the source code but also the service definition are important and useful.

Yes, the service definition is also (currently) contained in the source code as a mesg.yml file. But, in order to have a general purpose system and to fit in the long time vision of MESG, the source code should become optional (eg: pure workflow services), so services can be published / deployed / started with only a service definition.

The current implementation of the Marketplace requires a JSON payload called manifest data containing:

  • The service’s definition. Not the mesg.yml version but the protobuf version exposed by the GetService API:
    • The service’s hash is present (and not in the mesg.yml)
  • The service’s source code (as an IPFS hash)
  • The service’s documentation (currently the content of README.md)

By keeping only the service’s definition and the service’s source code, any Engine should be able to run the specified service without any other required data from the User.

Currently, to start a service, 2 steps / api calls are needed:

  • Deploy the service source code to the Engine by:
    • Generating the service definition from the mesg.yml
    • Calculating the service’s hash
    • Building the docker image
  • Start the actual service based on the service’s hash (or the service’s sid) by:
    • Creating the actual docker services and containers using the service definition, the hash and the build docker image

To publish on the marketplace, 5 steps / api calls are required that:

  • Generates the service definition and hash from the mesg.yml (by calling the deploy api)
  • Get the service definition (by calling the getService api)
  • Publish the source on IPFS
  • Create the manifest data from the service definition and the IPFS hash
  • Finally, publish the manifest data to the marketplace

Service definition

The service definition is a protobuf structure containing all necessary information to start a service on any Engine.

  • The only modification to do on the service definition is to add a link to the service’s source.

The service definition is the compiled version of the current mesg.yml. The Engine should be only compatible with service definition and not with mesg.yml anymore.

This gives a lot of flexibility to improve or even replace the mesg.yml. The service definition could be compiled from anything: HCL, syntactic analysis to generate it directly from source code, source code comments, GUI, etc…

Compile (path to source) -> service definition

The most different function is the compilation of a service.

This function compiles a service definition from local folder containing the service source.

This function is very similar to the compilation of an Ethereum Smart Contract: the source code is transformed to an ABI (in our case, service definition) and to bytecode (in our case, the link to the archive containing the source code).

This function should run completely independently from the Engine. A light software without internet and connection to an Engine should be able to compile a service.

The steps of this function are:

  • Generates the service definition from the mesg.yml
    • Validate the mesg.yml
  • Upload the source to IPFS
  • Injecting the source link to the service definition
  • Returning the service definition
    • Could be returned as raw protobuf (byte / hex representation) or as JSON for readability

Deploy (service definition) -> service hash

This api is very similar to the current deploy api except that it takes as parameter directly the service definition (instead of the tar of the service source code ) and will returned the service hash.

The steps of this api are:

  • Parse the service definition
  • Download the service source
  • Calculate the service hash from the service definition AND the source code
  • Save service definition in service DB (returns error if existing service definition)
  • Return the service hash

Note

  • The service hash is not calculated from the custom env anymore. This will come back with the introduction of the Instance DB. This has the consequence to allow only 1 service (same service definition) to run at a specific time even with different custom env variable.

Publish

The current command marketplace publish will be drastically simplified by receiving directly the service definition containing most of the required information to publish on the current Marketplace running on Ethereum.

The evolution of this function is that the future Marketplace will be directly integrated into the decentralized Service DB. I will not explain the details of this future function as we will implement it only when the decentralized Service DB is created. But basically, this future function will be able to publish on the whole network just based on the service definition generated by the Compile function.

Schema

In orange, step that are different than current implementation

Deploy

graph TD 1[Parse the service definition] --> 2 2[Download the service source code] --> 3 3[Calculate the service hash] --> 4 4{Check existing hash in Service DB} -- if no exist --> 5 4 -- if exist --> error 5[Save service definition in service DB] --> 6 6[Return the service hash] error[return error] classDef diff fill:orange; class 1,2,3 diff;

gRPC definition, server & sdk

This feature should not modify any existing gRPC definition, server or sdk functions but rather create new ones to start from a fresh and clean mind even if code duplication is necessary :wink:

gRPC

A new gRPC api should be created to introduce the beginning of the separation of the gRPC api by ressources.

EDIT #1

This new gRPC api should be created in /protobuf/api/service.proto and contain the following protobuf def:

syntax = "proto3";

import "protobuf/definition/service.proto";

package api;

service Service {
  rpc Create (definition.Service) returns (string) {}
}

This api Service will only manage resources in Service DB and use a CRUD-like design.

The current apis Core.DeployService, Core.DeleteService, Core.ListServices, Core.GetService will be “duplicated” to this new service.proto file and rename to a more CRUD-like style (eg: Create, Delete, List, Get) in another feature / PR.


EDIT of June 4th: I fix the service hash calculation. It should be calculated from the service definition and the source code.

A small precision on the service hash calculation from https://forum.mesg.com/t/instance-db/295/6: