Introducing Workflow Files

proposal

#1

Abstract

One of the goals of MESG is letting people to create applications without coding a single line of code!

Before digging into this idea, let’s go a little back in the history where computers and internet are just yet evolving. From the beginning of early times we never had a way of creating reusable services that communicates with each other with a standardized communication protocol. Because of this negative adoption from back, even if we use microservices in today’s world, they have to be refactored each time to be compatible with just another service communication protocol used in new projects. And this a great waste on human resources!

MESG solves this problem by defining a common communication protocol for services. This way, it’s now possible to use your existent services in totally different projects without a need of changing your source code.

MESG is an event-oriented framework where applications only needs to react to various events and task results published by services to execute another chain of tasks from various services.

We connect MESG services by creating applications which defines the way how data should flow between services. Currently, we do this by using application clients in any programming language. With the workflow files that’s now being introduced within this proposal, creating applications will be done in the most simplistic way just by creating a configuration file in yaml, without using an actual programming language!

You can even create this workflow files via a user interface where you connect some data dots from various services that exists in the market and it can create a workflow.yaml for you!

Workflow Files

Workflow files is a place where we define our constants, secrets, service ids that we want to use and the way how event data and task results should be piped to execution of tasks in these services.

As mentioned before, workflow files are all about describing how your data should flow between services. This is why we treat them as just configuration files. You don’t need an actual programming language to create your applications.

Proposal

In this proposal we need to discuss about the syntax of workflow files and how they should managed by core. As @Anthony suggested, workflow files doesn’t required to be run inside consumers’ computers. Instead they can directly run inside core (actually now as a system service). Since core can be run in a decentralized way as multiple peers, applications also can benefit from this and get load balanced naturally.

Running Workflows

Workflow System Service (WSS)

We’re going to create a new system service for core to manage and run workflows.

@Anthony suggested before to listen all events and results from all services that connected to core for once and then react to these data by following the rules defined inside workflow files to execute corresponding tasks. But doing it this way instead of figuring out service ids directly from workflows introduces unnecessary complexity. Because this requires to add more tasks to WSS and watch services connected to core to get live service id events which may require some code changes on core itself too. And I cannot see any pros of doing it that way.

So instead, we can figure out the service ids directly from the workflow files when they’re created and listen events and results on those services to make task executions.

For the running part part, I propose two different ways of running workflows:

  • Create the WSS to manage and run all workflows.
  • Create the WSS to manage all workflows but run them as different services like we do with MESG services. This is a good for isolation of workflows and better for scaling of different types of workflows.

WSS will have following tasks & events on itself:

  • task: createWorkflow
  • task: validateWorkflow
  • task: removeWorkflow
  • task: updateWorkflow
  • task: inspectWorkflow
  • task: listWorkflows
  • task: watchWorkflowLogs
    • Activates workflowLog event for a workflow. All previous logs messages will be sent in the first workflowLog event.
  • task: unWatchWorkflowLogs
    • Deactivates workflowLog event for a workflow.
  • event: workflowLog

gRPC APIs

We’ll add new gRPC APIs to coreapi for validating, creating, updating, removing, logging, inspecting and listing workflows. Each workflow can get a unique name defined by dev and have a second unique id (hash) calculated from workflow file. This way we’ll be able to identify them while updating, logging, inspecting and removing.

CLI Commands

We’ll have new cli commands as:

  • $ mesg-core workflow create --name optional workflow.yml

    • Saves and runs workflows.
    • Optional name used as a unique id for workflow.
  • $ mesg-core workflow validate workflow.yaml

    • Validates workflow file’s syntax, existence of service’s that defined with their ids, existence of events, results, tasks and validations of their inputs/outputs for these services.

    • It’ll also analyze any potential infinite workflow cycles. For example a service’s task result executes the same task on the same service which ends up with an infinite loop. (Maybe this check should also be made inside core when possible.)

  • $ mesg-core workflow dev workflow.yaml

    • This is a development mode. It creates and logs the workflow outputs to std. It’ll run until it’s canceled and workflow will be removed when quitting the command.
  • $ mesg-core workflow update id-or-name --file workflow.yml

    • Update any content of workflow.
  • $ mesg-core workflow inspect id-or-name

    • See info about workflow like it’s name, description, services, configs, events, results, tasks etc.
    • See the received event, result count and successful/failed execution counts.
  • $ mesg-core workflow list

    • List of workflows and their received event, result count and successful/failed execution counts.
  • $ mesg-core workflow remove id-or-name

    • Stop running workflow and completely remove it from core.
  • $ mesg-core workflow logs id-or-name

    • See the live log stream of workflow. Each workflow log will start with workflow name and description printed first. And all events, results and successful/unsuccessful task executions and their input datas will appear.
  • $ mesg-core workflow init

    • Creates an empty workflow file via a terminal dialog to get inputs from user for workflow name, description, service ids etc. or defaults will be used where it’s possible.

Defining The Workflow Syntax

We can have a custom syntax to create workflow configurations but for now it’s too complicated to have and we currently don’t know if we really need it. So I think, we should stick with an already well know language like yml at first as proposed before.

We currently have two types of workflow.yml syntaxes. One is from @Anthony which I really like to hear the concept behind it and other one is like more close to Application Client API Spec that we have which brings us to something like below:

Sample Service

If you’d like to test this service please install prototype workflow cli first and then deploy webhook and discord invite services.

Update the service ids, email & SendGrid api key configuration.

After that run your workflow with $ mesg-workflow run discord-invites-workflow.yml and watch the logs. You’ll receive an email when you run the curl command in the description!

name: discord-invites

description: |
  Send discord invites to your fellows.

  curl -d "email=your@email.com" -XPOST http://localhost:3000/webhook

services:
  webhook: 4f7891f77a6333787075e95b6d3d73ad50b5d1e9
  discord: 1daf16ca98322024824f307a9e11c88e0aba55e2

configs:
  sendgridAPIKey: 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

when:
  webhook:
    event:
      request:
        map:
          email: $data.data.email
          sendgridAPIKey: $configs.sendgridAPIKey
        execute:
          discord: send

Logs will look like this:

✔ discord-invites workflow started
Send discord invites to your fellows.

curl -d "email=ilkergoktugozturk@gmail.com" -XPOST http://localhost:3000/webhook

>> event request received on webhook service, execution data will be:  {
  "email": "ilkergoktugozturk@gmail.com",
  "sendgridAPIKey": "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"
}
<< execution successfully made for send task on discord service

Schema

Note that schema can be slightly different from the sample service above because of the improvements.

name: *display name of the workflow*

description: |
  *description of the workflow*

# services are constants.
# accessed via $services variable.
#  e.g.: $services.*alias*
services:
  *a chosen unique service name alias*: *service id*

# configs are constants.
# accessed via $configs variable.
#  e.g.: $configs.*value*.*can*.*be*.*nested*
# they can be in any type.
configs:
  *configuration key*: *configuration value/can be a nested object*

# start of workflow.
when:
  *service alias*:
    event:
      *event name*:
        map:
          # *data* can be set directly or it can be set from $services, $configs and $event.
          # $event is a special run time variable where it is filled with event info.
          # $event consists of $event.key and $event.data.    
          # use dot notation to access individual fields of $event.data.
          *task input data*: *data*
        tags:
          # associate tags with executions.
          # tags can be set directly or it can be set from $services, $configs and $event.
          # $event is a special run time variable where it is filled with event info.
          # $event consists of $event.key and $event.data.    
          # use dot notation to access individual fields of $event.data.
          - *tag*
        execute:
          *service alias*: *service task key*
    result:
      *result name*:
        tagFilters:
          - *tag filter*
        map:
          # *data* can be set directly or it can be set from $services, $configs and $result.
          # $result is a special run time variable where it is filled with result info.
          # $result consists of $result.key, $result.data, $result.taskKey and $result.executionTags.    
          # use dot notation to access individual fields of $result.data.
          *task input data*: *data*
        tags:
          # associate tags with executions.
          # tags can be set directly or it can be set from $services, $configs and $result.
          # $result is a special run time variable where it is filled with result info.
          # $result consists of $result.key, $result.data, $result.taskKey and $result.executionTags.    
          # use dot notation to access individual fields of $result.data.
          - *tag*
        execute:
          *service alias*: *service task key*

Conclusion

@core team please give feedbacks about the following topics:

  1. Which syntax we’re gonna pick? @Anthony’s or the other one we already have an example for or something else?

  2. How do you feel about these new gRPC APIs and CLI commands?

  3. Do we want to be able to execute multiple tasks on an event or result? We already have good syntax for this one, please check the execute attr in the yaml file, it can be naturally extended.

  4. How we should implement filters for events and results? Should we introduce comparison primitives like eq, gte for comparing values? Or should we avoid having a filtering syntax for a simpler use for non-programmers and create a special service that does these comparisations. For example, when an event or result arrived, it’s data can be forwarded to a special service to decide whether the actual task execution should be made or not. If it should, service can fire an event with the data for starting the task execution but this kinda use can add more complexity to event-oriented programming.

  5. How we should make it possible to compose multiple data together to create new values to be used as task inputs or execution tags. This is a similar question with having the filters. If we want to dynamically create new values from $event, $result, $services, $config or/and static values, what kind of primitives we should introduce to this syntax or do this kinda stuff with special services?

    • e.g. syntax for multiplying:
      • taskInputField: $event.data.x * $event.data.y.
    • e.g. combine static value with dynamic:
      • taskInputField: "string_prefix_" + $event.data.x.
    • e.g. combine constant value with dynamic:
      • taskInputField: $configs.xPrefix + $event.data.x.
  6. What do you think about the architecture/tasks/events of WSS? Should we watch events, results from all services in core for once or go more logicless and get service ids from workflows on creation and listen events & results dynamically as described there. Other question is, should we make WSS to manage and run all workflows or just leave the managing part on WSS and run workflows as separate docker services. Which way to go?

  7. Any other thing in your mind, let’s share ideas. :slight_smile:


#2

Thanks for this post it’s really nice :slight_smile:

Syntax

I like your syntax, it’s really close to the api that we have right now and this is something nice. I have one concern though. We need in the future to be able to chain the executions. In my first proposition I was thinking to flat all the executions and resolve the dependencies. I’m afraid with your syntax it might be hard to chain executions.
It’s something we should think about. Maybe something like that:

when:
  serviceX:
    event:
      eventX:
        execute:
          nameofexecution:
            serviceY: taskY
            result:
              resultY:
                map:
                  foo: $nameofexecution.resultY.outputY
                  bar: $event.dataX
                execute:
                  nameofexecution2:
                    serviceZ: taskZ
                    ...

With something like that we could even flatten all the executions and resolve them based on the data they need. This might be too much to implement for now but I just want to have a syntax that we will easily be able to migrate to adopt that.

New APIs

All good for that, I would just remove the update part, let’s keep it simple for now, we delete and create a new one like the services. We can have an id system on top of that later on to mimic an update.
I would be careful to have a consistant naming between the service and workflow actions like remove vs delete

Multiple task executions

Totally necessary, it’s kind of related to my first point but I think here you are more talking about executing them all in parallel and not chained which is something that we should cover too but we will have the same problems. The execution can be extended but the mapping might be totally different and this is why I think we should group the mapping inside the execution part.

Filters

This one is tricky, especially if we want something simple. We definitely need a filter system, filters that for me should be done based on all the data from the execution (data, tag, outputKey) but also the one from the parent execution (in case of nested executions). For the kind of filters at least the equal is necessary and all the other primitives should be perfect but for now, like you propose we can use services for that. Let’s make sure that we have something where we will be able to add the filters but we can now have special services for that.

Data composition

I think this one is too much, I would recommend to go with a service for that, we will never be able to cover the different needs for that so let’s not try I think

Architecture

I think we should have something always reacting from event’s services. For now we can have something simple and listen for the api that we already have based on the workflow informations, basically what you’ve already did. But we should have all these workflow informations in a database and for every events request this database to see if we need to execute a task. This way we remove all “listening” part that is not really scalable and hard to manage.


In conclusion, it’s really nice and for now we can use the system of listeners but we should keep in mind that this will evolve with a database (even distributed database) and also the syntax needs to be “future friendly”. I really think we should name the executions and do the processing inside these executions, that way we will be really flexible but I might be biased by my previous researches. Definitely open to rethink that.


#3

I would split inspect into two

  • first for getting workflow definition workflow get-def id-or-name
  • second for inspecting workflow inspect id-or-name

Because definition of workflow is static resouces and everthing else is more dynamic one.

Except that everything is ok.

We should provide an option to execute multiple tasks

See new proposition (triggers.when.outputs). Also, we should avoid manipulate outputs because we will need to create kind of DML for json (I haven’t seen a successful project for it).

First let’s set up some proposition on workflow file, then we could talk about the arch of it.

So I have such proposition:

# name of the workflow
name: email-notification

# description of the workflow
description: Workflow for notify when email is sent

# services aliasas that cloud be accessible in the action
services:
  - email: 6b0884a06e169c095ed8c412c3afc398
  - slack: 474cb31a6264142684d314d6f2ec650a
  - forum: 174cd4d4ba541fda5cf46d0d74e1102a

# triggers is a list of all services and its events mesg will listeing for.
triggers:
  # the name of trigger used in eventflows
  public-email:
    description: "email with topic and message"
    # id or name of service
    service: email # 6b0884a06e169c095ed8c412c3afc398
    # when filters the service events.
    when:
      # events name from services mesg.yml
      event: EmailSent
      # events tags
      tags:
        - t1 # simple tag name
        - /^\w+$/ # maybe regexp?

      # events outputs (from services mesg.yml)
      # NOTE: this is the core of workflow file
      # required outputs will be passed to execution 
      # of next service
      outputs:
        - topic
        - message

  # another service (here is the same but with diffrent outputs)
  private-email:
    description: "email with topic only"
    service: email
    when:
      event: EmailSent
      outputs:
        - topic

# list of services to execute
actions:
  # name of the action
  send-to-slack:
    description: "send to slack slack"
    # id or name os service
    service: slack 
    # name of the task from service mesg.yml
    task: send-to-channel
    # provide inputs (they will be combine with triggers outputs)
    inputs:
      apikey: "80676bd37b0636dc11828d7f23cdafbb3889aba8"
      channel: "notification"

  post-on-forum:
    description: "post on forum"
    service: forum 
    task: post
    inputs:
      user: "root"
      pasowrd: "pass"

# eventflows combines triggers with actions
eventflows:
  # trigger name
  public:
    # on has one trigger for actions
    on: public-email
    # execute contains action/list of actions to execute on given trigger
    execute: send-to-slack
  private:
    on: private-email
    execute:
      - send-to-slack
      - post-to-forum

The key features:

  • the syntax is not so nested
  • the triggers.outputs and actions.inputs correspond 1:1 with mesg.yml definitions
  • it has 3 main part : triggers, actions and bindings between them
  • you can chain triggers and actions (althouth you can’t create multiple chain - on a do b then c because such chain requires keeping the state).

#4

It’s good that if we can reduce nested executions for readability. In the functionality side, they all seem the same. We need be sure to have a nice syntax for serial(dependent) & parallel task executions.

I’m throwing an idea by extending the original syntax that I provided to cover both parallel and serial task executions without a nested syntax. I’m introducing the new dependsOn field and named executions pattern inspired from @Anthony’s.

@Anthony I think you mentioned about having map inside execution, this makes sense and it’s needed to make it possible doing multiple task executions with different input data. The below example also adopts that part.

And there is an example in the bottom about wildcard use for listening all events or results that I forgot about mentioning in the first post.

when:
  serviceA:
    event:
      eventX:
        execute:
          # this execution runs in parallel because it doesn't depend
          # on any other executions.
          execution1:
            map:
              field1: $event.data.fieldX
            serviceX: taskX
          # this execution runs in parallel because it doesn't depend
          # on any other executions.
          execution2:
            map:
              field1: $event.data.fieldY
            serviceY: taskY
          # this execution waits execution1 to complete with resultX &
          # execution2 to complete with resultY.
          execution3:
            dependsOn:
              execution1: resultX
              execution2: resultY
            map:
              foo: $execution1.result.data.fieldX
              bar: $execution2.result.data.fieldY
              baz: $event.data.fieldY
            serviceZ: taskZ
          # this execution waits execution3 to complete with resultX.
          execution4:
            dependsOn:
              execution3: resultX
              ...
  serviceB:
    result:
      # listens all results from serviceB.    
      '*':
        execute:
          logExecution:
            map:
              message: $result.data
              serviceName: serviceB
            logger: log

And there can be multiple executions that waits for the same execution to complete with the same or different output keys. There is also a big range of possibilities for doing dependent executions with this kind of syntax.

WSS will analyse all the dependent executions and run them in serial or parallel depending on how they’re defined and what executions that each execution depends on.


#5

@krhubert Yes, I agree that we may need to provide a command to user for showing the underlying workflow.yml. It’d be nice :slight_smile:.

I think we can use one of the command names below for this:

  • $ mesg-core workflow dump ID
  • $ mesg-core workflow definition ID

#6

I’m thinking about adding mongodb as dependency to WSS so we can query workflows depending on incoming event and results to execute tasks. Note that, we may not need this at this time and only query saved workflows on startup and keep their definitions in memory. We’ll see this by time while experimenting.

I’d like to see how a distributed database will work together with WSS. @Anthony already pointed that we’ll need a distributed database in future so mongodb can be good start and we can always change it in future if needed. I don’t want to use a simple key-value database like LevelDB for workflows because we’ll need some querying.

@core team please give feedbacks :slight_smile:.

This is my current TODO list:

create a base for workflow feature and add dummy create & delete features. #541
create the most simple VM implementation in WSS to run workflows. It should be able to run the sample workflow service in the first post. And use mongodb for saving and querying workflows. #559
implement workflow logs command so we can easily debug running workflows. #559

Later on, do improvements on syntax, VM that runs workflows and implement remaining cli commands / gRPC apis.


#7

Workflow Running Policy

This is an another thing that we need to discuss. We need to decide how to deal with disconnected services in order to run workflows stably.

Scenario #1

What to do if a service cannot be started that a workflow depends on, in the first creation time of the workflow?

Should workflow create feature return with an error or create the workflow and try starting and listening on the services within intervals until all of them are responsive? We can still log this process to workflows own log stream for devs to be aware of the life cycle of workflow.

I prefer the second way by leaving this management to WSS so it can deal with services under the hood instead of failing on workflow creation.

Scenario #2

What to do if a service is got disconnected that a workflow depends on?

This service could be the service that tasks are executed on or can be a service that listening results and events on it. There can be several services that a workflow depends on and some or all of them can be disconnected/unresponsive.

In this case, workflow cycle will not proceed properly because of the disconnected services. For example a task may be able to get executed after an event received from a service but some other tasks may not be executed because their services are down or some results or events may not be listened for the same reason.

Should we completely pause the execution of a workflow when at least one of the services it depends on is not responsive (I think yes)? In this case workflow can continue after its services are fully responsive again. To make this possible, we need to make sure that we’re keeping workflow’s state (unhandled events, results, inputs/outputs, executed/non-executed tasks etc.) correctly otherwise we can miss some task executions on the way and this can introduce weird behaviours to application.

I think WSS should manage all the services like this and have restarting and relistening policies on services. And log any info to workflow’s log stream about the status of services and listening/execution state of workflow.


#8

For now we don’t have some kind of registry to map service ids with their Git URLs. Because of that, in workflows, we’re not able to automatically deploy depended services. So, we’re thinking about supporting repo urls and local paths next to service ids in the definition.

e.g.

name: ...
description: ...

services:
  # with service id.
  serviceA: 5baa5a2f1ecdda9a25a15e350f0a94730ca2ad3b
  # with git host.
  serviceB: https://github.com/mesg-foundation/service-influxdb#also-supports-branches
  # with absolute path.
  serviceC: /Users/ilgooz/Programs/go/src/github.com/ilgooz/vuejsapp
  # with relative path.
  serviceD: ./another/service
...

#9

I really love the idea to have this deployment part directly in the workflow, like that we can just provide the workflow and this install and start everything.
Let’s definitely keep this in mind, maybe not a priority for now but really good, we could have a kind of workflow service resolver that for now is database resolver but later can be git, path, tar, ipfs…


#10

We may support configuring services inside workflows as mentioned in another proposal.


#11

Let’s keep the deployment idea but implement it later :wink:


#12

we can implement a lock file for locking service’s version. see: Improve relation between SID and Hashes


#13

Here is a proposal for the workflow UI for applications/workflows that will not surcharge the cli / core for now and let us experiment before adding this feature in the core.


#14

I remember that we had some ideas about executing tasks on pre-created applications like we do with MESG services. For example, an application can have some tasks/funcs that actually makes various task executions on services under the hood and produces a result.

This way it’s possible to create reusable & configurable applications as well.

@Anthony can you share your vision for this about how we should implement this feature in workflows?


#15

I’m not sure this is a really good thing to do, I think the best is to really keep it simple and if there is tasks for preprocessing or post processing we just put them before and after the task we want.

If it’s tasks really independent of the application that needs that for the deploy, they can just create a app that execute a task and then use the deploy api with the result but in that case we still don’t need any pre/post processing.


#16

@Anthony I created an another proposal for this and explained the idea a bit more. Reusable Workflows & Workflow Marketplace


#17

I’ve been looking at Github workflows, the config behind Github Actions.
You can actually create your own tasks based on docker with the connections you want, that might be a good source of inspiration.


#18

To let you know it looks like hcl syntax.

For syntax we need to first decide if we want to use yaml or hcl (json dosen’t have comment and it’s unreadable, toml has it’s own quirk). Other languages are not so popular.