Reusable Workflows & Workflow Marketplace

proposal

#1

Application = Workflow

Problem

Let’s say that we have a sendgrid service in the service marketplace to send emails.

And let’s say that a part of our workflow needs to send a specific type of emails to send verification codes to users to make it possible for them to verify their accounts, and let them to reset their passwords and so on.

Knowing that, think about a scenario that a user wants to reset its password. To do that we need to send a verification code to its email. After user receiving this email that contains the code, it should enter it in a mobile app or website and be able to set a new password for its account if the verification code is correct.

To make this possible, devs need to generate a verification code in their workflows maybe with a help of another service, keep this code in a database by using a database service and send the code to user with an email by using the sendgrid service. And check the code entered by the user while password is being changed.

As we see, to implement this feature we need these two sendgrind and mongodb services and introduce some business logic directly into the workflow.

Proposal In Short

  • Allow workflows to be reusable.
  • Create a marketplace for reusable workflows.

Proposal In Long

My proposal here is to allow devs to create reusable workflows so they’ll not be required to duplicate same logic in their different workflows(projects).

With this reusable workflows, we can simplify the code verification feature we talked about in the Problem section by creating a reusable workflow like below:

workflow id(wid): email-code-verification
workflow services (the services that workflow uses): sendgrid, mongodb
workflow tasks:

  • sendVerification ([ inputs> email string, timeoutDuration string ])
  • verifyCode ([ inputs> code string ]) ([ outputs> email string ]) // returns email if code is exists and not expired

Reusable workflows are just like reusable services. They have tasks, task results and events. They can be used by other workflows like how services are used inside workflows. They can also have deployment time configurations. They’re easy to create with a GUI and can be deployed to workflow marketplace.

This reusable workflows idea is very similar to service composition but there are some major differences:

  • It is possible to use multiple services in a reusable workflow and not just only one(this is the case in service compositions).
  • It’s easier to create workflows rather than services. Because a GUI will be available to create workflows for non-programmers.
  • We’ll have a service & a workflow marketplace growing naturally and separately. Services will stay more generic because we’ll have this strong separation between the services and workflows. On the other hand, service composition will result services to be less generic(opinionated) which is a very bad adoption and will create a noise in the service marketplace.

We can also make this reusable workflows to be compatible with the current service(listenTask, submitResult) & core(listenResult, listenEvent, executeTask) APIs so we don’t need to create more gRPC APIs or logic in the Core.

Details About Reusable Workflows

  • Reusable workflows can be created over a GUI.
  • Reusable workflows can use other reusable workflows or services and listen for their results or events and execute their tasks.
  • Reusable workflows can be used by regular workflows just like how regular workflows uses services.
  • Reusable workflows can receive configurations in the deploy time.
  • Reusable workflows acts like services so they’ll share the same service(listenTask, submitResult) & core(listenResult, listenEvent, executeTask) APIs with services.
  • There is no much difference between regular workflows and reusable ones. They are deployed/started/stopped/deleted in the same way. The difference is, workflow syntax will let devs to define some tasks and emit some results or events from a workflow. This way, they can be used just like services. Reusable workflows gives the possibility to combine multiple services together and create an opinionated one.

Introducing Workflow Files
#2

Great proposal! We wanted to implement a similar system once workflow and the marketplace are release.
Good that you came up with a similar idea :wink:


#3

Here is a real world example of reusable applications, please check the graphql service.

Note that, since we don’t officially support reusable workflows/service compositions yet, this is just an experimentation with what we have.

Normally, a service only connects to service gRPC API but this service also connect to core gRPC API. In another means, this service can react to events, results and execute tasks from other services likewise an application.

The service’s purpose is to run as a GraphQL server. To do that it depends on http-server to react GraphQL queries sent over http requests. When there is a new request on http-server, it receives the request event and checks the path and method of the http request to determine if it’s a GraphQL request.

If so, it emits a query event that contains information about the corresponding GraphQL request. Applications should listen for this event and prepare the data for it and then execute completeQuery on the service to send back a response.

After response received by the graphql service it forwards it to the http-server service by calling its completeSession task and then user gets its GraphQL query results.

In addition to this, graphql service also depends on graphql-introspection service to automatically send responses to GraphQL’s Introspection queries by using the SCHEMA which is set as an environment variable to the graphql service.

To check graphql service in action please check this PR on marketplace application.


#4

That’s a good way to do service composition with the current version of Core :slight_smile:

The first problem I see here is currently there is no way to set the dependency between the composed service in the service definition. We could add a new field in the service dependency to specify the composed service and deploy / start them automatically.

The second problem here is that the service is using the core api. That’s something we would like to remove in the future. This can be solved if the service could also have workflows that will connect the composed service to the main one. But in this case, the workflow will be executed by the Core itself. This will be much cleaner.