The Import Cycle Between API & SystemServices Packages

As we experimented, we have this incoming import cycle issue* between api and systemservices, systemservices/* packages. We currently use methods from api package inside system services and because of this, we’re not allowed to call methods of systemservices inside api pkg.

I’m not sure if this is an actual issue? It shows us that either we have a design issue or we don’t have a design issue and calling system services inside api pkg is not right anyway.

As I’m experimenting, I think that this is not an issue and we shouldn’t call methods of system services packages inside api package at this point.

Here is the reason why:
The idea behind of having api pkg is to have a single point of entry for accessing functionalities that core provides by combining low level internal packages. So it’s the api package that where we should interact with core, not the other internal packages. The api package is meant to be a wrapper for functionalities of core with a high level api.

To me, having the systemservices manager and its siblings (systemservices/*) shares the same idea with the existence of api package. We should treat systemservices like how we treat api package. systemservices/* packages are the wrappers around system services, like the api package being wrapper for internal pkgs of core. Remember when we talk about splitting parts of api pkg into smallar pkgs under api/*? It is the similar idea of having systemservices/*. So api pkg and system services pkgs are in the same level.

The idea of having system services is splitting core to smaller services in the first place. And this causes the same effect on api package. We required to move some methods of api package to different system services. Of course, splitting the whole core/api package into smaller system services may not be possible because at least for now we required keeping some logic like deploying services inside the core itself.

I’d like to give an example over Workflow System Service:
Let’s say that we’re not going to implement this workflow feature as a separate service but embed it in the core. In this case we were probably create an internal package called core/workflow, have some implementation there and expose this functionality with core/api package. This way intergace/grpc/core pkg can call methods like CreateWorkflow() from api package and expose this feature to network.

As a system service point of view, first, we’re going to have a mesg service as equlivent of core/workflow. Second, we’re going to have a high level wrapper for it under systemservice/workflow and this is the equlivent of code that we put under core/api for workflow. In this case, as third and last, intergace/grpc/core pkg will call the CreateWorkflow() method from systemservice/workflow.

Please note that it’s totally ok for system services pkgs calling methods from core/api because it’s the native part of core that we do service deploying and listening tasks etc. So calling api pkg inside systemservices is a valid approach. In future, if we can make it possible splitting all the parts from api package to different services then we can directly use the api package as a place where we connect all the system services together and expose them as high level features of core like we do today. But since we’re not able to do it in once, I think, it’s acceptable for now to have this two headed dragon. :slight_smile:

As a proposition, I think we can rename the api package to something called like native and create a new api package to call methods from native and systemservices. System services pkgs will call the methods from native pkg anymore, not the api. By time, we’ll move all the logic from native pkg as system services and native pkg will get deleted.

This is just a perspective that we can think about it. Let’s have a starting point from here to discuss and see how system services and core will evolve together.

Why not remove API from system service?

I’ve previously written about removing API from system service (move deploy to separate package). In this solution, system service deploys & start services.

Also, we can remove API package from resolver and keep it only for data mapping.

Simple and easy, no need to create 2 APIs.

Yes, yours is a really simple solution. But I’m actually not directly aiming to solve that problem. I’m more talking about in the architectural side about the direction of core and system services.

We must have a solid philosophy between the composition of core, splitting core to system services and the area where we do this transition by time. Otherwise we can even more complicate stuff by doing small workarounds.

It’s not workarounds, it is the solution.

In that case, you will have

API - which uses the deployer
system services - which uses the deployer

and in that system services will be something to manage services (not to manage tasks on services). I wanted to split them from the beginning, but as for now, system services use API pbm as we have right now, occurred. With the usage of deployer, you don’t need to change anything in API and split them into two.

And you propose to create abstract API on API, how many layers API of should be there? I think one API is enough and I have never seen API on API. It involves matters like open multiple sockets for listing for two APIs.

Also if native package shouldn’t be exposed via port then it just a package with general purpose which doesn’t tell anything about its responsibility.

I like the idea of removing the api from the systemservices package. If we want to do that there is more than the the deployer to change because we use the api for 3 things

  • Deploy the services
  • Start the services
  • Give the api to the different systemservices instances

The deployer is “easy” to fix but what about the start and the listen and execute that the services will do ?

The approach of another layer of abstraction seems something doable but I can’t really understand the part with the workflow and if it’s hard to understand already it’s not really a good sign for clarity of the code :stuck_out_tongue:
The part to have the low level api (I prefer over native) and high level api where the high level api will call the low level api AND the system services if needed and the system services will use only the low level api.
This might work but I think we will face the same problem. Let’s imagine that one systemservice wants to execute something that is done by another systemservice, then we have the same problem eg: calling a task through the network with the resolver.

I will put one more idea.

The idea is to totally remove the api from systemservices
For the deployment AND starting of the services we can extract this to the core/main itself and the systemservices doesn’t have the responsibility anymore to “create” these system services. It is just a map/list of systemservices with their instances.

For the different system services instances (that needs the api to listen and execute tasks) we can try to use the low level packages that we already have and bypass the api for that. We will loose some checks from the api but these are more verifications for user’s mistakes so it might be ok inside the core to bypass this.

  • tasks execution is handled by the execution package
  • event/result listening is handled by the pubsub package

With this we don’t use the api package anymore inside systemservices.

What do you think about that guys? Is there some stuff I’m missing?

As you said, deployer is easy
Start - https://github.com/mesg-foundation/core/blob/dev/api/start.go - it’s super easy to remove as well
About systemservices instances, I’ve already talked about it - “we can remove API package from resolver and keep it only for data mapping”

In that case, it’s just slice of services and we don’t need a package for it (append should be enough :))
I don’t like the idea of moving it to the main package, because main func should be as small as possible.

So

ss, err := systemservices.Init(config.SystemServicesPath) // deploy & start services
r := ss.Resolver() // get resolve instance for data mapping (could be skipped if someone want to
resp, err := api.ExecuteAndListen(r.AddPeersInput([]string{"8.8.8.8"})) // execute api call anywhere
err := r.AddPeersOutput(resp) // map output data 

// same as above but for Resolver
resp, err := api.ExecuteAndListen(r.ResolveInput("serviceID...."))
peerAddress, err := r.ReesolveOutput(resp)
1 Like

I like the idea of mapping just the parameters. I think this should solve the issue. Would like to have more feedbacks but this looks like the solution to go with.

I don’t agree on using systemservices pkg just as a mapper for core/api pkg. To me, systemservices should provide a high level api like an application instead of tightly coupling with the core/api package. It should be a logicfull package for interacting with system services. To me, turning systemservices to a logicless mapper is just a workaround in favor of core/api pkg, not a solution. It’s hard to use it that way and it’s low level, doesn’t reflect the idea in the first system services proposal. We can have 10s of different ways of solving import cycle problem but doing it this way kills the philosophy of how we should compose system services together in core.

The idea of having system services is to remove all the logic inside core as much as possible and split it to separate system services. In future, we want to use core only as a proxy for system services, as a place where they are all composed together. Like a mesg application that composes mesg services.

core/api pkg actually is just going to be an another system service by its own when we extract it from core. Even if we can’t completely remove it from core because of the possible complexity, it should be treated as one. At first, the purpose of having core/api pkg was to compose all the features of core in a single pkg with a high level api. But since we’re adopting system services now, core/api doesn’t need to wrap all the features. It has to have the most basic parts about services, other features should be splitted to different system services, like network, workflow and etc.

I think all system services should have access (able to import) to core/api pkg because, it provides the most basic features for services but not vice versa. And if a system service wants use an another system service to complete it’s tasks, it shouldn’t call that system service inside itself. That can be handled in the parent caller. This is the way how we should compose system services. The only case where a system service can directly access to another one is, the case when a dependent system service provides some low level functionalities like core/api pkg does. In this case, core/api pkg doesn’t even need to make calls to other system services. Because it shouldn’t be aware of them, it’s just a dependency. I’ve explained the similar thing in the first post as well. If we need to call systemservices inside core/api pkg, this would lead to a bad design.

Here is an another explanation about why we actually don’t need to call (import) system services inside core/api:

See CreateWorkflow gRPC API. Normally all the gRPC APIs calls methods from core/api. But in this case, creating a workflow is made by a dedicated system service. So the only thing we need to do is calling the Create method from the workflow system service and it’s done. So actually, we don’t only need to use core/api as a high level api bus anymore, we split the logic to different services and we call their wrappers to have a high level api for using them. If we required to call multiple system system services to complete a task, we can do it as well. But as I explained above, we should do it in the parent caller, in this case a gRPC handler. We should compose the data we got from different services and response it to user. We can even flow data from different system services, compose it and response to user. And as I explained before, some system services need to be able to call other system services -like calling core/api- to use them like dependencies, like go packages but as a service. The system services being used as a dependency, shouldn’t be aware of caller system services. This would lead to a bad design and it’s cycling.

My question is, what is the actual reason of calling systemservices inside core/api package? Can you provide a real world example? We can directly use system services apis from gRPC handlers like we do for CreateWorkflow.

And if we adopt that method we required to do mapping for other apis that core/api provides. Like the one used here by workflow system service. Which is not elegant to do.

The import cycle problem shows us that we shouldn’t call system services from core/api pkg at all, because it’s not needed and not proper. Think core/api package as a monolithic system service where we expose the most basic functionalities like deploying, starting services and listening events, result and executing tasks. core/api pkg doesn’t need to call other system services, only the rest of the system services wants to call core/api to communicate with other system services. So actually, there is no an import cycle problem. This is just how the design it is. In future, we’re going to split api package to it’s own system services, or to a one system service or keep it in the core as the native functionality. But the rest of the system services will depend on it.

My intention while starting this discussion was to show that to you guys. If this doesn’t make sense, I’ll be fine going with mapper idea but I don’t think it is the right way to go. If I’m missing something please lighten me. I’m giving this advices because I heavily worked on this feature and already implemented an actual system service.

I also like the mapping of parameter proposed by @krhubert.

One system service must not depend on another system service by its implementation in core. Eg @ilgooz’s example

If a system service needs to delegate to another system service (or any service), it should be done by the future service composition.

API package should delegate to system service a maximum of logic (not go package in the core, but the actual service itself).
But if API needs to always execute something from service A and then from service B, it’s also fine. But again, the Core’s implementation (in core’s go package) of a system service should not depend on another system service.

@ilgooz you implementation is not wrong but not in accordance with the vision of the Core. We need to explain more so you guys can implement stuff faster :wink:

Service composition is something really different. In that context, I agree that service dependency rules may differ.

For the core/api pkg part, I think you’re in favor about composing system services inside the core/api pkg instead of doing this in the intergace/grpc pkg. That’s also ok to me. The original idea in the first post to rename the current core/api pkg as core/api/native and put any api that generated from system service’s composition under core/api. @Nicolas I’m not sure what do you think about that.

Even with mapping, system services and core/api still are depending on each other but indirectly. I think this is a design issue. I’d rather to have the native pkg proposed in the first post to overcome this.

Anyways, I’ll not push anymore on this. As a reminder, if you prefer to keep the current version of core/api pkg as it is and have all the system services composed there, there are other ways of solving this import cycle problem without going low level in syntax with mapping. We can still maintain the original high level syntax in the first proposal. One way of doing is to have an interface like below that each system service’s wrapper can accept it in their New() func:


type Executor interface {
   ExecuteAndListen(serviceID, task string, inputs map[string]interface{}) (*execution.Execution, error)
}

I don’t like much this solution either, it’s just another hack but here it is.

I think nobody likes having both core/api/native and core/api pkgs to solve import cycle problem except me so, I’m eliminating this idea. To finalize the discussion about having high/low level apis in the wrappers I’m putting a poll below so, we can proceed much master on open PRs.

  • Use mapping and have low level api in system service’s wrappers.
  • Adopt another method, maybe interfaces (Executor) to be able to use methods of core/api still in the wrappers to provide a high level api without the import cycle problem.

0 voters