Listener system based on a persistent state


Right now when an application wants to listen for an event (or result) we create a stream and everytime we get events we go through all the different streams open and check if they match the event. This option is fine but if we need to scale to 100+, 1000+ listeners this will be really hard to have good performances and also with the introduction of the workflow file this is not really relevant. Another thing is that it will be kind of hard to syncronize this over a network.


We could store all the listeners with their filter in database and based on that for every events/result query the database to find all the listeners that need this event and then send them. The logic will move from the listener part (that filter the event) to the event part (that find the matching listeners).
This way we can be closer to what the workflow file will look like and have pseudo code like this when we receive event/result

func receiveEvent(event) {
  listeners := db.findListenersFromEvent(event)
  for listener := range listeners {
    // To manage the case of an application that listen the api stream
    if hasStream(listener) {
    // To manage the case of a workflow file
    if hasExecution(listener) {

This would be helpful for the following issues,

This will be fixed by the implementation of the workflow engine Workflow implementation