When we do add a new feature or do a small/big refactoring in the core, we always required to do some manual testing. Otherwise it’s really easy to miss newly introduced bugs to the system. Existent unit or integration tests may not powerful enough to help us to catch these bugs. We need to integration test core from outside of the core to be aware of these kind of bugs which is what we do with manual testing by spinning up dev-core and using dev-cli to cross test the behaviours of these two and see if they act as expected.
Currently, we have some integration tests that are actually not integration tests. They seem like they are but not fully. They don’t test the actual integration behaviour. The integration tests we have are running inside the core itself and they don’t directly interact with core through the gRPC server.
Lets bring up a sample to examine:
TestExecute is not a real integration test. Yes, the testing service is really started but it’s tasks never get executed. The actual reason of why the service is started here is because there is a RUNNING status check on services and to not get failed by this, services must be exists on Docker.
We’re only able to test a small portion of task execution process in this test by checking the generated execution id.
Like within this example, all the other integration tests that starts services doesn’t really carry about them. And they can’t too because services are not able to connect to gRPC server to get task execution requests, get events and submit task results because our tests doesn’t run in a container and not in the same network with the containers that are running the services. Also for the same limitation, we don’t have any tests that requires listening results and listening events in grpc/core pkg.
Other than previous example, the rest of the integration tests inside core are testing very simple cases like this one does. Which they are not needed as integration tests because we only need to test the application logic. So unit tests are enough for testing all the internal packages except container.
Correct Way of Creating Integration Tests
I first thought creating integration tests by using yaml files for simplicity where we can have table tests for testing various scenarios. But using yaml files for testing can quickly get out of control and limit us because we might need to deploy and start a few services in a row and assert some service ids, input/output datas by executing tasks and so on. To accomplish these kind of stuff with yaml test files, we need to introduce comparison and variable primitives which is too much work and another thing that requires us to maintain. So, this is a no.
We can use Go for integration testing and have a flexibility of a programming language. Still, it’s always better to have table tests as much us possible to not duplicate logic between similar tests and test lots of scenarios at once.
We can place our integration test under a package called integration, inside core’s root. And we can have integration/grpc & integration/cli subfolders for testing gRPC API and CLI separetely.
Please see feature/integration branch to get insights from the first implementation of new integration tests. I think this is a good start, we can add more tests by time. Note that integration tests are living under the integration folder and they should be run by ./run-integration. Also, it’s not required to have integration tests under the core repo because it only interacts with core through network and uses cli binary to test cli.
- We should be adding tests for each cli command including the ones about daemon.
- We don’t need to add tests for service gRPC API directly because we use real services and they use this API to interact with core. This way we also test if core is starting services correctly inside containers and giving the right service id as env variable to service’s containers. So having integration tests for core gRPC API is enough.
Lets get rid of all the integration tests inside the internal packages and create equivalent ones inside new integration folder.
Lets have the integration tests in the most higher (gRPC server, cli) level and this way we’ll also eliminate doing manual tests after each change on core. Also inc integration tests coverage by adding more tests by time.
Lets only keep the integration tests we currently have for container pkg and do unit testing for the rest of the core as suggested before. By having dedicated integration tests for container, we can easily keep track of bugs in container pkg in the low level, otherwise debugging bugs caught from the higher level integration tests can get time consuming.