ADR-008 Deployment environments
Date: 27-11-2023
Status
✅ Accepted
Context
When working across multiple teams, or switching between teams, it can be hard to follow what each deployment environment is for and what practices it keeps. This can lead to data ending up in an improper environment, or tests being run with incorrect assumptions of the state of the environment.
We sometimes need to test, demo or explore functionality between multiple connected services. To do so, we need persistent environments of each service that are connected together. Not all products have this, and those that do can be inconsistent, making it harder to perform these activities.
Decision
All services must have the following essential environments, which should be deployed sequentially as part of the path to live:
- Local environment
- Deployment testing environment
- Production environment
- Demo environment
Services may commonly have additional environments:
- Automatic PR environments
- Manual PR environments
- Preproduction environment
- Other persistent environments
Production environment
This is what our users access and it contains our real data. It must be the only environment in the production account.
Deployment testing environment
Every service must have at least one environment that is as close to production’s infrastructure and scale as feasible. This may contain a copy of production data and live in the preproduction account, or may contain synthetic data and live in the development account.
It is not accessible by the public. It is deployed in the pipeline before production to test that the release can be deployed successfully.
Demo environment
This is used to test, demo and explore functionality against the currently deployed release. It is in the development account, does not contain production data and is not accessible by the public. It is deployed in the pipeline after production to ensure it reflects the latest releaase.
It is connected to the demo environment of any dependencies, and is the target of the demo environemnt of any dependent services. This allows cross-service testing.
It is be available at demo.development.{production host}.
Local environment
All services must be available to stand-up on departmental laptops in some form for local development and testing. Local environments should not require any external connections to dependent services (though their build pipeline may require external connection to e.g. fetch code and packages).
Automatic PR environments
These are created automatically when a pull request is raised to use for testing and demo of changes before they are merged. They are in the development account and are not accessible by the public.
These environments should be used to test against, rather than using local containerised environments on the CI runner, to provide a higher level of confidence.
When the pull request is merged, the associated environment is automatically destroyed.
They are available at {automatically derived name}.development.{production host}
Manual PR environments
In some services it may be impractical to automatically create environments for every pull request. In this case, it should be possible to manually create an environment based on the pull request. These environments should have a “deletion date” on creation and automatically be destroyed when that date is passed.
They are available at {manually or automatically derived name}.development.{production host}
In all other regards, they are the same as automatic PR environments.
Preproduction environment
This has identical infrastructure and data to the production environment though is not connected to external services. It is in the preproduction account and is not accesible to the public.
It is used to test against a real data set, where synthetic data is not reflective or large enough to provide confidence. It may the deployment testing environment and deployed to as part of the main pipeline, or be deployed to manually whe needed.
It is be available at preproduction.{production host}.
Other persistent environments
Some services may require other persistent environments. For example, for training or user research. These environments may be created automatically or manually, depending on the team’s needs.
These environments are in the development account and do not contain production data.
Consequences
Having more consistent environment practices will make developing and maintaining services across multiple teams more predictable and less prone to confusion. Having consistent demo environments will make cross-service testing easier and safer.
This will be more challenging for services that make exceptions to this decision, as they might lead to incorrect assumptions. Service teams will need to clearly document their exceptions and explanations to avoid confusion.
Where services connect to each other, teams must document which environments are connected and what type of data (production, pseudonymised etc.) each uses. This is essential to ensure that private data does not incorrectly cross tiers.