Background

The City of Oslo is one of Norway’s largest employers with ~50 000 employees, and 50 organizational units, all varying in size, responsibilities, domain, IT-competence and funding. In 2003 they established an integration platform to be able to tie the units together. In 2016 there were approximately 200 microservices running in production and around 60 developers implementing digital services for the citizens. Around that time, some problems started to arise…

Scaling challenges

One challenge being a codebase too large to comprehend, as one of the developers stated:

“I don’t know what this microservice does?” ”What happens if I change X?”, ”Let’s add Y instead”.

At the time, the teams were organized as feature teams, building a feature making it all the way to production using DevOps-principles. A feature often involved changes in numerous microservices. Often developers from other teams were needed to understand the structure and logic of the components involved, when implementing a feature.

Numerous dependencies made a feature cumbersome to implement and test. As more features were added, the program became even more complex because different components often had different non-functional requirements. One set of components required a particular type of service level agreement. Another set of components had different requirements to scale independently of others and with a higher user load.

Strategy for enabling autonomous teams

So we tried to do something about this scaling-problem… The new strategy had two goals:

1. Teams were given responsibility for a collection of microservices within a business domain, instead of having responsibility for a set of features implemented all across the code base. A domain is a structure which comprise software elements. The main idea being that the team can focus on one business domain exposing a defined API, or other natural grouping of the microservices.
   
   
2. Aligning boundaries with team boundaries. By introducing an API-centric architecture (Fig. 1), teams are responsible for one or more domains and they apply explicit service boundaries between domains. The only interface into a domain is through the API-component, no exceptions are made. This is an API-centric architecture, which is somewhat analogous with the ”API-mandate” famously issued by Amazon CEO Jeff Bezos.

Each domain has its own version control of the team’s components, and only team members can deploy code to production. The code is open to other teams, and pull requests can be created by other teams, but the code has to be approved and deployed by the team responsible for the component. This architecture also has other benefits, e.g. teams can choose different cloud infrastructure and technology within their own domain, further adding to team autonomy.

While the architectural strategy is important we found that the self-service platform supported autonomy. In addition to deploy code to production, teams could then create their own test environments and experiment with different tools and technology.

Implications for practice

We interviewed team members and architects, and went through JIRA and Confluence-documentation. What did we find in our research that could be relevant for practitioners?

• Find the right type of domain: Three possible strategies are 1) A team is responsible for an end-user product and put all components related to the product in one domain or 2) mapping a team to an organizational unit. 3) Components may have different non-functional requirements, e.g. some have a 24/7 SLA. In this case, it may be necessary to group components with the same service level agreement in the same team domain.
  
  
• Modify the service boundaries continuously: evolving your team and organizational structure to promote your desired architecture. This is also referred to as the ”Inverse Conway manoeuvre”.
  
  
• Implement API versioning and management: Versioning and API-management of an API are critical, meaning that other teams can rely on the stability of external domains and its components.

Conclusion

We found that an architecture organized by domains with APIs and service boundaries, and a self-service platform enables team autonomy. As this is an ongoing study, we will continue exploring challenges such as coordination between teams, different types of domains, domains with shared functionality and how changes in domains influence team autonomy.

For the last 15 years the I’ve been working with the City Of Oslo, transforming the delivery platform from a Monolith to Microservices and further into an API-centric architecture. From Waterfall to DevOps, and recently into Autonomous Teams in large scale Agile. It’s been great fun, and I’ve been talking a lot about this at JavaZone and other conferences, please see https://gundelsby.com if interested.