Scaling your development team

This post is a collection of my thoughts and observations on what happens when you try and grow your development team. I want to apologise in advance for the rambling nature of this post but I wanted to put my recollections into text, I plan on refining the post over the coming weeks. If anything is unclear please ask for clarification on twitter or via email you can find both on my home page. These observations are anecdotal and are based on two of the teams I worked in.
The first I’ll call Team A, grew slowly over the course of several years. It was lead by a strong architect who had a clear vision for the system that the team were building, initially there were three other developers of which I was one and all three had considerably less development experience than the team’s leader. The development methodology used was waterfall with a release every couple of months. Requirements gathering was done by a member of the team who would meet with the client to discuss their needs then a design meeting would follow with the whole team present, to discuss how those requirements might be delivered. All members of the team would give their input but the leader would have the final say on how the solution would be developed. The team were collocated so once development started informal meetings would occur often if one of the team became blocked implementing the agreed design, if needed the design could be altered quickly based on the new information that had been gleaned. Over time this team built up a large amount of domain knowledge and the system matured. Over the course of five years the team grew to seven members. The newer members of the team needed a lot of assistance from the original four members to learn how the system worked and to learn how to add new features to it. The system was the classic tightly coupled ball of mud and the various modules were usually integrated via the database using stored procedures a classic anti pattern. There was limited documentation to help new developers understand the system. The system did not have any automated tests and there were no QA’s on the team. Developers would test their own work and then demo it to the client, when the client signed it off then the code would be deployed for user acceptance testing which would be carried out by the client. If any problems were uncovered by the client a patch would be deployed and retested. When all problems had been resolved then the code would be released to production.

The second team initially consisted of six members, it did not have a formal leader and it was tasked with rewriting the existing in house platform. Two of the team members were from an outsourcing company and were based in a different country and spoke english as a second language. The system was designed around SOA principles. The team set itself the goal of building a distributed system based on messaging, using continuous integration with automated deployment and automated testing. None of the six team members had built a system like this before, so an external consultant was brought in for the first few weeks to help get them on the right track. The team had a lot of domain knowledge because they had built the platform that was to be replaced. During the first six months many decisions had to be made about tooling and processes often the discussions ran on for many days about the best way of accomplishing a particular task, the lack of a formal leader meant that progress was impeded until there was a consensus, sometimes the various team members would be obstinate and it would take a long time to reach a consensus often the business would have to get involved and broker an agreement. The original estimate by the team for how long they would need to create an MVP was six months, this proved to be on the low side. When six months had elapsed it was clear that they were not close to delivering the new platform. At this point the business was suffering because the key developers of the original platform were unavailable to maintain and extend it. A decision was made to move three of the key developers back to the existing platform team and to hire new developers to work on the new platform. A decision was also made to expand the team and open an overseas office to hire new developers based there. This plan was executed aggressively and fifty new team members were hired over the next twelve months approximately half based at the HQ and the rest at the overseas office. The new recruits were experienced developers but they did not have any domain knowledge. A decision was also made to hire an architect to lead the team, he was based in the overseas office, he had lots of experience building distributed systems but had not worked in this domain before. Next a decision was made to adopt Scrum, none of the team had experience of scrum so an agile coach was brought in to help get the team up to speed. During the twelve months a lot of code was written but it lacked cohesion the three remaining original team members were overwhelmed by the number of new team members they could not hope to spend enough time explaining design decisions and rationale for how the system was built. The system was still being designed in places so sometimes that information was not available. This resulted in a code base that lacked cohesiveness, as new developers joined the team they were unable to deduce from the existing code the correct way to implement a feature because existing features did not follow the same patterns and conventions. This created a real problem because these new team members were unable to ask experienced team members as their time was already oversubscribed.
Lessons learned.
Team A was tight knitted they were very stable and had a common understanding of the “right” way to implement new features. The environment around them was relatively stable so they did not have to repeatedly implement features in new unfamiliar domains. The codebase they created was tightly coupled this made it difficult for new developers to write code unless they were given lots of support from experienced team members and it also meant that it would take several years before they would feel confident working in the codebase, especially as they had no tests or documentation to help them. Thankfully the need to rapidly scale this team up did not occur because it would have been impossible.
Team B was distributed between two sites and grew very quickly from six to fifty developers and QA’s. It had continuous integration, it also built automated and manual testing. As the team grew the codebase became in-cohesive, developers were doing their best but had no real way to know if the code the wrote followed the correct conventions. There were very few developers on the team who had any domain knowledge. For the majority of the first eighteen months there was no outright leader who could make decisions if a consensus could not be reached via debate. The majority of the team were inexperienced with the chosen SOA architecture. The result was a system where the team had to continuously fight to push code and were constantly trying to figure out why the build was red. Build, test and deploy times started to increase and the code base became tightly coupled when testing and deploying, soon developers were unable to test locally as there desktops did not have enough resources to deploy the full system. Later on when these problems became obvious and the team started trying to split the code out into separate repos and also to formalise dependencies this process reduced productivity even further. The end result was it took four years to deliver the the project to production and the business was unable to do any significant innovation during this time. During this time there was a very real chance of management losing all faith in the development team and canning the project, thankfully this didn’t happen. The highly unproductive environment did cause developers to leave which had a negative impact on the team.

A few points to consider
– When creating a new system always ponder how quickly the business intends to create new products and enter new markets. Provide feedback to the managment about the readiness of the development team and codebase to undergo expansion. If you can make the management team understand what you will have to do to expand the team then hopefully they will factor that into their plans for global domination.
– Money can get you access to the best engineers but it will not help new developers gel any quicker.
– If you are contemplating a complete re-write then consider who will support and develop the existing system and who will write the new system, how will you migrate existing customers and how will you run both in parallel whilst you migrate. Will hard won domain knowledge be available in the new team. These are really hard problems to solve. Always consider evolution away from the existing platform to the new design, it will often be quicker, less disruptive to the business and less risky.
Final thoughts
As software developers we often focus our attention on the technical challenges after all these are the sort of problems we enjoy solving. If you are working in a small team of six or less then that will normally be fine. The moment you are faced with the challenge of growing beyond that number you should start spending a considerable percentage of your time thinking about how the team will function well and how you need to organise the codebase to ensure the developers can remain productive. If you fail to do this you’ll lose the chance to work on solving the technical problems and you likely not enjoy your job.