What is an internal developer platform?
A suite of tools or applications, utilities or the likes that is tailored to the needs of developers of an organization that enables them to build, deploy and manage software in an opinionated way that is often considered as the golden path to building and releasing a product to customers.
Companies generally invest in these as they start to scale. They tend to realize the benefits, which, I think, can be summarized as the following:
- Standardization: consistent set of tools for everyone in the organization — comes with many other benefits, elaborated in the following points
- Scalability: centralizing the duplicated effort of setting up common infrastructure, creating quick start templates for developers to start developing on, provides the ability to onboard new engineers quicker as everyone uses the same tool there would be plenty of help around also improving productivity and enhancing the ability to mobilize engineers where capacity is necessary. This makes collaboration easier.
- Governance: Security engineering teams and other operational and compliance teams are usually much smaller than the number of software engineers for a company. Thus, standardizing tooling, ensure that the small number of members in these teams can focus all their effort on specializing and observing just that central standard set of tools and applications. Also, keeping an eye on cloud costs — essential before it becomes the big gaping hole in your budget.
Internal developer platforms are meant to reduce cognitive load on developers and enable them to focus on just what they were hired to do, write efficient software that solves business needs.
Over the past decade or so, there has been a slow evolution, something that potentially evolved from the DevOps movement — which unfortunately have been marketed as a role, where instead of developers and operations working together in a team, the operations engineers build all the tools necessary for the developers to do the right thing. This has created the advent of internal developer platforms as a thing.
So what?
Let’s say, over the years, your organization decided to build an IDP. Your IDP probably has a long-term vision to become as developer friendly as anything built on the underlying complex technology can ever be. But you haven’t been able to realize that vision yet, not even close. So far, you have been in a mad rush to consolidate all the poorly architected and fragmented legacy cloud infrastructure or applications owned by the various teams at your rapidly prototyped, fast-growing organization under one roof. While you were at it, you also decided to centralize datastores and are now wondering if you can integrate serverless technologies. That’s a long product roadmap, or maybe just an ambition for now. But who are you building it for and how long have you been doing this?
In this post, I wanted to share some of the invaluable lessons I learned from my experience leading a platform engineering team building the foundations of the product that my organization sells. We have made a fantastic internal developer platform that does numerous awesome things for the organization, saving millions along the way. I’ve learned plenty of invaluable lessons and have enjoyed the ride thus far. However, I wanted to also share the not so fun parts or the downsides.
I’m using an imaginary example here where an IDP was created many years after the money-making product was built. This could pose a unique challenge in the journey of developing a great developer platform.
What do developers think of your IDP?
The investment so far in your organization has been to consolidate infrastructure targeting cost optimization, which you have definitely achieved by migrating as many applications as possible to your IDP. However, very little thought has gone into the real software engineering efficiency that your organization needs, to move as fast as possible in this competitive market.
You assumed that consolidating infrastructure would do it. However, you have never really considered why developers would want to use it over the many tools that they have at their disposal today.
Getting all the engineering teams to follow a golden path is a noble goal but at the same time, highly ambitious if the organization has a history of having heavily invested in non-standard, peculiar and custom technology or stacks. Such a goal requires many bright engineers who not only focus on the mechanism and the patterns of cloud deployments but also ones who focus on abstracting the complex layers of the underlying platform, like Kubernetes from an average software engineer, so that deploying an application doesn’t become a side-quest that they now have to complete, while under pressure to deliver something for a demanding client. This is where you unlock real value. This is how you get developers wanting to deploy and develop applications that run on your IDP.
Let me guess, your IDP doesn’t even have a unified user interface with role-based access control to ensure that only authorized personnel can do CRUD operations on the systems that they own. Pretty bare bones, huh?
What were the costs so far?
So you have used all the strategies that you could possibly use to get the expensive legacy services in your organization onto your IDP, whether it is by hiring contractors locally or nearshore or offshore, it all came with a cost.
Let’s explore some of these costs.
Collaboration
Communication is hard. Whether it is nearshore or offshore organizations that you are working with, they are excellent at delivering. But to get to the same page takes longer due to factors like cultural, time-zone and organizational policy differences
Onboarding
It takes a considerable effort to onboard a new engineer into your fast-growing organization that is probably changing priorities as often as you shelve your side projects. It takes even more effort to understand a legacy system and then translate it into the golden path approach as recommended by your brand-new developer platform
Monetary
Finding the right balance in terms of the cost of an engineer to the value they return is hard.
You may get inexperienced but bright engineers in large numbers overseas for a cheaper rate, costing you money in — accidental incidents or distractions to the development team. Oh wait, but wasn’t this the team you were trying to protect from distractions by hiring these engineers in the first place. But now you’ve ended up inadvertently using that very precious capacity to onboard and guide these offshore engineers.
You have also hired many such offshore engineers to parallelize work. So many that you now need a team of managers to keep everyone on the same page and coordinate work. Like a delivery manager, a program manager and so on.
Local contractors are so expensive that they eat into the budget so quickly, that it makes it really difficult to hire more than a couple.
Cost of existing employee time
To keep everyone on the same page, you now have 4 new expensive meetings every week, mostly managers — engineering and product from various teams talking about how and when they can coordinate to begin this mass migration of legacy services. This is work in addition to their actual day jobs.
That is a lot.
Cost of the platform engineering team
But wait, didn’t you say, you’ve invested like 3 years into building this with a team of 6 engineers and a product manager and engineer manager? That sounds pretty expensive already!
However, this is the cost you pay to clean your foundations to build the next big thing. So this is NOT a sunk cost, it is going to deliver long-term value.
At least that is what you and your team tell one another. It is also your job to ensure it does because the whole damn organization is counting on it.
What are the challenges in funding an internal developer platform?
If your organization’s primary line of business is not building developer tooling or products, then you’ll struggle to get any money to expand on your awesome internal developer platform, no matter how awesome it is.
Internal platforms aren’t moneymakers, at least not directly. Thus, they aren’t the primary choice when it comes to the C-Suite thinking of putting money into. They are strategically useful vehicles to multiply impact and do things quicker in the future. However, when you are working in a spin-off or start-up that is yet to make a profit, the odds are likely stacked against you. You need to first close that deal, or deliver that feature that makes it happen.
Operations are generally here to support the development of the product that sells and brings in the big bucks. At least that is how the C-Suite understands it. They don’t care if you only have 2 engineers building the platform that supports the whole thing, so long as it all just works. In their mind, it is a one-time investment, like setting up a network or buying some servers in an imaginary data centre. You only need an engineer to turn things on and off every once in a while and push some upgrades or patches when necessary. How far from the truth!
This is unfortunately a really hard battle to fight, if you are accountable for delivering the outstanding internal developer platform, especially in an organization that is making strides in a non-technical industry. Unless those making the budgetary choices understand that a developer platform is very much a product that has to be developed, fixed, maintained, and enhanced and is indeed a force multiplier, which is the truth, when it is done right.
What do you need to get to the most loved platform?
So the large migration effort has helped you get some real customers for your IDP, and this brings you feedback — much needed feedback. This is also going to put the system through some real tests.
But you have never gone to your customers — the developers in your organization, trying to understand their pain points. Have you?
What do you need to know to build a good platform? Many things really, let us start with some:
- What is the current developer stack? What do they need to run this stack efficiently? How common are the parts of the stack in the rest of the org?
- What does a developer workflow look like today? What are the common patterns there?
- What do developers consider as distractions to their work — obstacles that get in the way of delivering features? Could these be solved with the right tooling? What do they wish they didn’t have to do in the first place?
- How much do they understand your IDP?
- What do they wish they didn’t have to do to deploy their app on the IDP?
These are just some questions that will give you an insight into the drift between what you think your software engineers need and what they really need. These insights are gathered by organizing customer empathy sessions in larger organizations, where there is a bigger budget for such things.
These insights will help you build a roadmap that would benefit your engineers, turning your IDP product into something they look forward to working with. This is unlike what you have been doing so far - i.e. focus on cost savings primarily, which is unfortunately not their concern. Remember the high delivery pressure, your customers, the software engineers, couldn’t care any less about costs of deploying an app on the cloud or leaving it running there. They barely have time to understand their client’s requirements, let alone implement those features.
These conversations are essentially baked into what Teresa Torres formulates into continuous discovery and has to happen, as she said - continuously to ensure that the platform engineers do not lose touch with how developer workflows have changed over time.
What does good vs great look like?
A good platform needs to make security and compliance seamless for anything that runs on it.
For a good operational platform, concerns like security, networking setup and policies, compliance and related non-functional requirements must be baked into it rather than be an afterthought. This is something that your IDP has done well, because it was primarily born out of the security and reliability engineering function of the organization. Fantastic! You suddenly feel a pride that you didn’t realize you had.
However, most engineers on the operational side of the fence are obviously experts in operations, and hence more knowledgeable in systems, network, security, etc. These are secondary concerns for software engineers, and quite often they lack expertise in these very things. All software engineers care about are things like:
- I need a key value store for caching these values, so the app can access them frequently
- We can store this data in a relational database
- This operation is CPU intensive, so we probably need to run these on beefy hosts
- This application does memory intensive operations, so we need big memory on the hosts for this one
- This app has to talk to this external vendor’s API to make a successful payment
You see how the thoughts are almost high-level talk? Software engineers don’t want to know about the complexities of networking or any of the other things mentioned earlier. Those are distractions for them. They may be interested in learning about these things, but practically do not have the time to invest in such things during working hours if they have to deliver what they were hired for.
They may not want to know how to run a relational database, but they do want to be able to use Postgres with their application. So long as it is up and running, they are happy because that is what enables them to do their job - writing code in their favourite programming language that solves a real business problem.
Realistically, software engineers in a fast-growing company, that’s also rapidly iterating to meet client needs to strengthen its foothold in the unique industry they are in, do not have the time to focus on non-functional requirements. They were rightly hired to be experts in expressing the business problems and solutions in the programming language that they are familiar with or has been chosen as the standard by the organization they work for. Everything else is noise that they need the help of operational engineers, i.e - platform engineers, security engineers, network engineers, so on to solve. I have reiterated this point in so many ways in the last few paragraphs.
This is why a good platform is one that has all the NOISE - taken care of by platform engineers.
A great platform, is one where it isn’t just the noise but also the developer experience that is streamlined and made more efficient.
What have companies on a similar journey learned from platform engineering?
- Adopt a product mindset: an IDP is the platform engineering team’s product, and developers are your customers. Treat your developers like you would treat any external customer. You want your developer platform to go viral, not to be stunted or ignored or forced upon. Prioritize developer needs and experience, and you’ll build a great platform.
- IAM, KMS, Networking, etc are complex and less interesting things for software engineers to deal with when focusing on solving business problems. Abstracting these complexities in a way that software engineers can realize what they’ve designed on paper/virtual white-board to working applications that just work without having to learn all about the underlying technology that would host those applications is where a platform engineering team wins.
- Autonomy within guardrails - this is the experience that developers expect from a platform. They can do something that complies with security and data protection relatively easily and intuitively on the cloud platform, why would the platform engineering team make it convoluted for them?
- Some DORA Metrics
- Burnout prevention: in a fast-paced, rapidly growing company, the influx of change is at a rate that is not common. The need to iterate and deliver on time repeatedly can take its toll on everyone. Imagine if the platform that the software engineers were to deploy on, created additional cognitive overload or friction - as the engineers might have to onboard or learn a new tech stack to be able to do their job or the platform has several leaky abstractions that the average software engineer isn’t concerned about. A good platform is one that is only visible when there is a genuine problem/incident. Otherwise, it seamlessly blends into the background as part of a developer’s workflow.
- Flow: the state of flow measures how much focus a person tends to achieve during development tasks. A platform that enables software engineers to achieve a longer state of flow, is a great platform. Platforms that are designed to have getting started templates, right level of abstraction for the majority of the use-cases, enables developers to be in the flow state longer as they can think and dream of the solution to the real business problem, without having to be concerned about how to secure or connect to something
- Organizational and team performance: With more engineers that are in a state of flow for longer and where burnouts are a thing of the past, the teams, and the organization does better overall. It is not rocket science.
- Platform customers rule: your developers are your most important customers, and you don’t want to lose them because of an average platform. A great platform helps teams thrive, and the company grow at pace. A bad one, slows teams down, causes attrition and burnout. Weigh the various ways to meet developers where they are. Some things just cannot be unified or consolidated. However, think about where you can build plugins or extensions or libraries or wrappers that can help you gain the observability that the organization needs centrally while still enabling your developers to do their best
What if your organization doesn’t have the budget for an IDP?
Great question!
You work in an organization that is just starting its product development journey.
This is when you don’t build your own platform from scratch. Please don’t.
Use something that’s already out there. Checkout Internal Developer Platform.org to learn more. I don’t sponsor that project or anything. But it is a great starting point, collectively built by the many good folks from various organizations sharing a wealth of knowledge on this. By the way, that website is not a product, just guidelines and best practices for an IDP, but it also has links to products that are readily available to use.
This is the best way to get started when it comes to standardizing things if your budget is tight - use what’s out there, that’s battle tested, well maintained and probably even open sourced but has some well worth paying for features that can help you achieve exactly what you want without having to build your own.
Usually, this is the best choice, especially in the short term (a few years) when getting started, at least until your organization turns a profit, proves its concept.
When you feel like you are turning a profit, and you need that extra boost, some customization that would give you an edge, or you have pushed the limits of what’s already out there but cannot scale to the limits that you need to, then and only then, think of going all in to build your own, preferably on top of the solid foundations you got from adopting the solution that was already out there.
Bottom line
You may think your developer platform is unique. Believe it or not, you aren’t doing anything unique, at least not internal developer platform-wise. The vast majority of organizations have their custom internal developer platform because of many years of various consolidations and standardizations and often end up finally resembling something that is remotely comparable to a well-thought-out and purpose built from scratch product, that could potentially have been used in the first place to save all the headache and yet not do what an IDP would ideally do.