We Accidentally Built a Second Codebase

A few weeks ago, I deleted 93 skills in a single pull request.

A skill is a bundle of instructions for a model to follow when doing a task, such as investigating a flaky test or gathering context for an incident. Teams package skills into plugins and publish them to a shared marketplace. Engineers can install whichever plugins they want.

People had started pointing out that we had no quality control on what went into ours. Anyone could publish a skill. I opened the list expecting to find a few duplicates and tidy them up.

I assumed someone depended on each one. Deleting a skill might break a workflow I’d never heard of, and that uncertainty had kept me from touching them before. We’d also worried that deleting people’s contributions would put them off writing more.

The 93 skills were all in one plugin, which had 96 in total. Each of those 93 had been invoked zero times in the previous 60 days. I removed them and merged the PR.

Nothing broke, and nobody messaged me about it. We’d spent a while debating whether anyone would care. As far as I could tell, nobody noticed.

I wanted to understand how we’d ended up keeping so many unused skills, and what else in the catalogue needed attention.

The outdated skill still in use #

One example was a skill I left in place. It told the model where to find things in our codebase and how the different parts fit together.

The codebase had changed since someone wrote it. The instructions hadn’t. Even when they were new, they described the part of the codebase the author knew, so an engineer working elsewhere could get incorrect guidance.

I don’t think we should have written this one. The model can open the repo and look around, but we’d told it where to look based on the part one person knew. Now someone has to keep checking those instructions against the code. I’d rather delete them and let the model work it out.

I left it there, though. People were using it, and I still hadn’t worked out what removing it would do to their workflows. With the 93 unused skills, I didn’t have to answer that question.

How the maintenance accumulated #

A team writes a skill for a task they repeat, finds it useful and shares it. Another team does the same. Someone combines several skills into a longer workflow. Each team has a reason for adding its skills, and the catalogue keeps growing.

Our marketplace had grown to more than 600 skills across nearly 80 plugins. Deleting 93 still left hundreds to look after.

Meanwhile, the models and tools kept changing. A workaround for something a model struggled with might become unnecessary after an upgrade. A tool might handle steps that a skill still described as manual work. Code moved, teams reorganised, and instructions could become wrong without anyone editing them.

That’s awkward to review. The change that invalidates a skill might be in another repository or outside our code altogether. Looking at the skill’s own history won’t tell you whether its assumptions still hold.

Roughly half the skills in our catalogue had a single commit, and most had only one author. Those counts don’t prove a skill is outdated, but they made me wonder how often anyone checked.

I’d been treating skills as documentation, something to write and refer back to. Yet they influenced what the model did, depended on tools and other skills, and could interact in ways their authors hadn’t intended. We had to maintain them as those dependencies changed. We’d built a second codebase, mostly in English.

An engineering leader had raised the same concern months before my deletion PR, in a Slack thread I found later:

Our skills etc. are now our “code”. Have we discussed what a quality ensuring pipeline would look like here?

What needs deciding #

I think we should expect to delete skills as the reasons for writing them go away. Some compensate for a model’s limitations or a missing tool. Others describe a workflow or record knowledge that only one person had. We need to check whether those instructions are still needed, and whether the current model does better with them than without them.

I’ve suggested a quality bar for publishing, named owners and scheduled reviews for retirement. I don’t yet know which of those would help enough to justify the work. Benchmarks might tell us whether a skill improves results, but we’d still have to decide what to test and keep those tests relevant.

The usage figures made my deletion PR straightforward. They won’t settle every case. A skill for an unusual incident might go unused for months and still be worth keeping.

I’d like us to work through a few questions:

• How do we distinguish an unused skill from one that’s rarely needed but important when it is?
• What evidence justifies deleting a skill, and who makes that decision?
• How do we check quality without making it difficult for engineers to share something useful?

I could check 60 days of usage and remove the 93. I still don’t know how we should review the hundreds that remain, especially the ones people use while their instructions become less reliable.