This article is about how one can lead by being an enabler of experimentation and innovation without exerting authority or imposing mandates.
When Generative AI tools exploded into the software engineering landscape, the reaction across my teams did not really mirror what was happening in the broader industry.
The hype didn’t really inspire my engineers to explore and adopt these tools. Their negative past experience with Gen AI tools made them conclude that it was a waste of time.
So although I was excited and hyped about experimenting with GenAI tools, my team didn’t really share that excitement.
The other side of this is that there was genuine delivery pressure and hence they didn’t really have the head-space to think about experimenting with something that they felt was already a waste of time. Imagine having signed delivery SLAs with clients and hence negotiating scope was not an option without significant monetary consequences! So the question of adopting something new in the middle was difficult.
The overall feeling was that the hype is good for the vendors but it wasn’t going to bring us any good. The first part of that belief is absolutely true. But if we find the right use-cases, i.e. the things that we would rather let someone else do, we could potentially get GenAI tools to handle these effectively! Imagine that! All the tasks that drain you, done by autonomous agents while you are free to do the things you love! That was the best case vision I had in mind.
So I realised that neither a mandate from the top nor ad-hoc fiddling would lead to sustainable value - our ROI on AI. I knew that my team would appreciate it if they got the time, at work, to test and prove their hypotheses of various GenAI use-cases themselves. To maintain our standard of engineering craftsmanship and hit our ambitious delivery and throughput goals, we first needed dedicated time to experiment and validate the impact of these tools.
So I set out to build something to enable this with the sponsorship of the higher echelon of Engineering Leadership.
The Scientific Method
Rather than asking my engineers to forcefully “use more AI” alongside tight delivery deadlines, I got to formalise and implement a dedicated space and structure:
- A Protected Cadence (“the second Thursday of a 2-week-sprint”): One dedicated day per sprint specifically for GenAI experimentation and learning. This gave engineers protected time, and it cushioned the pressure of delivery and provided the psychological safety needed to explore without fear of falling behind. Although it provided cushioning, there were times when some engineers had to skip this experimentation to get on with delivery for the sake of deadlines. This was a delicate compromise. But those few engineers who focussed on delivery, helped the rest of the engineers have the dedicated time.
- A Hypothesis-Driven Logging: Every experiment started with a clear hypothesis - a statement predicting the expected outcome of the experiment. Using a shared hypothesis template, engineers defined what they were testing, the tools involved (such as Claude Code, GitHub Copilot, Gemini, or Rovo), and specific evaluation metrics.
- I avoided measuring success purely by “time saved”. Instead, I wanted the evaluation to give us a holistic view of the impact of GenAI Tools:
- Correctness & Accuracy: Did it produce working, compliant code/output?
- Adherence to Architectural Standards: Did it match our conventions and avoid hidden tech debt?
- Cognitive Load: Was the prompt-engineering and verification effort worth the outcome?
- A “Show & Tell”: In each sprint at the end of the dedicated experimentation day, we reserved time for team members to share what they tried, demo it and categorize it under Adopt, Trial, or Discard.
When I look back, maybe I could have invented a metric to consolidate the various dimensions and give a unified score for the impact. But I didn’t follow that path. You could though.
Discovering the High-Leverage use-cases for GenAI
So I did my thing of applying the principle of context-not-control and getting out of the way of smart engineers after giving them the outcome we were aiming for with the ring-fenced time for it.
What happened thereafter was a real treat! The team applied GenAI to broadly the following areas:
- Complex Architecture & Flow Discovery: Using AI agents to parse legacy repositories and produce granular flow diagrams (e.g. mapping charging calculation flows with Excalidraw skills) to de-risk refactoring efforts. This had a huge impact on the newer engineers who joined the domain and team. They were finally able to understand the workings of the system, which they had only a high level understanding of.
- Contextual Documentation & Runbook Synthesis: Generating operational runbooks from incidents/bugs, PRs, and team discussions to streamline cross-timezone handovers.
- Technical Discovery & Prototyping: Accelerating discovery by breaking down complex technical requirements into PRD drafts and building quick reference Proof of Concepts, i.e. user interfaces to help demo a functionality was a big winner.
- Sharing what one learned: Through regular Show and Tells and a shared “Skills Scratch Pad”, engineers began sharing reusable prompts, tool skills, and CLI tricks across squads.
What did we find out
Speed without correctness is just accelerated technical debt
If an AI tool saves 30 minutes in drafting code but takes 45 minutes to debug subtle edge cases, it’s not an efficiency gain. Measuring correctness, test coverage, and maintainability is vital.
Dedicated space beats ad-hoc experimentation
This was obviously necessary in the high pressure delivery environment that we were in. Engineers need slack and this was the only way to provide the headspace. Setting aside protected time turned shallow tests into deep, actionable insights.
Codifying knowledge scales the impact
An individual engineer discovering a clever prompt or CLI workflow only benefits that engineer. By documenting prompts as reusable “skills” and logging results centrally, the whole engineering organisation levels up together. This was a really powerful force multiplier.
“Discard” is a successful outcome
Proving that a specific tool or use case is not ready for production is just as valuable as finding a winning one. Celebrating “Discard” decisions saved us from pain.
Looking Ahead
Generative AI isn’t a silver bullet, but when approached with an engineering mindset—testing hypotheses, measuring outcomes, and codifying knowledge—it becomes a reliable catalyst for engineering excellence. As we continue refining our workflows to hit our delivery milestones and scale our products globally, our commitment remains the same: keep questioning, keep experimenting, and let evidence guide our toolkit.
Some Real-World Experiments and Outcomes
I thought I’d share what I felt were the most compelling outcomes from my team’s Innovation Days:
1. Architectural Safety & Preventing Outages
The “Data Migration Audit” Skill
| The Problem | A database migration had shipped with 5 passing integration tests but caused two production reverts because empty objects {} broke downstream strict consumer schemas and NULLs caused silent data loss. |
| The Experiment | Created a specialized migration-audit skill using SQL equivalence-class framing and full-row schema parsing (paper-only reasoning across schemas and code). |
| The Outcome (Big Win) | When tested against the original migration, the AI surfaced all 6 findings (including 3 High-severity breakages and data-loss risks) before running any code. |
| Takeaway | GenAI acts as semantic architectural reviewer that catches edge cases that traditional unit tests miss. |
2. De-risking refactoring inLegacy Codebases
Automated Charging Flow Architecture Mapping
| The Problem | The core charging calculation flow across multiple markets was deeply layered and hard to onboard into or refactor safely. |
| The Experiment | Fed the charging codebase into Claude with an Excalidraw skill (~100k token context) to map both high-level coordination and granular calculator execution. |
| The Outcome | Successfully generated end-to-end, visual flow diagrams and identified concrete refactoring paths that directly fed into the Charging technical roadmap. |
| Takeaway | Using AI to map and visualize legacy systems dramatically reduces cognitive load and accelerates technical discovery. |
3. Pragmatic Realities of AI-Assisted Operations
3-Way Merge Conflict Resolution
| The Problem | Stale PRs with multi-branch merge conflicts are tedious and interrupt feature work. |
| The Experiment | Tested Claude / Copilot Agent mode on synthetic 3-way merge conflicts across complex calculation logic. |
| The Outcome (Honest Realism) | Scored ~70% accuracy / neutral time saved. The agent resolved syntax and type-checked successfully, but initially picked one branch blindly rather than “merging intents” without explicit engineer guidance. |
| Takeaway | AI agents need human guidance. Without knowing the “why” behind conflicting branches, the agent defaults to syntactic fixes rather than semantic correctness. |
4. Cultural Evolution: How Ways of Working Adapted
Redefining Pair Programming in the Agentic Era
| The Problem | How does a high-collaboration team preserve its pair programming culture when an agent writes the code? |
| The Experiment | Paired engineers on active epics using Live Share alongside Claude CLI and VS Code Agent Chat. |
| The Outcome | Found that pairing while waiting for an agent to type code was disengaging. Instead, the team redefined pairing: pair intensely on upfront architecture/prompt design and post-generation code review, while letting the agent do the asynchronous code synthesis in between. |
| Takeaway | AI doesn’t eliminate pair programming; it shifts the human collaboration to higher-leverage design, architectural intent, and rigorous review. |
5. Knowledge Transfer & Handover Across Timezones
Automated Runbook Generation from Slack History & Docs
| The Problem | Handing over Australian EnCom services from the Global team to the AU team risked losing operational context. Writing runbooks manually was high toil. |
| The Experiment | Queried Slack AI across #dev-energy-commerce-core and used Atlassian Rovo to synthesize past incident discussions, PR links, and primary contacts into structured runbooks. |
| The Outcome & Safety Learning | Created actionable runbook drafts, while uncovering a vital security guardrail: Slack AI accesses private channels of the prompt author, requiring mandatory human review before publishing. |
| Takeaway | GenAI can collate institutional memory for team handovers, but enterprise guardrails and verification remain essential. |
6. Codifying Best Practices into Reusable “Skills”
Standardizing Agent Workflows
| The Problem | Ad-hoc agent workflows led to inconsistent results and high cognitive load for engineers. |
| The Experiment | Built the Copilot Feature Implementation Plan Skill (breakdown-feature-implementation), transforming Jira tickets into architecture-aware EnCom monorepo implementation plans. Institutionalized CLI efficiency practices (e.g. leveraging grep/awk to reduce token consumption, adopting 4-sentence structured prompting, and enforcing a strict “plan-first, execute-second” workflow). |
| The Outcome | Standardized workflows reduced friction and improved the quality of agent-generated outputs. |
| Takeaway | Codifying best practices into reusable skills and templates creates compounding engineering leverage. |
Conclusion
The team embraced GenAI where it made most sense. It helped augment our capacity. Skills had a big impact on how we triaged support and reduced cognitive load on engineers who were on support rota. Agent instructions helped ensure consistent outcomes from our AI assisted workflows.
And most importantly we learned a lot.
Before I go, I’ll leave you with something that resembles the template I used for recording our hypotheses and experiment results.
Experiment Template
| Field | What it meant |
|---|---|
| The Problem | The concrete pain point the experiment was aimed at, stated in business or engineering terms. E.g. “Handing over Australian Regional services from the Core team in the UK to the Australian team risked losing operational context.” |
| The Hypothesis | The falsifiable claim being tested about a tool or workflow. E.g. “Slack AI + Rovo can synthesize past incident threads into a usable runbook faster than an engineer writing one from memory.” |
| The Tools Used | The specific products and the model where applicable that was used, named precisely. E.g. Slack AI, Atlassian Rovo, GitHub Copilot, Claude Code. |
| Date of Experiment | When it was run, so results can be revisited as models and tools change quickly — what “works” today may not hold six months later. |
| Status | Like tracking work in JIRA, we could list hypotheses to be tested in the next innovation day or conclude if it was worth adopting or discarding. E.g. todo, in progress, in review, adopted, rejected. This turns a pile of anecdotes into a tracked backlog, e.g. “Codifying Best Practices into Skills” moving from in progress to adopted once it reduced friction team-wide. |
| Methodology | The step-by-step approach taken, so someone else could rerun it. E.g. “queried Slack AI across #dev-energy-commerce-core, fed transcripts + PR links into Rovo, asked it to draft a structured runbook, then had a human review before publishing.” |
| Observations — Advantages | What genuinely worked well or saved effort, described concretely rather than as a vibe. E.g. producing actionable runbook drafts from scattered Slack history. |
| Observations — Hallucinations or Wastage | Where the tool made things up, went down a wrong path, or burned time/tokens without payoff. E.g. a merge conflict resolution that needed human intent the model couldn’t infer. |
| Results — Accuracy | A number, not a feeling: % of output usable as-is, defects caught, correctness rate against a known-good baseline. |
| Results — The Time Saved | Before/after effort comparison, e.g. “runbook drafting went from a day of digging through Slack to under an hour of review.” |
| Results — Usability | Whether the tool made the day-to-day work easier or harder once integrated, e.g. pair programming shifting from mechanical typing to architectural co-design. |
| Results — Risk | Any new risk the GenAI tool itself introduced. E.g. Slack AI surfacing private channels of the prompt author, which meant mandatory human review became a hard guardrail before publishing anything. |
| Demo Recording | A link to a recording of the experiment in action, so the outcome could be judged by someone who wasn’t there — evidence over anecdote. |