In this article, Ed addresses some key questions on delivery processes and how they are run.
Director of Product at FutureLearn.
Ex Depop & Rocket Internet.
The core of a product function is the way that the delivery process is defined and run. How do you go from objectives through to working software that delivers value to users and the desired outcomes for the business?
A great delivery process has numerous benefits:
- Aligns business around the problems the product teams are working on, and the solutions they deliver to solve these
- Ensures that feedback from stakeholders is appropriate to the development state the feature is at (i.e. discussing the right problems to solve whilst setting OKRs, and the best designs after user research and wireframing)
- Increases visibility of new features across the company, and makes sure non product teams aren’t blindsided by releases
- Increases the chance of the feature delivering the expected impact, by requiring each problem to be properly researched
- Ensures that each feature has good accompanying documentation for reference by stakeholders, the team and any new joiners.
At the heart of this process are five key check in points:
- OKR setting (handled in separate meetings on a quarterly basis)
- Kick Off
- Implementation Review
- Release Notes
- Impact Assessment
Kick-Off, Implementation Review, and Impact Assessment all take place within a weekly Product Review. This is typically an hour-long and divided into 15-minute slots (for smaller features PMs can skip the meeting and send out details via email, for larger features, a dedicated meeting can be called). PMs claim a slot and present their initiatives as they pass a check-in point. Product Review is attended by representatives from across the business: leadership, product, marketing, and customer experience. The agenda is sent out in advance, and materials afterward (often Confluence pages), so that people can drop in for the sessions they want, without needing to attend the whole meeting.
- OKR setting
OKR setting happens in the month preceding the start of the new quarter (i.e. Q2 planning happens in March). Product Managers drive a series of meetings with their teams and key stakeholders with the objective of aligning everyone around a high level plan for the upcoming quarter, and increasing everyone’s confidence in the team’s ability to deliver against that plan.
The output is an OKR and prioritised backlog / roadmap for each team.
OKR = Objective + Key Results:
- Objective (what we want to achieve)
e.g. Streamline the process for creating new listings
- Key results (pacing for how we reach the objective)
e.g. Release 5 major improvements to the listing flow; Increase listings per lister by 10%
OKRs are set for each theme of work, with a product team typically undertaking 1–3 themes for the quarter. (For more on setting effective OKRs read Perfect Pacing: The Secret to Great Product OKRs)
This is a prioritised list of initiatives — individual problems or opportunities that sit within a theme. At this stage, each initiative is usually just a line in a spreadsheet, but with the following details:
- Description / hypothesis
- Impact estimate (quantified in a high level business metric such as revenue)
- Effort estimate (number of sprints to tackle this problem; can be a range)
With this backlog in place, we have a rough idea of what the team will get through in the upcoming quarter, based on the estimated effort for each initiative.
Critical at this stage is not to forget any of the maintenance work or operational load that the team has to carry. Tech debt and other necessary work that the team will undertake but won’t move the primary OKR needs to be explicitly called out — ideally in its own technical OKR — so that stakeholders have visibility of it.
Of course, there’s a degree of uncertainty, as we don’t know exactly what we’re going to build at this point, but it’s good enough to set stakeholder expectations and ensure the team is set up for success (generally this means it has enough people with the right skillsets). Make sure that stakeholders realise the team is NOT committing to delivering the roadmap — it’s committing to delivering the results, and the roadmap is the best guess at this point in time of how it will do that.
It follows that there’s likely to be things you don’t get round to in the quarter. This isn’t a problem if the team has delivered the impact it committed to. These features can flow straight into the next quarter’s planning, or be shelved for a later date if the company objectives have changed.
2. Kick Off
With the quarter underway, each initiative has a Kick Off meeting before the team starts discovery. The Kick Off will include:
- Summary — what is the problem we’re trying to solve, and why is this important?
- OKR — how will this help us achieve our goal for the quarter?
- Impact estimate — back-of-envelope calculation or “what would need to believe for this to be worth building” type sensitivity analysis
- Success metrics — i.e. what are the leading metrics that will drive the OKR
- Supporting insights — qualitative, quantitative and competitor research that provides context on the problem we’re trying to solve
- Expected approach — current high level thinking on how we might solve the problem, if known
- Scope — roughly how big a solution are we considering? What is in and out of scope?
- RACI — clarification of all the stakeholders involved
- Next steps
The kick off will also give designers, analysts and engineers and high quality brief to work from, and so provides a foundation for good discovery.
3. Implementation Review
After the team has done discovery, the PM brings the proposed solution back to Product Review for Implementation Review. By this point, the team should have final designs and high level tickets ready to go. Implementation Review allows any final input from stakeholders before the team starts to build the solution, and includes:
- Recap of Kick-Off details (OKR/impact estimate/scope)
- Insights from user research and deeper analysis that lead to design
- Final designs and user flows, including copy and content
- Engineering estimate based on the tickets created. The team should also be in a position to give a projected release date that follows from this
- Roll out and product marketing plans, as well as testing strategy
4. Release notes
We send out release notes each week to accompany our app store release trains. These get sent to the entire company and for each feature cover:
- Objective of the feature
- Which OKR it supports
- The success metrics for the feature
- Brief description of how it works
- Roll out details (which platforms / AB test groups)
5. Impact assessment
After release, the teams will report back on how much impact the feature had. Exactly when this happens depends on how quickly we can see results, but usually 2–4 weeks after release:
- Decision on rollout/rollback for AB tests, with rationale
- Change in success metric since launch (and statement on confidence whether this can be attributed to the feature if it wasn’t AB tested)
- Qualitative feedback from the community on feature
- Any other insights/analysis deep dives that we’ve done to understand the impact in more detail
- Note on whether any follow-up features are planned (and link to relevant documentation)
Subscribe to our newsletter below to receive our latest insights, career tips and industry experts interviews into your inbox every month:
Find your next career opportunity on Movemeon.com