The Confidence Quotient
There is also a cost to experimentation. Variant testing, user experience research and other methods of product experiments are time-consuming and costly. PMs are not academics and do not have the luxury of spending months or even years proving a hypothesis. Product Managers serve the commercial interests of the company and therefore need to bring products to market quickly while mitigating the risk of failure. PMs must balance these conflicting agendas, ultimately assessing the cost/benefit ratio and degree of acceptable risk.
A useful concept and tool in assessing risk is a “Confidence Quotient.” It is a variable that reflects one’s confidence in a potential solution hypothesis. It is not an exact formula, but rather a scale, from 0 to 5, with 0 representing no evidence and 5 representing highly compelling, verified evidence. The Confidence Quotient is not a proxy for whether or not you think a solution will work, but rather an estimate in the confidence of the data informing decisions.
Taking risks and making bets is an integral part of making products, but having a common vernacular to honestly and transparently evaluate and communicate risk is vital to establishing a culture of trust and learning, and provides practical day-to-day value in decision making and prioritization.
RICE scoring is one example where the use of a confidence quotient is particularly useful. R.I.C.E. stands for Reach, Impact, Confidence, and Effort and is a lightweight method for quickly prioritizing initiatives.
Let’s say there are five projects the team wants to get done, but the engineering capacity can only accomplish 4 of them. RICE scoring makes it possible to stack rank and prioritize the projects. Most methods use value and cost as the key variables to drive prioritization, with a goal towards doing projects with the biggest impact and lowest effort. Confidence, however, provides a valuable input when assessing relative cost and impact among multiple initiatives. Here is how it works:
Reach = How many users are affected by this feature
Impact = How will each user be impacted by this feature (on a scale of 1–3)
Confidence = Our confidence in the validity of the first 2 numbers (on a scale of 1–5)? If no research has been done and we’re working from pure gut, confidence should be a 1. If we have run multiple studies and have empirical data to indicate a high probability of success, confidence would be a 4 or 5.
Effort = How many engineering weeks are required to build the feature
Using these variables as inputs, the following simple formula gives you a RICE score:
(Reach * Impact * Confidence) / Effort
Projects are then stack ranked based on their relative score. While far from perfect, RICE scoring provides a best guess of which projects have the highest probability of success.
Thanks for reading. In Part 4 we’ll take at aligning Objectives to customer needs.