Define success metrics and risks

With the team deep in delivery, there are many discussions around the importance of scope. Complexity grows, backlogs increase, and money slowly burns away … naturally, the discussion of reprioritizing surfaces. In other words—someone has to give something up. These sessions are, by design, meant to deal with conflicts of interest.

Establishing a set of hard metrics before the effort starts is a good way to build a foundation for critical decisions and steering stakeholders in the right direction. Metrics also serve as a reality check for when additional data becomes available. For example, a user-testing session on a functionality set revealed that the requirements were only partially accurate. This is no fault of anyone on the team—the client made the best attempt to guess the needed functionality and we did our best to implement it. But the reality of the product was different, and the collective needed to change direction mid-flight to make the product viable.

Analytics and product metrics should be fundamental requirements in the story map.

Tools range in complexity and price (e.g., Google mobile analytics, Adobe, and Flurry). It’s important to see metrics on sessions, active users, and feature use. Without these data points, it is really challenging to discuss metrics to effectively measure product use.

Selecting the right metrics

It’s appropriate to revisit the vision statement and the product charter when you’re thinking about metrics and analytics. Discuss how to quantify the goals.

If the product is an upgrade or replacement of an existing platform, consider extracting metrics that may be relevant and using those as the bedrock for future goals. For example, you could track the time to accomplish a task or number of errors for quantity of data captured.

Goals should be specific, measurable, and achievable.

If solid historical data hasn’t been collected, consider framing goals by establishing a hypothesis for the ideal state for an initial target and discussing the behavior necessary to move from current state to the ideal. Furthermore, revisit the story map and double-check what the shortest path to success looks like—this should be the highest-priority story in the release plan.

Consider measuring the following for success metrics.

• State change: The client signs a contract using a digital signature. They previously would have had to sign a paper copy.
• User adoption: Track the number of users who have actively switched to the new platform.
• User engagement: This could include the number of sessions, median session duration, or number of key actions during a session. This metric may be useful to track between releases of versions versus the first release, as the baseline may not be available for a greenfield application. Cohort analysis and time normalization is important for these metrics—otherwise the story may be lost.
• User efficiency: Establish metrics for key workflows in the system, such as onboarding, registration, checkout, work order completion. You could use either completion percentage (e.g., 60 percent of users complete the workflow) or completion duration.
• Volume of support requests: Ideally, the new product lowers the overhead for support and the volume decreases over time.
• Net Promoter Score (NPS): If the platform is a B2C tool, leverage NPS over time to improve the customer experience.

The majority of Devbridge’s product work is B2B and often internally facing, so these SaaS metrics rarely apply in our projects. As a result, metrics such as customer lifetime value, CAC, and average revenue per account are not included.

Pinpointing risks

Why revisit risks? The short answer is to build influence. As the workshop wraps up, people are energized. With goals in mind, the stakeholders are already visualizing a future in which their lives are easier and better because of the product that is about to be created. This is the perfect time to link the risks and goals for the client. This exercise further positions Devbridge as a trusted partner and helps define necessary steps to de-risk the delivery.

Once the goals are identified on the whiteboard, circle back to all the risks that were identified throughout the workshop and discuss risk impacts for each metric/goal. Brainstorm suggested steps to lower the risks. Keep a record of the steps for reference to address.

Informing outcomes with analytics

Product analytics empower teams to make informed decisions about the product roadmap. The tools available today capture top-level metrics such as adoption, granular insight around feature usability, high-friction application areas that demotivate customers, and many others. Analytics are often deprioritized in favor of more features. The Devbridge approach is instead to mandate implementation of analytics as a means to determine which features generate the most ROI.

Take, for example, an account registration page that requires users to create a password. There’s a balance between forcing users to create a password that is difficult to hack and allowing passwords that are easy to use. Analytics could measure the frequency of error messages triggered by passwords that don’t pass validation as well as user drop-off on the page. With these measurements, it becomes possible to A/B test toward the optimal outcome.

Consider the following types of data to capture:

• Application performance: page load times, feature response time, etc.
• Application engagement: concurrent users, weekly active users, etc.
• Application usability: user workflow funnels, user drop-off, conversions, etc.

Proof: Analytics drive product funding

The company overview

A physician-driven company dedicated to helping hospitals and providers deliver high-quality patient care decided to introduce software automation to its workflow. On an annual basis, the client treats more than eight million patients. With thousands of clinicians and hundreds of medical facilities nationwide, effective communication was an area of opportunity.

The opportunity

The client’s doctors and employees had been interacting via various platforms: an out-of-the-box text messaging system, email, SMS, iMessage, and telephone—a far from ideal situation. For instance, instead of being able to send one group message to book one shift, the schedulers reached out to several doctors individually via several platforms generating several lines of disconnected communication touchpoints.

The client envisioned a new mobile and web-based platform that would allow automated scheduling, direct and group messaging, doctor onboarding, and many other features. Because this was a new venture for the organization, additional insight was necessary to inform their investment and to identify the highest ROI features.

The approach

Analytics were implemented in the first test run of the application and released to a group of early adopters within a couple of months—minimizing the financial exposure for the client. The approach used was not unlike the one in the popular TV show Shark Tank. Investments were released in chunks of $250,000, which allowed the team to release, capture data, learn, and improve future releases.

The result

By reviewing the data captured by these analytics, the client was able to

As a result, the client was able to rapidly ship an incredible amount of high-value functionality.