Understanding Customers – as a VP of Engineering

What role does understanding the customer and the market play in the VPE’s job? For VPE’s in technology driven companies, it might seem obvious that it is important to understand the people who purchase your products or services. However there is actually a lot of debate on this topic. While even large corporations will say their companies are “customer driven”, not everyone one agrees on who the customer is. At both large companies and startups, I’ve been in arguments about whether Engineering’s customers are the people who purchase the products or services, or if they are the internal business groups, such as Product Management, Marketing, Sales or Professional Services. This is a fundamental philosophical difference between a company where all departments try to understand the end customers versus a compartmentalized company where each department focuses on their “internal customer” and only a few departments such as Marketing, Sales & Customer Support focusing on the end customers and the market.

As a practical matter, this philosophical difference really has to do with how a company is organized and how decisions are made. The arguments for having only customer facing departments concerned with understanding the market and customers is that they have the most expertise. This view compartmentalizes departments into focusing on their specialty areas, with their major concern satisfying the internal groups they work with most immediately. This view tends to be present in larger companies, where each department tends toward protecting their own turf. In those environments, the group that represents the end customer and the market has powerful leverage in decision making, which can make them reluctant to share. In this view, having every department express their views on the market and end customers would be chaotic. While this sort of rigid separation of responsibilities can look efficient and well organized, it fails to provide any checks and balances against internal departments having biases. For example if the business groups are comfortable selling and supporting 1 type of product, they might be reluctant to transition to new products, even if more neutral groups see this as necessary.

For the VPE, the definition of who their customer is mainly affects how much say they have in product strategies, product management and requirements. If a company views that all employees should strive to understand the customer and the market, then Engineering can bring this points up in product discussions. While this can lead to more arguments, proponents of this view would say that this dynamic leads to better products and services. Certainly it is more enjoyable, not just for the VPE but for all Engineering. If Engineering is limited in how much it can discuss the market or customers, it becomes more like a service group or a vendor. When that happens and Engineering is not really a full a partner in product development: the best people will leave, and the organization is starting down the road of believing they can outsource all the technology.

The rest of this section will assume that it is important for the VPE to understand the market and the end customer. For those of you in a customer driven organization that supports this, it is important to learn how to see the business from more than just the technology perspective. For those of you in a compartmentalized company, it is still critical for your personal development and career that you understand the market and the end customers. While you might not get to express these views as often in decision making, it is important you have your own opinions on whether your business groups are making the right decisions. Also, someday you may well work for a company that expects their VPE to understand the market, so this is a good time to learn how.

The Customer Perspective

What is the goal of understanding the customer: why is this important? The fundamental issue is to understand the customer perspective and their business and their market. Companies are successful when their products and services address the needs of their customers, which is more complicated than it first appears.

Who really is the customer? An anecdote: One of my first engineering jobs was with a medical equipment manufacturer, developing an X-ray image processing system specifically designed for cardiologists. It worked well technically, it did well in testing with cardiologists, and as a dedicated device for cardiologists, and it was much cheaper than the general purpose X-ray systems. However it was a complete flop on the market. Why? Because during that time period, the purchasing decisions for all X-ray equipment were made by radiologists. It wasn’t clear to radiologists that a dedicated cardiology system made financial sense and the product approach emphasize ease of use for cardiologists, rather than technical capabilities which appeal to radiologists. The company thought they understood the customer by spending a lot of time with cardiologists, but they didn’t understand the complexity of the market. The customer was really the purchasing hospital and clinic, which had in it different stakeholders with independent perspectives.

To understand each stakeholder’s perspective, one exercise is to consider it from their perspective. What’s going to get a stakeholder their next promotion? What is going to keep them from being fired? What is painful or difficult enough in their lives to spend money to address the problem? How are the purchasing decisions made?

Customer Understanding is key to the entire product Lifecycle

Customer adoption makes or breaks a business and it’s critical to understand the product requirements from the customer perspective. Recently, Mark Leslie (ex-CEO Veritas) documented this viewpoint in his book “The Sales Learning Curve”. For startups with limited cash or ‘runway’ this is especially important. They have limited time to find solution(s) that resonate with customers and solve a critical pain point. They need to hit their customer target audience as quickly as possible without burning valuable cash by planning on a few product iterations to test assumptions about customer needs and assist sales in learning the product ‘sweet spot’. Building a large Sales Team before product market validation is complete is a waste of money. There are similarities between this principle and that of the Agile model of merciless re-factoring, i.e. learning from product prototyping early.

The process of a company engaging its customers tying to find the key pain points is not easy to be managed since there is a chance that customers will focus on the short term instead of long term goals. In order to allow the customers to share the product vision with you, the right context has to be set. There is a technique called “buy a feature” that can be used to engage in customer discussions where they will feel part of the decision making process. This requires that you invite a substantial amount of customers to a meeting. At the beginning of the meeting you distribute a limited amount of fake money to each customer and present them with a list of all possible new features or improvements on your product/service. For each feature there is a price that corresponds to the cost of implementing it. There will be somebody facilitating the discussion but it’s important to let the customers to lead the process. They can buy a feature as a group or individually. The ideal scenario is to spend the whole day in this process, but there are cases where its possible to implement this process at a conference with potential customers or even during user training depending of the type of product.

One common error in gathering customer feedback is to ask open ended questions like: Do you want a free ice cream? A more useful question would be do you prefer ice cream or coffee after lunch. The second set a context and exposes a trade off.

There is a book called “Innovation Games” that describes several techniques to help better evaluate the customer needs for a particular product/service.

Product Requirements

Product Management plays the key role in gathering representative requirements use cases and stories. They have the primary responsibility of understanding the customer role(s) and audience(s). Product requirements gathering/documentation means more than feature requirements. Information related to non feature requirements/assumptions must also be captured such as:

• Quality expectations – What is an acceptable failure rate or MTTR (mean time too repair) and MTBF (mean time between failures)
• Performance expectations – What is an acceptable latency for customer interaction?, what is acceptable ‘task’ performance?
• Key requirements such as diagnosability/maintainability/failover, etc. are often overlooked in an initial product definition cycle but are critical to success.
• Using the Agile model, the collection of simple customer ‘stories’ or ‘use cases’ is invaluable, but they also have to be collected based on the concept of a user class or ‘role’. Is this user an Admin or end-user, DBA or Executive?

Of course creating an accurate document of customer requirements/stories is of no good if there are process problems in the development cycle. That’s where the value of requirements traceability becomes important. It is feasible to formally track each individual requirement through the development process (Definition, Design, Implementation and Test) but it comes at a high price. and few companies actually use a formalize requirements tracing process. At a minimum, most companies can perform some sort of simple Root Cause Analysis of all customer bugs and determine what percentage are related to requirements gaps? versus design to coding bugs (a key measure Product Management success). If the percentage of customer issues related to requirements gaps is high, then you either have a problem with the requirements gathering phase or else the traceability process.

Gathering Customer Feedback

Like the proverbial blind men and the elephant, customer feedback gathered from different parts of a company is heavily influenced by the perspective and concerns of that group. This makes it especially valuable to gather feedback from a wide variety of sources, so that the information can be weighed and evaluated with the hopes of achieving a relatively unbiased understanding of the customer. For example, Sales might say the new features are too confusing, yet Marketing loves them and Customer Service isn’t finding a high rate of complaints. This could be because the features make it harder to sell but are actually easy to use, or it could be the early adopters are more sophisticated, and the customer service issues will show up later. Marketing tends to always loves new features, and might feel it’s just a matter of communication. Balancing these views and seeking other forms of information can help bring clarity to customer understanding.

Sales Feedback

• Anecdotal evidence from Sales discussions
  • Typically sales feedback can come from a number of different channels that are typically managed by the sales function. These include Support, Professional Services and Sales (Sales Engineering and Pre-sales). Each of these organizations has a different perspective or product interaction with the customer
    • Support: Feedback from Support is invaluable as it includes real world feedback from paying customers regarding both defects as well as areas of product enhancement. This information needs to be directed to the right organization, based on severity and prioritized effectively. This process is critical to company success and it’s highly recommended that you create a detailed flow or process chart for steps through the entire process. It should document all steps from customer case filing through Support, Engineering, QA, back to support and back on to the customer. Responsibilities and response times should be clearly identified/set for each phase of the process in order to meet SLA’s (Service Level Agreements) with the customer.
    • Professional Services: The PS or Professional Services team typically is involved with customizing the product ‘above’ the APIs in order to enable customer success. As such they collect feedback (their own or the customer’s) on a regular basis regarding product extensibility, API documentation, API reliability, etc. This feedback, much like support feedback has to be documented and logged in a tracking system for future resolution.
    • Presales/SE: The pre-sales function (including Sales Engineering) typically deals with a wide variety of leads or sales prospects. Typically some percentage of these prospects will buy the product and statistics should be kept that document why folks are buying, (price, features, performance, etc.) and likewise, a root cause analysis should be done that captures why folks are NOT buying your product. This info should be fed directly through Product Management in order to prioritize/influence future roadmap discussions.
• Product Adoption rates
• Competitive concerns

Marketing Feedback

• Competitive & Market Analysis
• Analysts and Media Product Reviews

Customer Service Feedback

• Quantitative Information
  • gather statistics from trouble ticket system
  • monitor trends with new releases
  • item for the VPE Dashboard: trouble ticket trends: counts, count by category and severity

• Qualitative Feedback
  • asking customer service reps for their impressions of customer concerns
  • request feedback when CSR’s suspect an issue: this can give faster feedback than waiting for quantitative results
  • establish a regular program for gathering this input

Gathering feedback as part of the Product Lifecycle

‘Customer Understanding’ is needed at nearly all phases of the Product Lifecycle not just during requirements gathering. Once again, the advantage of the Agile model (release early and often) allows you to quickly tune a product to customer needs, assuming the customer has the patience. Other tools and processes useful in gathering this feedback at various stages of product development include;

• Development of UI prototypes and surveys (important during design stage)
• Formalized ‘Beta’ phase is highly encouraged, assuming you can get a representative set of beta testers and with the various roles listed above.
• Role of Human Factors and Usability Testing is critical in customer advocacy & formal Usability Lab testing with real customers. there is a fine-art to professional usability testing. At Portal Software we were fortunate to have a dedicated team of Human Factors experts who videotaped the customers/prospects performing task based product evaluations. These tapes were later analyzed in detail for user interface efficiency, frustration factors, documentation and online help effectiveness, etc.

One major advantages of having customer understanding permeate the product lifecycle is that there are explicit discussions and decisions on how customer will react to the product. These models can then be tested against the customer feedback after the product is released. This should help to refine the customer understanding, and help point the way to future improvements.

Customer Advisory Boards

Customer Advisory Boards can also play a key role within startups that don’t have existing customers. The trick is to source and retain members that are representative of your target customer base. If you’re a company targeting Fortune 500 companies as customers, it doesn’t make sense to have an advisory board composed of small medium business representatives.

Once product development is done and the product has been in the customer hands for some period of months/years, another tool for getting customer feedback is the ‘Prognostics Surveys’ which are popular in Silicon Valley. These surveys can be somewhat expensive but track customer feedback on issues like product fit to customer needs.

How else can we get customer feedback?

• How to capture this information?
• How to share it with your team?
• How to evaluate one individual vs others?
• How to extrapolate from a single individual?
• How to mine individuals for collateral input
  • Budget authority
  • Who else is a stakeholder?
  • Regulations / constraints?
  • Sources of trust / reviews / referrals?
  • Ways to stay technically current?
  • Peer groups – what do peers do?

Next Steps

We can answer all these questions much more deeply by taking a closer look at Requirements Management, and specifically introducing the notion of the Application Scenario. This will give us a framework for formalizing not only requirements, but also of customer understanding itself.

Note: This is an excerpt from “Leading and Managing in Silicon Valley“, a book co-authored by Sam Hahn and other VPEs.