Introduction

Managing the trade off between encouraging innovation and meeting tactical objectives is a continual struggle for every Engineering organization. Caution will dictate to avoid innovation and take the tried-and-tested path. A high achieving development organization however will always be pushing to innovate and implement new or cutting edge techniques. In this chapter we look at ways of institutionalizing innovation in your organization. We’ll look at two main areas. First, how does innovation factor in to the day to day operation of the organization, especially for smaller organizations. Then, we’ll look at how we can institutionalize certain kinds of innovation via dedicated research labs.

Innovation and Your Team

When should you encourage innovation and when should you be cautious? Reining in your development team can be demotivating but may be the right choice. On the other hand there are times when innovation can produce breakout results and set you up to meet longer term strategic goals. The answer depends on a number of factors including:

1. how much slack do you have in the schedule?
2. can you schedule the time for innovation?
3. do you trust the developer(s) to meet the schedule?
4. can you mitigate the risk around the deliverable?
5. are the costs fully understood and can the company afford it?
6. are other executive stakeholders bought in?
7. innovation usually comes at the price of product quality (but not always), can you afford the risks? In the industry, the Strategic ‘.0’ class releases either make or break a company.
8. Some companies are driven by true R&D effort where innovation must be factored into nearly every new product line to survive. Chip foundries and Disk drive makers come to mind here. What kind of company do you work for?

Promoting Innovation

Lab or No Lab

Putting commitments to solve not-understood problems on a product roadmap is a behavior that introduces and increases risk to a product roadmap. I prefer to identify problems that have no known solution, and invest in discovering solutions, through an explicit activity track I refer to as Research. When done in the context of iterative development, there are frequent timely occasions (at iteration boundaries) to review the results of research, and decide whether to put those results on the product roadmap.

There are a variety of ways of managing research. In a larger organization you can create a dedicated research lab, which we discuss more below.

Fellows programs (senior contributors whose judgment and alignment with company objectives has been proven and is unquestioned) can be excellent ways to encourage relevant research, but is typically not possible in smaller organizations.

Some accommodation to individual research directions should be allowed in reasonably-sized organizations, but must be reviewed and authorized with an eye toward relevance to the business.

Engaging The Engineers

In smaller organizations where would not be possible to create a “Lab organization”, the VPE should encourage the exploration of possible innovations on the spare time as long as there is no compromise on the milestones. There should be a clear distinction between committed work and innovation efforts.

Typically, the CTO at a given company has the responsibility to drive these efforts, with visibility and oversight from the VPE. They have the responsibility to look out several years across the technology horizon and look at trends in technologies, tools, competition, etc. in order to provide direction to the R&D staff.

If each engineer spends some time thinking in ways of improving the product, out of lots of ideas it will be likely that some new approaches to improve or solve problems will arise. These ideas should be documented in a informal way like and shared with the team. These documents will form a “pool of ideas” that could be reviewed in meetings with the product manager. This process gives engineering the chance to explore ideas without adding risk to releases. In any case it’s important to separate efforts related with possible innovations with efforts required with fulfill milestones.

The VPE has to make clear to the engineering team that the business side will be responsible to schedule the features needed in the product. The engineering team has to understand its role in the process. The interaction between people focusing on addressing business need with people capable to innovate creates a powerful cycle. This allows engineering to participate and the product managers to validate new ideas with customers.

Engaging your engineers also means tasking them (through weekly 1:1 meetings and personnel reviews, etc.) with staying on top of the latest technologies that affect the company. This takes time and $. Encourage your team to attend relevant conferences and to allocate x% of their time to research and not implementation (actually build this time into your staffing schedules).

Also have a reward system in place for innovation, whether it’s a bonus for patents or spot bonuses for great ideas.

The Lead Horse

One technique that I have used successfully in the past is to have a “lead horse” function. Assume a group of 7 developers working on a product. The product releases approximately every 6 months. The “lead horse” starts the next cycle (project) 2 months prior to ship of the current project. The lead horse can be a single person or a team of two. They are charged with staying within the product vision but not held tightly to a priority list of features. They can innovate and prototype during this time. It is expected that by the time the remainder of the team comes off of the last project, the lead horse team will have figured out some cool stuff and will have reduced the risk of implementing the cool stuff through prototypes. These new items can be offered up to PM or management for consideration in addition to the top down requirements. The lead horse role is a plum role and can be used to reward particularly great contributors on the prior project.

This technique requires that you have 1) enough excess capacity (10% or more) 2) a large enough team (doesn’t work well with a product team of 3 people) 3) long enough development cycles to actually get something interesting done.

Benefits include 1) “researchers” are still working on and responsible for tactical delivery of some items so they stay close to the mainline – no ivory tower syndrome 2) ability to “hide” innovation costs in the product development budget. 3) ability for anyone who makes a big enough impact do cool stuff and try out ideas.

Innovation Pipeline

There is a similar technique that I’ve used to allow the refinement of product features in a way that resembles a pipeline where on the first stage the PrdM and VPE with help of prototyping developers create and recreate simulations of what the implementation should look like. The team repeats this process until everybody is happy with the idea and the simulation of how it’s going to work.

The engineers would work as consultants helping identify risk factors and engineering challenges. They also can be source of ideas to be explored and validated with management.

The work produced on the first stage serves as input on the traditional development stage where real applications are being built. The main advantage of this approach is that requirement gets validated by management and refined by the several prototyping iterations. This approach also helps the business select the ideas with more potential of returning real value for customers instead of just “gold plating” the product with nice features but no strong value behind them.

It’s important that the engineers understand that the goal at the first phase is prototyping and not architecture planning. This helps the engineers embrace the several changes requests and feature refinement efforts. As efforts to design any type of infrastructure increase there will be a natural resistance of throwing out the old designs and starting with a new approach

Risks

The major risk with encouraging innovation is to spend effort on activities that don’t actually contribute to the bottom line. The principle issue is that almost all innovation exists on a somewhat longer time horizon, increasing the possibility that the end point will no longer be aligned with the company’s needs by the time it appears.

For example, I was managing the development organization of an enterprise sw company when I was asked to take up a small research team to work with the CTO to prototype a next generation predictive analytics engine to replace the existing one. One of the challenges faced by the company was addressing the needs of customers in new verticals with the existing engine. A new vertical generally required one major release and one minor release for general acceptance–a huge development cost. There was also a big professional services cost to customers from configuring the current engine which was brittle to noisy data and generalized poorly outside the original industry vertical. This new engine promised to vastly simplify the configuration and be a lot more robust to data quality issues. It was believed that the business was about to take off and the current system system would quickly become inadequate as we grew the customer base and number of industry verticals we served.

We set an aggressive 6 month schedule and devoted two star developers to the project. However, concurrent to this effort the company was trying to deliver a major release to a key customer. The release fell behind schedule so after 3 months we had to stop the project to help with the release. After the release was done we finished the prototype of the new engine and presented it to the company. Although the results were every bit as good as promised, the new engine was never productized and the effort was essentially wasted. What went wrong?

1. The company was not really in a position to invest the necessary resources to re-architect the software. The company was losing money and having trouble acquiring new customers in a stagnant economy. The new engine was not a drop-in replacement for the old engine, and coming on the heels of a difficult and poorly planned major release there was no appetite for another big release with no new features for users. Setting up a research team to develop next generation infrastructure can be successful only if the business itself is sufficiently successful to afford it. Otherwise, innovation can only be factored in from within the common engineering schedule. Generally, in a negative business environment caution should rule.

2. There wasn’t the necessary stakeholder buy in at the end of the project. The professional services organization felt like it had its hands full with the current system and all its warts, and was not ready to think about retraining to a new system. The sales organization understood the need to bring the services cost down, but their greatest difficulty was just closing deals on feature/functions, so a big effort for feature/function parity was not very compelling. Product management was struggling with usability issues and also focused primarily on the ui. They already had a big laundry list of features for the next release, so they also were not bought in. Engineering understood pretty clearly what the cost to productize would be, and this should have been communicated with the other parts of the organization earlier so the decision not to invest could have been made before wasting six months of effort. Keeping the other executive stakeholders informed and bought in should happen continuously throughout the project. If the business climate deteriorates, you will very likely lose that buy in and should consider cutting the project. Completing a successful prototype does not mean the project was successful. Cutting your losses early can be a better outcome than completing the prototype and not productizing it.

3. The decision to invest in the effort was made based on optimistic predictions on the business. The underlying assumption was that the business was about to turn the corner. Even if we were “early” with the prototype, it was felt that productizing it could be delayed until the business picked up and greater resources were available. Unfortunately this rosy business scenario did not materialize, nor was delaying the productization of the prototype a useful mitigation. The main codeline was a moving target, so the value of the prototype depreciated with every product release as it further diverged from the target it was meant to replace. Mothballing a prototype is a feasible risk mitigation only if the requirements are static, a somewhat rare occurrence. If you cannot afford to productize the prototype, then you should seriously question why you are building it.

Not all innovation efforts have as much cost or risk associated with them as this example, however in all cases a very disciplined decision making process should be applied to ensure winning outcomes. This chapter will help formulate the framework in which those kinds of decisions can best be made.

Research – Finding Solutions to Hard Problems

For purposes of this discussion, we define Research as technical work that goes beyond the normal scope of development, which requires some novelty or greater depth in it’s solution and is part of Product Development. From a management perspective, it is a technical challenge, whose solution involves either major risk or substantial rewards, or both. Research is a greater unknown than the normal product development: the time lines are more non-linear and un-predictable because it’s hard to know when insight will lead to solutions.

Clearly it is very risky to depend on the uncertain timing of Research for product schedules. However, there are cases where this is unavoidable. Research can give a company a big competitive advantage by providing intellectual property, patents and trade secrets. Start ups often make Research the crown jewel that justifies their entire existence. Sometimes an entire path of development hinges on a novel solution in a very specific area. For a variety of reasons, Research may be required and here are ways to mitigate the risk.

The easiest way to manage Research in a Product Development environment, is to separate the Research so it is independent of the Product Timelines. This requires a Product Roadmap which can proceed without the Research, but be improved or accelerated if the Research works out. There are still challenges to this approach: it is important that the team doing Research stays engaged with the Development team. When the Research is not explicitly part of the Product Roadmap, there is a tendency for the Research to lose touch with the Product needs. I worked in the corporate research labs of a multinational corporation and at times the Research lost complete touch with the Product Development needs. Research itself became the “Product”, but it was a product without customers. However most companies won’t have the problem of setting up independent Research groups, they don’t have the luxury of product Product Roadmaps.

The best way to structure is the relationship between the research group and Engineering is through the regular roadmap process. Generally, the roadmap will describe both long term and short term objectives. Within this context the research group will identify long term but not short term objectives. The long term objectives take the form of specific areas of focus in the research, leaving the group some latitude to decide on specifically what projects to undertake at any particular time. The objective of each group in the research group is to produce some deliverable, whether it’s a paper or a prototype. Once a project gets to the point of producing a prototype, the research group needs to transition the project to the regular development groups, where at that point its further development can be managed via the normal roadmap process within development. The key metric to watch out for is that in fact a useful number of projects from research are actually making this transition. The subtlety here is that with radical research a great many dead ends need to be considered before success is to be had, thus making it hard to quantify what an ‘acceptable’ success rate. A good way of mitigating this risk is to partner with a university so lowering the exposure to potentially huge numbers of unproductive efforts, while not closing the door to the successes when they do happen.

Notice that this model hopes for but does not guarantee that a particular piece of research makes it into the development roadmap at any given time. Making regular development cycles dependent on particular pieces of research is extraordinarily risky and should only be undertaken if there’s really no alternative, and/or one has some a priori reason for believing the effort will be successful. The business team needs to adjust their expectations and timelines to reflect the risk the Research introduces. Marketing and Sales needs to recognize the risk of heavily promoting features which are at risk. Production, Customer Service and QA teams need to understand where the additional risk is and how they can help deal with it. If it’s so important, it may also be useful, budget permitting, to run multiple efforts in parallel, in the hopes that one succeeds. While this obviously takes more staff, it reduces the risk by not forcing a premature choice between likely possibilities. Managing teams which are essentially in competition requires everyone to focus on the larger goal of advancing the Product Roadmap. Teams working on the Research path that is not used need to know their contribution was important and valued.

Research labs can be extremely valuable by providing an opportunity to invest in long term projects that will become crucial in the future, but are far enough out that they would never survive any tactically based short term investment plan. Research labs can also establish partnerships with universities that can bring more innovation into the organization as well as make recruiting easier by raising the profile of the organization with new graduates. This is particularly valuable when starting new sites in other countries. A research partnership with a local university can greatly improve the applicant flow.

Finally, to make sure the lab gets adequately funded, you should as part of your business planning, determine what percentage of your spending you want to spend on the lab, and make sure that funding doesn’t get cannibalized by tactical considerations. The big factors you want to look at as a manager in starting a lab are:

• Do you need long term innovation ?
• Are academic contacts valuable to you ?
• Do you already have researchers who can benefit from working in a lab environment, but will make responsible use of it ?
• Do you have adequate ability to manage this

Success metrics include number of journal articles published, patents filed, number of academic partnerships, and after a while, number of innovations fed back into product development.

Note: This is an excerpt from “Leading and Managing in Silicon Valley“, a book co-authored by Sam Hahn and other VPEs.