Advice for Program Managers: 6-Quantified Agile for Hardware

Summary: Applying Agile methodologies to Programs that include hardware development.

This Series, “Advice for Program Managers”: Modern Program Management requires skills and methods specialized to characteristics of its industry; to technologies of its produced products and services; and to management of its organization. In execution, Program Management envisions, creates, organizes, and rationalizes information and leadership across all these domains. The leadership and information created make Program Management an integral part of management of a majority of work flowing through an organization.

BACK to Series Start — Advice for Program Managers: 1-Program Management Opportunity

The Series also solicits contributions to this Blog site, to extend coverage of PM Best Practices and Core Program Structure to a broadening set of industries and technologies.

Why Agile for Hardware?

System programs managed using Phased methodology as described on this blog site and embedding multiple methodologies including Agile/Scrum, are quite successful for mixed-technology development. They’ve achieved on-time, on-budget, fully-featured programs. They implement Agile Mindset even if not Agile methodologies throughout, using cyclical development iterations and using continuously updated planning for tasks, dates, logistics and finances as needed for hardware development. These Programs can include Agile methodology just-in-time planning for certain product elements. What could be improved by incorporating Agile/Scrum methodologies for use in hardware development?

Improved cross-coordination: across product elements, across product element technologies, across organizations, across operation with partners.

Improved logistics, finance and operations better incorporating needs of software development efforts into hardware and manufacturing operations, and vice-versa.

Improved management of mutual dependencies between software and hardware. Hardware doesn’t exist without software, and software efforts may need hardware.

Hardware companies are increasingly dominated by software development effort to manage and to use their hardware product, to lock-in customers, and to integrate with foreign product. We can better homogenize internal development methods across software and hardware.

A software company with broad current commitment to Agile methodology might acquire or create a hardware effort (maybe VR, Phone, Console) and desire to homogenize their internal development methods.

Common hardware development methodology is based on design by individuals coupled with personal interactions, weekly project meetings and design reviews. This works well and people like it; but this methodology is highly dependent on individuals. Agile methodology enforces cross-domain interaction and increases transparency of progress via timeboxed short-term plans, just-in-time decision-making, and continuous review of progress.

Current Proposals

Several organizations have attempted to describe use of Agile/Sprint/Scrum methodologies for development of hardware.

CPrime

Agile Processes for Hardware Development: proposes 2–8-week sprints for HW prototyping.

Agile Vox: Scrum for Hardware?

Eleven Lessons from Agile Hardware Development

Plantronics Applies Agile for R&D Teams: Agile for Hardware Case Study

Modified Agile for Hardware Development (MAHD)

Identifies the right problems to address for hardware development (lead time, etc.)

Describes extended Sprints (IPAC) to accommodate HW prototyping: 2-8 weeks.

Supports hierarchical sprints within a project to facilitate long Sprints for hardware logistics processes.

Supports concurrent development of multiple sub-projects. This is mentioned, although coordination or integration processes among sub-projects are not described in detail.

These both identify real problems to be solved for hardware development, and both have good new ideas about applying Agile methodology. They also bend some common Agile practices, a necessary step for this application. Much more is needed, though, to develop commercial hardware product.

The key hard element of these proposals is longer sprints on the order of 8-10 weeks to handle hardware logistics. The identified proposals also describe updates to other Agile artifacts including User Stories, Release Plan et. al.

No doubt it’s possible to develop hardware prototype units within 8-10 weeks.

Local quick-turn board fabrication, on-hand components, build at local Contract Manufacturer.

Quick fab is dependent on board-size and geometry, pitch, layers, availability of key components. It’s also dependent on board fab and assembly technologies incorporated.

Ahead of a build, one must include planning and materials acquisition, and following a build include test and qual activities, customer experience, and evaluation tasks.

Proposed methodology focuses on product design and prototyping. Full productization is not directly addressed.

But fast builds, User Stories and Release Plans can’t be the full answer.

Why not? Let’s review what it takes to develop a product that includes hardware elements, and to productize it for commercial or consumer use. Following that, a proposal for how we can apply Agile methodologies to that. A bottom-up approach rather than top-down.

We ask, “How can we incorporate Agile methodologies to benefit a Program that includes development of hardware?” instead of asking “How can Agile methodologies be used for hardware development?”

The article is going to go a bit beyond “Agile for Hardware”, to illustrate how this would be done in practice.

An article in the MPUG (Microsoft Project User Group) July 2024 online publication outlines scenarios for which more is required than only core Agile methodology, and suggests how to proceed. This is pretty consistent with this softtoyssoftware.com article.
What if Agile is Not Enough for Project Success? 4 Scenarios When Agile Isn’t Enough and What to Do Instead
by Vartika Kashyap

Hardware Is Gonna Hardware

Engineering vs. Production Fast builds support functional design checkout to accomplish engineering proof-of-concept, and potentially could be used for experimental product or low-volume production.

However, they are inadequate for most activities of a commercial development program, at least for products intended for enterprise-level use or for mass-consumers. For these, build cycles are longer than the proposals allow. Builds are further embedded between preparation and qual tasks that add significantly to time required. For some specifics, the article cited above, Phased methodology, describes time required and the reasons behind that. Here’s a summary.

Validation of Production Product The hardware product itself that’s ultimately validated, qualified, and certified must have been produced in its production environment. Release to manufacturing doesn’t happen after engineering checkout; engineering of the product itself, plus manufacturing sources, procedures, test, and metrics are intimately connected, affecting the development process from start to finish.

“Prototypes are easy, production is hard.” – Elon Musk re: Tesla

Parallel and Iterative Development of Product and Manufacturability Hardware and software both need functional testing during development that covers normal operation, corner conditions, stress, capacity, performance and more. This is done by product development organizations in bringup and directed testing. SQA effectively analyzes these for integrated software and hardware during development and then into production.

Beyond development of function, hardware must be physically realized. Internal and external technical margins are inherent characteristics of physical hardware and affect product operation and reliability. Hardware also is sensitive to both its manufacturing and operating physical environments. Producibility and cost are tied in too. All of these are evaluated and adjusted iteratively during development and manufacturability setup. Changes to hardware product design and test design are inevitably required.

To meet reasonable FCS schedule (First Customer Ship) for enterprise-quality use, production capability is brought up concurrently with development and validation of the product design. From the tasks outlined, it’s clear that product design and manufacturing capability are tightly linked so that completing design cycles and then releasing to Manufacturing is not an adequate plan.

Production capability setup is sketched in the article Phased methodology on this blog site and consists of progression along “Tracks” across several Integration Cycles. These Tracks execute material acquisition, fabrication for EE and ME parts; element build, integration, assembly, and test; margin adjustment, multiple sourcing, and yield as capabilities are developed, moving from quick prototypes to demonstrated production capability.

Manufacturing setup specifics developed in prototyping cycles

Setup of business processes for orderability, component sourcing and purchase setup, MRP (Materials Requirements Planning) database, build procedures, assembly procedures, test sequence, test environment, fixture development and propagation, test automation, product software and key installation, sourcing of programmed parts.

Specifics Doesn’t the manufacturer already have these capabilities? In a general sense of infrastructure, yes. But each must be tested and evolved with detailed specifics of the product in development. For all processes and tools mentioned here we’re identifying work specific to developing production capability for all elements of that particular product.

Setup required for Physical build processes including component placement, SMT; physical test (AOI (Automated Optical Inspection) and Flying Probe); board-executable test; firmware update and software load; assembly and test; ICT (In-Circuit Test) fixture and test; adjustment of test coverage. Mechanical bending, molding, casting, machining, inspection.

These all progress along Tracks through a sequence of prototype cycles, updated as progress is made. Examples of progression: initially local suppliers, later production suppliers; initially local quick-turn fabs, later remote production fabs; initially local proto manufacturer, later remote production supplier; initially flying probe, later ICT; initially only boards produced, later assemblies as well; initially soft tooling and sandcasting, later hard tooling.

Multiple Sourcing Product is built within prototype cycles to qualify parts from alternate source suppliers.

Production yield is evaluated, and sources and methods tweaked, and product re-evaluated to achieve required metrics. Yield affects product cost, reliability of operation, and failure rate.

Engineering Qual Cycle Specifics for Manufactured Prototype Builds

PVT (process/voltage/temperature) evaluation of EE and ME margin corners, resulting in adjustment of component choices, suppliers, and values; layout for manufacturability, testability, and signal integrity; firmware setup of device configurations, voltages, and clocks. Mechanical tolerance evaluation and design update. These, plus test coverage and test cases at the manufacturer, are adjusted and re-evaluated throughout the succession of prototype cycles.

Environment In Qual cycles, Engineering evaluates and mitigates failures found in reliability testing; Environmental corner testing; and EMC conducted susceptibility and EMI emitted interference. Safety inspection and test. Mitigations generally require design and manufacturing adjustments just as for other tasks identified here.

Test development and implementation at manufacturing partners quickly become the critical path for hardware development, inevitably requiring product design iteration. Iterations of margin, testability, and manufacturability, are major activity for hardware development, interacting with manufacturing setup, and evolving through the prototype build cycles.

Supply Chain Adversity Hardware development and production are subject to supply chain adversities, needing mitigation. Shipment shortages or damaged parts. New silicon limited availability. Component end-of-life may directly affect a product or may cause a last-time-buy fixed quantity that limits subsequent production, setting a date requirement for product replacement. Chip technology node obsolescence, or chip version succession (e.g., CPU annual introduction cycle) commonly push product replacement with date requirements. Adverse events commonly and unexpectedly affect availability of specific components, like crystals, Zener diodes. And broad supply chain disruption of the 2021 era, or regional earthquake, force unexpected variances to product shipment capability. Mitigations are very dependent on situation specifics but inevitably introduce new work with elapsed time, date and quantity requirements.

Logistics of Integration An enterprise product likely has multiple EE and ME elements to be constructed and tested, then integrated with other element sub-assemblies, tested, and shipped. There are integration functional and schedule dependencies among hardware, firmware, test, and software. Assembly processes and Tests must be developed, then implemented within the Manufacturing flow. Integrations are usually paced by the slowest-built elements and uncover the most obscure anomalies.

Customer Feedback requires hardware assembly integration, hardware and software; followed by physical distribution and installation at customer sites. Then, integration with customer applications and customer environment usage and reporting.

Offshore Shipments For volume and for production cost, production units will likely be produced remotely requiring shipment time within the build progression (e.g., fab material to fabs, components to CM), for delivery to the development site, and to partners and customers. Production suppliers take longer than local quick-turn but must be included in the prototyping and qual sequence toward production, to assure technical margins and yield for real product.

Hardware Requires Quantification

Quantification includes time, dependencies, quantities, configurations, and financials.

Partners Hardware development and productization requires working with partners to perform design tasks, to supply material, to fab EE and ME parts, to assemble, build, integrate, and test. All of these require quantification of numbers of units, time of initiation and need, and cost. Material and build capabilities must be arranged in advance with suppliers: partner capacity is scheduled among all their customers. Material must be marshalled prior to build. Build and test equipment require setup time.

Synchronization is needed for realization of multiple hardware elements in assemblies, requiring quantification of availability and quantity. Purchase of components considering lead time, and timing of aggregated cost are predicted based on planned timing of builds. For all elements, quantity of units to be built, and arrangement for fab and build via partners at coordinated times must be determined. Tests must be available, some at partner sites, synchronized with builds. Supply chain deliveries are often inconsistent. Configurations of built units based on intended use, and acquisition of peripheral equipment such as cables, plug-in cards, power supplies etc. must be determined for acquisition ahead of build and integration. Firmware for test and operation is needed for builds, and platform-dependent software is needed for integration and qual testing at the time of each prototype cycle.

Product Completeness Hardware elements are not as capable of subsequent adds or changes as are software elements. Working with concurrent development efforts requires synchronization of at least key dependencies among hardware elements; they can’t be missed and added later.

Financial Hardware prototypes cost significant incremental spending for materials and services, as each proto build cycle spikes spending. Early proto cycles build few units, that are relatively expensive; later cycles build many more units at lower unit cost but at higher aggregated cost. Prototype spending is a significant component of Program cost, to be predicted every time and well in advance. It doesn’t work to come in with a spend of a million dollars that wasn’t predicted due to lack of planning.

NPV (Net Present Value) For most hardware products, the organization won’t receive revenue until that product is delivered for customer commercial use. That makes schedule to FCS (First Customer Ship) important for hardware products. Production metrics including yield, cost, delivery date and shipment volume all affect Program NPV inherently expressing commercial value of the Program. Programs usually won’t be started without a visible path to achieve NPV normal to the organization. Some organizations track NPV during Program execution and may stop the Program if NPV becomes insufficient.

Propagation of Dependencies Quantification of hardware propagates through all program developed elements and processes due to functional and timing dependencies. Phased methodology describes a method of synchronizing minimal sets of high-level dependencies among elements involved in Integration Cycles and Quals. Detailed dependencies must also be resolved within each element development project.

Demand drives quantification. Each Integration Cycle must enumerate units to be built, specifying configuration of each. Some units may be built to advance development along design and production Tracks, and more may be built to support development of other parts of the Program. Builds usually supply many more units to software development tasks than to any other, needing predictions for those units.

Because of material and supplier lead times and availabilities, plan changes become increasingly difficult as a build approaches and may require workarounds or other mitigations. Also, the moment of irrevocable commitment of material to a build process is of key importance.

Adequacy of Program Methodology For all the above, the methodologies identified for Agile for hardware seem inadequate for commercial programs. And beyond adequacy of steps proposed, they must also fit with structure for manufacturing and quantification needed for hardware technology development of product and production capability.

Quantification is Superimposed onto Methodology

Truth be told, all Program Management methodology descriptions are at too high a level to address these very real process and quantification topics. Nevertheless, Program Managers address them every day. The result is little common knowledge, and no training on what each PM dealing with hardware and systems spends a great deal of effort on. Each PM, each PMO or organization, must develop their own ways of dealing with quantification and logistics of hardware product development and productization. Each technology has its own consequences for sequencing and quantification by Program Management. There is little or no sharing of techniques found effective to deal with common problems, that might broaden opportunities for operational improvement. Consequently, tools are developed by each PM, varying widely in scope and adequacy. No wonder programs fail.

The structure of Phased methodology described in this blog site lends itself to quantification, described in articles linked toward the end of this post. Phased methodology can embed multiple operational PM processes. But since PM methodologies across the board don’t prescribe quantification of technology consequences, any needed quantification is tacked-on to the side of the PM methodology used.

That all sounds unfortunate but, turning lemons into lemonade, that leads to how we can execute hardware development with Agile methodology and still handle the coordination and quantification required by the technologies.

Proposal: Making Agile Methodology Work for Hardware

To make this work, we want to gain the collaboration, adaptability, and current decision-making of Agile, while maintaining or even improving quantification needed to handle precision execution of hardware technology productization.

Remember the advice in 3-Program Management Career Skills that “It’s never all or nothing”? Divide and conquer: De-Construct, Parse, Re-Factor, Piecewise improvement.

Hardware development tasks of greatest commonality with software development are those for product structure and design, test design, and for engineering bringup and test execution. On the other hand, Hardware fab, build, assembly, and manufacturing test have the least commonality with any software development tasks.

Instead of trying to use Agile methodology to manage facets of a technology that it doesn’t fit well, we can re-factor hardware element development projects to separate out tasks for structure and design, bringup and engineering test into hardware development-focused projects. In principle, we identify tasks that don’t need quantification. These tasks are the most like software development and testing tasks.

Then, prototyping tasks for material acquisition, build, assembly, manufacturing test, and other dissimilar tasks (in principle, tasks that need quantification) can be organized into a project providing prototyping service to all Program hardware development projects.

With that, we can manage the engineering/non-quantified projects using Agile methodology since those tasks don’t need quantification and are the most likely to benefit from collaborative and just-in-time properties of Agile methodologies across numerous technologies and organizations.

The Prototyping/quantified project can maintain quantification and methodologies more suitable to its tasks. This prototyping project would involve Dev Leads, Supply Chain and Ops for continued collaboration between development and production.

To add Agile methodology to hardware development we’re using sprint structure suggested by CPrime and MAHD. Importantly, we’re applying them selectively to project activities, and we’re providing infrastructure in other areas of the Program to maintain practices that are also key parts of hardware realization and productization.

Such a Program would still be “Hybrid”, involving other methodologies along with Agile. But much more Program development work and planning, involving both hardware and software, would be done within the domain of Agile just-in-time collaborative methodology.

There could be methodology tweaks on the Agile management side as well to facilitate program management especially of co-development with hardware – pervasive use of Program Increment (PI) planning over a series of sprints, sequencing of initial design and porting tasks and dependencies among elements, sequencing integrations and migrations for support in more environments and platforms, and participation in planning automation to facilitate plan updates to continuously steer and re-sequence both software and hardware tasks in order to follow “the best path from here”. Scheduling of detailed day-to-day tasks may not be accurate but planning a coarser sequence of functionality development and integration, updated frequently, is almost always possible and can keep the project coordinated, aligned, and on-track.

Program Management Structures

The Program runs as a set of projects developing product elements with several technologies and methodologies. Those elements are combined into system product iterations in a hierarchy of integration projects, and each iteration is evaluated by associated qual (“qualification”) projects. The hierarchical and cyclical Program structure is described in the linked article on Phased methodology.

The Program concurrently develops the capability for production and business of the system product, involving a distributed set of internal and partner organizations and customers.

The methodology of breaking down the Program to a set of projects and applying Agile methodology where applicable, now including management of hardware development tasks, and employing quantified methodology where that’s needed instead, is likely applicable to many technologies and mixes.

The next major sections of this article describe in increasing detail how Program Management methodologies including Agile, Phased, DevOps, and Quantified can be used in a system project. The descriptions are based on broader use of Agile in hardware projects divided into major parts managed appropriately by Agile methodology for collaborative coordination, and separate major parts managed in other appropriate domains.

Consider the breakdown method described here, re-factoring tasks into un-quantified projects separate from quantified projects and managing accordingly, as a template to be mapped to your Program.

Sprint Structure for Quantified Agile Hardware Development

Agile-Sprint-Scrum Methodologies Applied

Timeboxed Sprints

Hierarchical and Extended sprints used but not required

Program Increment (PI) Planning

Sprint objectives and planning

Backlog, Estimation, Burndown, Metrics

Periodic Standups

Demo (or review) and Retrospective

Operational prototype is NOT necessarily produced in each hardware Execution sprint.

Advancement of artifacts of design, material, or process may instead be planned and achieved over several sprints. Hardware production of physical prototypes is handed off to a specialized project running outside of the Agile framework.

Agile methodology for hardware development is executed in Hardware Element development projects. Each Hardware Element development project scope could contain development of one or more associated Elements, one of more Assemblies.  These projects contain tasks focused on Element structure and design (EE, ME, FW, Test, Driver), Element bringup and test execution. Build tasks are handed off to a separate Hardware Build project. Qual efforts for Hardware Element-level (bringup and build checkout, margin corner mitigations etc.) are performed in Hardware Element development projects, on prototypes returned from the Hardware Build project.

Element development projects each execute a series of Integration Cycles planned by the System Program Management project. Each Integration Cycle for each project includes design for all technologies included in the project, handoff to the Hardware Build project, and element-specific testing on prototypes returned.

Tasks for the Element that require system-level quantification for prototype allocations, builds, dependencies, timing coordination, and Finance are handed off to a Hardware Builds project that is quantitatively managed. Tasks for qual of elements integrated as Assemblies are performed in separate Qual projects.

Each Element Development project manages each Integration Cycle as an extended-duration Hierarchical Sprint that contains multiple Execution sprints. The Hierarchical sprint plans and manages a story achieving requirements set by the System Program Management project for its Integration Cycle, through contained Execution sprints. Each Execution sprint defines and progresses timeboxed deliverables set by the story.

On handoff to the Build project for prototypes, the Element project prepares for evaluation of returned prototypes, and initiates work of the next Extended Sprint for the next Integration Cycle with planning, designing of updates and deferred function, and by initiating acquisition of materials for the next cycle by collaboration with the Hardware Build project.

Note that an Element development project may work simultaneously among several Hierarchical Sprints, managing overlapping Integration Cycles. For example, the project may concurrently address qual test error mitigation known from a preceding Integration Cycle, and initiate material lead time purchase for a following Integration Cycle. This is normal for hardware development because of material lead times, and durations of prototype builds and of Qual Cycles. The article on Phased methodology shows how this works. This is hardware!

Execution sprints may commonly be 2 weeks duration with flexibility. An extended Hierarchical sprint for an Integration Cycle could take many months, containing many Execution Sprints – for example, nine months for initial architecture, partitioning, logic-, mechanical-, and physical design, and handoffs to the Build project for materials and prototype build. Integration Cycle duration typically declines as the project progresses, to three months for cycles late in the Program.

Hierarchical and Execution sprints perform Agile methodology just-in-time sprint definition, work quantification, backlog, metrics, retrospective etc. The progression of Execution sprints accomplishes objectives set by their parent Hierarchical Sprint. Execution sprints do not each result in a working prototype since work for builds is handed off to another project. Execution Sprints deliver intermediate objectives planned. Hierarchical Sprints on the other hand do produce prototypes at Element level and Assembly level, through the Hardware Build project, for distribution planned and for Qual Cycles associated with their parent Integration Cycle.

A separate System Program Management project defines Integration Cycles, specifying outcomes for Element development projects to achieve for each Integration Cycle, and progress along development Tracks. These outcomes quantify system-level functional dependencies, time, logistics, and finances to be addressed in Integration Cycle work in development projects and in qual projects.

At any time, Element development projects may also develop function beyond these defined outcomes, to be integrated into the Program independently of requirements and timing of the Integration Cycles.

Hardware Builds Hardware Element development projects hand off prototype builds to a separate common Hardware Builds project that runs in parallel with Element development projects. Hardware development tasks requiring quantification (materials, fab, builds, test, integration, warehousing, distribution, dependencies, costs) are executed in the Hardware Builds project. This Build project provides a quantified Service handling material and build tasks, and returns built prototypes at allocated quantity, configuration, integration, and test levels to Element development projects, to distribution, and to Qual projects.

The Hardware Build project further produces Assembly Element integrations of multiple boards, enclosures, and peripheral components (power supplies, cables, plug-in boards etc.)

The Hardware Build project also produces System integration including Assembly Elements and purchased system elements (switches, storage, cabling, rack equipment, etc.), integrated with software Releases by IT or Lab.

Software Development Software function tied to hardware dependencies participates in sprints for the associated Hardware Element development projects. This includes firmware, test, and driver needed for test.

Agile methodology can be used for all software development in the Program. Each software development effort must adopt quantifications, set for progression of Integration Cycles by the System Program Management project.

Quantification is greater for software elements tightly tied to hardware, and lesser for elements with limited mutual dependencies on hardware development. Software function “virtualized” with limited hardware mutual dependency is developed within Software Element development projects, and software with tight dependencies with hardware is developed within Hardware Element development project structures.

Software development sprints deliver functionality and test to a Software Integration, SQA and DevOps project that manages branching, software integration, SQA, releases, bugfix triage, and software endgame.

Program Quantification is handled for all execution methodologies (hardware, software, ops, supply chain) within a System Program Management project that determines System-level quantified parameters (schedule, dependencies, logistics, finance, and Integration Cycle dependencies by Element and Track) for all projects contained in the Program, and covering for all technologies and methodologies including hardware, software, build, integration, DevOps.

Quantification determined by the System Program Management project (Integration Cycle deliverables, schedule, logistics, finance) are estimated from history on prior projects, or due diligence estimates; and kept up to date in the Program plan incorporating actuals and project updates. Quantification is continually updated with these updates reflecting actual state, accommodating estimation error, unforeseen events, changes etc. and planning “the best-known path from here”. The Program plan is related to business, so mitigation of adverse impacts is incorporated into the plan to overcome adversities; changes to key metrics are mitigated, not accepted lightly.

Quantification is done for all projects and services of the Program. Application to hardware prototyping has been discussed. Application could also cover software development projects, for licensing, contracting, or for hardware bought for development such as systems for developers, SQA, Performance and limit testing etc.

In fact, quantification needed for pretty much any Program (item allocation, product structure, schedule, logistics, finance; and including both planning and operation) can be handled by a quantification planning project, running in parallel with design projects, and backed up by operational services (Supply Chain, Ops, Finance, IT, Lab).

Each project plans sprint task content to achieve objectives of its hierarchical task that manages its Integration Cycle work. This includes sequencing tasks to handle dependencies and planned integration points, allocating work into sprints based on task sizes and work velocity etc. Plan, actuals, and metrics are updated to the Program Management project to be incorporated in the Program plan.

Connected Personnel Overlap across projects within this described hierarchy of planning, elements, builds, integrations, qual, and release, accomplishes Program-wide continuous communication and coordination of requirements and dependencies in all dimensions.

Project Structure for Quantified Agile Hardware Development

Now we describe the Program structure, fitting together all the parts to handle Agile methodology used to develop hardware product elements. We summarize project structures, and interactions across multiple projects for function and quantification dependencies.

Hardware Element Development Projects

Project Scope System Programs contain multiple Hardware Element development projects.

Typically, each Element project manages a hardware entity such as a board assembly or a chassis assembly, with tight dependencies among EE, ME, Firmware, Test and Driver elements.

Another example might encompass a group of elements with tight interdependencies among mechanical packaging/skins, midplane/backplane/riser board, and interdependencies for assembly, power, cooling, storage, interconnect, display, and peripherals.

A higher-level example could be a Rack/housing including functional elements, switches and routers, management elements, interconnect, power distribution and cooling.

Project Responsibilities Each hardware element development project is responsible for all elements of its own functional design, EE and ME physical design, release to build, manufacturability, bringup, board test and board qual, including artifacts necessary for material procurement, build handoff, manufacturing, release, and maintenance.

Each project is responsible for firmware for board configuration, management, and product function; for included software including libraries or other Intellectual Property, on-board kernels, manufacturing test and for associated SQA tests; and for artifacts necessary for source management, release, and maintenance.

Element projects are responsible for interdependencies among included project elements and must deliver minimum requirements set within the System Program Management project including integration and delivery of quantifications and required dependencies, with delivery date requirements, to internal and external Release users. Exceptions need to be resolved in the System Program Management project.

Project Management Methodology Hardware Element projects focus on design, test, and qual tasks for which we can introduce Agile methodology to coordinate cross-functional design including EE/ME/FW/Test/Driver/SW dependencies.

Participants in each Hardware Element project include Program or Project Management, Dev Leads for EE, ME, Physical Design; Manufacturing Test, Firmware, entity-local Kernels, Host Driver, System Management, SQA; and Supply Chain and Manufacturing Ops. Some participants may participate periodically or part-time.

Software Element Development Projects

Scope A system Program contains multiple Software Element development projects, each developing a software entity such as messaging system, transport, applications, system management. Each Element project is probably staffed by developers with specialized knowledge. These Elements have very limited dependencies on hardware, and often can run on prior generations or on platforms outside the scope of the targeted system development Program.

Responsibilities Each Element project is responsible for development of operational function, for element manageability function, and SQA test development for function developed in each sprint.

Element projects are responsible for interdependencies among included project elements and must deliver minimum requirements determined within the System Program Management project including quantifications, integration, and delivery of required dependencies, with delivery date requirements, to internal and external Release users. Exceptions need to be resolved in the System Program Management project.

Project Management Methodology Project methodology can be Agile or other continuous development methodologies.

Participants Software element development project participants include developers, dev leads, Agile methodology management roles, and Project or Program Management.

Software Integration, SQA, DevOps Project

Scope Specifies and implements the system development Program branch plan including PI (Program Increment) plans; private branches as needed for software element development projects; periodic or CI/CD continuous branch integration and test; tools and facilities for SCM (configuration management and branch management) and repository; change promotion; test environments and SQA testing; and build distribution internally and to partner and external organizations.

Drives bugfix triage and endgame including finalization of release candidate branch, and internal and external distribution facilities. Endgame includes SQA resulting in an Early Adopter Release for hardware and software platform certifications, in parallel with customer re-certification with applications; a Certified Release for customer production; and Post-FCS Releases for bugfix, feature add, and platform re-certification.

Responsibilities This project is responsible for interdependencies among included project elements and must deliver minimum requirements determined within the System Program Management project including quantifications, integration, and delivery of required dependencies, with delivery date requirements, to internal and external Release users. Exceptions need to be resolved in the System Program Management project.

Project Management Methodology Project artifacts delivered integrate with project management methodologies in use for all element development projects. This includes deliverables for hardware-build operations at partners, including firmware installation by component suppliers; and installation of firmware, software, and security and license keys, by a CM (system Contract Manufacturer). These must support authorized software availability for installation at customer sites by Field Support for installation and software updates.

Participants include DevOps and SQA, Dev Leads, Agile roles and Program or Project Management, and Product Management.

Hardware Element Build, Chassis Integration, System Integration Project

Scope This project handles day-to-day execution of prototype build cycles for Board-level and Chassis-level CM operation for all hardware-element, Assembly, and System Integration projects.

This project includes Supply Chain, CM (Contract Manufacturer or Ops Proxy), and Ops (Manufacturing Operations) and works as a service for Hardware Element projects. This project provides physical material acquisition, build, assembly, integration, and test Services to multiple hardware element projects, for each of their Integration Cycles. The project handles overlapping material and build tasks within and across multiple Hardware Element projects and across multiple overlapping build phases.

Tasks include EE/ ME Materials acquisition, EE/ME Fab, EE/ME Board Assembly and Test; Chassis materials acquisition, Assembly, integration, and test; System materials acquisition, System build, IT integration and test.

This project manages board-level materials cost and contract management, ICT (In-Circuit Test) development, Manufacturing Test environment and implementation, and First-Pass and Net Yield common manufacturing requirements.

Manufacturing Operations involve planning and execution with fab partners, CM scheduling and management, Incoming material warehousing and built-element warehousing, physical integration, IT and release integration, and distribution and shipment to using-projects as set by System Program Management planning.

Responsibilities Must deliver minimum dependency requirements determined within the System Program Management project.

Project Management Methodology Tasks in this project are heavily quantified and managed. For each hardware element, Integration Cycle dependency requirements, Schedule, Logistics, and Finance are set within the System Program Management project and required exceptions and changes are resolved in that project.

Participants include Program Management, Supply Chain, Manufacturing Ops, Manufacturing Test, and hardware Element development project Dev Leads. IT participates for chassis and system builds.

Integration Cycle Qual Projects

Scope These Qual efforts and more are usually defined by corporation standards addressing Quality, Branding and Legal product delivery requirements.

EE/ME Environmental Qual margin assurance including PVT (process-voltage-temperature margin corner management), RDT (reliability test), shake (think shipment), clock skew setup/hold margin management etc.

Electro-magnetic conducted susceptibility (EMC).

Safety/UL/CSA compliance (international).

Electro-Magnetic Interference standards compliance (EMI) (international).

Privacy, Security, standards certifications (international).

Responsibilities For these Qual projects, project management methodologies; Cycle requirements, Schedule, Logistics, and Finance are determined within the System Project Management project.

Must deliver minimum dependency and quantification requirements set within the System Program Management project. Exceptions are resolved within the System Program Management project.

Participation Participants in each Qual project include Program or Project Management, Qual Project Dev Lead, and ME or EE Dev Lead for quals involving hardware, Software Dev Leads for quals involving certifications.

System Program Management Project

Scope Overall system-level quantified capture and planning for the Program across all projects, across technologies, organizations, multiple-methodology integrations, and across methodologies specific to contained project technologies.

System Tasks and Schedule Plan based on quantified due-diligence element plans, in cooperation with Dev Leads and organizational Line Management.

Logistics plan: Equipment usage/configuration/cost/build/distribution.

Finance plan, detail, and aggregation, with required plan/actual update and review.

Defines quantified Integration Cycle and Track Dependency target matrix, Schedule, Logistics, Finance.

Quantifications applying to logistics of Hardware and software Element design, build and test, System-build-integration cycles, and Qual cycles.

Continuous update and reporting of plan and actuals, forward planning.

Examples of these are illustrated in the Phased methodology article on this blog site, and further articles are linked toward the end of this post.

Participation includes Program Management and Project Management interacting heavily with Dev Leads for hardware, software, SQA, DevOps, Product Management, Finance, Supply Chain, Ops, and organizational Line Management and Senior Management.

System Architecture Project

Scope There are numerous architectures involved in a System development program, some are enumerated below. Some exist independently, many of them touch adjacent architectures.

Architectures may pre-exist, or they may be developed outside of Program Management constructs – they are seldom scheduled or quantified in advance.

Even a single system element exists within multiple architectures: physical partitioning; interconnect and networking; partitioning of function among elements, clients, and servers; messaging, streaming, real and virtual memory and storage allocation and access; management of elements, platform, developed; integration with management of co-existing system elements from other suppliers Open Source and other IP. And more…

Responsibilities A system development project collects proposals for new architectures and adaptations of existing architectures at the start of a program, to be incorporated in the product to be developed, for mutual presentation and review to ensure completeness, compatibility, work requirement, fit with product requirements, and solid product definition.

Conclusion of a System Architecture project defines significant content leading to an exciting Program Kickoff event.

Participation in Architecture Review includes Program Management, Product Management, organizational management, and functional element Architects.

Conclusions

In a Program managing projects including both hardware and software product development, some task types can benefit more than others from using Agile methodologies.

Tasks that are relatively un-structured have the most to gain from Agile. For development of hardware, these tasks center around design and test efforts. Examples of high-level design work include logical and physical architecture and partitioning, and examples of low-level design work include logic, mechanical and firmware design, physical design, design verification, representation in design tools, and specification of parts and other tangible representations of product. Tasks for test include design and execution for manufacturing validation, engineering design validation, and for formal qual tests covering design corner analysis, reliability, EMI, Safety etc.

These design and test tasks cross boundaries of technologies, organizations, and individuals, all requiring collaborative effort. A plan may represent this work with limited detail, but detailed execution of this work is the domain of Agile methodology.

On the other hand, tasks for hardware for prototyping, and ultimately for production, must be defined in substantial detail so that partners can produce units and to support organizational operating requirements. Paragraphs above in this article enumerate broad requirements for quantification. Production of physical units requires plans predicting details long in advance. For these tasks, there is limited benefit from Agile methodology.

Further experience may show more nuances of affinities of work types with management methodologies.

The proposal in this article is to classify Program efforts as quantified and un-quantified, and to organize work management around elements in these classifications. Agile methodology then nicely fits un-quantified, less-structured efforts, and is supported by quantified services. The less-structured work benefits from collaboration and flexibility of Agile, and the structured work results in precision realization of that work.