Quality Management

What Is Software Quality?

Everyone wants "high-quality software", but you cannot plan, check or pay for something nobody has defined. The lecture's central message is that quality cannot be bolted on after the system is built; it has to be built in from the start (W06 slides). Software Quality Management is the planning, assurance and control work that builds it in. Before any of that work can start, the team needs to agree what "quality" means for this product.

Three definitions to quote exam

The course gives three definitions. They are worth learning word for word, because each one points at a different yardstick you can measure the product against.

Two viewpoints: the end user and the developer

The slides contrast how the two groups who live with the software see quality. An end user wants software that is fit for purpose, reliable and of reasonable performance, easy to learn and use, and that helps them reach their goals. Users will even accept a hard-to-learn function if it is important enough to them. These are external characteristics: they are about the software's behaviour, which you can see without opening it up.

A developer looks mainly at internal characteristics, which are about the software's structure: the number of faults, how easy it is to modify and test, whether the design is understandable, how reusable it is, whether it conforms to the requirements, its resource usage and its performance.

Many views, one definition per project

Because quality can be seen in several ways, CISQ (the Consortium for Information & Software Quality, 2022) suggests building a definition for each project from four aspects. The two famous definitions above are the first two; the third brings in published standards; the fourth is the quality of the code itself.

Pressman also reports Garvin's five views of quality, which are useful for recognising which view a speaker is taking:

Quality is relative to the application

Whichever view you take, quality is relative to the application (Cavano & McCall 1978). Spaceflight software needs far more reliability than a teaching program, and being "10% faster" is not "better" if memory, not speed, is the real constraint. That is why a project must make its quality goals explicit and measurable: "good" means nothing until you say good at what, and how good.

The Cost of Quality

Quality work is not free: reviews, tests and training all take people's time. But skipping that work is not free either, because defects that get through must be fixed later, often at a much higher price. The cost of quality puts both sides on one ledger so you can argue about the trade-off with numbers instead of opinions.

Conformance and nonconformance exam

The slides, following Schwalbe, split the total in two:

Each part divides into two categories. Prevention stops defects being made in the first place; appraisal finds defects by checking the work. On the failure side, internal failure and external failure differ only in when the defect is found: before release or after it.

Schwalbe adds a fifth category, measurement and test equipment. The timing matters a great deal: CISQ (2022) puts external deficiency costs an order of magnitude (about ten times) above internal ones.

Worked example: does a requirements inspection pay for itself?

This example puts numbers on the trade-off. Assume a project has 50 requirements defects, and measure effort in a unit X = 2 hours. The relative cost of fixing a requirements defect depends on the stage where it is found (NIST 2002): 1X during requirements, 5X in coding, 10X in system test, 15X in beta and 30X post-release.

Sensitivity: the answer depends on the curve

The saving comes entirely from the rising cost curve. If the cost to fix were flat, 1X at every stage (which is roughly what Menzies and colleagues found, §7.3), scenario A would cost 50 × 1X = 50X = 100 h, and B would cost 40 h + 50 × 1X = 140 h. The inspection would then cost more than it saves, and could be justified only by the fewer escaped defects and the reduced harm to users. So in an answer like this, state your assumptions.

Why Late Defects Cost More (and When They Don't)

The worked example in §7.2 assumed that a defect costs more the later it is found. That assumption drives most arguments for early reviews, so it is worth knowing why it might be true, what evidence supports it, and what evidence questions it. The exam rewards a balanced answer here.

Where the extra effort comes from

The slides split the extra effort a late issue causes into three parts. The later the issue is found, the bigger the first two parts grow, because more work has been built on top of the mistake.

1. Effort already spent getting it wrong. The design, code and tests built on a wrong requirement may all have to be undone.
2. Resolution-process effort, such as getting a change approved through change control (Part X).
3. Fix effort, which may be minor (changing a display colour), significant (rebuilding a workflow) or a project-stopper (the architecture cannot handle the transaction throughput).

All of this assumes the issue is fixed. If it is left unfixed, the cost does not vanish: it becomes nonconformance, paid by users and by the organisation's reputation.

Evidence that early detection pays exam

Several sources put numbers on the cost of poor quality. You do not need every figure, but you should be able to quote one or two from each source and say what they show.

CISQ 2022 (US) estimates that the cost of poor software quality is at least $2.41 trillion, about 10.32% of GDP ($23.35 T). Finding and fixing defects accounts for $607 billion of that. Over a large system's life, nearly half of every dollar goes on finding and fixing bugs. Technical debt, the future cost of shortcuts already taken in the code, is about $1.52 T and is reported separately.

NIST 2002 found that inadequate testing costs the US $59.5 B a year, of which $22.2 B is avoidable. According to the slides, a bug found in coding or unit testing takes about 4.9 h to fix, while one found after release takes about 15.3 h, roughly three times as long.

Boehm (1981) reported a factor of up to about 100× for requirements errors found in operation on large projects; the curve is flatter on small projects. Finally, the slides point to the human cost of defects: the 1991 Patriot missile bug and the troop deaths that followed, and radiation-therapy overdoses that killed patients. Some external failures cannot be "fixed" at any price.

The counter-evidence: Menzies et al. (2016) exam

Menzies, Nichols, Shull & Layman (2016) tested the delayed issue effect (DIE): the idea that the longer an issue lingers, the harder it is to fix. They studied 171 projects (2006–2014) that used the TSP (Team Software Process, a disciplined, heavily measured way of working). They found no consistent or substantial increase in resolution time for issues fixed later. They also point out that nearly all citations of the DIE trace back to Boehm's 1970s data.

Menzies and colleagues offer five explanations for why the DIE did not show up. The DIE may be:

1. a relic of the punch-card era, when any change was slow and costly;
2. intermittent rather than a law;
3. confined to very large systems;
4. reduced by change-friendly methods such as agile;
5. reduced by modern tools such as refactoring, version control and CI.

Planning, Assurance and Control exam

"Doing quality" is really three different jobs: deciding what good looks like, setting up the way of working that produces it, and checking that the results actually meet it. Mixing these up is one of the commonest ways to lose marks, so the course gives each its own name, definition and output. The definitions come from Schwalbe.

The three processes form a loop. Planning sets standards and measurable goals; assurance puts processes, V&V and testing in place to meet them; control checks the results. Control then sends corrective action back to planning (change the plan or the standard) and defects back to assurance to be fixed.

The slides also draw QC watching each step of the work from three sides: its input (is what we were given good enough?), its process (are we doing it the agreed way?) and its output (is the result good?).

QA vs QC exam

The single most important contrast: QA looks at the process and tries to prevent defects; QC looks at the product and detects and corrects them. QA happens before and during the work ("are we set up to do this well?"); QC happens on the results ("did this particular thing come out right?").

An SRS (software requirements specification) is the document that records the requirements. Notice that QA defines the review process, but inspecting this particular SRS is QC: one is setting up the way of working, the other is checking one result.

Where do audits belong? This course's answer

An audit is an independent check that work conforms to defined standards (§7.8). Textbooks often file process audits under QA, because they are about the process. This course files them under QC: the W06 slides list audits, both product and process audits, under "Quality Control and Monitoring" as a type of technical review. The quality plan's process description still names audits among the QA processes, so the safe split is:

Why QA concentrates on process (2000 Q8a)

A predecessor subject's exam (433-443, 2000 Q8a) asked why QA focuses on process rather than product. The guide's answer has four parts:

QA: Standards, V&V and Testing

Quality assurance needs concrete tools, or it is just good intentions. It works through three things: the standards the team adopts, verification and validation, and testing.

Product standards and process standards

A standard is an agreed, written rule for how something must look or be done. The slides distinguish two kinds. Product standards apply to what the team produces; process standards apply to how the work is done.

Verification vs validation exam

V&V is the pair of checks every artefact faces. Verification asks whether you are building the system right: does each artefact faithfully follow the one before it? Validation asks whether you are building the right system: does the product actually meet the stakeholders' needs? A perfectly verified system can still fail validation if the requirements themselves were wrong.

Testing at four levels

The slides call testing "one form of verification", done in every phase, not just at the end. It works at four levels, each with a wider scope than the last:

To these four levels you can add regression testing (re-running existing tests after every change to catch anything that broke; it is the core of CI, §7.10), as well as performance, security, usability and exploratory testing. The V-model (Part II) pairs each specification level with a test level, so each test level has a document to verify against.

The Agile Testing Quadrants

Agile teams use a 2 × 2 grid to make sure their tests cover more than unit tests. The quadrants sort tests along two axes: do they face the business or the technology, and do they support the team (guide development) or critique the product (find what is wrong with it)? Each quadrant has a typical way of being run.

Quality Planning: the Software Quality Plan

Planning quality produces one document, the Software Quality Plan (SQP, or SQAP for Software Quality Assurance Plan). It is the contract the rest of the quality work follows: assurance implements it and control checks against it. Without it, a reviewer has nothing objective to check against, and "quality" turns back into opinion.

The SQA Plan template exam

The slides' template has six parts. Part 3 is the heart of it: the plan must not just list quality attributes but justify which ones are critical for this product.

1. The product overview describes the product, its market and the quality expectations.
2. The product plan gives the critical release dates and responsibilities.
3. The quality goals identify the critical quality attributes and justify why they are critical.
4. The process description sets out the QA processes, such as reviews, audits and testing.
5. The document and coding standards are stated.
6. Risks and risk management covers the risks to quality and their responses, linked to the risk register (Part IV).

The course's PMP template (Part V) adds two things. The first is quality roles: the sponsor (the senior person who funds and champions the project) approves the standards and signs off; the PM implements the plan; and quality specialists keep the QA/QC logs. The second is a quality baseline table, which lists each item with its acceptable level, for example "recognition ≥ 98%". A baseline here is the agreed reference level that results are measured against.

Reviews and Inspections

Testing needs something that runs. Most expensive defects, though, are born in requirements and designs, long before there is code to run. Reviews, where people examine an artefact, are quality control's tool for those early artefacts, and they turn out to catch many code faults too. Together with audits and defect analysis (§7.8), they make up QC.

Three kinds of review

The slides describe three kinds of review, which differ in what is being checked and who is checking it.

Technical reviews, and why they pay exam

In a technical review, artefacts are reviewed by peers, with the author involved. It is called a "soft" method because nothing is executed: people read and reason. Reviews have their sceptics, but the empirical evidence says the scepticism is unjustified. The slides give five advantages:

1. Reviews work on any artefact, whereas testing and measurement need executable artefacts.
2. They detect problems earlier, and earlier detection means lower resolution cost (§7.2).
3. Code reviews find 30–70% of faults (up to 80% at IBM), and reviews and testing each find faults the other misses.
4. A review finds the actual fault, whereas testing only shows that one exists somewhere.
5. Under release pressure, fixing faults found by testing introduces more mistakes than fixing faults found by review.

The disadvantages: reviews take time and resources, and they need careful planning. Advantage 4 is easiest to see in a picture. A fault is the mistake in the artefact; a failure is the wrong behaviour it causes when the program runs.

Informal and formal reviews

The slides divide reviews into informal and formal reviews, and divide formal reviews further into walkthroughs, inspections / code reviews and audits.

An informal review is a desk check or a chat with a colleague, with no procedure. It is much less effective than a formal review, because one reviewer brings no group diversity, but it is cheaper. Checklists of generic questions for that type of artefact help.

A formal review, in general, is a meeting of several stakeholders, such as developers, testers and the client. It has 3–5 members and lasts at most 90 minutes (attention fades after that). It has four roles and three possible outcomes:

Walkthrough vs inspection vs audit

The three kinds of formal review differ in who leads, how much preparation they need and what they are for. This is a favourite near-miss MCQ topic.

Inspection checklists

A code inspection uses a checklist based on the coding standard. The slides' requirements-inspection checklist covers five areas:

The Fagan inspection process

The guide gives the standard six-stage Fagan inspection process (standard knowledge rather than a slide). Major rework sends the artefact all the way back to planning for a fresh inspection.

Peer-review roles exam

These are the classic role names, with the slides' names in brackets. A predecessor exam (1999 Q11) asked for exactly this list.

Defects vs adequacy (1999 Q10) exam

A review looks for more than defects. A defect is a specific error. Adequacy is about the artefact as a whole: whether it is fit for its purpose and for the next phase, which means it is complete, has the right level of detail, is consistent, complies with standards and is verifiable.

When the author strongly disagrees (1999 Q12) exam

Reviews can get heated. When the author strongly disagrees that something is a defect, there is no vote and no endless debate. The meeting's job is to find defects, not to settle arguments, so the dispute is moved out of the room:

Audits and Pareto Analysis

Reviews look at artefacts one at a time. Quality control also needs two wider views: is the team actually following its agreed procedures (an audit), and where do most of our defects come from (defect analysis, such as a Pareto chart)? Both turn scattered evidence into a decision about what to fix first.

Product audits and process audits

An audit is led by an independent auditor, possibly an external team, and the authors are not involved. It examines a sample (random or targeted) of objective evidence and checks conformance to defined standards. There are two kinds, and in this course both are QC (§7.4).

The seven audit steps

An audit follows seven steps (an older exam topic). The stepper runs them on the predecessor exams' scenario (1999 Q13 and Q17): auditing a documentation procedure that covers the project's main documents (SPMP project plan, SRS, SDD design, STD test document, UM user manual, VDD version description).

Pareto analysis

Pareto analysis is an older exam topic, not in the 2026 slides, but predecessor exams asked about it (2001 Q7). It rests on the 80/20 rule: about 80% of defects come from about 20% of causes. A Pareto diagram is a bar chart of defect counts by category, sorted in descending order, with a line showing the cumulative percentage. It picks out the "vital few" causes to fix first, and when it is re-plotted after action it shows the improvement.

Category                Count  Cumulative
Requirements ambiguity    70     35.0%   (70/200)
Interface errors          50     60.0%   (120/200)
Logic errors              30     75.0%   (150/200)
UI layout                 20     85.0%   (170/200)
Documentation             15     92.5%   (185/200)
Other                     15    100.0%   (200/200)

The method, in order:

1. Classify the defects into categories.
2. Count the defects in each category.
3. Sort the categories in descending order.
4. Plot the bars and the cumulative-percentage line.
5. Act on the top categories.
6. Re-measure, re-plot and check the improvement.

Agile Quality: Testing First

In a plan-driven project, testing can become a phase at the end, squeezed when the schedule slips. Agile teams cannot afford that, because they release every sprint. So they do not leave quality to a separate phase: in Scrum, each feature is tested within the sprint by the whole team. Acceptance criteria define "done" from the user's point of view, and the sprint review acts as a recurring acceptance check (T06; see Part VI).

The agile QA mindset: Crispin & Gregory's ten principles exam

Alongside the mindset, the slides give three working agile QA principles: (1) test early and often, which is called shift left; (2) automation; and (3) open feedback channels.

Shift-left testing

Draw the lifecycle left to right, and traditional testing sits near the right-hand end. Shift-left testing (IBM) moves testing earlier and more often, and involves testers from requirements onwards. Its practices include TDD, automation in CI/CD, early performance and security testing (static analysis), exploratory testing, and production-like environments. Shift-right testing, which means testing and monitoring in production, complements it.

The benefits are cheaper fixes (§7.2), the confidence to deploy often, and a more testable architecture. The danger is that testing implementation details makes brittle tests that break whenever the code is restructured, even though its behaviour is unchanged. So test what the code does, not how it does it.

Three test-first approaches

Three approaches put "test early" into practice by writing the test before the code. They work at different levels, from the business goal down to a single unit.

1 · ATDD: Acceptance Test-Driven Development

In ATDD, stakeholders and the team first agree the acceptance criteria, which decides what to test before any code exists. The acceptance tests are written next, and the code comes last. The criteria come out of a "three amigos" session (Uffizzi): the business states the problem, the developer proposes a solution, and the tester asks what could go wrong. The slides walk through the cycle for a login feature:

2 · BDD (sometimes "BTDD")

BDD (behaviour-driven development) is a goal-oriented approach that ties requirements to business goals. The Agile Alliance calls it a synthesis of TDD and ATDD that adds three things: the "Five Whys" asked of each story, so that its link to a business outcome is clear; outside-in thinking, which means building only the behaviours that serve those outcomes; and one notation shared by domain experts, testers and developers, namely Given–When–Then. BDD is also called specification by example, and its tools include Cucumber.

3 · TDD: Test-Driven Development

In TDD you write a failing unit test before the code. Robert Martin's three laws make the discipline precise:

1. Write no production code except to make a failing test pass.
2. Write no more of a test than is enough to fail; compilation failures count.
3. Write no more production code than is enough to pass that one test.

Following the laws produces a short cycle of red (a failing test), green (just enough code to pass) and refactor (tidy up while staying green). The three approaches then nest inside each other by level:

CI/CD and QA Metrics

Testing first only helps if the tests are actually run, every time, on every change. Agile teams hand that job to an automated pipeline, and then use a small set of metrics to see whether quality is improving. Both are quality assurance built into the daily routine rather than bolted on at the end.

CI/CD as QA

The slides draw CI/CD as an infinity loop. Plan, code and build make up CI (continuous integration); release, deploy, operate and monitor make up CD; and the two halves are joined by continuous testing. (The configuration-management side of CI/CD is in Part X.) The relevant Agile Manifesto principle 1 is to "satisfy the customer through early and continuous delivery of valuable software".

The three practices work as follows:

• In continuous integration, every commit builds the whole application and runs a comprehensive automated test suite. If the build fails, the team stops and fixes it immediately. The goal is software that is always working. Because bugs are caught early and cheaply, CI teams deliver faster with fewer bugs (Humble & Farley).
• In continuous delivery, every change that passes the pipeline is releasable, possibly with a manual business gate. Continuous deployment goes one step further and releases every passing change automatically.
• Continuous testing (Atlassian) tests at every scale, as the figure below shows. CISQ warns against CI/CD without continuous quality engineering: automating the release of untested code just ships defects faster.

The pipeline below shows CI's stop-the-line rule: a failure at any test stage sends the team back to fix it straight away, before anyone builds on broken code.

Agile QA metrics

A metric is a number tracked over time to show how quality is going. The slides (with formulas from TestRail) list both kinds:

Pick 5–10 metrics that you can act on. Pair leading indicators, which warn before problems arrive (such as test coverage), with lagging ones, which confirm what already happened (such as escaped defects). Avoid vanity metrics, numbers that look good but drive no decision.

Worked example: a release's quality metrics

A 12 KLOC release (KLOC = thousand lines of code) has 400 planned tests, of which 380 were executed and 361 passed. Of the 58 defects, 48 were found before UAT, 6 in UAT and 4 in production, and fixing them took 174 h at $50/h.

Defect leakage = 9 / 75           = 12%
Defect density = 90 / 20 KLOC     = 4.5 defects/KLOC
DRE            = (75 + 9) / 90    = 84 / 90 = 93.3%
MTTR           = 270 h / 90       = 3 h
Cost per fix   = 3 h × $60        = $180

Quality Standards and CMM

Every organisation could invent its own idea of a good process, but then a client could never compare suppliers, and nobody could learn from anyone else. Published standards and maturity models capture what has worked elsewhere. This section covers documentation standards, the ISO 9000 family, the IEEE standards and the CMM/CMMI maturity levels.

Documentation standards

Documents are the tangible form of the software, so they have standards of their own. Documentation process standards define how documents are developed, validated and maintained. Document standards cover identification, structure, presentation, change highlighting and interchange.

Standards: pros and cons exam

The ISO 9000 family

A QMS (quality management system) is an organisation's documented set of policies, processes and records for managing quality. The ISO 9000 family defines what a good QMS looks like:

The family rests on seven principles: customer focus, leadership, engagement of people, process approach, improvement, evidence-based decision making and relationship management.

IEEE standards

The guide lists the main IEEE software-quality standards as standard knowledge:

CMM / CMMI maturity levels exam

The Capability Maturity Model (CMM) was developed by the SEI (Software Engineering Institute) at Carnegie Mellon. It describes the key elements of an effective process and a path from ad hoc and immature to mature and disciplined, and organisations are rated at Level 1–5. Each level adds key process areas (KPAs), the practices an organisation must have in place to reach it. Climb the stairs one level at a time:

Reading down the Focus column shows the progression: attention moves from individuals, to project management, to the engineering process, to product and process quality, and finally to continual improvement.

Problems implementing CMM, and remedies (1998 Q9)

A predecessor exam (433-443, 1998 Q9) asked why CMM programmes fail. Each common problem has a matching remedy:

Quality Models: McCall and ISO

"Make it high quality" is not a requirement anyone can test. Quality models solve this by breaking "quality" into a fixed list of named characteristics, so a team can pick the ones that matter, define each, and attach a measurable target. The course covers three: McCall's model, ISO/IEC 9126 and its replacement ISO/IEC 25010.

McCall's model (Cavano & McCall 1978)

McCall's model is a hierarchy. At the top are factors, which are external and reflect the user's or management's view. Each factor is broken into criteria, which are internal attributes of the software. Each criterion is measured by metrics taken during development. It has 11 factors grouped into 3 perspectives, usually drawn as the sides of a triangle. The user question beside each factor is a quick way to remember what it is about.

ISO/IEC 9126 (2001, now withdrawn)

ISO 9126 was withdrawn and replaced by ISO/IEC 25010, but it still appears in older material. It had six characteristics; each also had a "compliance" sub-characteristic alongside those listed here:

ISO/IEC 25010:2023 (SQuaRE): nine characteristics exam

The current model, part of the SQuaRE family of standards, has nine characteristics. They are used for requirements, test objectives, QC criteria and acceptance criteria. Two were renamed in 2023 and one is new:

Comparing the models

McCall groups its factors by lifecycle use: operation, revision and transition. ISO's models, by contrast, are consensus-based lists of characteristics and sub-characteristics. What all three have in common is that they turn "quality" into specific, measurable attributes that can go into requirements and quality plans, which is exactly what part 3 of the SQP needs (§7.6).

Applying It to the Medic Case

The tutorials' running case brings the whole Part together. Medic is a haptic virtual-reality trainer for temporal-bone surgery, built for the hospital client RVEEH. Trainee surgeons "drill" a 3D model of the ear bone using a haptic device, a robotic arm that pushes back on the hand so that drilling feels real, while wearing stereo glasses. It must be reliable (reliability being the probability of not failing over a given time), it must meet strict performance criteria on execution time, and it must be usable by surgeons.

The team is 12 engineers who are new to C++, 3D rendering and haptics. The case addendum adds a test lead with 5 test engineers, daily collaboration from the client's surgeons, and an RVEEH support lead who takes over maintenance after delivery.

A mini Software Quality Plan for Medic

Using the slides' six-part template (§7.6):

1. Product overview. Medic is a haptic VR temporal-bone surgery trainer for RVEEH. It is expected to be realistic, responsive, stable, easy to learn and maintainable.
2. Product plan. The plan follows the release plan. The PM owns the SQP, the development lead is responsible for code quality and the test lead for the test strategy. The two surgeons are the acceptance authority for realism and usability, and RVEEH IT gives operational acceptance.
3. Quality goals (ISO/IEC 25010). Each critical attribute, why it is critical for Medic, and an example measurable target:

4. Process description. The QA processes, drawn below: requirements inspections with the surgeons; early haptics and rendering prototypes to retire the performance risk; an architecture review focused on throughput; code inspections against the C++ standard (which also train the inexperienced team); TDD for the force algorithms; CI running unit, integration and performance-regression tests on reference hardware, alongside hardware-in-the-loop tests; sprint reviews and usability tests with the surgeons for validation; UAT; and a mid-project process audit.
5. Standards. A C++ coding standard, document templates, ISO/IEC 25010 and the DoD. The team should also record a decision on whether medical-device rules (ISO 13485) apply.
6. Risks (linked to the risk register, Part IV), each with a response, shown in the table after the figure.

Metrics. The plan tracks escaped defects per sprint, defect density per module, defects found by review versus by test, the performance-test pass rate and latency trend, surgeon satisfaction and team morale: a mix of quantitative and qualitative, leading and lagging (§7.10).

Tutorial Team Activity 1: acceptance criteria for "restore a saved model within 10 seconds"

Three criteria cover the normal case, the correctness of the restored state, and failure:

Tutorial Activities 3 and 4: which DoD criteria are invalid?

A Definition of Done must be objective, verifiable and consistent, and it must apply to all increments (Part VI). Judged by that test:

• Activity 3. Valid: C5 (peer reviewed, merged), C6 (all automated tests pass) and C8 (deployed to the test environment, documentation updated). Invalid: C1 (only MUST-HAVE stories: a DoD applies to every item, whatever its priority), C2 ("compiles on my machine": not verifiable by anyone else), C3 (the PO's subjective approval), C4 (known bugs knowingly left) and C7 (a partly done story; the sprint goal is not the DoD).
• Activity 4. C1, C2 and C3 are valid, because they are automated, measurable and verifiable. C4 is invalid because closing Jira tasks is activity, not quality. C6 (90% pass) and C7 (documentation deferred) are invalid too, and so is C8, because "realistic enough" is subjective: that is validation feedback, not a DoD criterion. C5 is questionable: it is valid only if those two environments are all the supported ones.

Pitfalls and Self-Test

This last section gathers the mistakes that most often cost marks, then works through every self-test question in the guide. Questions tagged with a year are real questions from predecessor subjects' exams (433-443 and 433-341), not SWEN90016 papers; the answers are the guide's model answers, not official ones.

Common pitfalls

Self-test

Without reviews:
  system test  40 × 10X = 400X = 400 h
  production   10 × 30X = 300X = 300 h
  total                         = 700 h

With reviews (1X = 1 h):
  review effort                 =  60 h   (appraisal)
  found in review   30 × 1X     =  30 h
  system test       16 × 10X    = 160 h   (internal failure)
  production         4 × 30X    = 120 h   (external failure)
  total  60 + 30 + 160 + 120    = 370 h

Saving = 700 − 370 = 330 h; escaped defects 10 → 4