| Episodes | 31 |
| Mentions | 45 |
| Cited here | 35 |
| First — last | #76 — #714 |
| Top guests | Jason Huggins, Matt Liberty, Pete Staples |
| Related | calibration · continuous integration · python · test equipment · firmware |
Automated testing is the use of scripted procedures, fixtures, and instruments to verify hardware or software without continuous human judgment, spanning bench verification during development through end-of-line production test. Its economics are driven by volume: at high production rates the capital for purpose-built automation is defensible, while at low volume the design and debugging cost of a fixture is never recovered.[96][182] The discipline divides naturally into development testing, where regression suites guard against reintroduced defects, and manufacturing test, where the optimisation targets are throughput, test time per unit, and deciding how much needs to be tested at all.[505][714]
Economics and volume
Production volume sets the test-time budget directly. Building five million greeting-card sound modules in three months leaves only seconds per unit, which forces automation regardless of how simple the device is.[424] At volumes in the hundreds of thousands of units, a capital budget on the order of a couple hundred thousand dollars for the test system is defensible, covering pogo-pin fixtures, hydraulics that push boards along a jig, and flippers that sort units into pass and fail bins.[96] A single high-speed production tester can cover both digital and analog acceptance in one insertion, verifying the entire ROM contents of a chip alongside standby current, operating current, and output drive in roughly eight to ten seconds per unit.[424]
At low volume the calculation inverts. Building an automated pass-fail jig costs a couple of weeks of design plus an open-ended debugging tail, an effort not recovered at low production volume, and the decision worsens when the fixture must be shipped to a remote assembler who will need troubleshooting guides for it.[182] Whether a board receives manual or automated test is therefore decided by volume: low-volume boards are tested by hand, and the same board moves to an automated fixture once volume rises.[522]
Once a factory’s automated test equipment is fully depreciated, the marginal cost of testing each unit falls close to zero, and the binding constraint on test capacity becomes tester uptime rather than test cost.[71] On the services side, test-automation packages priced as a fraction of an engineer’s salary land in the tens to hundreds of thousands of dollars, which lets a small vendor stay profitable on engineering services rather than scaling to thousands of units of hardware.[369]
Production test strategy
Production test and design verification answer different questions. Once a design is qualified, the automated test on the line checks the assembly of that individual unit, so the failure modes it must catch are those introduced during build rather than those inherent to the schematic.[363] Production test can be scoped as a spot-value verification against specification at a single operating point rather than a sweep across the full range, which is far cheaper than calibration; calibration exists to correct nonlinearities in precision instruments and carries a large maintenance burden.[182] Test coverage strategy differs sharply by industry: a large-scale consumer product typically ships with only a built-in self test and no calibration step, whereas at a test-equipment maker the calibration setup is one of the largest pieces of the production test system.[369]
Staging automated test at each level of assembly keeps fault-finding cheap. In one high-current power supply programme, the contract manufacturer drives a mini variant of the power module board from a 24 volt AC input while its switching pulses are measured; the assembled module then passes through an automated module tester, and only then is it built into a stack.[522] When every subassembly has already passed its own automated test, the final system test should pass without incident; a final test that regularly finds faults is evidence that coverage is missing further upstream.[522] Final system test can remain automated in its measurement while still needing an operator, because connecting many cables and the cooling water for a water-cooled stack cannot be automated economically.[522]
Exhaustive testing of a configurable instrument is physically impossible, because the number of permutations of settings grows far faster than any test budget; coverage decisions therefore select which configurations matter rather than aiming for completeness.[619] A fully automated board line can chain surface mount, depanel, jig loading, electrical test, and packaging with no manual handling, though even a mature high-volume operation may still have people loading parts into test sockets while the last robot cell is brought online.[529]
Fixtures and mechanical automation
Fixtures are designed into the product. Designing a card edge connector into the edge of the production panel makes an entire panel of boards, ten at a time, accessible to a single test fixture without individual handling.[182] Fitting each board with a unique laser-programmed identity chip lets automated test software read the serial number and file a test report against that specific unit, turning the test database into a troubleshooting resource for returns years later.[174] Because the unit is already electrically accessible in the end-of-line jig at the moment of programming, that is the cheapest point to do everything else to it and emit one consolidated test report for that serial number, rather than splitting programming and test across separate stations and vendors.[544]
Cheap mechanical actuation covers both endurance testing and design experiments: a rig that powers a unit up, presses a button several hundred times, and powers it back down can qualify a cross-referenced or second-source part before it enters the line.[167] Connector life can be characterised with an automated insertion jig that mates and unmates a sample a thousand times at varying approach angles while gold plating wear is measured and service life predicted.[178] A second-source part built to be as close a copy as physically possible still behaves differently in endurance test, so a copied connector must be fully re-characterised rather than assumed equivalent to the original.[178] In RF work, manual characterisation requiring roughly 900 connector matings was reduced to about 30 connections by an automated switching system, which also removed the repetitive strain injury risk from screwing precision connectors on and off.[533]
Bed-of-nails fixtures can themselves cause the defect they fail to detect: pogo pin pressure flexes the board, and the crack in a ceramic capacitor forms as the pins lift and the board springs back, after the measurement has already been taken.[585] A board damaged by its own fixture ships as fully tested and then exhibits a long-term failure in the field, sometimes appearing only with temperature, mechanical stress, or when the unit is screwed down at its mounting points.[585]
Firmware and software test automation
Automated testing of hardware changes has a hard floor on cycle time, because each iteration requires physically building and reworking boards; firmware changes have no such floor and are the better first target for automation.[161] A firmware test rig needs four pieces a pure software project does not: the target hardware physically attached, a programming system, an automated tool that pulls whatever was committed to the repository, and a scripted procedure that compiles, installs, and exercises the build.[161] In a hardware-in-the-loop setup at a 3D printer company, the build server pushed each new firmware version to the machine and ran a unit test suite that jogged the printer to every coordinate to confirm no axis crashed.[505]
The case for automation rests on the decay of manual process. Manual test procedures are not actually executed on every build: engineers forget, run out of time, or decide an item is not worth checking, so a manual regression plan silently decays while an automated one does not.[556] Running a physical sequence once or twice by hand is acceptable, but repetition introduces human error, making repeated manual verification the natural first candidate for automation.[599] Attaching a regression test to the change that fixes a defect means the condition is re-checked on every future version, so the suite accumulates protection against exactly the failures the product has already exhibited.[505] A large software product’s daily build can be gated by automated test scripts to the point that a build failing enough of them receives no manual testing at all.[627]
A low-effort entry point is to exercise the serial or USB command line most embedded projects already have: a Python script opens the serial connection, issues commands, and checks for the expected responses.[556] Running the same C test suite twice, once compiled for x86 on the development machine and once cross-compiled and executed on the target, uses the second run purely as a cross-check that the port itself succeeded, since porting reliably introduces its own defects.[556]
Simulation-based testing
Firmware automated testing is rare in practice because setting it up against real hardware is complicated, which is the gap device simulation targets.[599] Renode, an open-source embedded systems simulator that has existed in some form since 2010, lets a team execute tests and gather metrics against a simulated target, removing physical hardware from the loop for the bulk of test runs.[519] Running each new build against a simulated device covers most of the behaviour of interest and is a reasonable gate before promoting a build to integration-level test on real hardware.[599]
Physical testing dominates development time for embedded control systems, consuming roughly 80 percent of the effort on university drone and sensor work, because each scenario requires holding hardware and mounting things in the real world.[691] Field testing is always incomplete, not for lack of ability but because the effort per additional scenario is high enough that coverage stops well short of what the design warrants.[691] Simulated targets change the economics: additional scenarios cost only compute and parallelise across many servers, so a thousand simulation runs can execute simultaneously.[691] A simulated system that spends most of its time reading, writing, and sleeping can have its time base compressed, exercising a week of real behaviour in about fifteen minutes; the technique breaks down when the system is busy-looping, which is hard to simulate faster than real time.[691] Simulation also collapses manual environmental matrices, such as tuning firmware constants by physically carrying a device to different rooms at different times of day, into repeatable automated runs.[599]
Instrumentation and interfaces
The target state for manufacturing test instrumentation is full scriptability: probes land on the unit, the instrument walks a scripted sequence, and it returns a pass-fail result without operator judgment, which also allows scopes and function generators to be tied into a pogo-pin fixture.[355] The economic case for scripting a bench procedure is that writing the script takes roughly the same time as performing the procedure once by hand, after which it can be repeated indefinitely in identical order, which matters for calibration and any sequence whose validity depends on ordering.[527]
The test instrument industry is historically closed, and its established control standards, SCPI over GPIB and the later LXI, were designed for command-and-response measurement rather than streaming data, so they perform poorly when a test script needs a continuous sample stream.[527] Interface design determines a rig’s longevity. Earlier factory tools exposed APIs only as a linkable DLL, pinning the test station to Windows, or offered access only from C; driving a tool from a Python library or plain ASCII commands over a serial port keeps it usable from MATLAB, LabVIEW, Python, a Linux single-board computer, a Windows laptop, or a Mac.[461] USB is a poor transport for sensors on a mobile robot because of connector and cabling reliability, but it is well suited to bench-side automated testing of embedded electronics, where per-port current monitoring converts a silent bus collapse into a diagnosable event: a sensor that drew a hundred milliamps and now draws an amp indicates a cable short, and that port can be switched off before the whole bus goes down.[425] A general end-of-line test instrument needs a command-and-measure loop over UART, CAN bus stimulus, voltage, current and resistance measurement, and a frequency counter, since those cover the interactions most projects require at the end of the line.[544]
In a hardware-in-the-loop flow, a precision current measurement instrument can be folded into the same run that executes the functional tests, so current and energy consumption are recorded as tracked metrics over time rather than measured only when someone suspects a problem.[607] Where a company uses its own programmable instruments to manufacture and calibrate the next generation, a separately traceable calibrated multimeter is placed in series as the reference to prevent generational calibration drift.[640]
Constraints and failure modes
Automating hardware test is harder to specify than automating software test because pass criteria depend on real-world physical characteristics that are difficult to measure, requiring test stands and instruments rather than an assertion in code.[505] The gap between hardware and software change review is functional confirmation: nothing automatically tells a reviewer that a changed feedback resistor still behaves as intended, whereas software review has cheap access to test frameworks that answer that question.[577] A system not architected for testability can feel impossible to bring under test later, because hardware dependencies tangled through the code prevent any part of it from running on a development machine.[556] Treating test data as first-class product data means designing instrumentation into the firmware rather than hanging a UART-to-USB adapter on the board and printing at the first bug, because the ad hoc route yields data no automated test can consume.[584]
Some scenarios can only be run by a machine because their duration makes them unsafe or unlawful for a person, such as a twelve-hour overnight run repeatedly navigating into and out of screens to trigger a buffer overflow.[369] Conversely, teams routinely hold a list of scenarios they label untestable, typically involving breaking a physical or wireless link, such as forcing a device to lose its Bluetooth connection to a phone on demand.[369] Buying a test robot does not remove the engineering labour: a skilled engineer must still consume the robot’s API and author the tests, so the capability remains labour-intensive even when the mechanism is off the shelf, and the purchaser is usually the engineer’s manager rather than the engineer.[369] Vendor FPGA toolchains resist automation in two specific ways: they are GUI-driven rather than scriptable, and even free licences are node-locked to a MAC address, which blocks packaging the toolchain into a portable container image for automated builds and tests.[423] Configuration-dependent performance regressions are a real failure class in instrument firmware: enabling a single horizontal measurement on an oscilloscope dropped waveform processing and update rate by about three orders of magnitude, a defect invisible to any test that does not exercise that combination.[619] Standing up a remote hardware test setup once is straightforward; keeping it available to a whole team indefinitely is a different and much harder class of problem, and that operational burden is the real cost of hardware test infrastructure.[537]
Moving from prototype to high-volume manufacturing has prerequisites in the board itself, since a prototype-level design cannot be run through a high-volume setup unchanged; building the test automation is part of that transition work.[661] Production test software has an extremely long service life and pins the factory to obsolete platforms: test racks built around Windows 3.1.1 remained in production use into the late 2000s.[647] Earlier still, vacuum tube burn-in was automated with an electromechanical timer sequencing racks and rows, holding heaters on continuously and switching each rack under load overnight, after which an automated tester ran each tube individually through a cycle of measurements whose parameter data was used to match tubes into sets rather than only to pass or fail them.[647]