The evidence

An emulator that looks right and an emulator that is right are different things, and the difference surfaces years later, in the one game nobody tried. So every claim here has a check behind it, and wherever possible the check comes from outside — written by someone who did not know what we believe.

That last part is the whole of it. A test written from our own understanding agrees with our own mistakes. The suites that matter are the ones that can disagree.

The tests come in tiers, separated by what they answer and what they cost. The fast tier is hermetic and runs on every change; the others fetch what they need on first use.

The fast tier

It covers only what no external suite can see: the reset contract, the invariants of our own machinery, and the proof that every opcode on every prefix page finishes without outgrowing its micro-program.

Since the machine began it also grades the parts against their documentation.

The memory map is proved against the board’s: the eight banking configurations, the ROM paging and the I/O decode, each exercised through the bus by a program running from a fabricated ROM rather than by reaching into the machine’s fields.

The 6845 is held against the Compendium’s frame — 312 scanlines of 64 microseconds, the syncs where the registers put them, the video pointer walking the documented rows, the last line decided while the character counter is still 0 or 1, and a register written at a line’s second character still reaching that decision on the third, the counters running to their own tops when a program writes a limit beneath them, the vertical adjustment that brings a runaway row counter home, the display and the border as the latches the equalities throw rather than comparisons standing, a sync width of zero, which leaves the first two types no line sync at all and gives the other three one sixteen characters long, and the third type’s last line as its own chapter decides it, save one of its rules, down to the case study that holds the counters at 0 line after line by moving the row height through each sync. The monitor is timed from the middle of the line sync, which is what turns a trimmed pulse into half a character of scroll.

The Gate Array is held against chapter 27 and its own documentation: the interrupt counter looping at 52, bit 5 dying at the acknowledge — and outliving it on the one count that sets it, where the acknowledge arrives in front of the count and the next interrupt comes twenty lines on instead of fifty-two — the two-line rule after a frame sync, which falls on the same line behind the two ASICs’ later syncs as behind a type 0, the microsecond an interrupt waits after the line sync that asked for it — which that chapter draws at four widths for every type, every row transcribed: at a width of nothing no interrupt on types 0 and 1, and on a type 2 an interrupt seventeen microseconds after the character that asks for the sync, eighteen on the ASICs, whose interrupts come a microsecond after the others’ at every width, less the quarter of one the same test pins — and a byte becoming pixels in each of the four modes. Grimware states the rule for that microsecond inverted; the Compendium is the one tested on silicon, and it is the one we follow.

The video path is proved end to end — a screen of pixels through the whole machine, landing 640 by 200 exactly where the syncs put it.

And the duration of seventy-odd instructions in microseconds, against two tables of measurements made independently of each other and of us — with, for two of them, the microsecond inside their own run on which the write reaches the video chip, which no duration can say and which every trick that moves a register on the character it is read at depends on; and the cost of taking an interrupt, set against the documentation’s own comparison with the RST instruction that does the same thing from code. Both halves agree — four microseconds from code, five for the interrupt. The quarter of a microsecond on which the Gate Array hears the processor acknowledge is pinned as well, which no duration can say and on which the race for bit 5 is run; what fixes it is the suite rather than the chapter.

The tape reader is held to the same tables: the pilot’s 8063 pulses before a header and 3223 before data, the two syncs, the two bit lengths, and the second of silence between blocks — each transcribed from the TZX specification rather than worked out from the code that plays them. The cassette’s leader tape is held to the standards it is taken from, restated in the test rather than read from the reader: played first, less whatever silence the image kept, and not at all where the image kept more. Loops are held to the specification’s own words for them, and to the three loop images libspectrum keeps for bugs its own reader once had: the two it plays play to their end here too, and the third, a loop’s end with no start, is refused where libspectrum steps over it. The deck is graded apart from it, on pulses written out by hand, so what is proved there is the timing and nothing else. And the CPC’s own wiring is graded: the motor line turning the reel and stopping it 50ms late, a drop shorter than that delaying the reel without stopping it, and a pulse arriving at bit 7 of port B held for the longer count a 4MHz board owes it.

The ULA’s charge for the bus is held against the tables published for it: the delay owed at each T-state of a Spectrum’s frame, and the length of instructions whose operands lie in the screen’s memory — which fixes where each charge falls inside an instruction, including the internal T-states these ULAs charge for and the four rows by which a port is charged instead. That is not a handful of positions but a walk: twenty-seven shapes of instruction — one for every place a cycle can fall, each of the four rows an I/O access is charged by, a conditional taken and not taken, and the internal T-states that hold the interrupt register rather than an address the instruction chose — each begun at every T-state of the first line to carry picture, a line in the middle, the last, and the border either side. 23,004 durations, every one worked out from the published rule and none of them asked of the chip being graded.

The interrupt is graded as the machine delivers it rather than as the processor takes it: what an acceptance costs in each mode, where it sends the processor, what it pushes and what it leaves in the refresh counter and the two flip-flops — and the window, walked T-state by T-state with instructions of two lengths, so the last one that catches a frame’s interrupt and the first one that misses it are both pinned. A halted processor is woken by it, and a repeating instruction is interrupted between its iterations and pushed at its own address.

The disc is graded in three parts. The medium is proved to lay a track out where a formatter would, to answer every position on it — the identity checks it computes were checked against a CRC-16 computed outside this code — and to read back as the same disc after being written out. The µPD765 is driven through its own handshake by a loop that plays the processor, one microsecond of disc at a time, and judged against its datasheet: the bytes crossing every 32µs and the overrun when one is late, the sector found only when its identity comes round, the end of the cylinder a read without terminal count runs into, the flags for a sector that is not there and a track that has nothing on it, the seek stepping at the specified rate and the interrupt it leaves, and the change of READY that polling reports. What the datasheet leaves open was settled by the AMSDOS ROM’s own listing: a read is one sector with EOT set to R, and success is an abnormal end carrying EN.

The command line is run as a user runs it, its binary handed firmware the test writes. A CRTC type outside 0 to 4 is refused, and on a Spectrum any type, even 0; a CPC given none is built as a type 0; and each of the five reaches the machine, named the way a program names its CRTC — by what the chip answers on its read port and its status port, as chapter 21 of the Compendium has each type answer — save types 3 and 4, which answer alike and are told apart by the colour a write to the Gate Array shows half a byte sooner on a type 3. That half byte also shows a snapshot given to run going into the type named, and the answers it carries, still in memory after the run, show its program picked up rather than begun again.

The tier deliberately restates nothing an external suite already proves. The one place it cannot defer is timing: the corpus below runs the processor with its wait pin released throughout, so it proves nothing whatever about wait states.

The sanitized tier

The fast tier again, compiled under the address and undefined-behaviour sanitizers. It is hermetic like the fast tier but slow enough to be its own target, and it exists for one class of fault the other tiers cannot see: a read that strays outside the bytes it was given. Tape images, disc images, snapshots and the firmware all arrive from outside and are none of them ours, and the checks that keep a truncated or hostile one inside its own buffer are exactly the checks a passing test agrees with — without a sanitizer, a read past the end of an image returns a number like any other. This is where those are graded.

The machine tier

This one boots the real thing. It runs each of the four machines from reset, reads the screen back as text and checks it says what Amstrad, Locomotive Software and Sinclair wrote, then types PRINT 2+2 at the prompt and insists BASIC answers 4. The Spectrum’s screen is read twice over, once out of the display file and once out of the framebuffer the beam painted, so the serialiser and the sync separator are in the path that is checked and not merely the memory beneath them.

That one line is the strictest test here. The key matrix, the 8255’s direction flipping, the sound chip, the fifty-times-a-second scan and the interrupt that drives it must all be right at once, and none of it is graded by us — the firmware is the judge, and it was written in 1984 by people who had the hardware.

The letters are identified by looking each glyph up in the character table the ROM itself carries. A test that recognised letters by our own table would only prove we agree with ourselves.

A tape is judged the same way, and more sharply. The deck plays a block and the Spectrum’s own LD-BYTES reads it — the routine that measures the time between edges on the ear line and has no other way of knowing what it has been handed. It returns with the carry set only if every bit arrived and the checksum agreed, so the sixteen bytes that turn up in memory were put there by Sinclair’s arithmetic and not by ours. Two more blocks are wrapped the way images of protected tapes wrap theirs: one whose leader is a loop of short tones, and one that runs with no pause into a stop on a low pulse, whose last edge only the reader’s finishing millisecond supplies. LD-BYTES takes both whole.

A CPC’s tape is judged by its own firmware. A program is written in the format chapter 8 of SOFT 968 gives for the cassette manager’s records, behind a leader tone too short to outlast the 464’s two and a half seconds of waiting for its motor and with nothing in front of it. The 464 has to find it behind the cassette’s leader tape, check every CRC and run it; what prints TAPE OK is Locomotive’s code, not ours.

Sound is held to what the firmware plays. The 6128’s SOUND 1,142 must turn channel A over every 1,136 microseconds, a whole wave at 440.14Hz, the international A of its user manual. The Spectrum’s BEEP 1,0 must bring the speaker up once every 13,372 T-states, which is what the ROM’s own loop comes to when its instructions are counted out at their documented timings, and is middle C to within a twentieth of a percent; a contended write may stretch a cycle, but never shorten one.

Then it gives the 6128 a disc — Shaker’s, written by somebody else’s tool — and types CAT. The names and sizes AMSDOS prints are set against what a reader written in the test, which knows nothing of floppies or controllers and walks the image’s own directory, finds in the same bytes; a file is then loaded through the controller, the ROM and BASIC and compared byte for byte with that reader’s copy of it. Two routes to the same 27,004 bytes, only one of them through the chip. Then RUN"SHAKE27A must leave the machine in mode 2 with its program counter inside the module. The menu it draws there is in the ROM’s own characters, but mode 2 halves their width on the screen, so this tier can tell whose code is running without being able to read a word of it. Shaker’s own tier, below, reads it in mode 2. With no disc at all, the ROM must find the drive not ready and say so.

It needs the firmware and disc images, so it fetches them first.

The conformance tier

SingleStepTests: 1,604 files, one per opcode across every prefix page, a thousand randomised cases each — 1,604,000 in all. Every case fixes the processor and memory before and after, and the state of the bus after each individual clock cycle. That last part is what earns its 1.3 GB, because it tests timing rather than only results.

It matters just as much that we did not write it. It has already caught a real error: we had concluded that an undocumented behaviour under the DD CB prefix did not exist, and 168 files said otherwise. The corpus is pinned to a commit, so “passes the complete suite” names something exact that cannot shift underneath us.

Today every instruction the Z80 knows passes it, per cycle.

The tier grades one thing the corpus does not mark: which reads take an instruction’s own operands. A case’s memory holds the instruction’s bytes from its program counter on, among the bytes it reads and writes as data, so the fewest bytes any case of a file holds there are the instruction that file tests. Every one of them must be fetched once and in order, by an opcode fetch or by a read the processor calls an operand, and no other read may be called one.

The first run downloads the corpus, which takes a few minutes; after that it is local.

The acceptance tier

Frank Cringle’s Z80 instruction set exerciser, from 1994, in both its forms: ZEXDOC checks the documented flags, ZEXALL all eight bits including the undocumented two. It sweeps each instruction across long runs of operands and flags and checks the result against a CRC recorded from real hardware.

Its method is independent of the corpus, but the deeper difference is that it is a program. Millions of instructions in sequence, each inheriting whatever the last one left behind, where the corpus tests each instruction alone from a clean state. That is the failure the corpus cannot see, and the one that decides whether real software runs.

It is slow — hours of 4MHz machine time, minutes of ours — so it runs by the group, and running all sixty-seven is a deliberate act rather than the default: EXERCISER_GROUPS=0.

The Shaker tier

Shaker, Longshot’s CRTC acid tests, was written against real machines of every CRTC type and says what their silicon answered. make test-shaker boots its disc, reads each module’s own menu to find out what it offers, presses every key the menu offers, turns each group’s pages with the space bar until the menu comes back, and writes down what the screens show — as text, read through the character table the ROM carries, and as the beam’s own path. Each group runs in a process of its own, from the machine as its module’s menu left it, and every sample is read over a picture wiped two frames before, so a group reads only what the machine drew for it. Each page is watched until five hundred frames pass with nothing new on it, and for fifteen hundred at most; a group with a page still drawing then is recorded as cut off at the cap, a mark that follows what the group draws rather than what it grades. A line is read only once it has stood still, so one a group shows for a frame or two is never read.

The verdict is read off each group’s own output, because Shaker states it differently in every group. Some print the machine’s value beside silicon’s, in nine different hands: eight keep both on the one line and one puts silicon’s on a title and the machine’s on the rows beneath it; five of the nine never write the word WRONG and leave the disagreement to the two numbers, one of those naming a value for each CRTC type and leaving the reader to find the clause that speaks for the machine in front of it, while another writes silicon’s value after that word, and only where the machine is wrong. A few make the comparison themselves and print the word alone, naming nothing of silicon’s — one of them ringing the word it prints for a failure, printing a plain OK for a pass and handing the passing outcome, in a bracket beside it, to the CRTC types it names — and one says it in a whole sentence, drawing one line where the machine is right and another where it is wrong. Most print a legend and leave the comparison to a picture, and some ask the reader to watch a line flash. So a group is graded only once its convention has been read off its own output, and the rest are recorded until it is. Which machine a group speaks for is read the same way. A module’s menu gives each group to a CRTC type, and most groups honour it by drawing nothing on the others; but a few read the type for themselves and grade whichever machine they find, so every group is tried, and the ones whose own screens name this machine are graded whatever the menu said.

What each group said is set against a copy kept on record, one for each version of the suite and each CRTC type the chip can be built as, ten in all. A change to the emulator that moves Shaker’s answer moves a line of a file a human has to read before keeping it, which pins the ungraded groups before anyone can grade them; so the tier passing means one thing only, that nothing moved. Two versions are walked: 2.7, and 2.6 before it, which Logon System’s older site still serves. Shakerland’s photographs of real machines are of 2.6 wherever their text can tell the two apart — the group checking the registers that end a frame, on all five types, and the group on a type 2’s last line — so a photograph is set against the version that drew it; make test-shaker SHAKER=26 walks 2.6. A group named by its module and key alone, here and in the code, is 2.7’s where a run of it is meant, and a photograph of it may be of either version. The image of 2.7 pinned is the one the site has served since 18 September 2026, whose module E differs from the image it replaced.

Every line the suite grades itself on agrees with silicon, on all ten records: under 2.7, 132, 87, 94, 75 and 81 lines on types 0 to 4, and under 2.6, 103, 89, 94, 75 and 81. Most of them stand past a group’s first page. On types 1 to 4 they are lines about a chip that is type 0 save the exceptions the machine sets out, so their agreeing says the groups that grade those types find nothing wrong, not that the chip is whole. Nor are the counts a measure of coverage: some groups name silicon’s value, or print their verdict, only while the machine is wrong, so a line that comes right leaves the tally rather than joining its agreements. One line comes and goes with the phase the disc boots at, and is no evidence either way. The fast tier’s durations are whole microseconds; where inside one an instruction ends, which decides whether an interrupt landing then is taken before or after it, nothing grades but two groups of the suite.

The photographs answer what the records cannot: the groups that say their piece in a picture. A group’s photographs are mostly its pages, so each is set beside the same page of a run — by eye, in surveys no tier repeats, the builds hidden behind letters wherever two were weighed. Of the 1,858 photographs over 243 pairs of a group and a type, about half match the machine’s page; about a quarter differ, most of those because the monitor has no flywheel (the machine says what that costs); a fifth are of pages the tier never reaches; and the rest cannot be read either way. Three rules the documentation states were read off them, and each is kept with a test of the documentation’s own: a byte written into the screen is read for the picture in the microsecond it is written, whichever instruction writes it, on every type (ch. 8); types 1 and 2 take no display delay from the register that also asks for interlace (ch. 19.1); and a change of mode is taken where the line sync’s first six microseconds end, or where the sync ends if it is shorter, and never from a sync of one (ch. 9.3). None of the three moves a line the suite grades.

The software tier

A demo, which breaks on anything less than exact — the only kind of test written by people trying to make the hardware do something beautiful rather than something correct. make test-demos fetches Batman Forever, a 2011 demo that reprograms the video chip line by line, splits its frames and blanks its screen between parts; starts it from the second drive, as its image requires; and plays it to the end, eleven minutes of machine time. Every frame is taken where the monitor begins a new one, which is the only moment the picture is whole, give or take the character the beam has just started — read between two of them, the top of one frame stands above the rest of the one before, and anything redrawing the whole screen each frame looks broken when it is not. Two frames a second are kept as a hash of the entire raster, sync and blanking and all, and set against a record; a frame that moves is also written out as a picture, so the first thing a reader sees is what changed rather than a number.

The record grades nothing, and says so at its head. What it is worth is the comparison made when it was accepted: every part of the demo set beside a capture of the same demo running on a real CPC, part by part, and frame by frame where the two looked least alike, and beside CPCEC’s run of it at two samples a second across the whole eleven minutes. That comparison is the evidence; the record is only its shadow, kept so that the next change has to answer for moving it. The disc it plays is named and hashed in the record, because a different release of the same demo is a record of its own rather than a machine that has moved — the capture it was accepted against is one such release, and differs in its greetings. What it catches is what a suite of acid tests cannot: the rule that blanks a screen while a part loads is worth one line of Shaker and eleven minutes of this. A second record holds the demo played with the machine built as a type 1, and is worth less on purpose: no capture of the demo on a real machine of that type has been set beside it, which it says at its own head. It keeps the frames of a run nobody has judged, so that work on the type 1 rules has something to move. By hash none of its 1,360 frames is the type 0 record’s, the two chips bringing the machine up to the demo at a different phase besides drawing by rules of their own, so the two records cannot be compared frame for frame.