65816 Instruction Set – The M and X flags

Register widths, immediate operand lengths, and decoding without knowing the flags

Introduction

Two bits of the status register choose how wide the 65816’s registers and data are. Because an immediate operand is as wide as the register it loads, the length of an instruction depends on a flag that earlier instructions set. This is the one place where the bytes of a 65816 program cannot be read on their own.

What each flag controls

FlagSet (1)Clear (0)Instructions that follow it
M8-bit accumulator and memory16-bit accumulator and memoryADC, AND, BIT, CMP, EOR, LDA, ORA, SBC, STA, STZ; ASL, LSR, ROL, ROR, INC, DEC, TRB, TSB on memory; PHA, PLA; TXA, TYA (the width they transfer). The immediate forms of ADC, AND, BIT, CMP, EOR, LDA, ORA, SBC are 2 bytes long when M is set and 3 when it is clear.
X8-bit X and Y16-bit X and YCPX, CPY, LDX, LDY, STX, STY; INX, INY, DEX, DEY; PHX, PHY, PLX, PLY; TAX, TAY, TSX, TXY, TYX (the width they transfer). The immediate forms of CPX, CPY, LDX, LDY are 2 bytes long when X is set and 3 when it is clear. Setting X clears the high bytes of X and Y.

Nothing else changes length: REP, SEP, COP, BRK and WDM always take one operand byte; the block moves take two; PEA takes two.

The cycle counts follow too: a 16-bit memory operand takes one more cycle, a 16-bit read-modify-write two (see the cycle tags).

What changes them

InstructionEffect on M and X
rep #mask, sep #maskClear or set the bits of the mask. Bit 5 is M and bit 4 is X: rep #$30 makes both 16-bit, sep #$30 both 8-bit, rep #$20 only the accumulator. In emulation mode they cannot be changed.
xceSwaps carry and emulation. Into emulation mode: M and X become 1. Into native mode from emulation: M and X are 1 (the mode that was forced). Native to native: unchanged.
plpM and X come from the byte on the stack.
rtiM and X come from the pushed status on the stack.
interrupt entry (NMI, IRQ, BRK, COP) in native modeNot changed; the handler starts with the flags of the code that was interrupted. The decimal flag is cleared and the interrupt flag set.
resetEmulation mode, both 1.

A program that returns from a subroutine does not restore M and X unless it saved the status register with php. So the widths at a subroutine’s first instruction depend on its callers, which a disassembler cannot know without a call graph, and several callers may disagree.

Decoding without knowing: the explorer’s rules

The explorer decodes an instruction at every byte of a code region, not only at the ones the program runs, so the answer for each byte needs a state. The rules are fixed and shown where they apply:

  1. The starting state is an option. The options bar has Start in emulation mode (on by default, which is what a reset does: both widths 8 bits), Accumulator width and Index width (8 or 16 bits, used when emulation mode is off) and Follow REP, SEP and XCE (on by default).
  2. Tracking. When the option is on, the disassembly is followed from the start of the region in order, the first column. After a rep or sep the M and X bits it changes take effect at the next instruction, clc and sec set a carry the viewer remembers (any other instruction that changes carry makes it unknown), and xce with a known carry enters native or emulation mode. So clc xce rep #$30 lda #$1234 is read with a 16-bit immediate.
  3. PLP and RTI. They pull the flags from the stack, which a linear decode cannot know. The viewer keeps the widths it had, marks them not known, and an instruction whose length depends on an unknown flag is flagged with a problem that gives both lengths. The next rep, sep or xce that settles the flag makes it known again. An xce after a carry the viewer does not know leaves the mode as it was and makes both widths not known.
  4. Overlapping decodes. The viewer shows what an instruction would be at every byte, in extra columns. An instruction that starts inside another is decoded with the state the first column had reached at its own first byte (the change made by a rep counts after the rep’s last byte). Without tracking, every byte uses the starting state.
  5. Both widths shown. Every instruction whose length depends on M or X names the other width in its note (at 8 bits: lda #$34 (2 bytes)), and the detail strip under the disassembly lists it, so the reader can see what the bytes are in the other reading.
  6. Jumps are not followed. A jsr or jmp does not carry the state to its target and a code region that starts in the middle of a program starts in the option’s state. Use the options to decode a region in a state you know from the code that calls it.

This is a limit of reading code without running it, not of the viewer: there is no state that is right for every byte of a program. The viewer therefore never guesses silently; it shows the state it used, and lets the reader change it.

An example

The ten bytes C2 30 A9 34 12 E2 20 A9 56 EA, which a programmer wrote as rep #$30, lda #$1234, sep #$20, lda #$56, nop, decoded three ways (the explorer’s own output):

StartReading
M and X 8 bits, not trackingrep #$30; lda #$34; ora ($e2); jsr $56a9; nop
M and X 16 bits, not trackingrep #$30; lda #$1234; sep #$20; lda #$ea56
M and X 8 bits, trackingrep #$30; lda #$1234; sep #$20; lda #$56; nop

Only the last reading is the one the author wrote, and it needed the tracking: the rep #$30 in front clears M, so the first lda takes a 16-bit operand, and the sep #$20 sets it again for the second. The first reading decodes the same bytes as four different instructions after the rep, none of them what the programmer meant, and nothing in the bytes says so.

Not described

Known, not yet covered

  • Viewer: following a jump or call to carry the M and X state to its target, and choosing a state per region by hand (a region override); the options give one starting state for all regions.
  • Viewer: the status register saved by php and restored by plp in the same routine is not paired up.

Links

Reference