x86 Instruction Set – Transactions and synchronisation

Introduction

These instructions help threads, cores and devices work together. They came in several separate sets, each with its own CPUID bit:

TSX (Transactional Synchronization Extensions), from Intel’s Haswell in 2013: RTM runs a block of code as a transaction that either takes effect as a whole or not at all; HLE does the same for an existing lock, through two prefixes. Intel has since turned TSX off on many processors, some through microcode updates, and HLE is deprecated.
Waiting: monitor and mwait (Intel’s Prescott, 2004) let a core sleep until a line of memory is written; AMD’s monitorx and mwaitx and Intel’s WAITPKG do the same in user mode, with a time limit.
User interrupts (Intel’s Sapphire Rapids, 2023): one thread interrupts another without the operating system, in 64-bit mode. They use a user interrupt flag, UIF, which only these instructions read and write.
Atomics: CMPccXADD compares and adds in one locked step, and RAO-INT adds or combines a value in memory without bringing the line to the core.
Direct stores to devices: MOVDIRI, MOVDIR64B and ENQCMD write to memory-mapped device registers in single, uncached writes.
SERIALIZE, which waits until every earlier instruction has completed.

The lock prefix, xchg, cmpxchg and xadd are on the Data transfer page; the fences are on the Cache and memory ordering page.

The instructions

In the forms, r32 and r64 are general registers and m32 and m64 memory operands. NP means that no 66, F2 or F3 prefix may be present.

Transactional memory (RTM and TSXLDTRK)

CPUID leaf 7: EBX bit 11 for RTM, bit 4 for HLE, EDX bit 16 for TSXLDTRK.

Instruction	Forms	Encoding	What it does	Set
`xbegin`	`rel16` `rel32`	`C7 F8 cw` `C7 F8 cd`	Starts a transaction. If it aborts, every change it made is undone, and execution goes on at the fallback address `rel`, with the reason in EAX.	RTM (TSX) (Intel Haswell, 2013)
`xend`	–	`NP 0F 01 D5`	Commits the transaction: its changes become visible to the other cores at once.	RTM (TSX) (Intel Haswell, 2013)
`xabort`	`imm8`	`C6 F8 ib`	Aborts the transaction. The immediate goes into bits 31:24 of EAX, for the fallback code to read. Outside a transaction it does nothing.	RTM (TSX) (Intel Haswell, 2013)
`xtest`	–	`NP 0F 01 D6`	Clears ZF when the core is running a transaction (RTM or HLE), and sets it otherwise.	RTM (TSX) (Intel Haswell, 2013)
`xsusldtrk`	–	`F2 0F 01 E8`	Suspends the tracking of load addresses inside a transaction, so that the loads that follow do not make it abort when another core writes to them.	TSXLDTRK (Intel Sapphire Rapids, 2023)
`xresldtrk`	–	`F2 0F 01 E9`	Resumes the tracking of load addresses.	TSXLDTRK (Intel Sapphire Rapids, 2023)

HLE adds no instructions, only two prefixes on locked instructions: xacquire (F2) where a lock is taken and xrelease (F3) where it is released. The core then skips the write to the lock and runs the section as a transaction. Processors without HLE ignore the prefixes. The disassembler shows them by those names.

Waiting

Instruction	Forms	Encoding	What it does	Set
`monitor`	`rAX, ECX, EDX` (implicit)	`0F 01 C8`	Arms the address-monitoring hardware for the range that holds DS:rAX. ECX gives extensions and EDX hints. Usually privilege level 0 only. CPUID leaf 1: ECX bit 3.	MONITOR and MWAIT (Intel Pentium 4 (Prescott), 2004)
`mwait`	`EAX, ECX` (implicit)	`0F 01 C9`	Waits, in the power state that EAX asks for, until the monitored range is written or an interrupt or other event comes.	MONITOR and MWAIT (Intel Pentium 4 (Prescott), 2004)
`monitorx`	`rAX, ECX, EDX` (implicit)	`0F 01 FA`	AMD’s `monitor` that also runs at any privilege level. CPUID leaf 8000_0001h: ECX bit 29.	MONITORX and MWAITX (AMD Excavator, 2015)
`mwaitx`	`EAX, EBX, ECX` (implicit)	`0F 01 FB`	AMD’s `mwait` for any privilege level. When ECX bit 1 is set, EBX gives a time limit in TSC ticks.	MONITORX and MWAITX (AMD Excavator, 2015)
`umonitor`	`r16/r32/r64`	`F3 0F AE /6` (ModRM.mod 11)	Arms address monitoring, in user mode, for the address in the register (its size is the address size). CPUID leaf 7: ECX bit 5.	WAITPKG (Intel Atom P5900 (Tremont), 2020)
`umwait`	`r32`	`F2 0F AE /6` (ModRM.mod 11)	Waits until the monitored address is written, or until the TSC reaches EDX:EAX. Bit 0 of the register picks the lighter (1) or deeper (0) of two waiting states. Sets CF if the wait ended because the operating system’s limit expired.	WAITPKG (Intel Atom P5900 (Tremont), 2020)
`tpause`	`r32`	`66 0F AE /6` (ModRM.mod 11)	Waits until the TSC reaches EDX:EAX, in the state that bit 0 of the register picks. Sets CF as `umwait` does.	WAITPKG (Intel Atom P5900 (Tremont), 2020)

User interrupts

64-bit mode only. CPUID leaf 7: EDX bit 5.

Instruction	Forms	Encoding	What it does	Set
`senduipi`	`r64`	`F3 0F C7 /6` (ModRM.mod 11)	Sends a user interrupt. The register is an index into the sender’s table of targets, which the operating system set up.	User interrupts (Intel Sapphire Rapids, 2023)
`uiret`	–	`F3 0F 01 EC`	Returns from a user-interrupt handler: pops the return address, flags and stack pointer, and sets UIF.	User interrupts (Intel Sapphire Rapids, 2023)
`clui`	–	`F3 0F 01 EE`	Clears UIF, holding off user interrupts.	User interrupts (Intel Sapphire Rapids, 2023)
`stui`	–	`F3 0F 01 EF`	Sets UIF, so that user interrupts are taken.	User interrupts (Intel Sapphire Rapids, 2023)
`testui`	–	`F3 0F 01 ED`	Copies UIF into CF and clears OF, SF, ZF, AF and PF.	User interrupts (Intel Sapphire Rapids, 2023)

Compare and add (CMPccXADD)

cmpccxadd mem, reg1, reg2 is one locked operation: it compares the value in memory with reg1 (setting the flags as cmp mem, reg1 would), adds reg2 to memory if the condition cc holds, and always loads the old memory value into reg1. 64-bit mode only. CPUID leaf 7, sub-leaf 1: EAX bit 7. The encoding is VEX.128.66.0F38.W0 for 32 bits and W1 for 64, with reg1 in ModRM.reg and reg2 in VEX.vvvv.

The disassembler uses LLVM’s names, which spell the condition as jcc does; Intel’s manual uses others. Both are given.

Instruction	Forms	Encoding	What it does	Set
`cmpoxadd`	`m32, r32, r32` `m64, r64, r64`	`VEX.128.66.0F38 E0 /r`	Adds if overflow (OF = 1). Intel: CMPOXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpnoxadd`	as above	`VEX.128.66.0F38 E1 /r`	Adds if no overflow (OF = 0). Intel: CMPNOXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpbxadd`	as above	`VEX.128.66.0F38 E2 /r`	Adds if below, unsigned (CF = 1). Intel: CMPBXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpaexadd`	as above	`VEX.128.66.0F38 E3 /r`	Adds if above or equal, unsigned (CF = 0). Intel: CMPNBXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpexadd`	as above	`VEX.128.66.0F38 E4 /r`	Adds if equal (ZF = 1). Intel: CMPZXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpnexadd`	as above	`VEX.128.66.0F38 E5 /r`	Adds if not equal (ZF = 0). Intel: CMPNZXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpbexadd`	as above	`VEX.128.66.0F38 E6 /r`	Adds if below or equal, unsigned (CF = 1 or ZF = 1). Intel: CMPBEXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpaxadd`	as above	`VEX.128.66.0F38 E7 /r`	Adds if above, unsigned (CF = 0 and ZF = 0). Intel: CMPNBEXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpsxadd`	as above	`VEX.128.66.0F38 E8 /r`	Adds if sign (SF = 1). Intel: CMPSXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpnsxadd`	as above	`VEX.128.66.0F38 E9 /r`	Adds if no sign (SF = 0). Intel: CMPNSXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmppxadd`	as above	`VEX.128.66.0F38 EA /r`	Adds if parity (PF = 1). Intel: CMPPXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpnpxadd`	as above	`VEX.128.66.0F38 EB /r`	Adds if no parity (PF = 0). Intel: CMPNPXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmplxadd`	as above	`VEX.128.66.0F38 EC /r`	Adds if less, signed (SF ≠ OF). Intel: CMPLXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpgexadd`	as above	`VEX.128.66.0F38 ED /r`	Adds if greater or equal, signed (SF = OF). Intel: CMPNLXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmplexadd`	as above	`VEX.128.66.0F38 EE /r`	Adds if less or equal, signed (ZF = 1 or SF ≠ OF). Intel: CMPLEXADD.	CMPccXADD (Intel Sierra Forest, 2024)
`cmpgxadd`	as above	`VEX.128.66.0F38 EF /r`	Adds if greater, signed (ZF = 0 and SF = OF). Intel: CMPNLEXADD.	CMPccXADD (Intel Sierra Forest, 2024)

Remote atomics (RAO-INT)

Each one combines a register into memory atomically, without returning the old value and without changing the flags. They are weakly ordered, like write-combining stores, so the processor may carry them out away from the core, near the data. Memory forms only. CPUID leaf 7, sub-leaf 1: EAX bit 3.

Instruction	Forms	Encoding	What it does	Set
`aadd`	`m32, r32` `m64, r64`	`NP 0F 38 FC /r` APX: `EVEX map 4, NP FC /r`	Atomically adds the register to memory.	RAO-INT (Intel, first named for Grand Ridge, now for later processors)
`aand`	as `aadd`	`66 0F 38 FC /r`	Atomically ANDs the register into memory.	RAO-INT (Intel, first named for Grand Ridge, now for later processors)
`aor`	as `aadd`	`F2 0F 38 FC /r`	Atomically ORs the register into memory.	RAO-INT (Intel, first named for Grand Ridge, now for later processors)
`axor`	as `aadd`	`F3 0F 38 FC /r`	Atomically exclusive-ORs the register into memory.	RAO-INT (Intel, first named for Grand Ridge, now for later processors)

Direct stores to devices

Instruction	Forms	Encoding	What it does	Set
`movdiri`	`m32, r32` `m64, r64`	`NP 0F 38 F9 /r` APX: `EVEX map 4, F9 /r`	Stores a doubleword or quadword as one write that bypasses the caches (write-combining), so a device register sees it whole. CPUID leaf 7: ECX bit 27.	MOVDIRI (Intel Atom P5900 (Tremont) and Tiger Lake, 2020)
`movdir64b`	`r16/r32/r64, m512`	`66 0F 38 F8 /r` APX: `EVEX map 4, 66 F8 /r`	Copies 64 bytes from memory to the 64-byte-aligned address in the register (at ES), as a single 64-byte write. The register’s size is the address size. CPUID leaf 7: ECX bit 28.	MOVDIR64B (Intel Atom P5900 (Tremont) and Tiger Lake, 2020)
`enqcmd`	`r16/r32/r64, m512`	`F2 0F 38 F8 /r` APX: `EVEX map 4, F2 F8 /r`	Sends a 64-byte command to a device’s queue register at the address in the register, filling in the process’s address-space ID (PASID) from the IA32_PASID MSR. The device answers at once: ZF is cleared if it took the command, and set if it was full. CPUID leaf 7: ECX bit 29.	ENQCMD (Intel Sapphire Rapids, 2023)
`enqcmds`	`r16/r32/r64, m512`	`F3 0F 38 F8 /r` APX: `EVEX map 4, F3 F8 /r`	The supervisor form: the PASID comes from the command itself. Privilege level 0 only.	ENQCMD (Intel Sapphire Rapids, 2023)

Serialisation

Instruction	Forms	Encoding	What it does	Set
`serialize`	–	`NP 0F 01 E8`	Waits until every earlier instruction has completed and every buffered store has been written, before the next instruction is fetched. Unlike `cpuid`, it changes no registers. CPUID leaf 7: EDX bit 14.	SERIALIZE (Intel Alder Lake, 2021)

Notes

RTM aborts. A transaction aborts on a conflict with another core, when it runs out of room to track its memory, on most exceptions and interrupts, and on instructions it may not run (such as cpuid and many system instructions). EAX then holds bit flags for the cause, and with xabort the immediate.
Prefixes. xbegin and xabort sit in the MOV group (C6 and C7) as ModRM F8, which is why the disassembler checks the whole ModRM byte there. With a 66 prefix xbegin takes a 16-bit offset.
Ordering. CMPccXADD is always locked and needs no lock prefix. MOVDIRI and MOVDIR64B are not ordered with other stores; use a fence when the order matters.
Waiting. The operating system limits how long umwait and tpause may wait (IA32_UMWAIT_CONTROL). With CR4.TSD set, the WAITPKG instructions raise #UD outside privilege level 0.