Shorten seek tables

A Shorten stream has no index: to reach the middle a player has to decode from the start, because every block is predicted from the samples before it and the bits of one command run straight into the next. Version 3 of the program (shorten 3.x, which writes a stream of format version 2 or 3) can add a seek table: a list of places in the stream, each with the state of the decoder at that place, so decoding can start there. The table is either appended to the .shn file or kept in a file of its own with the suffix .skt. This page follows the shorten 3.6.1 source and was checked against the two seek tables that come with it.

The two forms

Form	Where	Layout
Appended	At the very end of the `.shn` file, after the QUIT command’s word	Header, entries, trailer
External	A file named like the stream with the suffix `.skt` (or any name given with `-S`; `stdin.skt` when the stream is read from standard input)	Header, entries; no trailer

The program tells an external table from a stream by the first four bytes (SEEK), and finds an appended one by looking at the last 12 bytes of a file for SHNAMPSK. Appending is done by the -k option, and -e removes a table again by cutting the file back by the table’s length. The encoder also makes an appended table itself when it compresses (a second pass over the finished file). A decoder that does not know seek tables ignores the extra bytes, because it stops at the QUIT command.

Header

Offset	Size	Field	Meaning
0	4	Signature	`SEEK`
4	4	Revision	Little-endian. The only revision written is 1
8	4	File size	Little-endian. The length in bytes of the `.shn` file the table was made for, not counting the table

All numbers in a seek table are little-endian, whatever machine wrote it.

Entries

An entry is 80 bytes and describes one place in the stream: the start of a block of the first channel. The program writes one for every hundredth block of the first channel, beginning with the first (blocks 0, 100, 200 and so on, counting the all-zero blocks too). The entries are in stream order.

Offset	Size	Field	Meaning
0	4	Sample number	The number of the first sample of the block, counted in frames (one sample of every channel)
4	4	File position	The file position just after the 512-byte buffer that held the place, which is what the program’s own `ftell` said
8	4	Buffer start	The file position where that buffer began
12	2	Bytes left in buffer	The bytes of the buffer not yet taken
14	2	Buffer offset	The bytes of the buffer already taken, always a multiple of 4: the stream is read as 32-bit words
16	2	Bits left in word	The unread bits of the word being read, 0 when it is used up
18	4	Word being read	The 32-bit word the decoder held (its unread bits are the low ones)
22	2	Bit shift	The bit shift in force
24	12	Channel 0 history	Three signed 32-bit numbers: the last three samples of channel 0 before the block, newest first, as the predictor holds them (before the bit shift is applied)
36	12	Channel 1 history	The same for channel 1; zeros in a mono file
48	16	Channel 0 means	Four signed 32-bit numbers: the first four of the block means the mean estimator holds, oldest first
64	16	Channel 1 means	The same for channel 1; zeros in a mono file

The place is a bit position in the stream: the stream’s byte at buffer start + buffer offset is the next word to read, so the bit where the block’s command begins is (buffer start + buffer offset) × 8 − bits left in word, counted from the start of the file. The command begins with its function code; it is a block of channel 0 (DIFF0 to DIFF3, QLPC or ZERO), never a setting change, so any BITSHIFT or BLOCKSIZE command before it has been read already. The buffers are filled from the start of the stream (the byte after the version byte) in steps of 512 bytes, so the buffer start is that offset plus a multiple of 512.

The entry holds what the decoder needs to carry on from the place, with these limits, which follow from what the program saves:

Only channels 0 and 1 have a history and means: a stream of more than two channels cannot be resumed from its table.
Only the first four means are saved. With the default of 4 mean blocks that is all of them; with more, the newest are missing; with fewer, the unused slots hold whatever was in memory next to the table the program kept (the program reads past the end of its array).
The block size is not saved. It is the header’s unless a BLOCKSIZE command has changed it, which the encoder does only for the last, shorter block.

Trailer of an appended table

Offset from the end	Size	Field	Meaning
−12	4	Table length	Little-endian: the length of the whole table, header and trailer included. The table starts this many bytes before the end of the file
−8	8	Signature	`SHNAMPSK`

So the number of entries is (table length − 24) ÷ 80.

Worked example

The shorten 3.6.1 sources include a small mono stream tests/mono/test.shn (11,781 bytes) and its table test.skt (92 bytes). The whole table:

53 45 45 4B  01 00 00 00  05 2E 00 00            SEEK, revision 1, file size 0x2E05 = 11,781
00 00 00 00                                         sample number 0
05 02 00 00                                         file position 0x205 = 517
05 00 00 00                                         buffer start 5 (the byte after the version byte)
C4 01                                               bytes left in buffer 452
3C 00                                               buffer offset 60
1E 00                                               bits left in word 30
AE 13 60 2C                                         word being read 0x2C6013AE
00 00                                               bit shift 0
(zeros: both channels’ histories, the means, and channel 1)

The place is bit (5 + 60) × 8 − 30 = 490. The stream’s first command is the 44-byte stored WAV header (a VERBATIM command from bit 82 to bit 490), and the command that starts at bit 490 is the first block, a DIFF1 block of 256 samples. The decoder had taken 15 words (60 ÷ 4) from its first buffer, and the last of them, the word in hand, still had 30 unread bits. The first buffer holds a full 512 bytes (the stream is 11,776 bytes after the version byte), so 512 − 60 = 452 bytes were left in it.

The stereo stream tests/stereo/test.shn has a table of 172 bytes, so 2 entries (blocks 0 and 100 of channel 0). seekable.shn in each folder is the same stream with the table and its trailer appended (11,885 and 46,285 bytes, 104 and 184 bytes longer: the table and its 12-byte trailer); the appended table is byte for byte the .skt file.

Notes

A table is only good for the exact stream it was made for. Its header holds the stream’s length, which a program can compare with the file (the file is longer by the appended table and anything after it).
Tags added after a table, or an appended table after tags, hide it from the check for SHNAMPSK in the last 12 bytes: the explorer looks for a table and for tags at the end in turn.
FFmpeg and the original decoder ignore the table; the explorer shows it and checks every entry against the state its own decoder has at that place.

.SHN File Format – Seek tables