zzzip/docs/zzz_format.md

ZZZip Archive Format Specification

• File extension: .zzz
• Mime type: application/x-zzz

[[TOC]]

If the file starts with gzip, bzip2, xz, lz4, zstd etc. magic bytes, then it's stream compressed (like tar, cpio) and the filenames and attributes are also compressed. Otherwise if it starts with ZZZip header magic, it's compressed per file (like zip). The former offers better compression ratio, the latter offers easier parsing and handling.

The format was designed in a way so that it can be used in streaming, that is, compressed and decompressed without the need of seeking (unlike zip). But if archive can be seeked, the last 48 bytes contains an overall information. It was also designed to be a full archive format (capable of backup and restore system specific attributes), but also to be interoperable (capable of sharing files between different operating systems reliably).

All integers are in little-endian format. The CRC uses the same ANSI CRC method as the gzip format (also used by EFI GPT, PNG etc.), and not the Castagnoli CRC (used by bzip2 and network packets). All strings are UTF-8 encoded and zero terminated. In the archive, all directory separators are converted to a slash '/', regardless to the native separator on the OS.

The format was designed to be future-proof, file sizes are stored on 127 bits (however the current SDK only implements 63 bits), and the format is extensible without changing the structure of the archive, meaning it will remain backwards compatible.

The basic structure of the format is as follows:

Block	Description
Encryption	Optional, may not exists
Entity	At least one entity block
Entity	More entity blocks may follow
End	Terminating block (fixed size)

An optional encryption block, then multiple entity blocks (which can be compressed individually by separate threads and then concatenated) one for each compressed file, then a terminating end block.

Encryption Block

This block is optional. If it exists, there can be only one, and it must be the first block. It is differentiated from normal data blocks by file name length being 0 so that Block Header magic can be used to identify the file format. The magic consist of two uppercase 'Z' letters, one lowercase 'z' letter, and an ASCII 26 (EOF).

Offset	Length	Description
0	4	magic, 'ZZz\032' 0x1A7A5A5A
4	2	size of the header (h)
6	2	must be zero
8	4	CRC of the password
12	1	content padding in power of 2
13	h-13	list of encryption filters

Regardless to the encryption used, the CRC of the password is always stored for quick universal verification.

Padding is stored in power of two: 0 - byte, 1 - word (2 bytes), 2 - dword (4 bytes), 3 - qword (8 bytes), 4 - 16 bytes, 5 - 32 bytes, etc. It is the biggests of the applied filters' paddings.

EncType	Description	Padding
0	AES-256-CBC	4 (16 bytes)
1	SHA-256-XOR	0 (no alignment)

With AES, first an SHA256 is calculated from the password. 16 bytes of the result starting from the offset (hash's first byte modulo 16) is used as the AES iv, then SHA256 is used again on the hash to get the AES key. It uses 16 bytes padding.

With SHA, a tricky XOR method is used. First B is calculated by summing the characters in the password modulo 256. Then SHA256 calculated for the password, each byte of the hash XOR'd with B, and SHA256 is used again on that. This gives the iv mask. As it's a symmetric cipher, both encoding and decoding uses the same method: each byte is first XOR'd with a running counter, starting from B overflowing at 256. Then another running counter starting from 0 and overflowing at 32 is used to get a byte from the mask and XOR'd with that too. If this second running counter reaches 32, it's resetted and the mask is replaced by the SHA256 of the mask with each byte XOR'd with B. This way the first counter gives a running XOR mask repeating after every 256th byte, while the second counter uses a hash which is recalculated after every 32 byte. This is fast and strong enough by itself for most applications, however it's purpose is to use it in conjuntion with AES.

When multiple filters specified, they are stored in extraction order, eg. reverse creation order. The minimum size of the block is 14, meaning at least one filter must be specified.

(Note: the same cipher can be used multiple times to strengthen encryption, or you can mix those. Do not use SHA-256-XOR twice in a row, as that's a symmetric cipher it will decrypt. An AES + SHA + AES combination is paranoid enough to store state secrets and nuclear missile launch codes.)

Entity Block

Each entity (file, directory etc.) starts with a fixed size block header, followed by varying length fields and the content's data, ending in two checksum values.

Offset	Length	Description
0	4	magic, 'ZZz\032' 0x1A7A5A5A
4	2	size of the header (h)
6	2	must be non-zero, file name length (n)
8	2	file modification year
10	1	file modification month (1 - 12)
11	1	file modification day (1 - 31)
12	1	file modification hour (0 - 23)
13	1	file modification minute (0 - 59)
14	1	file modification second (0 - 60)
15	1	block type (file format, numfilter c)
16	16	uncompressed size
32	16	content size (s)
48	2*c	compression filters
48+2*c	h-2*c-48	extra or OS-specific fields
h	n	file name (UTF-8, dir separator '/')
h+n	s	file content
h+n+s	x	optional padding (only when encrypted)
h+n+s+x	4	CRC of the uncompressed file
h+n+s+x+4	4	CRC of the block for bytes 0 - h+n+s+x+4

The highest bit of file name length gives header size's 17th bit. This means header can be 128k in length, and file name 32767 bytes (which is a lot bigger than what most OS implements, MAX_PATH under Windows is 260, and under UNIX the PATH_MAX limit is typically 4096 for example).

When encryption is used, then extra fields are padded with a special field and bytes (48+2*c) to (h) are encrypted, as well as the file content field. The size of the padding is defined by the encryption methods used, stored in the encryption block (if it exists).

File Modification Date and Time

This is a packed, OS-independent representation of last file modification time. It must be in UTC (GMT + 0) without timezones and daylight saving applied. For separate creation, modification and access times with nanosec precision, use the POSIX Timestamps extra field.

Block Type

The higher tetrad (block type & 0xF0) encodes the file format.

Block Type	Description
0x0?	regular file
0x10	hard link
0x20	symbolic link
0x30	union
0x40	character device
0x50	block device
0x60	directory
0x70	named FIFO
0x80 - 0xF0	reserved

Depending on the file format, file content differs. For file format 0x10 and above, number of compression filters and uncompressed size must be 0.

The lower tetrad (block type & 0x0F) encodes the number of compression filters used on data.

Uncompressed Size

This is the uncompressed size of the file. It's 0 for non-regular files. It's stored on 127 bits, but current implementation only uses 63 bits.

For text files this might be bigger than the actual size on UNIX, because all newline characters are calculated as 2 bytes.

Content Size

This is the size of the file content as stored in the file (minus the optional padding). For regular files, it is the compressed data size. It's stored on 127 bits, but current implementation only uses 63 bits.

Compression Filters

The number of compression filters in block type byte encodes how many filters applied. 0 means no compression, file is stored as-is (uncompressed size = content size). This is the only option when overall stream compression used on the entire archive. Otherwise that many words follow, describing filters in decompress order, or with other words in reverse compression order. The first byte in the word is the filter, the second byte is the compression level (specific to filters, it is only informative as most compression filters encode the level in their headers and that's what really counts). When encryption is used, it's always applied to the final compressed data, meaning when extracting files, a decompressor must iterate on the encryption filters first, then iterating on compression filters.

Filter	Level	Description
0	0	ASCII newline conversion
1	0-2	executable bcj (level=arch)
2	1-9	zlib deflate
3	1-9	bzip2
4	0-9	lzma
5	0-9	xz (lzma2)
6	7-8	PPMd
7	1-22	zstd
8 - 31	x	reserved

Extra Fields

All extra fields have a common, 4 bytes long header:

Offset	Length	Description
0	1	magic, field type
1	1	magic, field format revision (0 for now)
2	2	size of field (h, including header)
4	h-4	field specific data

With OS-Specific fields (field type >= 0x10) the field format revision's most significant bit is set when saved on a big-endian machine (that's because OS-Specific field data is not converted to little-endian, unlike all the other integers in the archive).

With encryption there might be a padding field as the last field, and the entire extra fields block is encrypted.

Padding

Offset	Length	Description
0	2	magic, 0x0000
2	2	size of field (h, including header)
4	h-4	must be zero

Comment

Offset	Length	Description
0	2	magic, 0x0001
2	2	size of field (h, including header)
4	h-4	zero terminated UTF-8 text

Mime Type

Offset	Length	Description
0	2	magic, 0x0002
2	2	size of field (h, including header)
4	h-4	mime type (like 'text/html')

Mime type might contain optional character set information of the content, like "text/html;charset=latin-1", however converting text into UTF-8 is strongly recommended instead.

File Icon

Offset	Length	Description
0	2	magic, 0x0003
2	2	size of field (h, including header)
4	h-4	compressed icon (preferably PNG or SVG)

Icon can be in any format as long as the format can be determined by magic bytes, but for interoperability reasons PNG (bitmap) or SVG (vector) strongly recommended. Keep the icon's size small, around 32k tops.

Meta Info (XAttr)

Offset	Length	Description
0	2	magic, 0x0004
2	2	size of field (h, including header)
4	h-4	key value pairs

Each pair is stored with the following record format:

Offset	Length	Description
0	2	size of the attribute (s)
2	n	zero terminated UTF-8 attribute name
2+n	s	attribute data (could be binary)

This record is repeated as many times as the number of meta (XAttr) keys on the file.

POSIX Timestamps

Offset	Length	Description
0	2	magic, 0x0005
2	2	size of field (12, 20, 28 or 36)
4	8	last modify time in nanosec, UNIX UTC Epoch
12	8	last access time in nanosec (optional)
20	8	last stat change time in nanosec (optional)
28	8	file creation time in nanosec (optional)

These should be used for all OS, even on non-UNIX systems the time specification (like MSDateTime) can be converted into POSIX timestamps. Regardless other OS-Specific fields might contain OS-Specific representation of times.

Note that POSIX timestamps will wrap around after 2554-07-21, however combining with the year from the OS-independent modification time (and using other timestamp's year in relative to modification timestamp's year) these can be up to 65535-12-31.

POSIX Access Rights

Offset	Length	Description
0	2	magic, 0x0006
2	2	size of field (h, including header)
4	4	st_mode
8	8	st_uid
16	8	st_gid
24	h-24	user name UTF-8, zero, group name, zero

The user name and group name both are converted to UTF-8, terminated by a zero, and concatenated.

POSIX Access Control List

Offset	Length	Description
0	2	magic, 0x0007
2	2	size of field (h, including header)
4	h-4	UTF-8 ACL data

Stores POSIX compatible access control lists as defined by NFSv4 ACL. Each ACL is stored in a newline terminated string, with the following format: "type:flags:principal:permissions\n".

Text-based Access Control List

Offset	Length	Description
0	2	magic, 0x0008
2	2	size of field (h, including header)
4	h-4	ACL data

Used to store OS/2 ACLs, but non-NFS POSIX ACLs (POSIX 1003e draft standard 17 mostly) can be stored with this extra field as well.

UUID Access Control List

Offset	Length	Description
0	2	magic, 0x0009
2	2	size of field (h, including header)
4	16	owner UUID and rights
20	h-20	access UUID and rights

Stores access control lists, each access control entry is a UUID where the last (15th) byte stores the right bits:

Bit	Description
0	read
1	write
2	execute
3	append
4	delete
5	group ace
6	setuid
7	setgid, inherit ACL

OS-Specific Extra

Magic 0x000A to 0x000F are reserved. For a full list of other fields, see "Appendix: List of OS-Specific Extra Fields" below.

File Name

Encoded in UTF-8. Directory separator is '/', regardless to the OS. It must not start with '/', and terminated by a zero. If the block's file format is a directory, then the file name must end in '/', otherwise it must not. Cannot be empty, meaning the file name size field in the block's header must be at least 2 (one character plus the terminating zero).

File Content

Depends on file format field. For text files OS-Specific newline (CR or CR LF) must be converted to ASCII 10 (LF, '\n'). With binary no conversion is used. If compression filters are used, this field contains the compressed data.

For hard links, symbolic links and unions the content is a zero terminated UTF-8 path, the target.

For character and block devices, the content is 8 bytes long:

Offset	Length	Description
0	4	device major
4	4	device minor

For directories and named FIFOs the content is empty.

When encryption is used and the padding field isn't 0, then there are additional bytes to the content. For example if padding is 4 in encryption block (means 16 bytes) and content size is 119, then 128 bytes will be stored in the archive.

CRCs

The Entity Block is ended with two ANSI CRC values. The first one is the CRC of the uncompressed data if file type is text or binary. It is the content's CRC for all the other types. The second CRC is calculated from the beginning of the block to the end, including the uncompressed CRC but not including the second CRC value itself.

End Block

A ZZZip archive is always terminated with these 48 bytes:

Offset	Length	Description
0	4	magic, 'ZEnd'
4	2	size of the header (48)
6	2	mask of file types used
8	2	archive creation year
10	1	archive creation month (1 - 12)
11	1	archive creation day (1 - 31)
12	1	archive creation hour (0 - 23)
13	1	archive creation minute (0 - 59)
14	1	archive creation second (0 - 60)
15	1	file format version
16	16	sum of uncompressed size fields
32	8	total number of blocks
40	4	mask of compression filters used
44	4	CRC of the archive from 0 - this byte

Mask of File Type Used

This is a bitmask representing all file formats in the archive, (1 << ((block type) >> 4)). For example, if the archive contains at least one character device, then 4th bit in the mask will be set.

Archive Creation Date and Time

This is a packed, OS-independent representation of the creation time. It must be in UTC (GMT + 0) without timezones or daylight saving applied.

File Format Version

Currently 0. This only refers to the file format structures, adding new encryption methods, compression filters or field types should not bump the file format version.

Mask of Compression Filters Used

This is a bitmask representing all compression filters in the archive, (1 << filter). For example, if the archive uses zstd on at least one file, then 7th bit in the mask will be set.

CRC

This is the checksum for the entire archive from the beginning up to the End Block's 44th byte.

Appendix: List of OS-Specific Extra Fields

Extra fields with magic 0x0010 to 0x00FF are OS-specific fields. If they were saved on a big-endian machine, field revision byte will have the most significant bit set, so 0x8010 to 0x80FF. They are expected to be stored in key-value pairs, however an OS is allowed to use its own format.

10 OS/2

OS/2 specific data (magic = 0x0010) is a FEA2LIST structure.

11 OpenVMS

OpenVMS specific data (magic = 0x0011) is a standard key-value pair list of attribute records:

Offset	Length	Description
0	2	tag, as in ATR$C_XXXX and ATR$S_XXXX
2	2	size of the attribute (s, including header)
4	s-4	attribute data

12 MacOS

MacOS specific data (magic = 0x0012) is in RFC 1740 AppleDouble format. Consider using the meta info field instead.

13 Windows NT

Windows specific data (magic = 0x0013) used to store ACL: a binary ACL structure followed by DWORD aligned ACE entries. Currently all the other WinNT attributes can be described with common extra fields.

14 TargetFour

TBD.

15 OS/390

TBD.

16 AS/400

TBD.

17 Amiga / MorphOS

TBD.

18 BeOS / Haiku

TBD.

Other

Other OS specific data yet to be defined, but preferably using a similar tag attribute list like OpenVMS' with the same tags as in zip if possible. It is preferable to avoid OS-Specific extra fields, and attributes should be stored in common extra fields instead whenever possible.

Third party developers are welcome to contribute to this spec and add more Operating Systems and their attributes. OS type magic id will be assigned in PR filing order.