Inside Storage Systems - Storage Layout (1): Data and Metadata

Data are stored on some physical medium like hard drives or SSDs, but they are not simply placed there arbitrarily. A certain layout is defined by storage systems to organize data in the disk. For example, we need to know where of a disk is available for the data to be stored, and when we need some data where to find it.


A storage system keeps more than the data users want to store. The additional information include the one describing the data or the system status. They are called metadata. Typical metadata include the pointers to the data (i.e., where to get the data) and the available free spaces of the storage system.

If some metadata are used to locate certain data, then how to locate those metadata? Do we need metadata of the metadata? To avoid such infinite loop, metadata are usually stored in the special location of a storage system, for example, the beginning or the end of a storage unit, where they can be directly accessed without additional pointers.

Because of metadata, the data storage efficiency is not 100%. However, the size of metadata is fixed and is typically small compared to the normal data. The majority of a storage system is still used by user data, especially today when data tend to become larger and larger.

I/O unit

Data are stored in disk as either 0 or 1. Therefore, the finest unit of I/O can be in bit. However, it’s rarely for a storage system to read or write one bit at a time. Single bit doesn’t really tell much information and typical I/O requests are much larger than that. On the other hand, using a very large unit size may not fit either. When we are only interested in a small set of data, a too large granularity just wastes I/O resources by reading or writing large portion of unused data. With such consideration, storage systems usually perform disk I/O at KB.

Storage unit

It’s a good idea for storage systems to organize the disk space allocated to them based on the I/O unit, for example, by dividing the disk space into fixed size of units with the same size as the I/O unit. With that, each I/O would operate on a whole storage unit. Because of the fixed size, it’s also convenient to locate each storage unit.

Example - SQL Server

The disk space allocated to data storage in SQL Server is organized in pages. The size of a page is 8KB. I/O unit is also at the page level. That is, each I/O operation reads or writes a whole page.

Pages contain the metadata that can be directly located. Each page begins with a page header that tells the information about the page, such as the page number, page type, free space status, and etc. At the end of the page, a set of row offsets point to the location of each row in the page. For the type of data page, data are stored as rows.

Source: Pages and Extents Architecture Guide

Some pages contain only metadata. For example, A Page Free Space (PFS) page records the allocation and free space status of each page. It uses 1 byte for each page and thus a PFS page tracks about 8000 pages.

More details about the page organization in SQL Server can be referred to the official document: Pages and Extents Architecture Guide