analyst@unixshadow:~$ lvm-thin –layers –thick-vs-thin # seven layers between a RAID array and your files
LVM Thin Pools, Explained From the Metal Up
RAID to Datastore to VMDK to Thin Pool — and Thick vs Thin at the VM Level
A file on a virtual machine sits at the bottom of a stack of abstractions, and every layer in that stack can lie to the layer above it about how much space exists. Thin pools are where that lie becomes a feature — and where it becomes a failure. This guide walks the whole stack, then builds a thin pool, then explains thick vs thin at the VM level, because the two kinds of thin do not know about each other.
# the whole stack, top to bottom — what your file actually sits on
Read it bottom to top when you build it, top to bottom when something breaks. Each layer hides the one below it, and each layer can overpromise to the one above it.
your files
↓
filesystem — XFS or ext4, on a logical volume
↓
thin LV — e.g. 100G logical size, 10G actually written
↓
thin pool — the shared bucket of chunks all thin LVs draw from
↓
volume group (VG) — e.g. vg0, the pool’s home
↓
physical volume (PV) — LVM metadata on a partition
↓
partition — e.g. /dev/sda3, or the whole disk
↓
disk as the guest sees it — /dev/sda
↓
VMDK — the virtual disk file on the datastore
↓
datastore — VMFS or NFS, on the host’s storage
↓
physical disks in RAID — the metal
The confusion is not the commands. It is the stack. A sysadmin who can picture these ten layers can debug any of them; a sysadmin who cannot will grow the filesystem when the problem is the pool, or grow the pool when the problem is the datastore.
# thick vs thin at the VM level — the first overpromise in the stack
The VMDK is the first place space gets thin. When you create a virtual disk in vSphere you choose thick or thin, and that choice is about when the space is allocated, not about how much the guest thinks it has. The guest always sees the full provisioned size — a 100 GB thin VMDK looks like a 100 GB disk inside the VM from the first second. What differs is what sits on the datastore underneath it.
Thick, lazy zeroed. All 100 GB is reserved on the datastore at creation, but the blocks are only zeroed as the guest first writes to them. Creation is fast, the space is guaranteed, and the first write to a fresh block pays a small zeroing cost. This is the default most admins should start from: no surprises, no overprovisioning, and the datastore cannot fill up underneath a VM that never grew.
Thick, eager zeroed. All 100 GB is reserved and zeroed at creation. Creation is slow — vSphere writes zeros across the whole disk — but every later write is full speed, because no block ever needs zeroing on first use. This is the choice for the workloads that care: heavy databases, anything with latency-sensitive writes, and anything where a first-write penalty would show up in production. It is also the required format for some features — fault tolerance historically required eager zeroed, and some clustering setups still do. The cost is time at provisioning and space you cannot get back.
Thin. The VMDK starts near zero on the datastore and grows as the guest writes, up to its provisioned size. Creation is instant, a hundred VMs can share a datastore that could never hold all of them at full size, and a VM that only ever uses 10 GB only ever costs 10 GB. That is the reason thin exists: utilisation. Most VMs never fill their disks, and thin provisioning stops you paying for space nobody uses.
The reason thin bites. The guest does not know the VMDK is thin, and the datastore does not know how much the guest intends to write. If every thin VM on a datastore grows at once, the datastore fills up, and a full datastore does not slow VMs down — it stops them. Writes fail, VMs pause or crash, and the failure is at the host layer, not in the guest, so the guest’s logs show nothing useful. Thin at the VM level is a bet that not everything grows at once, and the bet is only safe if somebody is watching the datastore’s free space — an alarm at 80%, a hard look at 90%, and a plan before 100%. The same bet, made twice, is the theme of this whole post: thin VMDK on top of a thin pool means two layers overpromising at once, and neither one can see the other doing it.
Thick, lazy zeroed. All 100 GB is reserved on the datastore at creation, but the blocks are only zeroed as the guest first writes to them. Creation is fast, the space is guaranteed, and the first write to a fresh block pays a small zeroing cost. This is the default most admins should start from: no surprises, no overprovisioning, and the datastore cannot fill up underneath a VM that never grew.
Thick, eager zeroed. All 100 GB is reserved and zeroed at creation. Creation is slow — vSphere writes zeros across the whole disk — but every later write is full speed, because no block ever needs zeroing on first use. This is the choice for the workloads that care: heavy databases, anything with latency-sensitive writes, and anything where a first-write penalty would show up in production. It is also the required format for some features — fault tolerance historically required eager zeroed, and some clustering setups still do. The cost is time at provisioning and space you cannot get back.
Thin. The VMDK starts near zero on the datastore and grows as the guest writes, up to its provisioned size. Creation is instant, a hundred VMs can share a datastore that could never hold all of them at full size, and a VM that only ever uses 10 GB only ever costs 10 GB. That is the reason thin exists: utilisation. Most VMs never fill their disks, and thin provisioning stops you paying for space nobody uses.
The reason thin bites. The guest does not know the VMDK is thin, and the datastore does not know how much the guest intends to write. If every thin VM on a datastore grows at once, the datastore fills up, and a full datastore does not slow VMs down — it stops them. Writes fail, VMs pause or crash, and the failure is at the host layer, not in the guest, so the guest’s logs show nothing useful. Thin at the VM level is a bet that not everything grows at once, and the bet is only safe if somebody is watching the datastore’s free space — an alarm at 80%, a hard look at 90%, and a plan before 100%. The same bet, made twice, is the theme of this whole post: thin VMDK on top of a thin pool means two layers overpromising at once, and neither one can see the other doing it.
# what a thin pool actually is — the second overpromise in the stack
A thin pool is a logical volume with a job: it holds a shared bucket of storage, and thin LVs draw from it only when they actually write data. It is made of two LVs, not one — a data LV that holds the chunks, and a metadata LV that tracks which chunk belongs to which thin volume. You never write to either of them directly; you write to the thin LVs, and the pool hands out chunks on demand. The chunk size is the granularity: the default is 64 KiB, which is fine for general use; smaller chunks waste metadata on tiny writes, larger chunks waste space when a volume writes a little in many places.
Why
Why a full pool is catastrophic, not inconvenient. A thick LV that fills up fails one filesystem. A thin pool that fills up fails every thin LV that draws from it, at once, because there are no chunks left to hand out — writes fail across all of them, filesystems go read-only or worse, and the failure is in the shared layer, not in any one volume. That is why thin pools are monitored, not just created: Data% and Meta% are the two numbers that matter, and the metadata LV filling up is just as fatal as the data LV filling up, which is the part people forget — the metadata LV is small, and a pool with many snapshots or heavy churn can fill it while Data% still looks comfortable.
Why
lvs looks wrong, and is not. A thin LV created with -V 100G on a 50 GB pool shows 100 GB in lvs, and that is correct — 100 GB is its logical size, the most it can ever hold, not what it holds now. What it holds now is in the pool: lvs -a shows the pool’s Data% (how much of the pool’s chunks are handed out) and Meta% (how much of the metadata LV is used). The pool is the real number. The LV sizes are promises, and the sum of the promises can exceed the pool — that is overprovisioning, and it is the entire point. A pool with three 100 GB thin LVs on 50 GB of real space is not a mistake, it is the design, right up until all three try to fill up at once.Why a full pool is catastrophic, not inconvenient. A thick LV that fills up fails one filesystem. A thin pool that fills up fails every thin LV that draws from it, at once, because there are no chunks left to hand out — writes fail across all of them, filesystems go read-only or worse, and the failure is in the shared layer, not in any one volume. That is why thin pools are monitored, not just created: Data% and Meta% are the two numbers that matter, and the metadata LV filling up is just as fatal as the data LV filling up, which is the part people forget — the metadata LV is small, and a pool with many snapshots or heavy churn can fill it while Data% still looks comfortable.
Thin provisioning is not compression and it is not magic. It is a shared bucket with a ledger, and the ledger is as much a point of failure as the bucket.
# building a thin pool — the implementation, start to finish
1. The volume group, as usual. A thin pool lives in a volume group, so the bottom of the stack is the same as any LVM setup — a PV on a partition or a whole disk, in a VG with free space:
2. Create the pool. One command makes both LVs — the data LV and the metadata LV — and the pool itself:
That carves 50 GB out of vg0 for the pool. The metadata LV is created automatically at a size LVM picks from the pool size; if you want to set it explicitly,
3. Create thin volumes in the pool.
Two 100 GB thin LVs on a 50 GB pool. Both are empty. Both look 100 GB to the guest.
4. Filesystem and mount, as usual. A thin LV takes a filesystem exactly like a thick one:
Add it to /etc/fstab by UUID, as always —
5. Watch it. This is not optional, it is the operation:
Set autoextend so the pool grows before it fills: in
6. Snapshots, the reason thin pools exist in the first place. A snapshot of a thin LV is cheap — it shares chunks with the original and only stores what changes — which is why thin pools are the standard backing for VM images and container storage:
pvcreate /dev/sdb; vgcreate vg0 /dev/sdb2. Create the pool. One command makes both LVs — the data LV and the metadata LV — and the pool itself:
lvcreate --type thin-pool -L 50G -n thinpool vg0That carves 50 GB out of vg0 for the pool. The metadata LV is created automatically at a size LVM picks from the pool size; if you want to set it explicitly,
--poolmetadatasize 1G does it, and on a pool that will hold many snapshots or see heavy churn, sizing metadata up front is cheaper than fixing it later — growing a metadata LV afterwards is possible (lvextend on the hidden _tmeta volume), but it is fiddly and best avoided by sizing it generously at creation.3. Create thin volumes in the pool.
-V is the virtual size — the promise — and it can exceed the pool:lvcreate --type thin -V 100G -n lv_data --thinpool thinpool vg0lvcreate --type thin -V 100G -n lv_logs --thinpool thinpool vg0Two 100 GB thin LVs on a 50 GB pool. Both are empty. Both look 100 GB to the guest.
4. Filesystem and mount, as usual. A thin LV takes a filesystem exactly like a thick one:
mkfs.xfs /dev/vg0/lv_data; mkdir -p /data; mount /dev/vg0/lv_data /dataAdd it to /etc/fstab by UUID, as always —
blkid gives you the UUID.5. Watch it. This is not optional, it is the operation:
lvs -a -o +devices,seg_monitor vg0 — shows the pool, its hidden data and metadata LVs, and every thin LV with its Data% and Meta%.lvs -o lv_name,data_percent,metadata_percent vg0/thinpool — the two numbers that matter, for the pool itself.Set autoextend so the pool grows before it fills: in
/etc/lvm/lvm.conf, thin_pool_autoextend_threshold (default 100 — set it to 70 or 80) and thin_pool_autoextend_percent (default 20). When Data% crosses the threshold, LVM extends the pool automatically from the VG’s free space — which only works if the VG has free space, so a pool with autoextend and a full VG is a pool with a safety net made of air. Keep free space in the VG, or the autoextend has nothing to extend into.6. Snapshots, the reason thin pools exist in the first place. A snapshot of a thin LV is cheap — it shares chunks with the original and only stores what changes — which is why thin pools are the standard backing for VM images and container storage:
lvcreate --type thin -s -n snap_data /dev/vg0/lv_data. The cost is metadata churn and pool usage that grows with every change to the original, so a snapshot left for months is a pool slowly filling up for no reason. Snapshots are for hours and days, not for archiving.# the two thins, stacked — the trap this post exists to name
Thin VMDK plus thin pool is the default shape of a modern VM fleet, and it is two overpromises stacked on each other. The datastore overpromises to the VMDKs, the pool overpromises to the thin LVs, and neither layer can see the other. A datastore at 95% and a pool at 95% are both fine on their own and a disaster together, because the failure in either layer takes down everything above it — and the layer that fails is not the layer that reports the error. When a VM’s writes start failing, check both: datastore free space on the host, and pool Data% and Meta% in the guest. One of them is the answer, and
The rule that covers both: overprovisioning is a capacity-planning decision, not a default. Thin is the right choice when utilisation is genuinely low and somebody is watching the real numbers — datastore free space, pool Data%, pool Meta% — with alarms set before the numbers get exciting. Thick is the right choice when the workload cannot tolerate a full datastore or a full pool, when the write pattern is heavy and latency-sensitive, or when nobody is watching. The most expensive disk in a fleet is the thin one nobody monitored.
lsblk in the guest will not show you either one.The rule that covers both: overprovisioning is a capacity-planning decision, not a default. Thin is the right choice when utilisation is genuinely low and somebody is watching the real numbers — datastore free space, pool Data%, pool Meta% — with alarms set before the numbers get exciting. Thick is the right choice when the workload cannot tolerate a full datastore or a full pool, when the write pattern is heavy and latency-sensitive, or when nobody is watching. The most expensive disk in a fleet is the thin one nobody monitored.
Every layer in the stack is somebody’s abstraction, and every abstraction leaks at the worst time. The fix is not fewer layers — it is knowing which layer you are standing on when something breaks.
# case study: thin-pool exhaustion mistaken for XFS corruption — a real AlmaLinux 9 VM, and the order that matters
Symptoms. The VM dropped into dracut emergency mode with
What happened. The VMDK was thick-provisioned and the datastore had plenty of room, so the hypervisor was ruled out early. The first read — “no thin pools here” — was wrong, and it stayed wrong until the boot log showed
In the dracut shell the LVM tools only worked in their long form —
Lessons. XFS error 28 on LVM thin storage should prompt you to check pool usage with
XFS (dm-3): metadata I/O error ... error 28, followed by “Shutting down filesystem”. Error 28 is ENOSPC — no space left — which pointed away from a disk media fault and toward a space problem, even though nothing in the guest had reported the disk full.What happened. The VMDK was thick-provisioned and the datastore had plenty of room, so the hypervisor was ruled out early. The first read — “no thin pools here” — was wrong, and it stayed wrong until the boot log showed
device-mapper: thin: ... out-of-data-space lines. The pool was full; the filesystem was just the messenger.xfs_repair -n showed many orphaned inodes. A normal xfs_repair refused because of a dirty log, and mounting to replay the log failed with the same ENOSPC — there was no space to write the replay into. xfs_repair -L then completed and the filesystem mounted, but the VM crashed again on the next boot, because the pool was still full. The repair had fixed the filesystem; nothing had fixed the pool.In the dracut shell the LVM tools only worked in their long form —
lvm lvs -a — which showed pool00 (~36 GB) at 100% data usage. Root (89 GB) and swap (6 GB) were thin volumes on it, and the VG still had ~59 GB free. The fix was one command: lvm lvextend -L +30G /dev/mapper/almalinux-pool00 grew the pool, Data% dropped to ~55%, and a new xfs_repair -n showed more structural damage from the second crash. Once the pool had space, a plain mount replayed the log and the VM came back.Lessons. XFS error 28 on LVM thin storage should prompt you to check pool usage with
lvs -a — even when df inside the guest shows free space, because df reports the filesystem, not the pool underneath it. Do not use xfs_repair -L until the pool has space: a destroyed log is not reversible, and the repair cannot fix the underlying exhaustion — it can only clear the symptom, which is why the VM crashed again on the next boot. And a thin pool whose virtual volumes total far more than the pool is a latent outage, not a configuration: set thin_pool_autoextend_threshold and thin_pool_autoextend_percent in lvm.conf, and monitor pool usage, because the pool filling up is the one failure in this stack that takes every volume on it down at once.The order is the whole case study: extend the pool first, then repair the filesystem. Repairing first feels productive and changes nothing — the next boot proves it.
# quick reference — the whole post in six lines
Stack: RAID → datastore → VMDK → /dev/sda → partition → PV → VG → thin pool → thin LV → filesystem.
Thick vs thin (VM): thick reserves the space, thin allocates on write; thin is efficient and needs datastore monitoring, eager-zeroed thick is fastest for heavy writes.
Create a pool:
Watch:
Trap: thin VMDK on a thin pool is two overpromises stacked; a full pool fails every volume in it at once.
Snapshots: cheap on thin LVs, for hours and days — not archiving.
Case study: XFS error 28 on thin storage means check the pool with
Thick vs thin (VM): thick reserves the space, thin allocates on write; thin is efficient and needs datastore monitoring, eager-zeroed thick is fastest for heavy writes.
Create a pool:
lvcreate --type thin-pool -L 50G -n thinpool vg0, then lvcreate --type thin -V 100G -n lv_data --thinpool thinpool vg0.Watch:
lvs -a — Data% and Meta% are the real numbers; set thin_pool_autoextend_threshold to 70–80 and keep free space in the VG.Trap: thin VMDK on a thin pool is two overpromises stacked; a full pool fails every volume in it at once.
Snapshots: cheap on thin LVs, for hours and days — not archiving.
Case study: XFS error 28 on thin storage means check the pool with
lvs -a first — extend the pool, then repair the filesystem, never the reverse.Series: GuidesStatus: Published 8 Nov 2026
