Day 5 – Disk Management & LVM in Linux

๐Ÿ“Œ Introduction

Welcome to Day 5 of our Linux Administration series!

Disk and storage management is one of the most important responsibilities of a Linux System Administrator. In enterprise environments, administrators regularly perform tasks such as:

  • Checking disk and filesystem utilization
  • Creating and managing partitions
  • Creating and extending LVM volumes
  • Increasing filesystem capacity
  • Migrating data between disks
  • Replacing disks without moving application data manually
  • Resizing cloud disks in AWS
  • Troubleshooting filesystem and mount-related issues

In this article, we will start with traditional partitioning and then move to LVM (Logical Volume Manager), which provides much greater flexibility for enterprise storage management.

Let's begin! ๐Ÿš€


๐Ÿ“‚ 1️⃣ Types of File Systems in Linux

Linux supports many filesystem types. Some commonly encountered filesystems in enterprise environments are:

FilesystemCommon Usage
ext4General-purpose Linux filesystem
XFSCommon/default filesystem in RHEL-based enterprise systems
vfatEFI partitions and removable/USB storage
NFSNetwork-based Linux/Unix storage
CIFS/SMBWindows file shares

Check Filesystem Type

Use lsblk -f:

lsblk -f

Example:

NAME        FSTYPE      LABEL UUID                                 MOUNTPOINT
sda
├─sda1      xfs               1234-ABCD                            /boot
└─sda2      LVM2_member       xxxx-xxxx
  ├─rl-root xfs               xxxx-xxxx                            /
  └─rl-home xfs               xxxx-xxxx                            /home

Other useful commands:

blkid

and:

df -Th

df -Th is particularly useful because it displays both filesystem type and disk usage.


๐Ÿ“Š 2️⃣ Essential Disk Management Commands

๐Ÿ”น Check Disk Space

df -h

The -h option displays sizes in a human-readable format.

Example:

Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/rl-root      20G   12G  8.0G  60% /

๐Ÿ”น Check Filesystem Type and Usage

df -Th

๐Ÿ”น Check Directory Size

du -sh /var

For example:

du -sh /var/log

๐Ÿ”น Identify Large Directories

To check directories under /var:

du -h --max-depth=1 /var | sort -hr

This is extremely useful when troubleshooting a filesystem that is running out of space.

Example:

8.5G    /var
5.2G    /var/log
2.1G    /var/cache
800M    /var/lib

๐Ÿ”น Check Inode Usage

Sometimes a filesystem has free disk space but cannot create new files because all inodes have been consumed.

Check inode usage:

df -i

Example:

Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda2      5242880  524000 4720880   10% /

๐Ÿ”น List Block Devices

lsblk

For more information:

lsblk -f

You can also display disk topology:

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS

๐Ÿงฑ 3️⃣ Traditional Partitioning – fdisk

Before working with LVM, it is important to understand traditional disk partitioning.

fdisk can be used to create and manage partitions on disks.

First identify the disk:

lsblk

Suppose the new disk is /dev/sdb.

Start fdisk:

fdisk /dev/sdb

Common fdisk options include:

OptionFunction
nCreate a new partition
pDisplay partition table
dDelete a partition
tChange partition type
wWrite changes and exit
qQuit without saving

For example:

Command (m for help): n

Create the partition and then write the changes:

Command (m for help): w

Verify:

lsblk

You may now see:

sdb
└─sdb1

Create a Filesystem

For ext4:

mkfs.ext4 /dev/sdb1

For XFS:

mkfs.xfs /dev/sdb1

⚠️ Warning: mkfs formats the specified device. Formatting the wrong device can result in data loss. Always verify the device with lsblk before running the command.


Mount the Partition

Create a mount point:

mkdir /data

Mount:

mount /dev/sdb1 /data

Verify:

df -h /data

๐Ÿš€ 4️⃣ What is LVM?

LVM stands for Logical Volume Manager.

LVM provides a flexible way to manage storage by separating the physical disk from the logical filesystem layout.

Instead of directly creating a filesystem on a disk or partition, LVM introduces an additional storage-management layer.

LVM Architecture

Physical Disk
      ↓
Physical Volume (PV)
      ↓
Volume Group (VG)
      ↓
Logical Volume (LV)
      ↓
Filesystem
      ↓
Mount Point

For example:

/dev/sdb
   ↓
PV: /dev/sdb
   ↓
VG: vg01
   ↓
LV: lv_data
   ↓
XFS
   ↓
/data

Why Use LVM?

LVM provides:

✔ Flexible storage management
✔ Storage pooling
✔ Online filesystem expansion
✔ Logical volume resizing
✔ Disk migration using pvmove
✔ Snapshot capabilities
✔ Easier storage administration


๐Ÿ›  5️⃣ Create LVM Step-by-Step

Let's create an LVM configuration using /dev/sdb.

⚠️ Make sure /dev/sdb is the correct empty disk before starting.


๐Ÿ”น Step 1: Create Physical Volume

Create the PV:

pvcreate /dev/sdb

Verify:

pvs

Detailed information:

pvdisplay

๐Ÿ”น Step 2: Create Volume Group

Create a volume group named vg01:

vgcreate vg01 /dev/sdb

Verify:

vgs

Detailed information:

vgdisplay vg01

๐Ÿ”น Step 3: Create Logical Volumes

Create a 1 GB ext4 LV:

lvcreate -L 1G -n lv_ext4 vg01

Create another 1 GB LV:

lvcreate -L 1G -n lv_xfs vg01

Verify:

lvs

or:

lvdisplay

๐Ÿ“ 6️⃣ Create Filesystems

Create an ext4 filesystem:

mkfs.ext4 /dev/vg01/lv_ext4

Create an XFS filesystem:

mkfs.xfs /dev/vg01/lv_xfs

Verify:

lsblk -f

๐Ÿ“Œ 7️⃣ Mount Logical Volumes

Create mount points:

mkdir /lvm_ext4
mkdir /lvm_xfs

Mount the filesystems:

mount /dev/vg01/lv_ext4 /lvm_ext4
mount /dev/vg01/lv_xfs /lvm_xfs

Verify:

df -h

You can also verify with:

lsblk -f

๐Ÿ”„ 8️⃣ Make LVM Mounts Persistent Using /etc/fstab

A mount performed manually will not necessarily survive a reboot.

Edit:

vi /etc/fstab

Instead of relying on device names, using UUIDs is generally preferred.

Find the UUID:

blkid /dev/vg01/lv_ext4
blkid /dev/vg01/lv_xfs

Example:

/dev/vg01/lv_ext4: UUID="1111-2222" TYPE="ext4"
/dev/vg01/lv_xfs:  UUID="3333-4444" TYPE="xfs"

Add entries such as:

UUID=1111-2222 /lvm_ext4 ext4 defaults 0 0
UUID=3333-4444 /lvm_xfs  xfs  defaults 0 0

Test the configuration:

mount -a

Then verify:

df -h

๐Ÿ’ก Best Practice: Always run mount -a after modifying /etc/fstab. A syntax error in fstab can cause boot-related problems.


๐Ÿ“ˆ 9️⃣ Extend an LVM Volume

One of the biggest advantages of LVM is the ability to increase storage capacity.

Suppose /dev/sdc is a new disk.

First verify:

lsblk

๐Ÿ”น Step 1: Create a New Physical Volume

pvcreate /dev/sdc

Verify:

pvs

๐Ÿ”น Step 2: Add the PV to the Existing VG

vgextend vg01 /dev/sdc

Verify:

vgs

The additional capacity is now available inside vg01.


๐Ÿ”น Step 3: Extend the Logical Volume

For example, increase lv_ext4 by 1 GB:

lvextend -L +1G /dev/vg01/lv_ext4

Verify:

lvs

๐Ÿ“ 1️⃣0️⃣ Resize the Filesystem

Increasing the LV does not automatically mean the filesystem has been expanded unless you use an option such as lvextend -r.

For ext4:

resize2fs /dev/vg01/lv_ext4

For XFS:

xfs_growfs /lvm_xfs

Check the final size:

df -h

๐Ÿ’ก Easier Method

For supported filesystems, you can extend the LV and filesystem together:

lvextend -r -L +1G /dev/vg01/lv_ext4

This is convenient because the filesystem resize is performed as part of the operation.


๐Ÿ”„ 1️⃣1️⃣ Migrating Data Using pvmove

pvmove is extremely useful when replacing a disk in an LVM environment.

For example:

Old Disk                 New Disk
/dev/sdc                 /dev/sdd
    ↓                        ↓
   PV                       PV
    \                       /
     \                     /
        Volume Group

Suppose /dev/sdc needs to be replaced.

Step 1: Add the New Disk

pvcreate /dev/sdd

Step 2: Add It to the Volume Group

vgextend vg01 /dev/sdd

Step 3: Move LVM Extents

pvmove /dev/sdc /dev/sdd

This moves allocated extents from the old PV to the new PV.

Monitor the PVs:

pvs

You can also inspect LV-to-PV allocation:

lvs -a -o +devices

Step 4: Remove the Old Disk from the VG

Once all data has been moved:

vgreduce vg01 /dev/sdc

Verify:

pvs

Step 5: Remove the PV Signature

pvremove /dev/sdc

The disk can now be decommissioned or reused.

๐Ÿ’ก pvmove is particularly useful in enterprise environments when disks need to be replaced while keeping the logical volume configuration intact.


๐Ÿ—‘ 1️⃣2️⃣ Decommission LVM

When an LVM configuration is no longer required, it can be removed.

First unmount the filesystem:

umount /lvm_ext4

Remove the logical volume:

lvremove /dev/vg01/lv_ext4

If the volume group is no longer required:

vgremove vg01

Finally remove the PV:

pvremove /dev/sdb

Verify:

pvs
vgs
lvs

⚠️ Warning: Removing LVs, VGs, or PVs can destroy access to data. Always confirm the storage layout and obtain the required approval before performing decommissioning activities on production systems.


☁️ 1️⃣3️⃣ Resizing a Disk in AWS

Cloud environments make disk expansion easier because an EBS volume can be increased without physically replacing the disk.

A typical Linux workflow is:

AWS EBS Volume
      ↓
Operating System detects increased disk
      ↓
Partition / LVM extended
      ↓
Filesystem extended

๐Ÿ”น Step 1: Increase the EBS Volume

Increase the EBS volume size through the AWS management interface or approved AWS automation.


๐Ÿ”น Step 2: Verify the Disk

On the Linux server:

lsblk

You may see the disk size has increased while the partition is still smaller.

Example:

NAME   SIZE TYPE MOUNTPOINT
xvda    50G disk
└─xvda1 30G part /

๐Ÿ”น Step 3: Extend the Partition

If the root filesystem is on /dev/xvda1:

growpart /dev/xvda 1

Verify:

lsblk

⚠️ The exact device name may be /dev/xvda, /dev/nvme0n1, or another name depending on the EC2 instance and storage configuration.


๐Ÿ”น Step 4: Extend the Filesystem

For ext4:

resize2fs /dev/xvda1

For XFS:

xfs_growfs -d /

Verify:

df -h

๐Ÿงฉ AWS + LVM Example

If the AWS disk contains LVM, the process is slightly different.

For example:

AWS EBS
   ↓
/dev/nvme1n1
   ↓
Partition
   ↓
PV
   ↓
VG
   ↓
LV
   ↓
XFS
   ↓
/data

After increasing the EBS volume, you may need to extend each relevant layer:

lsblk

Then, depending on the layout:

growpart /dev/nvme1n1 1

Extend the PV:

pvresize /dev/nvme1n1p1

Extend the LV:

lvextend -r -L +10G /dev/vg01/lv_data

Finally verify:

df -h /data

๐Ÿ’ก The exact commands depend on whether the EBS volume is used as a whole-disk PV or contains a partition.


๐Ÿ”ฅ Troubleshooting Section

❌ Problem 1: Mount Fails

Check filesystem information:

blkid

Check /etc/fstab:

cat /etc/fstab

Check whether the mount point already exists:

mount | grep /data

Test:

mount -a

Check system logs:

journalctl -xe

❌ Problem 2: LV Removal Fails – Device Busy

If you receive:

device is busy

Check which processes are using the mount point:

lsof +D /lvm_ext4

or:

fuser -vm /lvm_ext4

Check whether the filesystem is mounted:

findmnt /lvm_ext4

After stopping the required processes, unmount:

umount /lvm_ext4

Then retry:

lvremove /dev/vg01/lv_ext4

❌ Problem 3: Filesystem Not Expanding

Check the disk layout:

lsblk

Check the LV:

lvdisplay

Check the filesystem:

df -Th

For LVM:

pvs
vgs
lvs

For XFS:

xfs_info /mountpoint

For ext4:

tune2fs -l /dev/vg01/lv_ext4 | grep -i 'block count'

๐Ÿง  Important Linux Storage Commands – Quick Reference

TaskCommand
List diskslsblk
List filesystemslsblk -f
Check disk usagedf -h
Check filesystem typedf -Th
Check inode usagedf -i
Check directory sizedu -sh /path
Check partitionsfdisk -l
Create PVpvcreate /dev/sdb
Display PVspvs
Create VGvgcreate vg01 /dev/sdb
Display VGsvgs
Create LVlvcreate -L 1G -n lv01 vg01
Display LVslvs
Extend VGvgextend vg01 /dev/sdc
Extend PVpvresize /dev/sdb
Extend LVlvextend -L +1G /dev/vg01/lv01
Extend LV + filesystemlvextend -r -L +1G /dev/vg01/lv01
Move data between PVspvmove /dev/sdc /dev/sdd
Remove PV from VGvgreduce vg01 /dev/sdc
Remove PVpvremove /dev/sdc
Grow ext4resize2fs
Grow XFSxfs_growfs

๐ŸŽฏ Key Takeaways

In this Day 5 Linux Administration lesson, we learned:

✅ Linux filesystem types
✅ Disk and filesystem monitoring commands
✅ Traditional partitioning with fdisk
✅ LVM architecture
✅ Creating PV, VG and LV
✅ Creating ext4 and XFS filesystems
✅ Mounting and configuring /etc/fstab
✅ Extending LVM storage
✅ Resizing ext4 and XFS filesystems
✅ Migrating data using pvmove
✅ Decommissioning LVM
✅ Resizing AWS EBS volumes
✅ Troubleshooting common storage problems


๐Ÿš€ Hands-On Practice

To reinforce today's concepts, create a Linux VM and practice the following:

Lab 1 – Traditional Partition

Create /dev/sdb1
      ↓
Create XFS filesystem
      ↓
Mount on /data
      ↓
Configure /etc/fstab

Lab 2 – LVM

/dev/sdc
   ↓
PV
   ↓
VG: vg01
   ↓
LV: lv_data
   ↓
XFS
   ↓
/data

Lab 3 – Extend LVM

Add /dev/sdd and:

Create PV
   ↓
Extend VG
   ↓
Extend LV
   ↓
Grow filesystem
   ↓
Verify with df -h

Lab 4 – Disk Migration

/dev/sdd → /dev/sde

Use:

pvmove

Then remove the old disk from the volume group.



๐Ÿงช Lab – Fill Inodes on a Linux Filesystem

A filesystem can have plenty of free disk space but still become unusable when all available inodes are consumed.

This happens when a very large number of small files are created.

Step 1 – Check Current Inode Usage

df -i /data

Example:

Filesystem           Inodes  IUsed  IFree IUse% Mounted on
/dev/mapper/vg01-lv_data
                    655360  12000 643360    2% /data

Step 2 – Create a Test Directory

⚠️ Perform this only on a lab/test filesystem. Do not run it on /, /var, /home, or a production filesystem.

mkdir -p /data/inode-test

Step 3 – Create a Large Number of Small Files

The following loop creates 100,000 empty files:

for i in $(seq 1 100000); do
    touch /data/inode-test/file_$i
done

Each file consumes an inode even though the files contain no data.

Check inode usage:

df -i /data

You should see the IUsed value increasing.


๐Ÿš€ Faster Method

For a larger test, use:

seq 1 500000 | xargs -n1 -P4 -I{} touch /data/inode-test/file_{}

Check:

df -i /data

The number of files required to exhaust the filesystem depends on how many inodes were created when the filesystem was formatted.


๐Ÿ” Step 4 – Find the Directory Consuming Inodes

Use:

find /data -xdev -type f | cut -d/ -f1-3 | sort | uniq -c | sort -nr | head

A simpler approach is:

find /data/inode-test -type f | wc -l

Example:

100000

๐Ÿงน Step 5 – Remove the Test Files

After completing the lab:

rm -rf /data/inode-test

Then verify:

df -i /data

The inode usage should drop back down.


๐Ÿ’ก Important Difference

Disk Space Exhaustion

Check:

df -h

Example:

Use% = 95%

There may still be free inodes.

Inode Exhaustion

Check:

df -i

Example:

IUse% = 100%

Even if:

df -h

shows:

Use% = 40%

you may still be unable to create new files.

This is because every file and directory requires an inode.


๐Ÿ›  Troubleshooting Inode Exhaustion

If a filesystem reaches 100% inode usage:

1. Identify the affected filesystem

df -i

2. Find directories containing many files

find /data -xdev -type f | cut -d/ -f1-3 | sort | uniq -c | sort -nr | head

3. Count files in a suspected directory

find /data/inode-test -type f | wc -l

4. Identify files that can be safely removed

For example:

find /data/inode-test -type f -mtime +30

5. Remove only after validation

find /data/inode-test -type f -mtime +30 -delete

Then check:

df -i /data

๐ŸŽฏ Interview Question

Q: Can a Linux filesystem be full even when df -h shows free space?

Answer: Yes.

A filesystem can run out of inodes even when it still has free disk blocks.

Check inode availability using:

df -i

A common cause is a huge number of small files, such as:

  • Application temporary files
  • Cache files
  • Session files
  • Mail queues
  • Log files
  • Application-generated metadata

Remember:

Disk Space → df -h
Inodes     → df -i

No comments:

Post a Comment