Multipath: Configuring Multiple Paths for External Disk Access
Introduction
We will discuss two topics here:
- Device mappers
- Multipathing
We need to understand how device mappers work before tackling multipathing, which is why there will be an explanation of both in this documentation.
In the Linux kernel, the device mapper serves as a generic framework for mapping one block device (“mapping” the device) to another. It is the foundation for LVM2 and EVMS, software RAIDs, or disk encryption; and offers additional features such as file system snapshots. The device mapper works by processing data transferred to it by a virtual block device (provided by itself), and passing the resulting data to another block device.
Multipathing allows you to have multiple paths to access the same data. This aims to increase data access capabilities if the storage equipment allows it (active/active) and to ensure redundancy in case of equipment failure, such as a controller. Here is what a multipath architecture looks like:
It also works very well with a single SAN.
Device Mapper
Device mappers are rarely used manually. They are generally used by higher layers such as LVM. Nevertheless, we’ll see how to use them.
To add a partition in device mapper:
|
|
- device: name of the device to create
- map_table: a file that must contain mapping rules, for example:
|
|
If I want to create a device mapper, I can also do it in a single command line without a file:
|
|
There are several types of mapping targets:
- linear: continuous allocation
- stripped: segmented allocation between all devices
- error: to generate errors (ideal for development and testing)
- snapshot - copy-on-write device
- snapshot-origin: mapping to an original volume
- zero - sparse block devices (equivalent to /dev/null)
- multipath: multiple routes for connection to a device
To see all available device mappers:
|
|
To delete a device mapper:
|
|
To list all device mappers as a tree:
|
|
Multipathing
Installation
Multipath is not installed by default, so we need to install a package:
|
|
Then we will load the modules and make the service persistent:
|
|
Configuration
If you don’t have a configuration file, get one from the documentation:
|
|
Multipath uses a group notion ranging from 0 to 1024 (from highest to lowest priority). Only one group is active at a time. A group can contain multiple paths.
Let’s proceed with configuring our multipathing service (I’m only showing the essential lines):
|
|
I strongly recommend checking the man pages for the options above.
We can now start our service:
|
|
Usage
Here is the order and tools to use for disk detection, this section is very important:
- Devices: partprobe /dev/
(e.g., sda) - Device-mappers: kpartx -a /dev/
(e.g., dm-1) - Multipath: partprobe /dev/mapper/
(e.g., mpath0)
To see active paths:
|
|
To create partitions on a multipathed device mapper, you must do it on the underlying disk (e.g., /dev/sda) and not on the multipathed device mapper! So the procedure is to create your partition with fdisk for example, then detect your new partition:
|
|
FAQ
I still don’t see my new LUNs, how do I refresh them?
It is possible that creating new LUNs/partitions requires a new scan to detect them. We will need this package:
|
|
Then let’s launch the scan:
|
|
Or alternatively, we can do it directly with /proc:
- If it’s on a SCSI type platform:
|
|
- If it’s for a Fiber Channel array:
|
|
I can’t see my new partition correctly, how should I proceed?
Here’s how to proceed when encountering an issue while creating a partition on a multipath. Let’s take this example:
- I don’t see mpath0p2 on one machine, whereas I can see it on other machines:
|
|
- I verify that I can see my partition on both paths (sda2 and sdb2):
|
|
Make sure there are no blacklist options in the configuration preventing you from properly seeing the devices. To do this, we’ll comment out all blacklist parts:
|
|
- We see that /dev/dm-1p1 is present, but not the 2nd one (at least there’s an error). I check again that there is no presence of my dm-2:
|
|
|
|
- We see that “253, 1” corresponds to /dev/dm-1. We’ll do a kpartx and partprobe on both to refresh the paths:
|
|
Even if you get errors like:
device-mapper: create ioctl failed: Device or resource busy
It’s not serious, it allows it to refresh the list of device mappers.
- And now it works:
|
|
Resources
Last updated 17 Apr 2012, 09:46 CEST.