Understanding Device Mapper Multipath
I wasn’t initially sure the best way to kickoff this blog but after following a particular thread on the Oracle RAC Sig forum (www.oracleracsig.com), I figured an introduction to DM-Multipath may be the way to go since there seems to be a bit of confusion for many on exactly where to begin.
For those who aren’t terribly familiar, DM-Multipath is the native multipathing utility for Linux. Based on performance tests I’ve run through in the past couple years, it definitely seems to hold its own against offerings similar to Veritas VxDMP and EMC PowerPath. In fact, I was able to get better performance than VxDMP and only marginally (about 1%) slower performance than PowerPath on both EMC CLARiiON and Symmetrix VMAX arrays using both Orion and Fio I/O testing utilities. In my book, it’s hard to argue against free, especially when the performance is right up there with the big boys.
Anyway, onto the basics.
The DM-Multipath config file is located at /etc/multipath.conf and is broken up into 5 major sections; defaults, blacklist_exceptions, blacklist, devices, and multipaths. The defaults section is in place to either specify or override any defaults explicitly written in /usr/share/doc/device-mapper-multipath-<version>/multipath.conf.defaults. An example of what you might see in the defaults section of the multipath.conf file might be something similar to this for Symmetrix arrays:
defaults {
user_friendly_names no
no_path_retry 5
}
Or similarly on a CLARiiON:
defaults {
user_friendly_names no
path_selector “queue-length 0”
prio alua
path_checker emc_clariion
no_path_retry 5
hardware_handler “1 alua”
}
As you can see in the above examples, I’ve opted (and recommend) turning off user friendly names. Ultimately what this does is tells DM-Multipath to specify the WWID inside of /dev/mapper (or /dev/mpath) as opposed to automatically naming devices such as mpatha, mpathb, mpathc, etc. At first this may seem counter intuitive, but I’d highly recommend coming up with your own device alias standards since I’ve found the automatic naming to be harder to manage than using a numerical system.
If you happened to notice from my example above, I’ve specified ALUA for the CLARiiONs. ALUA is an acronym for “Asymmetric Logical Unit Access” and without going into too much (or any) real detail, it is an optimization for some arrays to more efficiently handle how LUNs are seen and access data between a host system and the storage processor.
The next section of the multipath.conf file I mentioned above is blacklist_exceptions. This is exactly what it sounds like. In the blacklist section, you can specify specific devices to exclude from multipathing either directly or using a regex match, but you can override a disk if necessary. Probably the coolest thing about the blacklist section is that you can actually use a regex match similar to:
blacklist {
devnode “sda(^[0-9]| |$)”
}
You would generally do something like this in the event that /dev/sda is your root disk holding the OS. A benefit to using a regex pattern as in the example is the device sda, including any partitions listed on that device (sda1, sda2, etc.) are also excluded, however it won’t blacklist any disks such as sdaa, sdab, etc.
Keep in mind; you can spare yourself much of this trouble if you blacklist devices using the WWID such as:
blacklist {
wwid <wwid>
}
The next section is for devices a host is attached to. Here you would put any specific options for an array that you’d like, for example on a Symmetrix configuration, you might see something like:
devices {
device {
vendor “EMC”
product “SYMMETRIX”
features “0”
path_selector “round-robin 0”
path_grouping_policy multibus
failback immediate
rr_weight uniform
no_path_retry 5
rr_min_io 1
}
}
And on a CLARiiON:
devices {
device {
vendor “DGC”
product “.*”
product_blacklist “LUNZ”
features “0”
path_selector “queue-length 0”
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
no_path_retry 5
rr_min_io 1
path_checker emc_clariion
prio alua
}
}
A word of advice; setting the rr_min_io to 1 seems to provide the most visible performance in comparison with any other options across the board. Keep in mind the default for rr_min_io is 1000 up through RHEL 6.3 (if I recall correctly).
The last section I’m going to cover is for the multipath devices themselves:
multipaths {
multipath {
wwid <wwid>
alias multipath_d1
}
multipath {
wwid <wwid>
alias multipath_d2
}
multipath {
wwid <wwid>
alias multipath_d3
}
}
As you can see, I’ve standardized on an alias scheme such as multipath_d[number]. From my perspective with regard to management, this seems to be an effective way to handle disks as I tend to think numbers are simply easier to deal with than letters, especially when it comes to regex in scripts, as well as sorting.
One thing to note, you can override options in the defaults section here. This can particularly come in handy if you’d like to change the ownership of a given disk. This can be done by setting uid and gid options such as:
multipath {
wwid <wwid>
alias multipath_d1
uid 34
gid 34
}
NOTE: Setting the UID and GID is only supported on RHEL versions 5.4 and later.
That’s pretty much all there is to the multipath config. Personally, I would opt to not use the template and write (or script) your own multipath.conf from scratch since it’ll be easier to read and understand in the long run. Also, once you have DM-Multipath running, make sure it starts on boot. Generally if you’re using RHEL, chkconfig is the preferred method:
chkconfig –levels 2345 multipathd on
Hopefully this serves as a decent primer to anyone interested in using DM-Multipath. I’d also highly recommend looking at the Red Hat docs to further understand all the available options in the various sections as it can prove to be an invaluable resource when it comes to tuning.
June 5, 2012 at 1:25 pm
I agree, native Linux multipathing has come a long way. It’s much easier to use then in the old days and allows for a clean initial installation and easy implementation after installation as well, but there’s one small issue, which is not a fault of Linux multipathing. It’s support for 3rd party vendors; for example EMC. I work in a large shop and they have Clariions and VMAX, EMC will not support any disk related issues if we aren’t running powerpath. Sucks right, which includes one of the facts you stated; power path performs slightly better. So, my beef is with EMC. Linux (free) multipathing; you rock my World!
June 5, 2012 at 3:10 pm
That’s interesting. We actually submitted a service request not too long ago with EMC and and they were pretty helpful in our case (we were setting the path_checker as emc_clariion when it should have been directio). In fact, EMC and RH worked together to help solve the issue since both companies had conflicting documentation about the proper configs. You might have some luck pushing EMC a little harder, but as we all know in this industry, your mileage may vary.
Are you running one of the Linux distros from the supported OS list?