{"id":9,"date":"2012-05-29T16:19:16","date_gmt":"2012-05-29T16:19:16","guid":{"rendered":"http:\/\/www.relevantblogging.com\/?p=9"},"modified":"2012-05-29T19:19:53","modified_gmt":"2012-05-29T19:19:53","slug":"understanding-device-mapper-multipath","status":"publish","type":"post","link":"http:\/\/www.relevantblogging.com\/?p=9","title":{"rendered":"Understanding Device Mapper Multipath"},"content":{"rendered":"

I wasn\u2019t initially sure the best way to kickoff this blog but after following a particular thread on the Oracle RAC Sig forum (www.oracleracsig.com<\/a>), I figured an introduction to DM-Multipath may be the way to go since there seems to be a bit of confusion for many on exactly where to begin.<\/p>\n

For those who aren\u2019t terribly familiar, DM-Multipath is the native multipathing utility for Linux. Based on performance tests I\u2019ve run through in the past couple years, it definitely seems to hold its own against offerings similar to Veritas VxDMP and EMC PowerPath. In fact, I was able to get better performance than VxDMP and only marginally (about 1%) slower performance than PowerPath on both EMC CLARiiON and Symmetrix VMAX arrays using both Orion and Fio I\/O testing utilities. In my book, it\u2019s hard to argue against free, especially when the performance is right up there with the big boys.<\/p>\n

Anyway, onto the basics.<\/p>\n

The DM-Multipath config file is located at \/etc\/multipath.conf and is broken up into 5 major sections; defaults, blacklist_exceptions, blacklist, devices, and multipaths. The defaults section is in place to either specify or override any defaults explicitly written in \/usr\/share\/doc\/device-mapper-multipath-<version>\/multipath.conf.defaults. An example of what you might see in the defaults section of the multipath.conf file might be something similar to this for Symmetrix arrays:<\/p>\n

defaults {<\/p>\n

user_friendly_names\u00a0\u00a0 no<\/p>\n

no_path_retry 5<\/p>\n

}<\/p><\/blockquote>\n

Or similarly on a CLARiiON:<\/p>\n

defaults {<\/p>\n

user_friendly_names\u00a0\u00a0 no<\/p>\n

path_selector “queue-length 0”<\/p>\n

prio\u00a0 alua<\/p>\n

path_checker\u00a0 emc_clariion<\/p>\n

no_path_retry 5<\/p>\n

hardware_handler\u00a0\u00a0\u00a0\u00a0\u00a0 “1 alua”<\/p>\n

}<\/p><\/blockquote>\n

As you can see in the above examples, I\u2019ve opted (and recommend) turning off user friendly names. Ultimately what this does is tells DM-Multipath to specify the WWID inside of \/dev\/mapper (or \/dev\/mpath) as opposed to automatically naming devices such as mpatha, mpathb, mpathc, etc. At first this may seem counter intuitive, but I\u2019d highly recommend coming up with your own device alias standards since I\u2019ve found the automatic naming to be harder to manage than using a numerical system.<\/p>\n

If you happened to notice from my example above, I\u2019ve specified ALUA for the CLARiiONs. ALUA is an acronym for \u201cAsymmetric Logical Unit Access\u201d and without going into too much (or any) real detail, it is an optimization for some arrays to more efficiently handle how LUNs are seen and access data between a host system and the storage processor.<\/p>\n

The next section of the multipath.conf file I mentioned above is blacklist_exceptions. This is exactly what it sounds like. In the blacklist section, you can specify specific devices to exclude from multipathing either directly or using a regex match, but you can override a disk if necessary. Probably the coolest thing about the blacklist section is that you can actually use a regex match similar to:<\/p>\n

blacklist {<\/p>\n

devnode “sda(^[0-9]| |$)”<\/p>\n

}<\/p><\/blockquote>\n

You would generally do something like this in the event that \/dev\/sda is your root disk holding the OS. A benefit to using a regex pattern as in the example is the device sda, including any partitions listed on that device (sda1, sda2, etc.) are also excluded, however it won\u2019t blacklist any disks such as sdaa, sdab, etc.<\/p>\n

Keep in mind; you can spare yourself much of this trouble if you blacklist devices using the WWID such as:<\/p>\n

blacklist {<\/p>\n

wwid <wwid><\/p>\n

}<\/p><\/blockquote>\n

The next section is for devices a host is attached to. Here you would put any specific options for an array that you\u2019d like, for example on a Symmetrix configuration, you might see something like:<\/p>\n

devices {<\/p>\n

device {<\/p>\n

vendor\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “EMC”<\/p>\n

product\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “SYMMETRIX”<\/p>\n

features\u00a0\u00a0\u00a0 “0”<\/p>\n

path_selector\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “round-robin 0”<\/p>\n

path_grouping_policy\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 multibus<\/p>\n

failback\u00a0\u00a0\u00a0 immediate<\/p>\n

rr_weight\u00a0\u00a0 uniform<\/p>\n

no_path_retry\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 5<\/p>\n

rr_min_io\u00a0\u00a0 1<\/p>\n

}<\/p>\n

}<\/p><\/blockquote>\n

And on a CLARiiON:<\/p>\n

devices {<\/p>\n

device {<\/p>\n

vendor\u00a0\u00a0\u00a0\u00a0\u00a0 “DGC”<\/p>\n

product\u00a0\u00a0\u00a0\u00a0 “.*”<\/p>\n

product_blacklist\u00a0\u00a0 “LUNZ”<\/p>\n

features\u00a0\u00a0\u00a0 “0\u201d<\/p>\n

path_selector\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “queue-length 0”<\/p>\n

path_grouping_policy\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 group_by_prio<\/p>\n

failback\u00a0\u00a0\u00a0 immediate<\/p>\n

rr_weight\u00a0\u00a0 uniform<\/p>\n

no_path_retry\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 5<\/p>\n

rr_min_io\u00a0\u00a0 1<\/p>\n

path_checker\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 emc_clariion<\/p>\n

prio\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 alua<\/p>\n

}<\/p>\n

}<\/p><\/blockquote>\n

A word of advice; setting the rr_min_io to 1 seems to provide the most visible performance in comparison with any other options across the board. Keep in mind the default for rr_min_io is 1000 up through RHEL 6.3 (if I recall correctly).<\/p>\n

The last section I\u2019m going to cover is for the multipath devices themselves:<\/p>\n

multipaths {<\/p>\n

multipath {<\/p>\n

wwid <wwid><\/p>\n

alias multipath_d1<\/p>\n

}<\/p>\n

multipath {<\/p>\n

wwid <wwid><\/p>\n

alias multipath_d2<\/p>\n

}<\/p>\n

multipath {<\/p>\n

wwid <wwid><\/p>\n

alias multipath_d3<\/p>\n

}<\/p>\n

}<\/p><\/blockquote>\n

As you can see, I\u2019ve standardized on an alias scheme such as multipath_d[number]. From my perspective with regard to management, this seems to be an effective way to handle disks as I tend to think numbers are simply easier to deal with than letters, especially when it comes to regex in scripts, as well as sorting.<\/p>\n

One thing to note, you can override options in the defaults section here. This can particularly come in handy if you\u2019d like to change the ownership of a given disk. This can be done by setting uid and gid options such as:<\/p>\n

multipath {<\/p>\n

wwid <wwid><\/p>\n

alias multipath_d1<\/p>\n

uid 34<\/p>\n

gid 34<\/p>\n

}<\/p><\/blockquote>\n

NOTE: Setting the UID and GID is only supported on RHEL versions 5.4 and later.<\/strong><\/p>\n

That\u2019s pretty much all there is to the multipath config. Personally, I would opt to not use the template and write (or script) your own multipath.conf from scratch since it\u2019ll be easier to read and understand in the long run. Also, once you have DM-Multipath running, make sure it starts on boot. Generally if you\u2019re using RHEL, chkconfig is the preferred method:<\/p>\n

chkconfig –levels 2345 multipathd on<\/p><\/blockquote>\n

Hopefully this serves as a decent primer to anyone interested in using DM-Multipath. I\u2019d also highly recommend looking at the Red Hat docs to further understand all the available options in the various sections as it can prove to be an invaluable resource when it comes to tuning.<\/p>\n","protected":false},"excerpt":{"rendered":"

I wasn\u2019t initially sure the best way to kickoff this blog but after following a particular thread on the Oracle RAC Sig forum (www.oracleracsig.com), I figured an introduction to DM-Multipath may be the way to go since there seems to be a bit of confusion for many on exactly where to begin. For those who […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,1],"tags":[],"class_list":["post-9","post","type-post","status-publish","format-standard","hentry","category-linux","category-uncategorized"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"http:\/\/www.relevantblogging.com\/index.php?rest_route=\/wp\/v2\/posts\/9","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.relevantblogging.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.relevantblogging.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.relevantblogging.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.relevantblogging.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9"}],"version-history":[{"count":10,"href":"http:\/\/www.relevantblogging.com\/index.php?rest_route=\/wp\/v2\/posts\/9\/revisions"}],"predecessor-version":[{"id":21,"href":"http:\/\/www.relevantblogging.com\/index.php?rest_route=\/wp\/v2\/posts\/9\/revisions\/21"}],"wp:attachment":[{"href":"http:\/\/www.relevantblogging.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.relevantblogging.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.relevantblogging.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}