SSD硬盘配置最佳实践

本文原创作者杜霆，京东商城基础平台部MySQL DBA，经作者同意发表于本人博客，如需转载需经本人同意。一、测试背景服务器升级SSD后，速度和性能没有得到应有的提升，原因是没有正确使用SSD，主要是raid、linux、mysql数据库等相关参数没有正确的配置，本文对SSD的相关参数进行了对比性能测试，根据相关资料以及测试结果得出相关结论，了解实用场景以及对应优势。二、测试环境主机：172.20.166.52Cpu：32 * Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz内存：129G操作系统：CentOS release 6.6Sysbench：Sysbench 0.5SSD硬盘：800G*6 raid5主机：172.20.166.23Cpu：32 * Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz内存：129G操作系统：CentOS release 6.6Sysbench：Sysbench 0.5SSD硬盘：800G*6 raid10 /raid0三、测试工具及内容使用Sysbench对相同硬件、软件环境的raid、linux、mysql等相关参数进行io性能测试、mysql性能测试，对比io性能。IO性能测试Sysbench参数：参数名称参数值file-block-size16k混合读写比例1.5（读写比例3:2）file-num20file-total-size200Gmax-time（单位秒）3600MySQL性能测试Sysbench参数：参数名称参数值file-block-size16k混合读写比例1.5（读写比例3:2）oltp-tables-count15oltp-table-size50000000max-time（单位秒）3600四、测试结果1、Linux系统参数测试结果Linux对比测试参数如下：参数类型对比参数值1对比参数值2NUMA开启关闭IO调度算法DeadlineNoopRotational轮转模式01（默认值）read_ahead_kb预读大小0128（默认值）rq_affinity21（默认值）对Linux操作系统测试了5组参数，分别如上图表所示，由测试数据可得出如下结论：1、关闭NUMA，对SSD硬盘性能提高较大；2、IO调度算法采用Deadline和Noop性能差距不大，Noop调度算法性能略高于Deadline；3、rotational、read_ahead_kb、rq_affinity参数的调整对SSD硬盘性能几乎没有影响，建议使用默认配置。2、Raid参数测试结果Raid对比测试参数如下：参数类型对比参数值1对比参数值2Write PolicyWriteBack-WBWriteThrough-WTRead PolicyNormal（关闭预读功能）ReadAdaptiveIO PolicyDirectCachedBGI(Background initialization)Disabled（0%）30%BGI(Background initialization)：开启BGI，创建好Raid阵列后不必等待初始化完成就可以开始使用，比如分区和格式化，但是系统仍然在后台检查磁盘。对Raid配置测试了4组参数，分别如上图表所示，由测试数据可得出如下结论：1、Raid的Write Policy设置为WriteBack，硬盘性能有很大提高，而且性能比较稳定；2、Raid的Read Policy设置为ReadAdaptive，硬盘性能略高于Normal；3、Raid的IO Policy设置为Direct，性能明显高于Cached；4、BGI采用默认配置的30%或者关闭BGI，性能几乎没有变化，建议使用默认配置。3、Raid0的OP测试结果OP配置对比如下：参数类型对比参数值1对比参数值2OP（Over-provisioning：预留空间）NO-OP（硬盘空间全部分区）15%-OP（留下15%空间不分区）对比测试了不设置OP和设置15%的OP，如上图表所示，由测试数据可得出如下结论：1、io测试时线程数小于128时，15%的OP的性能明显高于不设置OP的性能，随着线程数的增加，15%的OP的性能优势逐渐减少，到1024线程时，两者性能很接近。2、mysql测试时15%的OP的性能略高于不设置OP的性能，两者性能很接近。4、MySQL参数测试结果MySQL对比测试参数如下：参数类型对比值1对比值2对比值3对比值4对比值5对比值6对比值7innodb_write_io_threads4816innodb_read_io_threads4816innodb_log_file_size512M1024M1536M2048Minnodb_io_capacity5001000200030004000800012000innodb_max_dirty_pages_pct25407595对MySQL测试了5组参数，分别如上图表所示，由测试数据可得出如下结论：1、innodb_write_io_threads/ innodb_read_io_threads设置为8性能比较理想；2、innodb_log_file_size设置为1536M或者2048M，性能差别不大；3、innodb_io_capacity测试了从500到12000的参数，性能差别不大，设置为3000时出现测试数据的最大值，从测试数据观察，该参数设置在2000-8000范围内性能相对理想；4、innodb_max_dirty_pages_pct设置为75性能较为理想。五、LINUX参数测试数据对比涉及到的linux对比测试参数如下：参数类型对比参数值1对比参数值2NUMA开启关闭IO调度算法DeadlineNoopRotational轮转模式01（默认值）read_ahead_kb预读大小0128（默认值）rq_affinity21（默认值）对Linux操作系统测试了5组参数，分别如上图表所示，由测试数据可得出如下结论：1、关闭NUMA，对SSD硬盘性能提高较大；2、IO调度算法采用Deadline和Noop性能差距不大，Noop调度算法性能略高于Deadline；3、rotational、read_ahead_kb、rq_affinity参数的调整对SSD硬盘性能几乎没有影响。1、NUMA测试数据对比1、IO性能测试threads关numa-吞吐量开numa-吞吐量关numa-iops开numa-iops关numa-load开numa-load关numa-iowait开numa-iowait8378.54266.4124226.7917050.516186.5616458.75360.7529359.8523087.83133111032553.27450.8235409.1428852.35351212064595.53523.4938113.7833503.585854238128613.85539.7639286.6134544.51381406365256605.66534.0838762.2934181.12602606572512603.14527.0438600.8533730.652052067732、MySQL性能测试threads关numa-tps开numa-tps关numa-qps开numa-qps关numa-load开numa-load关numa-iowait开numa-iowait82382.11363.0424995.5824534.76745162727.462608.0549094.31469457736324324.64053.6277842.7372965.156.534.5645317.554963.4195715.9689341.477.51.72.51284758.364553.2285650.528195810101.822564494.634364.4980903.3178560.920271.61.85124467.944411.6280422.8379409.135351.71.82、IO调度测试数据对比1、IO性能测试threadsdeadline-吞吐量noop-吞吐量deadline-iopsnoop-iopsdeadline-loadnoop-loaddeadline -iowaitnoop-iowait8255.89352.816377.0422581.81819.516.16.7116352.47465.1422557.9429768.73334.218.711.5132447.36558.6828631.1335755.75556.091822.5664536.64599.4834345.1938366.58683.543643.86128581.75609.0737231.8738980.8145138.376059.07256575.5626.536831.8740095.8265266.367662.77512550.95620.7735260.5339729.4520520.757864.542、MySQL性能测试threadsdeadline-tpsnoop-tpsdeadline-qpsnoop-qpsdeadline-loadnoop-loaddeadline-iowaitnoop-iowait81555.471173.7727998.4321127.9510.8921.57162723.712527.7349026.7645499.2611.752.52.35323989.453998.6771810.1671976.1512.782.52.96644743.534888.7985363.8887998.37.3910.572.682.391284587.234821.0882570.1986779.58.3713.531.872.152564446.684544.6580040.2381803.712.3917.131.671.615124340.154789.8278122.786216.835.7224.921.422.173、rotational测试数据对比threads0-吞吐量1-吞吐量0-iops1-iops8406.1406.1325990.4925992.0316462.09461.7229573.5229550.2832477.09477.6630533.5130569.9664484.68484.8831019.831032.59128487.97488.231229.8631244.88256488.26488.2531248.8631248.14512488.09488.2131237.4731245.541024487.96487.9631229.3931229.694、read_ahead_kb测试数据对比threads128-吞吐量0-吞吐量128-iops0-iops8406406.6125988.8126023.2316462.25462.529583.8229600.232477.61477.8930566.830584.8364484.92485.0531035.131043.36128488.22488.331245.9931251.39256488.27488.3131249.4731251.88512488.25488.1931247.8131244.061024488.18487.9831243.8231230.885、rq_affinity测试数据对比threads1-吞吐量2-吞吐量1-iops2-iops8406.36406.3526006.8926006.1216462.14462.142957729577.0232477.62477.5130567.630560.6564484.92485.0331034.8231041.8128488.21488.1831245.6131243.84256488.36488.3731254.9131255.43512488.2488.2931244.9431250.271024488488.0931232.0131237.56六、RAID参数测试数据对比涉及到的raid对比测试参数如下：参数类型对比参数值1对比参数值2Write PolicyWriteBack-WBWriteThrough-WTRead PolicyNormalReadAdaptiveIO PolicyDirectCachedBGI(Background initialization)Disabled（0%）30%BGI(Background initialization)：开启BGI，创建好Raid阵列后不必等待初始化完成就可以开始使用，比如分区和格式化，但是系统仍然在后台检查磁盘。对Raid配置测试了4组参数，分别如上图表所示，由测试数据可得出如下结论：1、Raid的Write Policy使用Writeback，硬盘性能有很大提高，而且性能比较稳定；2、Raid的Read Policy使用ReadAdaptive，硬盘性能略高于Normal；3、Raid的IO Policy使用Direct，性能明显高于Cached；4、BGI采用默认配置30%或者关闭BGI，性能几乎没有变化。1、Write Policy测试数据对比1、IO性能测试threadsWB-吞吐量WT-吞吐量WB-iopsWT-iops8352.8298.8922581.7819128.8216465.14397.5929768.7225445.9632558.68441.7435755.6728271.664599.48460.0938366.4729446.06128609.07459.9338980.829435.51256626.5465.3740095.8329783.48512620.77112.9239729.387227.122、MySQL性能测试threadsWB-tpsWT-tpsWB-qpsWT-qpsWB-loadWTB-loadWB-iowaitWT-iowait81173.771078.6521127.8719415.710.897.151.573.18162527.731778.845499.1732018.311.759.172.354.95323998.672632.4771976.1347384.412.7810.272.966.32644888.793974.1187998.277153410.5711.232.394.981284821.083498.1986779.4662967.313.5311.692.153.572564544.653495.2781803.6962914.817.1322.251.612.865124789.823358.7886216.756045824.9220.852.171.762、Read Policy测试数据对比1、IO性能测试threadsNormal-吞吐量ReadAdaptive-吞吐量Normal-iopsReadAdaptive-iops8312.87380.7220023.4624366.2516440.28479.3928177.6630680.932539.43566.2634523.6936240.6764591.11611.2837831.3439122.05128604.66619.2938698.539634.77256602.61621.3138566.8239763.55512606.65621.0938825.5539749.611024604.67619.2638698.9839632.652、MySQL性能测试threadsNormal-吞吐量ReadAdaptive-吞吐量Normal-iopsReadAdaptive-iops8312.87380.7220023.4624366.2516440.28479.3928177.6630680.932539.43566.2634523.6936240.6764591.11611.2837831.3439122.05128604.66619.2938698.539634.77256602.61621.3138566.8239763.55512606.65621.0938825.5539749.611024604.67619.2638698.9839632.653、IO Policy测试数据对比threadsDirect-吞吐量Cached-吞吐量Direct-iopsCached-iops8408.54375.526146.5724032.0316462.81411.9729619.7526366.3432487.86428.2731222.7527409.3364491.41432.0631450.3527651.82128494.34435.3131637.5627859.79256494.42435.9631642.9427901.3512494.1843631627.7327903.831024493.97436.1131614.1927911.184、BGI测试数据对比threads30%-吞吐量disabled-吞吐量30%-iopsdisabled-iops8413.43411.9526459.4526364.516469.9469.5930073.8630054.0132494.27493.531633.5231584.1564496.97496.431806.2331769.53128499.9499.4131993.4731962.09256499.97499.5931997.931973.49512499.69499.1331980.1431944.61024499.31498.831955.7831923.29七、Raid0的SSD OP测试数据对比OP是Over-provisioning（预留空间）的简称，是指用户不可操作的容量，就是留出一部分容量不使用不分区，OP预留的空间一般被用于优化操作。参数类型对比参数值1对比参数值2OP（Over-provisioning：预留空间）NO-OP（硬盘空间全部分区）15%-OP（留下15%空间不分区）对比测试了不设置OP和设置15%的OP，分别如上图表所示，由测试数据可得出如下结论：1、io测试时线程数小于128时，15%的OP的性能明显高于不设置OP的性能，随着线程数的增加，15%的OP的性能优势逐渐减少，到1024线程时，两者性能很接近。2、mysql测试时15%的OP的性能略高于不设置OP的性能，两者性能很接近。1、IO性能测试threadsNO_OP-吞吐量15%OP-吞吐量NO_OP-iops15%OP-iops8355.44400.8522748.0725654.7116496.2521.7431756.5433391.232601.71612.0938509.3239173.764635.39639.7940665.140946.7128641.7646.5641069.1141379.56256640.71644.6541005.1941257.91512640.51642.4440992.8141116.141024640.96641.7841021.3341074.222、MySQL性能测试threadsNO_OP-tps15%OP-tpsNO_OP-qps15%OP-qps81721.791745.6930992.2231422.41163070.812999.3655274.5153988.55325024.175087.0390435.0491706.48645769.85800.62103856.32104811.251286029.356047.59108528.26108856.712566163.546172.09110943.75111097.595125825.35870.96104855.43105377.3110245263.825302.4394748.7995443.74八、MySQL参数测试数据对比涉及到的MySQL对比测试参数如下：参数类型对比值1对比值2对比值3对比值4对比值5对比值6对比值7innodb_write_io_threads4816innodb_read_io_threads4816innodb_log_file_size512M1024M1536M2048Minnodb_io_capacity5001000200030004000800012000innodb_max_dirty_pages_pct25407595对MySQL测试了5组参数，分别如上图表所示，由测试数据可得出如下结论：1、innodb_write_io_threads/ innodb_read_io_threads设置为8性能比较理想；2、innodb_log_file_size设置为1536M或者2048M性能较理想，两者性能差别不大；3、innodb_io_capacity测试了从500到12000的参数，性能差别不大，设置为3000时出现测试数据的最大值，从测试数据观察，该参数设置在2000-8000范围内性能相对理解；4、innodb_max_dirty_pages_pct设置为75性能较为理想。1、innodb_write/read_io_threads测试数据threads4threads-tps8threads-tps16threads-tps4threads-qps8threads-qps16threads-qps81173.771492.431419.1121127.926863.7625543.96162527.732786.582503.4545499.250158.4345062.06323998.674089.444047.8871976.173609.9272861.79644888.795007.994906.4687998.390143.7488316.281284821.084910.924968.9586779.588796.5889441.112564544.654569.684576.2781803.782254.2382372.855124457.774471.444430.3280239.980485.8879745.692、innodb_log_file_size测试数据threads512M-tps1024M-tps1536M-tps2048M-tps512M-qps1024M-qps1536M-qps2048M-qps81492.431673.891624.91610.5326863.7630130.0529248.228989.6162786.582842.862883.262954.4150158.4351171.4551898.6553179.3324089.444400.774498.564537.5673609.9279213.8480974.1581676645007.995017.135095.415189.4990143.7490308.3691717.4293410.91284710.925286.195309.155350.9284796.5895151.495564.7896316.62564569.685223.625460.575465.2782254.2394025.1198290.2398374.95124471.445101.565403.775408.4580485.8891828.0497267.94973523、innodb_io_capacity测试数据threads500-tps1000-tps2000-tps3000-tps4000-tps6000-tps8000-tps12000-tps81658.261630.921567.111616.331587.681605.741616.981618.39162889.112887.322925.12946.262927.382946.932933.973038.03324485.994491.164560.834564.734536.174563.984563.864687.67645115.525083.295158.465158.745183.225209.635136.775341.221285323.145307.325324.145356.275350.895470.4954395461.052565477.865492.015505.185513.25473.415492.545494.195465.695125426.185414.675400.055404.585277.265363.15374.895364.9310244911.334884.974901.064924.874931.674924.814760.454936.01threads500-qps1000-qps2000-qps3000-qps4000-qps6000-qps8000-qps12000-qps829848.729356.628208.0329093.928578.3328903.2529105.55291311652003.951971.852651.8553032.652692.8753044.7252811.4354684.63280747.880840.982094.8982165.181651.1382151.5882149.5384378.16492079.491499.392852.392857.493298.0493773.4292461.839614212895816.595531.795834.4996412.996316.1198468.7697902.0198298.825698601.698856.295505.1899237.798521.4498865.6798895.3598382.451297671.297464.197200.9497282.594990.6696535.8596747.9996568.8102488403.987929.488219.1388647.688770.0788646.5185688.0588848.14、innodb_max_dirty_pages_pct测试数据threads25-tps40-tps75-tps95-tps81592.351635.881616.331604.2162926.662914.712946.262963.03324621.184669.544564.734640645183.435216.985158.745221.391285374.465418.15356.275346.032565467.125465.595513.25436.565125325.575394.825404.585347.1910244746.074899.434924.874769.11九、相同硬盘数不同raid测试数据对比使用6块ssd硬盘，分别配置了raid5、raid10、raid0、raid0-15%OP，进行了io混合随机读写性能测试。Sysbench测试参数：参数名称参数值file-block-size16k混合读写比例1.5（读写比例3:2）oltp-tables-count15oltp-table-size500000001、不同Raid对比分析结果1、硬盘利用率Raid0的硬盘空间使用率是100%，如果预留15%做OP，空间使用率只有全部硬盘大小的75%；Raid5的空间使用率是（n-1）/n（n是磁盘数量，磁盘数量越多，空间使用率越高）；Raid10的硬盘使用率只有50%；硬盘使用率顺序是Raid0 > raid0-15%OP > Raid5 > Raid102、读写性能raid0、raid0-15%OP的读写性能明显高于raid5、raid10，raid0-15%OP的性能略高于raid0。raid0-15%OP > raid0 > raid10 > raid53、稳定安全性Raid10是最安全稳定的，Raid5提供了一块盘的奇偶检验保证安全，Raid0没有安全保护措施。另外当Raid5一块盘出现故障时候，其性能明显下降。2、不同Raid测试数据1、io混合随机读写性能数据threadsraid5-吞吐量raid10-吞吐量raid0-NO_OP-吞吐量raid0-15%OP-吞吐量raid5-iopsraid10-iopsraid0-NO_OP-iopsraid0-15%OP-iops8486.81378.54355.44400.8531155.924226.7922748.0725654.7116531.31458.75496.2521.7434003.529359.8531756.5433391.232567.32553.27601.71612.0936308.535409.1438509.3239173.764588.14595.53635.39639.7937640.938113.7840665.140946.7128584.11613.85641.7646.5637383.239286.6141069.1141379.56256586.53605.66640.71644.6537537.838762.2941005.1941257.91512584.89603.14640.51642.4437432.738600.8540992.8141116.142、一块硬盘故障，raid10和raid5性能变化raid5MB_read/sMB_wrtn/s吞吐量raid10MB_read/sMB_wrtn/s吞吐量306.98244.34551.32342.13239.01581.14307.48240.34547.82420.18174.85595.03311.98243.11555.09210.94319.88530.82306.69242.92549.61396.29198.82595.11309.18241.09550.27455.8144.79600.59317.05243.94560.99321.38248.11569.49328.3247.53575.83269.85288.21558.06319.46257.87577.33478.38136.64615.02311.09253.59564.68460.5134.47594.97315.82256.95572.77377.16210.37587.53158.27156.9315.17368.72215.12583.84160.46166.41326.87291.59267.16558.75159.12163.47322.59414.14192.47606.61156.2162.23318.43459.39151.82611.21160.99163.99324.98215.77325.02540.79157.78161.77319.55391.12198.15589.27160.99163.99324.98365.72221.12586.84158.06160.86318.92250.39297.45547.84161.4162.81324.21457.67153.27610.94159.08161.38320.46360.43226.9587.33十、SSD推荐参数配置SSD对应linux、raid、mysql推荐参数以及推荐参数设置方法。1、Linux推荐参数设置1、关闭numa2、io调度算法设置为noop3、Rotational保持默认值14、read_ahead_kb保持默认值1285、rq_affinity保持默认值12、Linux推荐参数设置方法1、关闭numa进入System BIOS设置，选择Memory设置，选择Node Interteaving为Enable进行关闭numa，如下图所示。执行命令“numactl --hardware”进行检查，输出结果available: 1 nodes (0)时表示关闭成功，如下图。2、修改io调度算法noop执行命令：echo noop > /sys/block/sda/queue/scheduler检查命令：cat /sys/block/sda/queue/scheduler3、Rotational值的调整执行命令：echo 0 > /sys/block/sda/queue/rotational查看命令：cat /sys/block/sda/queue/rotational4、read_ahead_kb值的调整执行命令：echo 0 > /sys/block/sda/queue/read_ahead_kb查看命令：cat /sys/block/sda/queue/read_ahead_kb5、rq_affinity值的调整执行命令：echo 0 > /sys/block/sda/queue/rq_affinity查看命令：cat /sys/block/sda/queue/rq_affinity3、Raid推荐参数设置1、Write Policy设置为WriteBack（WB）2、Read Policy设置为ReadAdaptive3、IO Policy设置为Direct4、BGI(Background initialization)保持默认设置30%4、Raid推荐参数设置方法使用MegaCli命令可以修改Raid参数配置，该命名是官方提供的针对MegaRAID卡的设置命令，如果使用不同品牌Raid卡，可以参照官方提供的命令进行设置。MegaCli查看Raid参数命令：查看所有Raid：MegaCli -LDGetProp -Cache -LALL –aALL查看其中一个Raid：MegaCli -LDGetProp -Cache -L1 -a0MegaCli修改Raid参数命令：MegaCli -LDSetProp WT|WB|NORA|RA|ADRA -L1 -a0orMegaCli -LDSetProp -Cached|-Direct -L1 -a01、Write Policy设置为WriteBack命令MegaCli -LDSetProp WB -L1 -a02、Read Policy设置为ReadAdaptive命令MegaCli -LDSetProp ADRA -L1 -a03、IO Policy设置为Direct命令MegaCli -LDSetProp -Direct -L1 -a04、BGI调整方法进入Device Settings，选择Interated RAID Controller，选择Controller Managent，选择Change Controller Properties，可以修改BGI参数，如下图。5、SSD的OP推荐设置推荐保留15%的硬盘空间不进行分区，这部分空间即是OP的保留空间。设置方法：分区时候保留15%的硬盘空间不进行分区。4、MySQL推荐参数设置1、innodb_write_io_threads/ innodb_read_io_threads推荐设置为82、innodb_log_file_size推荐设置为2048M3、innodb_io_capacity推荐设置为30004、innodb_max_dirty_pages_pct设置为75欢迎关注京东商城基础平台部公众号