chef で Redis の master / slave 構成
構成
OS | Chef Version | redis verion |
---|---|---|
Centos6.5 | 11.16.0 | 2.8.17 |
EC2で作ります。 ElastiCache で Multi-AZ ができたので、作るだけアレだったかもしれない…。
前提
- redis はソースから make install
- master / slave として構成するのは自前の(しょぼい)スクリプト
- master / slave は相互生存監視するので master / slave / peer と 3 つのIPアドレスを判断しなければならないというアレな仕様
- フェイルオーバーは VPC Route Table を replace route で実現
概要
- node で environment を指定
- environment で role を指定
- role には master / slave を用意
- role で cookbooks/roles を指定
- アンチパターンに従順になってみました
- cookbooks/roles で cookbook/redis を include_recipe する
node
nodes/10.0.0.1.json
{ "environment": "development", "run_list": [ "role[redis_master]" ] }
nodes/10.0.0.2.json
{ "environment": "development", "run_list": [ "role[redis_slave]" ] }
environment
environments/development.rb
name "development" description "development environments" default_attributes( :redis => { :master_ip_address => "10.0.0.1", :slave_ip_address => "10.0.1.1", :port => "6379", :maxmemory => "200000000", :vip => "10.1.1.1/32" } )
role
roles/redis_master.rb
name "redis_master" description "redis master's role" run_list "recipe[roles::redis]"
roles/redis_slave.rb
name "redis_slave" description "redis slave's role" run_list "recipe[roles::redis]"
ここに master / slave を用意したのは、
ピアのIPアドレスが master_ip_address なのか slave_ip_address なのかを判断するためというだけのもの。
この判断は後述の cookbooks/attributes/default.rb が行う
cookbookとしてのrole
cookbooks/roles/recipe/redis.rb
include_recipe "redis"
cookbooks/roles/metadata.rb
name 'roles' maintainer 'YOUR_COMPANY_NAME' maintainer_email 'YOUR_EMAIL' license 'All rights reserved' description 'Installs/Configures roles' long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) version '0.1.0' depends redis
include_recipe する cookbook を depends に列挙しないと、その cookbook 内の attribute が使えない。
cookbooks/attributes/default.rb
if "#{node[:roles]}" == '["redis_master"]' default['redis']['peer_ip_address'] = "#{node['redis']['slave_ip_address']}" default['redis']['slaveof'] = 'no one' elsif "#{node[:roles]}" == '["redis_slave"]' default['redis']['peer_ip_address'] = "#{node['redis']['master_ip_address']}" default['redis']['slaveof'] = "#{node['redis']['master_ip_address']}" end
if "#{node[:roles]}" == 'redis_master' がダメで、
if "#{node[:roles]}" == '["redis_master"]' なら通った。
redisのcookbook
cookbooks/redis/default.rb
# init bash "sysctl" do user "root" code <<-EOC echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf sysctl -p EOC not_if 'grep "vm.overcommit_memory" /etc/sysctl.conf' end bash "binary_path" do user "root" code <<-EOC echo "export PATH=$PATH:#{node['redis']['binary_path']}" >> /etc/bashrc source /etc/bashrc EOC not_if 'grep "#{node['redis']['binary_path']}" /etc/bashrc' end # create directories %w{ "#{node['redis']['log_dir']}" "#{node['redis']['work_dir']}" "#{node['redis']['conf_dir']}" "#{node['redis']['dump_dir']}" }.each do |dir| directory dir do action :create end end # get latest redis source remote_file node['redis']['work_dir'] + node['redis']['source_file_name'] do source node['redis']['source_url_path'] + node['redis']['source_file_name'] not_if "File.exists? #{node['redis']['server_install_path']}" end # install redis bash "install_redis" do user "root" cwd node['redis']['work_dir'] code <<-EOC tar -zxf #{node['redis']['source_file_name']} cd #{::File.basename(node['redis']['source_file_name'], '.tar.gz')} make make install cp src/redis-server src/redis-cli #{node['redis']['binary_path']} EOC not_if "File.exists? #{node['redis']['server_install_path']}" end # config file template "redis.conf" do source "redis.conf.erb" path "#{node['redis']['redis_conf']}" owner "root" group "root" mode "0644" end # init script template "redis_server'" do source "redis_server.erb" path "#{node['redis']['init_script']}" owner "root" group "root" mode "0755" end # start redis service "redis_server-#{node['redis']['redis_port']}" do action :start end # master / slave bash "slaveof" do user "root" code <<-EOC #{node['redis']['cli_install_path']} slaveof #{node['redis']['slaveof']} EOC end # check script template "redis_check" do source "redis_check.sh.erb" path "#{node['redis']['check_script']}" owner "root" group "root" mode "755" end
cookbooks/redis/attributes/default.rb
# directories default['redis']['work_dir'] = '/usr/local/src/' default['redis']['dump_dir'] = '/var/lib/redis/' default['redis']['log_dir'] = '/var/log/redis/' default['redis']['conf_dir'] = '/etc/redis/' # Source Code URL default['redis']['source_ver_num'] = '2.8.17' default['redis']['source_url_path'] = 'http://download.redis.io/releases/' default['redis']['source_file_name'] = "redis-#{node['redis']['source_ver_num']}.tar.gz" # Redis Settings default['redis']['binary_path'] = '/usr/local/bin' default['redis']['server_install_path'] = '/usr/local/bin/redis-server' default['redis']['cli_install_path'] = '/usr/local/bin/redis-cli' default['redis']['redis_port'] = '6379' default['redis']['logfile'] = "#{node['redis']['log_dir']}redis-#{node['redis']['redis_port']}.log" default['redis']['redis_conf'] = "/etc/redis/redis-#{node['redis']['redis_port']}.conf" default['redis']['init_script'] = "/etc/init.d/redis_server-#{node['redis']['redis_port']}" # redis.conf default['redis']['maxmemory'] = '任意の値' # check script default['redis']['check_script'] = '/usr/local/sbin/redis_check.sh' default['redis']['check_log'] = "#{node['redis']['log_dir']}redis_check.log"
cookbooks/redis/templates/redis_server.erb
redis 2.4 系の起動スクリプトを流用しています。 変更点は2点
- redis のキャッシュ永続化をしないので、起動前に dump.rdb を削除
- master として動作させるために起動後に slaveof no one
#!/bin/sh # # redis - this script starts and stops the redis-server daemon # # chkconfig: - 85 15 # description: Redis is a persistent key-value database # processname: redis-server # config: /etc/redis/redis.conf # config: /etc/sysconfig/redis # pidfile: /var/run/redis.pid # Source function library. . /etc/rc.d/init.d/functions # Source networking configuration. . /etc/sysconfig/network # Check that networking is up. [ "$NETWORKING" = "no" ] && exit 0 redis="<%= node['redis']['server_install_path'] %>" redis_cli="<%= node['redis']['cli_install_path'] %>" prog=$(basename $redis) REDIS_CONF_FILE="<%= node['redis']['redis_conf'] %>" DUMP="<%= node['redis']['dump_dir'] %>dump.rdb" [ -f /etc/sysconfig/redis ] && . /etc/sysconfig/redis lockfile=/var/lock/subsys/redis start() { [ -x $redis ] || exit 5 [ -f $REDIS_CONF_FILE ] || exit 6 echo -n $";Starting $prog: " # remove dump file before starting. rm -f ${DUMP} daemon $redis $REDIS_CONF_FILE retval=$? [ $retval -eq 0 ] && touch $lockfile status $prog sleep 3 # start as master $redis_cli slaveof no one return $retval } stop() { echo -n $";Stopping $prog: " killproc $prog -QUIT retval=$? status $prog [ $retval -eq 0 ] && rm -f $lockfile return $retval } restart() { stop start } reload() { echo -n $";Reloading $prog: " killproc $redis -HUP RETVAL=$? echo } force_reload() { restart } rh_status() { status $prog } rh_status_q() { rh_status >/dev/null 2>&1 } case "$1" in start) rh_status_q && exit 0 $1 ;; stop) rh_status_q || exit 0 $1 ;; restart|configtest) $1 ;; reload) rh_status_q || exit 7 $1 ;; force-reload) force_reload ;; status) rh_status ;; condrestart|try-restart) rh_status_q || exit 0 ;; *) echo $";Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}" exit 2 esac
cookbooks/redis/templates/redis.conf.erb
- master として動作(master / slave で設定を同じにしたいため)
- 永続化はしない
daemonize yes pidfile /var/run/redis.pid port <%= node['redis']['redis_port'] %> timeout 0 loglevel notice logfile <%= node['redis']['logfile'] %> dir <%= node['redis']['dump_dir'] %> save "" slaveof no one slave-serve-stale-data yes slave-read-only yes slave-priority 100 maxmemory <%= node['redis']['maxmemory'] %> maxmemory-policy volatile-lru appendonly no
cookbooks/redis/templates/redis_check.sh.erb
redis の相互監視スクリプト
- peer の down を検知したら route table を変更してフェイルオーバーさせる
#!/bin/sh # redis-check.sh # monitoring peer redis. using redis-cli ping. # when ping failed ${RETRY_COUNT} , start failover. ### define ### # notice.this configration is different from peer. PEER="<%= node['redis']['peer_ip_address'] %>" instance_id=`curl -s 169.254.169.254/latest/meta-data/instance-id` PORT="<%= node['redis']['redis_port'] %>" VIP="<%= node['redis']['vip'] %>" route_table_names=("dmz" "lan" "mgmt" "db") # commands AWS="/usr/bin/aws ec2 --region ap-northeast-1" ENI_FILTER="--filters Name=attachment.instance-id,Values=${instance_id} Name=description,Values="Primary network interface"" RTB_FILTER="--filters Name=tag-value,Values=${route_table_name}" QUERY="--output text --query" REDIS="/usr/sbin/redis-cli" RUNFOR="/usr/sbin/runfor" LOGGER="logger -f <%= node['redis']['check_log'] %> -t $0" ### define end ### ### functions ### monitoring_peer(){ TRIES="0" RETRY_COUNT="3" while true do PING_RESULT=`${RUNFOR} 2 ${REDIS} -h ${PEER} -p ${PORT} ping` if [ "${PING_RESULT}" = "PONG" ]; then ${LOGGER} "INFO. peer alive. do nothing." exit 0 else TRIES=`expr ${TRIES} + 1` if [ ${TRIES} -lt ${RETRY_COUNT} ]; then ${LOGGER} "WARNING. redis-cli ping failed count => ${TRIES}." sleep 10 else ${LOGGER} "ERROR. no responce from ${PEER}." break fi fi done } promote_master(){ CHECK_ROLE=`${REDIS} info | grep role | cut -b6` if [ "${CHECK_ROLE}" = "m" ]; then ${LOGGER} "peer node is Dead." exit 0 elif [ "${CHECK_ROLE}" = "s" ] ; then ${LOGGER} "promote to master start." ${REDIS} slaveof no one else ${LOGGER} "STATUS UNKNOWN." exit 1 fi CHECK_ROLE=`${REDIS} info | grep role | cut -b6` if [ "${CHECK_ROLE}" = "m" ] ; then ${LOGGER} "promote complete. start replace route." else ${LOGGER} "ERROR. failed to promote master." exit 1 fi } replace_route(){ ### define eni eni_id=`${AWS} describe-network-interfaces ${ENI_FILTER} ${QUERY} '.NetworkInterfaces[].NetworkInterfaceId'` ### get route_table id then start replace route. for route_table_name in ${route_table_names[@]} do # get route table id route_table_id=`${AWS} describe-route-tables ${RTB_FILTER} ${QUERY} '.RouteTables[].RouteTableId'` # replace route ${AWS} replace-route --route-table-id ${route_table_id} --destination-cidr-block ${VIP} --network-interface-id ${eni_id} if [ $? = 0 ] ; then ${LOGGER} "replace vip for ${route_table_name} complete." exit 0 else ${LOGGER} "ERROR. failed replace vip for ${route_table_name}." exit 1 fi done } monitoring_peer promote_master replace_route
恥ずかしながら晒してみました。以上です。