set setting reset

インフラ関連の小ネタと備忘録

Chef で Redis の Master / Slave 構成

chef で Redis の master / slave 構成

構成

OS Chef Version redis verion
Centos6.5 11.16.0 2.8.17

EC2で作ります。 ElastiCache で Multi-AZ ができたので、作るだけアレだったかもしれない…。

前提

  • redis はソースから make install
  • master / slave として構成するのは自前の(しょぼい)スクリプト
    • redis-cli ping して master が応答がなければフェイルオーバー開始
    • redis-cli ping して slave が応答がなければメール送信だけする
  • master / slave は相互生存監視するので master / slave / peer と 3 つのIPアドレスを判断しなければならないというアレな仕様
  • フェイルオーバーは VPC Route Table を replace route で実現

概要

  • node で environment を指定
  • environment で role を指定
    • role には master / slave を用意
  • role で cookbooks/roles を指定
  • cookbooks/roles で cookbook/redis を include_recipe する

node

nodes/10.0.0.1.json

{
  "environment": "development",
 "run_list": [
    "role[redis_master]"
  ]
}

nodes/10.0.0.2.json

{
  "environment": "development",
 "run_list": [
    "role[redis_slave]"
  ]
}

environment

environments/development.rb

name "development"
description "development environments"
default_attributes(
  :redis => {
    :master_ip_address => "10.0.0.1",
    :slave_ip_address => "10.0.1.1",
    :port => "6379",
    :maxmemory => "200000000",
    :vip => "10.1.1.1/32"
  }
)

role

roles/redis_master.rb

name "redis_master"
description "redis master's role"
run_list "recipe[roles::redis]"

roles/redis_slave.rb

name "redis_slave"
description "redis slave's role"
run_list "recipe[roles::redis]"

ここに master / slave を用意したのは、
ピアのIPアドレスが master_ip_address なのか slave_ip_address なのかを判断するためというだけのもの。

この判断は後述の cookbooks/attributes/default.rb が行う

cookbookとしてのrole

cookbooks/roles/recipe/redis.rb

include_recipe "redis"

cookbooks/roles/metadata.rb

name             'roles'
maintainer       'YOUR_COMPANY_NAME'
maintainer_email 'YOUR_EMAIL'
license          'All rights reserved'
description      'Installs/Configures roles'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version          '0.1.0'

depends redis

include_recipe する cookbook を depends に列挙しないと、その cookbook 内の attribute が使えない。

cookbooks/attributes/default.rb

if "#{node[:roles]}" == '["redis_master"]'
    default['redis']['peer_ip_address'] = "#{node['redis']['slave_ip_address']}"
    default['redis']['slaveof'] = 'no one'
elsif "#{node[:roles]}" == '["redis_slave"]'
    default['redis']['peer_ip_address'] = "#{node['redis']['master_ip_address']}"
    default['redis']['slaveof'] = "#{node['redis']['master_ip_address']}"
end   

if "#{node[:roles]}" == 'redis_master' がダメで、
if "#{node[:roles]}" == '["redis_master"]' なら通った。

redisのcookbook

cookbooks/redis/default.rb

# init
bash "sysctl" do
  user "root"
  code <<-EOC
    echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf
    sysctl -p
  EOC
  not_if 'grep "vm.overcommit_memory" /etc/sysctl.conf'
end

bash "binary_path" do
  user "root"
  code <<-EOC
      echo "export PATH=$PATH:#{node['redis']['binary_path']}" >> /etc/bashrc
    source /etc/bashrc
  EOC
  not_if 'grep "#{node['redis']['binary_path']}" /etc/bashrc'
end

# create directories
%w{ "#{node['redis']['log_dir']}" "#{node['redis']['work_dir']}" "#{node['redis']['conf_dir']}" "#{node['redis']['dump_dir']}" }.each do |dir|
  directory dir do
    action :create 
  end
end

# get latest redis source 
remote_file node['redis']['work_dir'] + node['redis']['source_file_name'] do
  source node['redis']['source_url_path'] + node['redis']['source_file_name']
  not_if "File.exists? #{node['redis']['server_install_path']}"
end

# install redis
bash "install_redis" do
  user "root"
  cwd node['redis']['work_dir']
  code <<-EOC
    tar -zxf #{node['redis']['source_file_name']}
    cd #{::File.basename(node['redis']['source_file_name'], '.tar.gz')}
    make
    make install
    cp src/redis-server src/redis-cli #{node['redis']['binary_path']}
  EOC
  not_if "File.exists? #{node['redis']['server_install_path']}"
end

# config file
template "redis.conf" do
  source "redis.conf.erb"
  path "#{node['redis']['redis_conf']}"
  owner "root"
  group "root"
  mode "0644"
end

# init script
template "redis_server'" do
  source "redis_server.erb"
  path "#{node['redis']['init_script']}"
  owner "root"
  group "root"
  mode "0755"
end

# start redis
service "redis_server-#{node['redis']['redis_port']}" do
  action :start
end

# master / slave
bash "slaveof" do
  user "root"
  code <<-EOC
    #{node['redis']['cli_install_path']} slaveof #{node['redis']['slaveof']}
  EOC
end

# check script
template "redis_check" do
  source "redis_check.sh.erb"
  path "#{node['redis']['check_script']}"
  owner "root"
  group "root"
  mode "755"
end

cookbooks/redis/attributes/default.rb

# directories
default['redis']['work_dir'] = '/usr/local/src/'
default['redis']['dump_dir'] = '/var/lib/redis/'
default['redis']['log_dir'] = '/var/log/redis/'
default['redis']['conf_dir'] = '/etc/redis/'

# Source Code URL
default['redis']['source_ver_num'] = '2.8.17'
default['redis']['source_url_path'] = 'http://download.redis.io/releases/'
default['redis']['source_file_name'] = "redis-#{node['redis']['source_ver_num']}.tar.gz"

# Redis Settings
default['redis']['binary_path'] = '/usr/local/bin'
default['redis']['server_install_path'] = '/usr/local/bin/redis-server'
default['redis']['cli_install_path'] = '/usr/local/bin/redis-cli'
default['redis']['redis_port'] = '6379'
default['redis']['logfile'] = "#{node['redis']['log_dir']}redis-#{node['redis']['redis_port']}.log"
default['redis']['redis_conf'] = "/etc/redis/redis-#{node['redis']['redis_port']}.conf"
default['redis']['init_script'] = "/etc/init.d/redis_server-#{node['redis']['redis_port']}"

# redis.conf
default['redis']['maxmemory'] = '任意の値'

# check script
default['redis']['check_script'] = '/usr/local/sbin/redis_check.sh'
default['redis']['check_log'] = "#{node['redis']['log_dir']}redis_check.log"

cookbooks/redis/templates/redis_server.erb

redis 2.4 系の起動スクリプトを流用しています。 変更点は2点

  1. redis のキャッシュ永続化をしないので、起動前に dump.rdb を削除
  2. master として動作させるために起動後に slaveof no one
#!/bin/sh
#
# redis - this script starts and stops the redis-server daemon
#
# chkconfig: - 85 15
# description: Redis is a persistent key-value database
# processname: redis-server
# config: /etc/redis/redis.conf
# config: /etc/sysconfig/redis
# pidfile: /var/run/redis.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

redis="<%= node['redis']['server_install_path'] %>"
redis_cli="<%= node['redis']['cli_install_path'] %>"
prog=$(basename $redis)

REDIS_CONF_FILE="<%= node['redis']['redis_conf'] %>"
DUMP="<%= node['redis']['dump_dir'] %>dump.rdb"

[ -f /etc/sysconfig/redis ] && . /etc/sysconfig/redis

lockfile=/var/lock/subsys/redis

start() {
    [ -x $redis ] || exit 5
    [ -f $REDIS_CONF_FILE ] || exit 6
    echo -n $";Starting $prog: "
    # remove dump file before starting.
    rm -f ${DUMP}
    daemon $redis $REDIS_CONF_FILE
    retval=$?
    [ $retval -eq 0 ] && touch $lockfile
    status $prog
    sleep 3
    # start as master
    $redis_cli slaveof no one
    return $retval
}

stop() {
    echo -n $";Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    status $prog
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    stop
    start
}

reload() {
    echo -n $";Reloading $prog: "
    killproc $redis -HUP
    RETVAL=$?
    echo
}

force_reload() {
    restart
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
    ;;
    stop)
        rh_status_q || exit 0
        $1
    ;;
    restart|configtest)
        $1
    ;;
    reload)
    rh_status_q || exit 7
    $1
    ;;
    force-reload)
        force_reload
    ;;
    status)
        rh_status
    ;;
    condrestart|try-restart)
        rh_status_q || exit 0
    ;;
    *)

echo $";Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
exit 2
esac

cookbooks/redis/templates/redis.conf.erb

  • master として動作(master / slave で設定を同じにしたいため)
  • 永続化はしない
daemonize yes
pidfile /var/run/redis.pid
port <%= node['redis']['redis_port'] %>
timeout 0
loglevel notice
logfile <%= node['redis']['logfile'] %>
dir <%= node['redis']['dump_dir'] %>
save ""
slaveof no one
slave-serve-stale-data yes
slave-read-only yes
slave-priority 100
maxmemory <%= node['redis']['maxmemory'] %>
maxmemory-policy volatile-lru
appendonly no

cookbooks/redis/templates/redis_check.sh.erb

redis の相互監視スクリプト

  • peer の down を検知したら route table を変更してフェイルオーバーさせる
#!/bin/sh
# redis-check.sh
# monitoring peer redis. using redis-cli ping.
# when ping failed ${RETRY_COUNT} , start failover.

### define ###
# notice.this configration is different from peer.
PEER="<%= node['redis']['peer_ip_address'] %>"
instance_id=`curl -s 169.254.169.254/latest/meta-data/instance-id`
PORT="<%= node['redis']['redis_port']  %>"
VIP="<%= node['redis']['vip'] %>"
route_table_names=("dmz" "lan" "mgmt" "db")

# commands
AWS="/usr/bin/aws ec2 --region ap-northeast-1"
ENI_FILTER="--filters Name=attachment.instance-id,Values=${instance_id} Name=description,Values="Primary network interface""
RTB_FILTER="--filters Name=tag-value,Values=${route_table_name}"
QUERY="--output text --query"
REDIS="/usr/sbin/redis-cli"
RUNFOR="/usr/sbin/runfor"
LOGGER="logger -f <%= node['redis']['check_log'] %> -t $0"
### define end ###

### functions ###
monitoring_peer(){

    TRIES="0"
    RETRY_COUNT="3"
    while true
    do
        PING_RESULT=`${RUNFOR} 2 ${REDIS} -h ${PEER} -p ${PORT} ping`
        if [ "${PING_RESULT}" = "PONG" ]; then
            ${LOGGER} "INFO. peer alive. do nothing."
            exit 0
        else
            TRIES=`expr ${TRIES} + 1`
            if [ ${TRIES} -lt ${RETRY_COUNT} ]; then
                ${LOGGER} "WARNING. redis-cli ping failed count => ${TRIES}." 
                sleep 10
            else
                ${LOGGER} "ERROR. no responce from ${PEER}."
                break
            fi 
        fi
    done
}

promote_master(){

    CHECK_ROLE=`${REDIS} info | grep role | cut -b6`
    if [ "${CHECK_ROLE}" = "m" ]; then
        ${LOGGER} "peer node is Dead."
        exit 0
    elif [ "${CHECK_ROLE}" = "s" ] ; then
        ${LOGGER} "promote to master start."
        ${REDIS} slaveof no one
    else
        ${LOGGER} "STATUS UNKNOWN."
        exit 1
    fi

    CHECK_ROLE=`${REDIS} info | grep role | cut -b6`
    if [ "${CHECK_ROLE}" = "m" ] ; then
        ${LOGGER} "promote complete. start replace route."
    else
        ${LOGGER} "ERROR. failed to promote master."
        exit 1
    fi
}

replace_route(){

    ### define eni
    eni_id=`${AWS} describe-network-interfaces ${ENI_FILTER} ${QUERY} '.NetworkInterfaces[].NetworkInterfaceId'`

    ### get route_table id then start replace route.
    for route_table_name in ${route_table_names[@]}
    do
        # get route table id
        route_table_id=`${AWS} describe-route-tables ${RTB_FILTER} ${QUERY} '.RouteTables[].RouteTableId'`
        # replace route
        ${AWS} replace-route --route-table-id ${route_table_id} --destination-cidr-block ${VIP} --network-interface-id ${eni_id}
        if [ $? = 0 ] ; then
            ${LOGGER} "replace vip for ${route_table_name} complete." 
            exit 0
        else
            ${LOGGER} "ERROR. failed replace vip for ${route_table_name}."
            exit 1
        fi    
    done
}


monitoring_peer
promote_master
replace_route

恥ずかしながら晒してみました。以上です。