本篇文章给大家介绍一下用Shell和Ansible两种方式，去一键部署ES（ElasticSearch）集群，实现在集群部署中的批量化操作，享受自动化部署的魅力。
准备主机
| 主机名 | IP | CPU | disk | memory |
|---|---|---|---|---|
| Ansible | 10.0.0.61 | 2C | 20G | 2G |
| ELK104 | 10.0.0.104 | 2C | 20G | 2G |
| ELK105 | 10.0.0.105 | 2C | 20G | 2G |
| ELK106 | 10.0.0.106 | 2C | 20G | 2G |
Shell一键部署
我们在Ansible上准备好所需文件（无JDK的ES安装包和oracle的JDK），在Ansible编写Shell脚本，实现ES集群的一键部署，其实脚本编写的思路有很多，这里我使用先做免密钥，做完后写for循环，里面进行单点部署，也可以直接部署出来一个节点后进行拷贝
如果想修改目录的话需要修改脚本，elasticsearch.yml、es7.service里的文件，所以配置文件都打包放在了文末。
[root@Ansible ~]# cat deploy_ES.sh
#!/bin/bash
#1、先做免密钥，方便文件的推送以及远程执行命令
if [ -f /root/.ssh/id_rsa.pub ];then
echo "公钥已经存在"
else
ssh-keygen -t rsa
echo "公钥不存在，自动创建公钥"
fi
hosts=(
10.0.0.104
10.0.0.105
10.0.0.106
)
for i in ${hosts[*]}
do
expect <<EOF
set timeout 10
spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$i
expect {
"yes/no" { send "yes\n";exp_continue }
"password" { send "1\n" }
}
expect "password" { send "1\n" }
EOF
done
#2、准备配置文件
# 拼接IP字符串
ip_str=$(printf '", "%s' "${hosts[@]}")
ip_str=${ip_str:3}
# 替换文件中的IP地址
sed -i 's#^discovery.*#discovery.seed_hosts: \['"$ip_str"'"\]#g' elasticsearch.yml
sed -i 's#^cluster.initial_master_nodes.*#cluster.initial_master_nodes: \['"$ip_str"'"\]#g' elasticsearch.yml
#3、搭建ES集群
for q in ${hosts[*]}
do
scp -r elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz jdk-8u291-linux-x64.tar.gz elasticsearch.yml es7.service jvm.options jdk.sh $q:/root/
ssh $q "mkdir -p /koten/{softwares,data,logs}"
ssh $q "tar xf /root/elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz -C /koten/softwares/"
ssh $q "useradd koten"
ssh $q "chown koten:koten /koten/softwares/elasticsearch-7.17.5/ -R"
ssh $q "cat > /etc/security/limits.d/es.conf <<EOF
* soft nofile 65535
* hard nofile 131070
EOF
"
ssh $q "cat > /etc/sysctl.d/es.conf <<EOF
vm.max_map_count=262144
EOF"
ssh $q "sysctl -f /etc/sysctl.d/es.conf"
ssh $q "tar xf /root/jdk-8u291-linux-x64.tar.gz -C /koten/softwares/"
ssh $q "\cp /root/elasticsearch.yml /koten/softwares/elasticsearch-7.17.5/config/"
ssh $q "\cp /root/es7.service /usr/lib/systemd/system/"
ssh $q "\cp /root/jvm.options /koten/softwares/elasticsearch-7.17.5/config/"
ssh $q "\cp /root/jdk.sh /etc/profile.d/"
ssh $q "source /etc/profile.d/jdk.sh"
ssh $q "install -d /koten/{data,logs}/es7 -o koten -g koten"
done
#3、都部署好后再启动，防止脑裂
for e in ${hosts[*]}
do
ssh $e "systemctl enable --now es7"
if [ `ss -ntl | grep 19|wc -l` != 0 ];then
echo "$e 主机ES启动成功！"
fi
done
[root@Ansible ~]# sh deploy_ES.sh
......
Created symlink from /etc/systemd/system/multi-user.target.wants/es7.service to /usr/lib/systemd/system/es7.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/es7.service to /usr/lib/systemd/system/es7.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/es7.service to /usr/lib/systemd/system/es7.service.
[root@ELK104 ~]# netstat -tnulp|grep 19
tcp6 0 0 :::19200 :::* LISTEN 2252/java
tcp6 0 0 :::19300 :::* LISTEN 2252/java 
Ansible批量部署
1、初始化roles目录
[root@Ansible roles]# ansible-galaxy init es
- Role es was created successfully
[root@Ansible roles]# cd es
[root@Ansible es]# pwd
/ansible/roles/es
[root@Ansible es]# ls
defaults files handlers meta README.md tasks templates tests vars2、编写给10.0.0.104、10.0.0.105、10.0.0.106做免密钥的playbook
[root@Ansible es]# cat /etc/ansible/hosts | tail -4
[es_group]
es104 ansible_ssh_host=10.0.0.104 ansible_ssh_user='root' ansible_ssh_pass='1'
es105 ansible_ssh_host=10.0.0.105 ansible_ssh_user='root' ansible_ssh_pass='1'
es106 ansible_ssh_host=10.0.0.106 ansible_ssh_user='root' ansible_ssh_pass='1'
[root@Ansible es]# cat ssh.yaml
---
- hosts: es_group
gather_facts: no
tasks:
- name: install ssh key
authorized_key: user=root
key="{{ lookup('file','/root/.ssh/id_rsa.pub')}}"
state=present
[root@Ansible es]# ansible-playbook ssh.yaml
PLAY [es_group] ****************************************************************************************
TASK [install ssh key] *********************************************************************************
changed: [es106]
changed: [es105]
changed: [es104]
PLAY RECAP *********************************************************************************************
es104 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
es105 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
es106 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 3、编写数据推送和配置的playbook
如果需要修改主机，不仅要修改hosts文件，还需要修改elasticsearch.yml
[root@Ansible roles]# cat hosts | tail -4
[es_group]
10.0.0.104
10.0.0.105
10.0.0.106
[root@Ansible roles]# cat site.yml
- hosts: es_group
roles:
- role: es
[root@Ansible roles]# ls es/files/
elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz es7.service jdk.sh
elasticsearch.yml jdk-8u291-linux-x64.tar.gz jvm.options
[root@Ansible roles]# cat es/tasks/main.yml
- name: mkdir /koten/softwares /koten/data /koten/logs
file:
path: "{{ item }}"
state: directory
with_items:
- /koten/softwares
- /koten/data
- /koten/logs
- name: tar xf elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz jdk-8u291-linux-x64.tar.gz
unarchive:
src: "{{ item }}"
dest: /koten/softwares
with_items:
- elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz
- jdk-8u291-linux-x64.tar.gz
- name: Create koten Group
group:
name: koten
- name: Create koten User
user:
name: koten
- name: chown -R koten.koten /koten/softwares/elasticsearch-7.17.5/
file:
path: /koten/softwares/elasticsearch-7.17.5/
owner: koten
group: koten
recurse: yes
- name: copy content to /etc/security/limits.d/es.conf
copy:
content: |
* soft nofile 65535
* hard nofile 131070
dest: /etc/security/limits.d/es.conf
- name: copy test content to /etc/sysctl.d/es.conf
copy:
content: |
vm.max_map_count=262144
dest: /etc/sysctl.d/es.conf
- name: sysctl -f /etc/sysctl.d/es.conf
command: sysctl -f /etc/sysctl.d/es.conf
- name: push es7.service to /usr/lib/systemd/system/
copy:
src: es7.service
dest: /usr/lib/systemd/system/
- name: push jvm.options to /koten/softwares/elasticsearch-7.17.5/config/
copy:
src: jvm.options
dest: /koten/softwares/elasticsearch-7.17.5/config/
- name: push elasticsearch.yml to /koten/softwares/elasticsearch-7.17.5/config/
copy:
src: elasticsearch.yml
dest: /koten/softwares/elasticsearch-7.17.5/config/elasticsearch.yml
- name: push jdk.sh to /etc/profile.d/
copy:
src: jdk.sh
dest: /etc/profile.d/
- name: source /etc/profile.d/jdk.sh
command: bash -c 'source /etc/profile.d/jdk.sh'
- name: mkdir /koten/data/es7 /koten/logs/es7
file:
path: "{{ item }}"
state: directory
owner: koten
group: koten
with_items:
- /koten/data/es7
- /koten/logs/es7
- name: systemctl enable --now es7
systemd:
name: es7
state: started
enabled: yes4、执行数据推送和配置的playbook，部署es集群
[root@Ansible roles]# ansible-playbook site.yml
PLAY [es_group] ****************************************************************
TASK [Gathering Facts] *********************************************************
ok: [es105]
ok: [es106]
ok: [es104]
TASK [mkdir /koten/softwares /koten/data /koten/logs] **************************
changed: [es105] => (item=/koten/softwares)
changed: [es104] => (item=/koten/softwares)
changed: [es106] => (item=/koten/softwares)
changed: [es106] => (item=/koten/data)
changed: [es104] => (item=/koten/data)
changed: [es105] => (item=/koten/data)
changed: [es106] => (item=/koten/logs)
changed: [es105] => (item=/koten/logs)
changed: [es104] => (item=/koten/logs)
TASK [es : tar xf elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz jdk-8u291-linux-x64.tar.gz] ***
changed: [es105] => (item=elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz)
changed: [es104] => (item=elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz)
changed: [es106] => (item=elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz)
changed: [es105] => (item=jdk-8u291-linux-x64.tar.gz)
changed: [es106] => (item=jdk-8u291-linux-x64.tar.gz)
changed: [es104] => (item=jdk-8u291-linux-x64.tar.gz)
TASK [es : Create koten Group] *************************************************
changed: [es105]
changed: [es106]
changed: [es104]
TASK [es : Create koten User] **************************************************
changed: [es104]
changed: [es105]
changed: [es106]
TASK [chown -R koten.koten /koten/softwares/elasticsearch-7.17.5/] *************
changed: [es104]
changed: [es105]
changed: [es106]
TASK [copy content to /etc/security/limits.d/es.conf] **************************
changed: [es104]
changed: [es106]
changed: [es105]
TASK [copy test content to /etc/sysctl.d/es.conf] ******************************
changed: [es106]
changed: [es104]
changed: [es105]
TASK [sysctl -f /etc/sysctl.d/es.conf] *****************************************
changed: [es106]
changed: [es104]
changed: [es105]
TASK [push es7.service to /usr/lib/systemd/system/] ****************************
changed: [es104]
changed: [es105]
changed: [es106]
TASK [push jvm.options to /koten/softwares/elasticsearch-7.17.5/config/] *******
changed: [es104]
changed: [es105]
changed: [es106]
TASK [push elasticsearch.yml to /koten/softwares/elasticsearch-7.17.5/config/] ***
changed: [es104]
changed: [es106]
changed: [es105]
TASK [es : push jdk.sh to /etc/profile.d/] *************************************
changed: [es104]
changed: [es105]
changed: [es106]
TASK [es : source /etc/profile.d/jdk.sh] ***************************************
changed: [es104]
changed: [es106]
changed: [es105]
TASK [mkdir /koten/data/es7 /koten/logs/es7] ***********************************
changed: [es104] => (item=/koten/data/es7)
changed: [es105] => (item=/koten/data/es7)
changed: [es106] => (item=/koten/data/es7)
changed: [es105] => (item=/koten/logs/es7)
changed: [es104] => (item=/koten/logs/es7)
changed: [es106] => (item=/koten/logs/es7)
TASK [systemctl enable --now es7] **********************************************
changed: [es106]
changed: [es104]
changed: [es105]
PLAY RECAP *********************************************************************
es104 : ok=16 changed=15 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
es105 : ok=16 changed=15 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
es106 : ok=16 changed=15 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
# 查看es的状态
[root@ELK104 ~]# systemctl status es7.service
● es7.service - es7
Loaded: loaded (/usr/lib/systemd/system/es7.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-10-27 14:01:54 CST; 42s ago
Main PID: 3578 (java)
CGroup: /system.slice/es7.service
├─3578 /koten/softwares/jdk1.8.0_291/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=6...
└─3711 /koten/softwares/elasticsearch-7.17.5/modules/x-pack-ml/platform/linux-x86_64/bin/c...
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,716][INFO ][o.e.p.PluginsService ...wn]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,716][INFO ][o.e.p.PluginsService ...ql]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,716][INFO ][o.e.p.PluginsService ...ck]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,717][INFO ][o.e.p.PluginsService ...re]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,717][INFO ][o.e.p.PluginsService ...de]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,749][INFO ][o.e.p.PluginsService ...er]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,750][INFO ][o.e.p.PluginsService ...ded
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,958][INFO ][o.e.e.NodeEnvironment...fs]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,959][INFO ][o.e.e.NodeEnvironment...ue]
Oct 27 14:02:20 ELK104 elasticsearch[3578]: [2023-10-27T14:02:20,074][INFO ][o.e.n.Node ...st]
Hint: Some lines were ellipsized, use -l to show in full.
deploy_es_shell打包文件下载链接：https://pan.baidu.com/s/1l22I0aSB-50NXlh4lYZArw?pwd=umbm
deploy_es_ansible打包文件下载链接：https://pan.baidu.com/s/1lIfYWMnhsg7yoyzgkBSEDA?pwd=8hpl
