I’ve been learning my way through Ansible’s ad-hoc method of distributed management. It’s been a great start to re-familiarize myself with the important little details of Linux while getting the basics of Ansible down.
I’m also slowly working my way up in complexity into a distributed performance platform for performance analysis. But this post isn’t about that story. It’s about solving a problem.
The system architecture today is:
- Deploy VMs from template with DHCP addresses (via PowerShell)
- Install latest versions of Ansible and fio (via bash… for now)
- Auto-discover peer systems and separate them into zones inside /etc/ansible/hosts (via python thanks to a coworker)
- Set crontab with performance profile dictated by host zone (via Ansible)
- Reboot (via Ansible) and run the workload (via bash)
Then I/O begins. It’s pretty nifty, even in its duct tape and glue design right now.
But I kept finding myself in need of updating the fio workload profile and then restarting the bash script managing workloads. Until today, I’ve been:
- Using a shared NFS mount to distribute the updates:
ansible all -m shell -a "cp nfs1/*fio ." --ask-pass
- Rebooting all hosts but this one (learned that lesson):
ansible all:\!10.1.1.131 -m shell -a "reboot" --sudo --ask-pass
- Then rebooting:
Then I found that sometimes I just wanted to turn off all workloads in a rush. I knew the answer would have the Linux command
kill in it somewhere, but couldn’t figure out how to call it without knowing every system’s process ID (PID) for the command. So I asked on superuser.com and here’s what they taught me in just a few steps.
Step 1 – Find out the process name.
pgrep -lf bash [to find the bash shell script]
Let’s say it returns a process called
Step 2 – Kill it.
And that’s it! You can confirm you killed the process with
ps aux | grep bash, which shows you the process is no longer running.
I now have a single ad-hoc command to kill this process across 15 different VMs:
ansible all -m shell -a "pkill bash-fio" --ask-pass
This kind of distributed kill switch comes in handy. I hope you find good use of it as well.