I recently took a position at a new company and had the opportunity to stand up a greenfield environment. Part of this environment was obviously configuring monitoring. Since we’re pretty heavily tied to VMware here, we’re using vROps for our monitoring. For escalation of alerts, we’re using OpsGenie, and for normal collaboration we’re using Teams. Unfortunately, vROps versions older than 8.4 do not support sending alerts to OpsGenie or Teams out of the box. Fortunately, we can still get this working with a bit of work.

Overview

Approach

We’ll be running this whole project using Photon OS 4.

While we could simply run the python script in a screen, tmux, or similar session, but there are serious limitations in doing that. Admittedly, my python isn’t as strong as some of my other scripting languages, but after a bit of digging, it became clear that things like security, performance, and monitoring are either limited or unavailable if we do this. We’ll use a WSGI server called Gunicorn to run the python script, and use systemd to create a socket and service, and finally, use Nginx to connect directly to the socket. Whew! Even just reading that after completing this entire project feels like a lot, but don’t worry. As always, I’ll explain each step in detail and give you code to create this project.

Create Photon Packer Template

If you don’t know this about me by now, I’m a huge fan of packer. As such, we’ll be using it to create a template. You can find the source code for this packer template on my website’s github repository

To get a local copy, run the following command:

git clone https://github.com/jonhowe/Virtjunkie.com.git

You can find all source code in the directory Virtjunkie.com/Packer/PhotonOS4.

Create Custom ISO

Unfortunately, Photon OS, even in v4, doesn’t support providing the kickstart file via CD-ROM or floppy (see Issue#1113, Issue#798). Unfortunately, this means that if we want (or need) to use the vsphere-iso builder, we’ll need to create an ISO with the kickstart file embedded. I’m not going to duplicate documentation, so here’s the official documentation from VMware on how to add a kickstart file.

Instead of using the sample my_ks.cfg file provided in the ISO, use the version below.

IMPORTANT: Ensure that you replace [yourpassword] with your actual password

Use Packer To Create vSphere Template

IMPORTANT: Ensure that the password assigned to “root_password” in Virtjunkie.com/Packer/PhotonOS4/packer-photon.json.pkr.hcl matches what is set in the kickstart file above

packer build -var-file=nable.auto.pkrvars.hcl -var-file=vars iso-local-4.0GA.json -only=vsphere-iso.vmware-template packer-photon.json.pkr.hcl

Deploy Photon 4 VM

For production systems, I typically use Terraform to deploy my VMs, and that’s what I did. I’m not going to go over that, as I’ve done it before. If nothing else, just deploy the VM from a template, assign it a static IP in the guest customization specification, and then log in using credentials set in the kickstart file.

Configure Server

Install Packages

Execute the commands below to install required packages, and to clone the repository

Configure Shims

There’s a bit of config we’ll need to do in order to make this production ready.

Configure Teams

A fellow VMware vExpert, Shane Moore has an excellent article. So that I don’t recreate the wheel, please see his instructions for how to prep your environment to use Microsoft Teams.

In short, you’ll create a webhook connector in the Microsoft Teams channel, and plug the URL that you get from Teams into the teams shim file.

Configure OpsGenie

OpsGenie is pretty simple. All we’ll need to do is create an integration by selecting your team in Opsgenie, navigating to Integrations, then selecting Add Integration.

In the resulting list of available integrations, select API.

Then note the API key (you can grab it later, it doesn’t disappear when you navigate away). We’ll use that when we configure outbound settings in vROps to use OpsGenie.

Disable Unused Shims

In my environment, I’m only going to be using Teams and OpsGenie alerts. To reduce bloat and access points that could cause me security issues, I’m disabling all shims not in use.

Edit the file loginsightwebhookdemo/webhook-shims/loginsightwebhookdemo/__init__.py

Disable Welcome Page

This could probably be optional, but I appreciate not having a welcome/hello world webpage open for a production service. We’ll disable it.

Edit the file loginsightwebhookdemo/webhook-shims/loginsightwebhookdemo/__init__.py

Configure Systemd

Create two files: /etc/systemd/system/gunicorn.service and /etc/systemd/system/gunicorn.socket and populate with the text below.

/etc/systemd/system/gunicorn.service

This service file ensures that Gunicorn is started the same way, and is managed by systemd. You’ll note that it requires gunicorn.socket, which we will create later.

/etc/systemd/system/gunicorn.socket

This socket file sets permissions for the socket assigned to gunicorn

Configure Nginx

Replace the file /etc/nginx/nginx.conf with the contents below

This is telling Nginx to connect directly to the socket we’re creating with the systemd socket gunicorn.socket we created earlier.

Enable and Start All Services

Finally, enable, and start both services, and restart Nginx

systemctl enable gunicorn.socket
systemctl enable gunicorn.service
systemctl restart gunicorn.socket 
systemctl restart gunicorn.service
systemctl restart nginx.service

Configure vRealize Operations Manager

At this point, all we need to do is configure an outbound instance for Microsoft Teams and OpsGenie.

Add a New Rest Notification Plugin

OpsGenie

To do this, sign into vROps, and navigate to Administration ->Management->Outbound Settings, or https://[Your_vROps_URL]/ui/index.action#/administration/outbound-alert-settings

After you select the Rest Notification Plugin, there are 3 fields required in the Add/Edit Outbound Instance box

Finally, navigate to Alerts -> Configuration -> Notification (https://[Your_vROps_URL]/ui/index.action#/alerts/notifications) and add a new rule that targets the adapter you just created.

Teams

Again, I’m not in the business of recreating the wheel, so check out Shane’s blog for info on how to do this.

References

The post Configure vROps To Send Alerts to OpsGenie and Microsoft Teams appeared first on VirtJunkie.

Everything happening in the world these days started to get me down a bit starting a few months ago. I decided to embrace change in my life with things I could affect instead of letting the things that I couldn’t affect get me down. A leader in my organization once brought up the concept of “hard learning”, which is learning things that are outside of your comfort zone. I decided that hard learning is what I’ll embrace in order to get the serotonin boost I’ve been craving. 

Anyone that has talked to me about certification is aware that I haven’t placed a lot of value on achieving them in the past, and that I’ve relied on the ability to hit the ground running as proof of competency. What I didn’t think about until recently is that very few people have a certificate AND can hit the ground running. An even smaller group contains people that receive certs that actually have hands on experience with their particular certification area. I aim to be a part of the very small group of people that can hit the ground running, is certified, and has significant experience backing everything up.

Part of the hard learning for me here was realizing that I was wrong about certs. I needed to make them work for me, by studying and ensuring I have actual experience so they are not just a meaningless piece of (digital) paper. The other hard learning part is pushing myself to learn new things. It’s easy to get stuck in your comfort zone. Mario Andretti said, “If everything seems under control, you’re not going fast enough”. In tech, we die on the vine when we are not pushing and innovating ourselves. I’m working hard to apply this principle to myself by learning things that aren’t connected to my current skill set.

Anyhow, that’s all I’ve got right now. Big things have small beginnings. I appreciate you spending the time to read this.

The post Big Things have Small Beginnings – How I Beat 2020 appeared first on VirtJunkie.