Packer, Photon, Python, Flask, WSGI, vROps. Holy smokes folks.. I’m not sure I could make a more click-bait-y post if I tried!
I recently took a position at a new company and had the opportunity to stand up a greenfield environment. Part of this environment was obviously configuring monitoring. Since we’re pretty heavily tied to VMware here, we’re using vROps for our monitoring. For escalation of alerts, we’re using OpsGenie, and for normal collaboration we’re using Teams. Unfortunately, vROps versions older than 8.4 do not support sending alerts to OpsGenie or Teams out of the box. Fortunately, we can still get this working with a bit of work.
Table of Contents
Overview
Update – Starting in vROps 8.4 you can use the webhook plugin that is delivered with vROps to accomplish this. LINK
While vROps does have the ability to send outgoing notifications via REST API, it doesn’t have the ability to let you customize what fields are sent, or how they are formatted. Because of this limitation, some very thoughtful folks created a project called webhook-shims. This is essentially a Python Flask project that accepts web requests, translates them, then forwards the correctly formatted request off to whatever system you want notifications sent to.
This project hasn’t been updated in over 3 years, has a number of pending PR’s and Issues, so my spidey sense says it’s likely dead. So please be smart about how and where you use this, and consider the implications of running this in production. That said, from a functional standpoint, it seems to work well, and in the rest of this post I’ll go over how to set this up to run in a production fashion.
The list to the right are the applications that the project supports, and the bold items are what I’ll show example configuration for.
- bigpanda
- bugzilla
- groove
- hipchat
- jenkins
- jira
- kafkatopic
- opsgenie
- pagerduty
- pivotaltracker
- pushbullet
- servicenow
- slack
- socialcast
- travisci
- vrealizeorchestrator
- zendesk
- moogsoft
- msteams
Approach
We’ll be running this whole project using Photon OS 4.
While we could simply run the python script in a screen, tmux, or similar session, but there are serious limitations in doing that. Admittedly, my python isn’t as strong as some of my other scripting languages, but after a bit of digging, it became clear that things like security, performance, and monitoring are either limited or unavailable if we do this. We’ll use a WSGI server called Gunicorn to run the python script, and use systemd to create a socket and service, and finally, use Nginx to connect directly to the socket. Whew! Even just reading that after completing this entire project feels like a lot, but don’t worry. As always, I’ll explain each step in detail and give you code to create this project.
Create Photon Packer Template
If you don’t know this about me by now, I’m a huge fan of packer. As such, we’ll be using it to create a template. You can find the source code for this packer template on my website’s github repository
To get a local copy, run the following command:
git clone https://github.com/jonhowe/Virtjunkie.com.git
You can find all source code in the directory Virtjunkie.com/Packer/PhotonOS4.
Create Custom ISO
Unfortunately, Photon OS, even in v4, doesn’t support providing the kickstart file via CD-ROM or floppy (see Issue#1113, Issue#798). Unfortunately, this means that if we want (or need) to use the vsphere-iso builder, we’ll need to create an ISO with the kickstart file embedded. I’m not going to duplicate documentation, so here’s the official documentation from VMware on how to add a kickstart file.
Instead of using the sample my_ks.cfg file provided in the ISO, use the version below.
IMPORTANT: Ensure that you replace [yourpassword] with your actual password
| { | |
| "hostname": "photon", | |
| "password": { | |
| "crypted": false, | |
| "text": "[yourpassword]" | |
| }, | |
| "bootmode": "bios", | |
| "disk": "/dev/sda", | |
| "linux_flavor": "linux", | |
| "packagelist_file": "packages_minimal.json", | |
| "postinstall": [ | |
| "#!/bin/sh", | |
| "sed -i 's/PermitRootLogin no/PermitRootLogin yes/g' /etc/ssh/sshd_config", | |
| "systemctl restart sshd.service" | |
| ] | |
| } |
Use Packer To Create vSphere Template
IMPORTANT: Ensure that the password assigned to “root_password” in Virtjunkie.com/Packer/PhotonOS4/packer-photon.json.pkr.hcl matches what is set in the kickstart file above
Deploy Photon 4 VM
For production systems, I typically use Terraform to deploy my VMs, and that’s what I did. I’m not going to go over that, as I’ve done it before. If nothing else, just deploy the VM from a template, assign it a static IP in the guest customization specification, and then log in using credentials set in the kickstart file.
Configure Server
Install Packages
Execute the commands below to install required packages, and to clone the repository
| #Install Prerequisite Packages from Repo | |
| tdnf install wget curl python3 python3-xml python3-pip git nginx -y | |
| #Clone git repository | |
| git clone https://github.com/vmw-loginsight/webhook-shims.git | |
| #Install pip packages | |
| pip3 install uwsgi | |
| pip3 install gunicorn | |
| pip3 install -r ./loginsightwebhookdemo/webhook-shims/requirements.txt |
Configure Shims
There’s a bit of config we’ll need to do in order to make this production ready.
Configure Teams
A fellow VMware vExpert, Shane Moore has an excellent article. So that I don’t recreate the wheel, please see his instructions for how to prep your environment to use Microsoft Teams.
In short, you’ll create a webhook connector in the Microsoft Teams channel, and plug the URL that you get from Teams into the teams shim file.
Configure OpsGenie
OpsGenie is pretty simple. All we’ll need to do is create an integration by selecting your team in Opsgenie, navigating to Integrations, then selecting Add Integration.
In the resulting list of available integrations, select API.
Then note the API key (you can grab it later, it doesn’t disappear when you navigate away). We’ll use that when we configure outbound settings in vROps to use OpsGenie.
Disable Unused Shims
In my environment, I’m only going to be using Teams and OpsGenie alerts. To reduce bloat and access points that could cause me security issues, I’m disabling all shims not in use.
Edit the file loginsightwebhookdemo/webhook-shims/loginsightwebhookdemo/__init__.py
| #Starting at line 278, comment out all shims you aren't using | |
| 277 # Import individual shims | |
| 278 #import loginsightwebhookdemo.bigpanda | |
| 279 #import loginsightwebhookdemo.bugzilla | |
| 280 #import loginsightwebhookdemo.groove | |
| 281 #import loginsightwebhookdemo.hipchat | |
| 282 #import loginsightwebhookdemo.jenkins | |
| 283 #import loginsightwebhookdemo.jira | |
| 284 #import loginsightwebhookdemo.kafkatopic | |
| 285 import loginsightwebhookdemo.opsgenie | |
| 286 #import loginsightwebhookdemo.pagerduty | |
| 287 #import loginsightwebhookdemo.pivotaltracker | |
| 288 #import loginsightwebhookdemo.pushbullet | |
| 289 #import loginsightwebhookdemo.servicenow | |
| 290 #import loginsightwebhookdemo.slack | |
| 291 #import loginsightwebhookdemo.socialcast | |
| 292 #import loginsightwebhookdemo.template | |
| 293 #import loginsightwebhookdemo.travisci | |
| 294 #import loginsightwebhookdemo.vrealizeorchestrator | |
| 295 #import loginsightwebhookdemo.zendesk | |
| 296 #import loginsightwebhookdemo.moogsoft | |
| 297 import loginsightwebhookdemo.msteams |
Disable Welcome Page
This could probably be optional, but I appreciate not having a welcome/hello world webpage open for a production service. We’ll disable it.
Edit the file loginsightwebhookdemo/webhook-shims/loginsightwebhookdemo/__init__.py
| #Create a comment block starting at line 241, right before @app.route("/") and ending right before @app.route("/endpoint/test", methods=['POST']) | |
| 239 return ("%s" % r.text, r.status_code, None) | |
| 240 | |
| 241 ''' | |
| 242 @app.route("/") | |
| 243 def _introduction(): | |
| 244 """This help text.""" | |
| 245 ret = _minimal_markdown(Markup("<p>%s</p>") % __doc__) | |
| 246 ret += Markup("<dl>") | |
| 247 | |
| 248 for f in sorted(app.view_functions): | |
| 249 if f != 'static': | |
| 250 ret += Markup("<dt><b>%s()</b></dt>\n") % f | |
| 251 for r in app.url_map.iter_rules(): | |
| 252 if r.endpoint == f: | |
| 253 ret += Markup(" <dd><code>%s</code></dd>\n") % str(r) | |
| 254 ret += Markup(" <dd>%s</dd>\n") % _minimal_markdown(Markup.escape(str(app.view_functions[f].__doc__))) | |
| 255 ret += Markup("</dl>") | |
| 256 return ret | |
| 257 ''' | |
| 258 | |
| 259 @app.route("/endpoint/test", methods=['POST']) |
Configure Systemd
Create two files: /etc/systemd/system/gunicorn.service and /etc/systemd/system/gunicorn.socket and populate with the text below.
/etc/systemd/system/gunicorn.service
This service file ensures that Gunicorn is started the same way, and is managed by systemd. You’ll note that it requires gunicorn.socket, which we will create later.
| #/etc/systemd/system/gunicorn.service | |
| [Unit] | |
| Description=gunicorn daemon | |
| Requires=gunicorn.socket | |
| After=network.target | |
| [Service] | |
| Type=notify | |
| # the specific user that our service will run as | |
| User=root | |
| Group=root | |
| # another option for an even more restricted service is | |
| # DynamicUser=yes | |
| # see http://0pointer.net/blog/dynamic-users-with-systemd.html | |
| RuntimeDirectory=gunicorn | |
| WorkingDirectory=/root/loginsightwebhookdemo/webhook-shims/ | |
| ExecStart=gunicorn -b 127.0.0.1:5001 loginsightwebhookdemo:app | |
| ExecReload=/bin/kill -s HUP $MAINPID | |
| KillMode=mixed | |
| TimeoutStopSec=5 | |
| PrivateTmp=true | |
| [Install] | |
| WantedBy=multi-user.target |
/etc/systemd/system/gunicorn.socket
This socket file sets permissions for the socket assigned to gunicorn
| #/etc/systemd/system/gunicorn.socket | |
| [Unit] | |
| Description=gunicorn socket | |
| [Socket] | |
| ListenStream=/run/gunicorn.sock | |
| # Our service won't need permissions for the socket, since it | |
| # inherits the file descriptor by socket activation | |
| # only the nginx daemon will need access to the socket | |
| SocketUser=nobody | |
| # Optionally restrict the socket permissions even more. | |
| # SocketMode=600 | |
| [Install] | |
| WantedBy=sockets.target |
Configure Nginx
Replace the file /etc/nginx/nginx.conf with the contents below
This is telling Nginx to connect directly to the socket we’re creating with the systemd socket gunicorn.socket we created earlier.
| #/etc/nginx/nginx.conf | |
| worker_processes 1; | |
| events { | |
| worker_connections 1024; | |
| } | |
| http { | |
| include mime.types; | |
| default_type application/octet-stream; | |
| sendfile on; | |
| keepalive_timeout 65; | |
| server { | |
| listen 80; | |
| server_name localhost; | |
| location / { | |
| #proxy_pass http://127.0.0.1:5001; | |
| #proxy_set_header Host $host; | |
| #proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | |
| proxy_pass http://unix:/run/gunicorn.sock; | |
| } | |
| # redirect server error pages to the static page /50x.html | |
| # | |
| error_page 500 502 503 504 /50x.html; | |
| location = /50x.html { | |
| root html; | |
| } | |
| } | |
| } |
Enable and Start All Services
Finally, enable, and start both services, and restart Nginx
Configure vRealize Operations Manager
At this point, all we need to do is configure an outbound instance for Microsoft Teams and OpsGenie.
Add a New Rest Notification Plugin
OpsGenie
To do this, sign into vROps, and navigate to Administration ->Management->Outbound Settings, or https://[Your_vROps_URL]/ui/index.action#/administration/outbound-alert-settings
After you select the Rest Notification Plugin, there are 3 fields required in the Add/Edit Outbound Instance box
| Item | Description | Example |
| Instance Name | Name of the rest notification plugin created | Opsgenie |
| URL | URL to OpsGenie shim endpoint | http://<IP or Hostname of Photon VM>/endpoint/opsgenie/<api key> |
| Content Type | Content Type – Unless you’re working with a SOAP API (which we aren’t), this will always be application/json |
Finally, navigate to Alerts -> Configuration -> Notification (https://[Your_vROps_URL]/ui/index.action#/alerts/notifications) and add a new rule that targets the adapter you just created.
Teams
Again, I’m not in the business of recreating the wheel, so check out Shane’s blog for info on how to do this.
References
| https://www.virtuallyshane.com/posts/how-to-send-vrealize-operations-alerts-into-microsoft-teams | Fellow vExpert Shane Moore’s article for setting MS Teams forwarding |
| https://github.com/vmw-loginsight/webhook-shims | Official Github site for the webhook-shims project |
| https://blogs.vmware.com/management/2017/01/vrealize-webhooks-infinite-integrations.html | VMware Blog article that got me started on this journey |
| https://vmware.github.io/photon/docs/user-guide/working-with-kickstart/ | VMware Photon documentation |
| https://github.com/vmware/photon/issues/1113 | Github Issue for presenting Kickstart file to Photon over cdrom/floppy |
| https://github.com/vmware/photon/issues/798 | Another Github Issue for presenting Kickstart file to Photon over cdrom/floppy |
Hi Jon,
Excellent article, thanks! But since version 8.4, the Rest notification plugin is deprecated and replaced by the Webhook notification plugin, which can be configured as you wish/need.
You can create templates, input properties and much more. You can use the following variables OOTB :
${ENDPOINT_URL}
${CREATE_TIME}
${UPDATE_TIME}
${CANCEL_TIME}
${ALERT_STATUS}
${ALERT_ID}
${ALERT_DEFINITION}
${ALERT_DEFINITION_DESCRIPTION}
${ALERT_RECOMMENDATIONS}
${RESOURCE_NAME}
${RESOURCE_KIND}
${RESOURCE_ID}
${ADAPTER_KIND}
${RESOURCE_KIND_TYPE}
${ALERT_IMPACT}
${CONTROL_STATE}
${ALERT_CRITICALITY}
${ALERT_TYPE}
${ALERT_SUBTYPE}
${OBJECT_HEALTH_STATE}
${OBJECT_RISK_STATE}
${OBJECT_EFFICIENCY_STATE}
${SYMPTOMS}
${ADAPTER_INSTANCE}
${LINK_TO_ALERT}
Hope it helps! Cheers!
Great info SZ, I just read up on 8.4 (guess I’m a little out of the loop) and it definitely looks a lot more flexible. That said, I’m personally stuck on 8.2 until we upgrade some of our internal applications, but I can’t wait until we can use this!