Investigating Nutanix Pulse ASUP’s Caught in Email Filter

Nutanix cluster’s have an AutoSupport mechanism, called Pulse, that can be configured to sent back configuration/health/coredump information on a daily basis. This is used by Nutanix Support SRE’s to do proactive support, analyze common configurations, and so on.

This data can either be sent directly to Nutanix over the web, or be relayed through a customer controlled SMTP server. Many customers have message/attachment rules/restrictions, and sometimes Pulse ASUP’s can get snagged, especially with large clusters, and especially when the “ALL” setting on Pulse is configured, which grabs a very high amount of coredump and diagnostic data.

This data is zipped into a .gz and attached to an email, so even if it is chalked full of data, it is usually pretty small, maybe 6-10 MB daily.I recently ran into an issue where the customer’s email system was configured to drop messages with attachments greater than 100 MB, and also attachments that, when uncompressed, were greater than 100MB.

This filter rule, combined with the ALL level was causing ASUP’s to not be delivered.

To diagnose this, you can see the email data that the cluster is doing by checking out the /home/nutanix/data/email/ folder.

Since the process that generates the emails is a shared service, it is possible for more than one CVM to generate emails over the life of a cluster.

To track the most recent data down, try this command allssh ‘ls -lah /home/nutanix/data/email’ —- This will give you an output like this one from my lab.

Below, you can see that 10.1.222.62 is doing the most recent work, and there are both “.sent” files, where are JSON formatted files that contain all of the email info, except for the attachments.

In the attachments directory, you will find the .gz files that are being generated and, in this case, flagged.

This discovery method can be used to pull down files that were never delivered, and validate their contents to see why an email filter might be tripping them up.

nutanix@NTNX-14SM15040014-A-CVM:10.1.222.60:~$ allssh 'ls -lah /home/nutanix/data/email'
Executing ls -lah /home/nutanix/data/email on the cluster
================== 10.1.222.60 =================
total 48K
drwx------.  3 nutanix nutanix  20K Jan  4 01:15 .
drwxr-xr-x. 21 nutanix nutanix 4.0K Oct 29 22:54 ..
drwx------.  2 nutanix nutanix  20K Jan  4 01:15 attachments
================== 10.1.222.61 =================
total 32K
drwx------.  3 nutanix nutanix  12K Jan  3 23:15 .
drwxr-xr-x. 21 nutanix nutanix 4.0K Oct 29 22:50 ..
drwx------.  2 nutanix nutanix  12K Jan  3 23:15 attachments
-rwx------.  1 nutanix nutanix  125 Nov 24 02:02 autosupport.1416816003.174225
================== 10.1.222.62 =================
total 36K
drwx------.  3 nutanix nutanix  12K Jan  5 00:15 .
drwxr-xr-x. 21 nutanix nutanix 4.0K Oct 29 22:56 ..
-rwx------.  1 nutanix nutanix  376 Jan  4 00:03 1420351431.354287.sent
-rwx------.  1 nutanix nutanix  376 Jan  5 00:03 1420437834.801072.sent
drwx------.  2 nutanix nutanix  12K Jan  5 00:15 attachments

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: