vCenter Server Appliance 5.5 Syslog Collector Rotation

VMware vCenter Server has been shipping with an optional syslog collector for quite some time now, and has proven to be easy way to retain non-persistent logs from ESXi hosts.

The Windows Server variant of vCenter Server ships with a basic syslog collector, that can be optionally integrated into vCenter as a service, allowing you to do the following:

  • Pull syslog data into support bundles for better troubleshooting.
  • See what hosts are actively logging data to the collector.
  • See active configurations for the service, such as maximum log size, rotation, and log directory.

The Appliance variant of vCenter Server, based on SLES 11, DOES ship with a built in syslog collector, which is pre-installed and enabled; however, this variant does NOT have feature parity with the Windows version.

Instead, the Appliance ships with a variant of syslog-ng, and can not be integrated into vCenter as a service, meaning you lose the plugin visibility and functionality described above.

To pile on to this, it also does not come with Rotation pre-configured. This presents a large problem for those depending on vCenter Syslog to maintain copies of their non-persistent logs.

This is especially nasty, as the partition, /dev/sdb2, is in the middle of a non-dedicated disk, making it difficult to expand, and ships only at 20 GB.

To work around this limitation, you will need to configure rotation, which is quite easy, as logrotate.d is already configured as an active cronjob.

To enable basic log rotation, SSH to vCenter Server Appliance, Navigate to /etc/logrotate.d, and modify “syslog” to include the following statements:

/var/log/remote/*/*/* {
daily
compress
delaycompress
rotate 14
postrotate
/etc/init.d/syslog-collector reload > /dev/null
endscript
}

Advertisements

Recommended Syslog Configurations for ESXi

Syslog is an important service in any enterprise architecture, and can even be business critical in some applications (think security, time sensitive logging, etc). To that end, VMware has recently made a play in the big data / log collection game with Log Insight (which is awesome, and if you haven’t tried it, GO DOWNLOAD THE EVAL NOW).

Regardless of what Syslog collector you are using (vCenter Integrated, Log Insight, Splunk, etc), there are a few non-default things that you will need to do to ensure persistent and reliable logging from ESXi 5.x hosts.

First off, note that depending on your patch level, if any of the following happen, the syslog service may not reconnect to your syslog collector, and logs may be missed (sounds important, eh?)

  • The network connection has been interrupted.
  • The remote host has closed the connection.
  • A firewall is preventing the logs from being sent.
  • The remote syslog server is not available.

To remediate these issues, check out the two articles below, and patch accordingly.

vSphere ESXi 5 and Remote Syslog: Make Sure You Patch/Update

VMware ESXi 5.x host stops sending syslogs to remote server

Lastly, I want to highlight VMware’s recommendation in the first article:

Once you have updated all your hosts to the versions listed below, we recommend using TCP or SSL. Without TCP, log message loss due to buffer overflows in network devices and network stacks may happen without detection.

This also sounds important, and it is, as the default log transport in ESXi is UDP (i.e. if you just type the IP or host name, logging will default to udp://loghost.company.com:portNumber).

To remediate this, simply add the tcp:// (or SSL) prefix before your log FQDN or IP Address, so that your syslog.global.loghost entry will be as follows:

tcp://loghost.company.com:514

OR

ssl://loghost.company.com:1514

Final Note: You can configure multiple syslog servers by making syslog.global.loghost comma-delimited.