BBWin Xymon Client: Tips and Tricks

Recently I rolled out BBWin (v0.13) to over 20 new Windows servers for one of my clients and I ran into a few snags that I thought might be helpful to discuss here.

Read on to learn about some BBWin and Xymon tips and tricks I picked up along the way.

BBWin Crashes on ‘stats.dll’

The first thing I noticed is that the servers would occasionally go completely purple on me. It appeared to be random (it wasn’t just one particular server) and simply restarting the Xymon client within Services would fix the issue.

After it happened a couple times across different servers I checked the Application logs and noticed that each time it was crashing on ‘stats.dll’, which provides the .ports test within Xymon.

For now my fix has just been to comment out stats.dll in BBWin.cfg:

   <load name="msgs" value="msgs.dll"/>
   <load name="procs" value="procs.dll"/>
   <!-- <load name="stats" value="stats.dll"/> -->
   <load name="svcs" value="svcs.dll"/>
   <load name="uptime" value="uptime.dll"/>

My client wasn’t using the .ports test so it was a quick and easy fix for now. And of course I had to drop .ports for each Windows server I updated because it went purple:

  ../bin/xymon 127.0.0.1 "drop <servername> ports"

At this point my best guess is that on newer versions of Windows server (and even Windows 10 in my lab) there are enough ports listed that sometimes it’s too much for BBWin to process (???).

First Messages After BBWin Install Fails (Central Mode)

Another issue I ran into is that after installing BBWin one server in particular it wouldn’t populate within Xymon.

After letting it run a couple times I checked \BBWin\tmp\ and I saw that the msg.<servername>.txt file was 102MB!

Turns out the client had some extra security logs enabled on that Windows server and BBWin was trying to send it all up to the Xymon server.

Since they have so many servers I run Xymon in Central Mode (see below), and because the initial message was failing, BBWin never got a copy of clientlocal.cfg.

As a workaround I added some extra ignore lines to the ‘win32’ section of client-local.cfg on the Xymon server itself and waited for it to propagate to another Windows server. Then I copied the new clientlocal.cfg from that server to \BBWin\tmp\ on the server with the huge Security log file.

The next time it ran, the msg.<servername>.txt file dropped to 52KB…

Mind you, everything I exclude in client-local.cfg on the Xymon server are success messages, so that was over 100MB of non-error messages.

So if you install BBWin and it doesn’t populate in the Xymon server, check the size of your msg.<servername>.txt file.

Local vs Central Mode

Speaking of … what is Central Mode?

The short version is that configuring Central Mode in BBWin.cfg simply tells it to get its log and file monitoring settings from the Xymon server instead of using whatever is configured in BBWin.cfg itself.

This is super handy when you have a lot of servers to maintain.

In the example above with the 102MB security log file, when I first started working with the client the average msg.<servername>.txt file was 3-5MB in size, which is a lot of data to keep sending to the Xymon server every 5 minutes.

After examining the servers, pretty much every cause was a chatty Security event log.

So I took a look at the .msgs column for a few of the servers, found a few really common ‘success’ messages, and added them to client-local.cfg on the Xymon server, like this:

   [win32]
   eventlog:security:10240
   ignore Special privileges assigned to new logon
   ignore The description for Event ID
   ignore A user's local group membership was enumerated
   ignore An attempt was made to query the existence of a blank password
   ignore Special privileges assigned to new logon

After a couple polls, the average size of the msg.<servername>.txt files across the servers dropped to around 100KB.

And that was for all servers because they were all set to Central Mode. A single change on the Xymon server saved 3-5MB of data for every 5-minute poll.

That’s the power of Central Mode!

How To Easily Ask BBWin to ‘run right now’

This one is just a simple trick you can use when you’re testing something and you want to kick of BBWin right now instead of waiting the full five minutes.

Simply open BBWin.cfg in your favorite editor (Notepad anyone?) and save it again making no changes – this will trigger BBWin to scan the server right now, including any external scripts.

This is great when you’re trying to debug an external script, or even for figuring out the best settings for Central Mode.

How To Create External Scripts

And since I just mentioned it … what exactly is an external script?

Quite simply, BBWin (and other Xymon clients) let you run custom scripts that report on server activity that isn’t monitored with the default BBWin client.

The key is to have the script run and then report the results to \BBWin\tmp\<testname>.

For example, a file named mytest would show up as column mytest in Xymon, along with whatever data was placed in the \BBWin\tmp\mytest file.

Probably the two most common languages for Windows external scripts are VB Script and PowerShell. Either one can run a command or query WMI and parse the output.

One common test I’ve seen using an external script is to run a query to see if any Windows Updates are pending on the server and/or if a reboot is required.

Once the script is working as intended, you can have it run automatically either by adding it to the ‘externals’ section of BBWin.cfg or by creating a Task in the Task Scheduler.

My personal preference is to use BBWin.cfg whenever possible. The only time I use a Task is when the script needs full Admin rights to run (for example it is querying WMI on another server).

Your Turn

Now it’s your turn. Have you run into any caveats with BBWin, or do you have a cool tip or trick to share?

If so please add it to the comments below!

You can also contact me if you have any questions about my experience with BBWin.