The Nuclear Bunny Blog » Jobs

You Don’t Need Flash for Rich Graphs on a Web Page

chadwick — Sun, 11 Apr 2010 21:35:33 +0000

Adobe’s Flash has a lot of uses, but one of the most impressive to me has been the the creation of interactive graphs on a web page. One just has to visit Google Finance to see a great example of this in action; it’s fast, effective and fits seemlessly within the rest of the page.

Many times, however, Flash isn’t an appropriate technology to use. If you’re an open-source product like Zenoss, Flash presents a licensing issue. If you’re targeting mobile platforms like the iPhone, Flash isn’t available. And, sometimes, you may just not like Flash; it does have it’s own security problems and overhead, for example. What’s a web developer to do? Enter HTML5 to the rescue…

HTML5 is not yet an approved standard, but it’s well on its way and surprisingly well-supported by the browser community already. One of the nice new features provided in HTML5 is the canvas element which allows for two-dimensional drawing functionality. Between this new feature and JavaScript, we should be able to create a rich graph display.

To get started, we’ll first need to create an HTML document that has the structure we’re after and a place for our drawing canvas to reside.



  
    Rich Graph Example using HTML5's Canvas Element
  
  
    Rich Graph Example using HTML5's Canvas Element

Next, we’ll need some JavaScript to do the work of creating the actual graph display. First, we’ll need to get a reference to the canvas DOM element and then ask it for it’s 2D drawing context:

    var graph1 = document.getElementById('graph1');
    var context = graph1.getContext('2d');

Next, we’ll need to define the raw data used to display the graph. We’re going to assume a set of CPU processor time measurements with 5-minute measurement intervals, much like you would see in Zenoss or other systems management application.

    // Raw data, such as what might come back from a server query.
    data = [
      { value: 9.9430, time: new Date(2010, 4, 9, 7, 50, 0, 0) },
      { value: 0.0000, time: new Date(2010, 4, 9, 7, 55, 0, 0) },
      { value: 0.0000, time: new Date(2010, 4, 9, 8, 00, 0, 0) },
      { value: 0.0000, time: new Date(2010, 4, 9, 8, 05, 0, 0) },
      { value: 5.5828, time: new Date(2010, 4, 9, 8, 10, 0, 0) },
      { value: 1.0000, time: new Date(2010, 4, 9, 8, 15, 0, 0) },
      { value: 25.3120, time: new Date(2010, 4, 9, 8, 20, 0, 0) },
      { value: 45.8210, time: new Date(2010, 4, 9, 8, 25, 0, 0) },
      { value: 98.3211, time: new Date(2010, 4, 9, 8, 30, 0, 0) },
      { value: 96.1290, time: new Date(2010, 4, 9, 8, 35, 0, 0) },
      { value: 94.9128, time: new Date(2010, 4, 9, 8, 40, 0, 0) },
      { value: 98.3219, time: new Date(2010, 4, 9, 8, 45, 0, 0) },
      { value: 32.4912, time: new Date(2010, 4, 9, 8, 50, 0, 0) },
    ];
 
    // Calculate the maximums in the provided data.
    var maxY = 0.0;
    for (var i = 0; i < data.length; i++) {
      maxY = Math.max(data[i].value, maxY);
    }
    var dataMaxY = Math.ceil(maxY);
    var dataMaxX = data.length;

Now, we have to calculate where we can start drawing on the canvas and how many pixels equal one coordinate in our user space data. The more elaborate legends we’ll want to use on the graph, the more these calculations will matter as we leave space for drawing text around the actual graph.

    // define a font to use for legends
    var fontHeight = 20;
    var font = "12px/20px arial, sans-serif";
 
    var xStart = 0;
    var yStart = fontHeight * 2; // leave room for 2 lines of text at the bottom
    var xEnd = graph1.width;
    var yEnd = graph1.height;
 
    // Determine how many pixel units equal one user space coordinate.
    var xStep = dataMaxX / (xEnd - xStart);
    var yStep = dataMaxY / (yEnd - yStart);
 
    // Create an array of 2D points to represent the user space
    // coordinates of each data item.
    points = new Array(data.length);
    for (var i = 0; i < data.length; i++) {
      var x = i / xStep + xStart;
      var y = data[i].value / yStep + yStart;
      points[i] = { x: x, y: y };
    }

Now we get to the actual drawing of the graph. This is straight-forward 2D drawing that should be familiar to anyone who has done it with other technology.

    // Save the graphics context before we begin mucking with it.
    context.save();
 
    // Create a transform so that the graph portion of the chart can be
    // drawn using a bottom-left origin instead of the default upper-left.
    context.translate(0, yEnd);
    context.scale(1.0, -1.0);
 
    // Draw our path using the appropriate color.
    context.beginPath();
    context.strokeStyle = 'black';
    context.lineWidth = 5;
    context.moveTo(points[0].x, points[0].y);
    for (var i = 1; i < points.length; i++) {
      context.lineTo(points[i].x, points[i].y);
    }
    context.stroke();
    context.restore();

At this stage, we can render a simple line-graph using JavaScript and an HTML5 canvas element. Now about making it rich and interactive? We’ll need to add some mouse event handling. We’ll do something simple here: as the mouse moves along the x-axis of the graph we’ll find the current data point based solely on its x-position. This technique is more useful to an end-user and saves the calculation of finding how close a point is to a line segment.

Mouse events are still rather browser specific, so the calculations presented here to determine the offset of the mouse inside of the canvas are simplified and will not work in all scenarios.

    graph1.addEventListener('mousemove', mouseWatcher, false);
 
    function mouseWatcher(evt) {
      // Get the mouse position relative to the canvas element. This stuff
      // is still gnarly across browsers. Here we're assuming we need to
      // figure out the offset the canvas has in the client area, so our
      // mouse position is really relative to the canvas, not the client area.
      var x = graph1.offsetLeft + evt.clientX;
      var y = graph1.height + graph1.offsetTop - evt.clientY;
 
      // Get the data point at this X position
      var foundPoint = -1;
      var x1 = points[0].x;
      for (var i = 1; i < points.length; i++) {
        var x2 = points[i].x;
        if (x >= x1 && x <= x2) {
          foundPoint = i-1;
          break;
        }
      }
 
      if (foundPoint >= 0) {
        txt = "" + data[foundPoint].value.toFixed(2) + " @ " + data[foundPoint].time.toDateString();
 
        // draw a new information line in the legend
        context.save();
        context.font = font;
        context.clearRect(0, yEnd - fontHeight, xEnd, fontHeight);
        context.fillText(txt, 0, yEnd, xEnd);
        context.restore();
      }
    }

And that’s it. Over your Sunday morning coffee you can write a simple, yet effective, interactive graph display using HTML and JavaScript alone. Clearly, a tremendous amount of work remains to make the graphs beautiful and robust, but luckily a large number of people are already working on libraries to do this work for us.

Internet Explorer will not yet display canvas elements natively, but luckily this problem is being solved for us.

There are other problems, too. Today, for example, the font measurement abilities in the HTML5 specification are lacking.

If you want to see the example in action click here. View the source in your browser to see the final source code to the example. A screenshot of the example is below.

Share and Enjoy:

My Zenoss Development Environment – Part 3

chadwick — Thu, 22 Oct 2009 22:01:01 +0000

In Part 1 of this series we discussed getting an initial Zenoss environment checked out and running on a Mac OS X or Ubuntu system. In Part 2 we discussed how to configure Eclipse to use the Zenoss source. In this part, we’ll discuss how to handle day-to-day operations such as branch management and working with multiple versions.

Branch-Based Development

At Zenoss we do all development, including bug fixes and small maintenance tasks, in a private development branch within the subversion repository. This allows us to independently work-on changes, check-them into the repository for safe-keeping, and then perform code reviews with team members without having to share files or using pastebin style tools (even though we do both at times).

Create a branch within your user’s sandbox. In this example, I’ve decided to name the sandbox new-widget to identify what I’m working on. If I were fixing a defect, I’d use the defect number from the defect tracking system. Create the branch by copying the trunk folder to the sandbox branch. In Subversion this is a low-overhead operation and doesn’t actually copy files.svn copy http://dev.zenoss.org/svn/trunk http://dev.zenoss.org/svn/sandboxen/cgibbons/new-widget -m " * Copying trunk to sandbox branch."
Switch your working directory to use the new sandbox branch. You can do this either from the command-line or using Eclipse. From the command line, you’d do the following:cd $HOME/zenoss/core svn switch http://dev.zenoss.org/svn/sandboxen/cgibbons/new-widgetFrom within Eclipse, secondary-click on the core project and choose Switch to Another Branch/Tag/Revision… option from the Team menu. On the dialog that appears, enter in the sandbox URL. After switching, your Eclipse will show the new location next to the core project item.

Once your development environment has been switched, you can make changes and commit to the Subversion repository as desired. If you’re unsure if you are in the right branch, you can always use the svn info command to see which directory is being used.

Merging Branches

Once you have completed changes in a branch and have had them reviewed by a peer, it is time to merge them into trunk (or another branch, if using a maintenance release). Merging can be tricky, but a consistent process can make it much easier to handle.

Change your working directory to a checked-out and clean version of the branch you want to merge into. For example, I keep a $HOME/zenoss/clean-trunk directory that I never make changes to, except for merging.
Determine the base working revision of your working branch. There are a variety of ways to do this, but one of the best is to view the revision log graph within the Trac system directly. For example, for the branch discussed above we can browse to http://dev.zenoss.org/trac/log/sandboxen/cgibbons/new-widget/ and see that revision 15513 is the base.
Perform a dry-run on the merge to get a general idea of what the changes into the branch will be. You should see your expected changes, plus any conflicts from changes to the other branch while you have been working in the sandbox branch.svn merge --dry-run --revision 15513:HEAD http://dev.zenoss.org/svn/sandboxen/cgibbons/new-widget .
If the merge results look satisfactory, rerun the command without the dry-run argument.
Look at the final merge results using svn status and svn diff, and once you’re ready, issue an svn commit.

Multiple Branches and Zenoss Configuration

As you switch between branches you will often render your Zenoss configuration useless. Resetting your database after each branch switch is usually a good practice, and being able to quickly recreate any test data you may need makes this process less painful.

After switching a branch, my process is usually the following:

Shutdown zenoss and restart only zeo.zenoss stop zeoctl start
Run the zenwipe script from the inst source directory.$HOME/zenoss/inst/zenwipe.sh --no-prompt
Run zenmigrate to install any database changes available within the current branch.

Depending upon the task at hand, I may install additional ZenPacks and add new devices through the command-line if those are needed. Helper scripts, such as install-windows.sh to install of the Windows ZenPacks and create several local test devices in the instance, are useful tools to have for your typical configurations.

Share and Enjoy:

My Zenoss Development Environment – Part 2

chadwick — Thu, 22 Oct 2009 19:26:36 +0000

In Part 1 of this series we discussed getting an initial Zenoss environment checked out and running on a Mac OS X or Ubuntu system. In this part, we’ll discuss configuring Eclipse to use this environment.

Mac OS X Prerequisites

Install Eclipse Classic 3.5.x Mac Cocoa 32-bit.

Ubuntu Prerequisites

Install the Sun Java JDK. Why? Eclipse is a Java application and requires a full-fledged Java environment to run properly.sudo apt-get -y install sun-java6-jdk
Install Eclipse Classic 3.5.x Linux 32-bit.

Eclipse Configuration

In part one, we created a Zenoss root project directory of $HOME/zenoss – use that directory, or the one you created, for the rest of these steps.
Launch Eclipse and configure it to use the Zenoss root project directory. Why? Eclipse needs a workspace directory to keep track of configuration settings for a group of related projects. By placing the workspace directory inside of the Zenoss root workspace, we can separate the requirements for a Zenoss workspace from other projects you may be using.
Install the PyDev plug-in for Eclipse. Why? PyDev provides Python language support to Eclipse.
1. Go to the Help Menu and choose Install New Software.
2. In the Available Software dialog, choose Add and enter in http://pydev.org/updates/ as the Location and close the dialog. Eclipse will return to the Available Software dialog.
3. Software matching PyDev will appear in the dialog; pick the PyDev option but do not pick PyDev Mylyn Integration. Click Next and install the plug-in.
Install the Subclipse plug-in. Why? Subclipse allows you to work with the Subversion version control system directly from within Eclipse.
1. In the Available Software dialog add the Subclipse plug-in update site: http://subclipse.tigris.org/update_1.6.x
2. In the Available Software dialog, choose the Subclipse, Subversion Client Adapter, Subversion JavaHL Native Library Adapter and the Subversion Revision Graph items.
Go to the Window menu, choose Open Perspective and then SVN Repository Exploring (you may have to choose Other… to see this option).
Choose the New Repository Location button in the SVN Repositories panel. Add the Zenoss SVN site at http://dev.zenoss.org/svn
Open the SVN repository and select the trunk folder. Secondary-click the folder and choose the Checkout… option. Be sure to change the Depth option to be Only this item so that we don’t check out any of the sub-folders of trunk just yet (many of the folders within trunk are not needed for development, but we want to keep the directory structure). In the next dialog you will be asked to choose a Project Wizard. Open the Pydev tree item and select the Pydev Project option.
Create the project in the Pydev Project dialog:
1. Use core as the project name and use the default location. This should create a core project at $HOME/zenoss/core.
2. Make sure the Python project type is selected.
3. Select 2.4 as the Python Grammar version.
4. Uncheck the Create default ‘src’ folder... option.
5. Click the Click here to configure an interpreter not listed… option in order to add the python interpreter built by the Zenoss installation scripts.
  1. In the Preferences Dialog, choose the Interpreter – Python item underneath Pydev and select the New… button to add a new Python interpreter.
  2. Browse to the $ZENHOME/bin directory and choose the python2.4 executable from that directory.
  3. Name the interpreter python-2.4, zenoss-python or some other variant.
6. Update the core folder from the command-line SVN client so you can selectively pull the sub-folders of the core project and not all of them:cd $HOME/zenoss/core svn update Products
7. Move the Products directory the installation checked out and create a symbolic link to the one updated above. Why? This allows the Products source tree to be worked on from Eclipse and then also used by the Zenoss run-time.cd $ZENHOME mv Products Products.old ln -s $HOME/zenoss/core/Products Products
8. Return to Eclipse and choose the Refresh option from the File menu so that Eclipse notices the updated directory and builds necessary dependencies.
9. Secondary-click on the core project folder in Eclipse and choose Properties. Choose the PyDev – PYTHONPATH item and add source folders so PyDev can reference the project from within itself.
  1. In the Source Folders tab choose the Add source folder button and pick the root core folder from the provided list.
  2. In the External Libraries tab choose the Add source folder button and choose the $ZENHOME/lib/python directory.
At this point, your Eclipse project should allow you to navigate between dependencies within the Zenoss project. You can also simultaneously switch between using the Team feature within Eclipse to update and manage Subversion or do so using the command-line svn client.

Next…

In Part 3 of this series, we’ll discuss how to manage day-to-day activities such as creating sandbox branches for changes and dealing with multiple versions are done from within the same environment.

Share and Enjoy:

My Zenoss Development Environment – Part 1

chadwick — Thu, 22 Oct 2009 17:05:46 +0000

Over the past 18 months the developers at Zenoss have used a variety of development environments and methods to productively work with Zenoss, but there are a lot of best practices that have emerged out of this diversity.

I develop Zenoss primarily on Mac OS X, with some work done on Ubuntu when necessary. I normally use Eclipse as my development editor, although I’ll often just use vim when doing quick changes or bug fixes. In either case, careful setup of the Zenoss environment is key to being able to productively work with both the new development version and with the shipping versions that require maintenance.

The environment you get by default when you check out the source version of Zenoss from the source repository, or from a source tarball, is not necessarily setup in the best way to develop productively using tools like Eclipse.

The rest of this post will document how both my Mac OS X and Ubuntu environments are initially configured so that a working source Zenoss installation is realized.

Mac OS X Prerequisites

These prerequisite instructions assume Mac OS X 10.5 Leopard; 10.6 Snow Leopard will not be able to compile Zenoss’s third-party dependencies, so an additional work-around is required for that platform until Zenoss moves to Python 2.6.

Install Xcode. Why? Installs the GNU C/C++ compiler and other command-line development tools needed to build dependencies used by Zenoss.
Install Universal Subversion 1.6.x from CollabNet Community.Why? Leopard only comes with Subversion 1.4.x. This version is not compatible with the Subclipse plug-in for Eclipse that will be used later. In order to be able to use both the command-line and Eclipse Subversion versions simultaneously on the same checked our source, the release of subversion should match. This installation will place the Subversion binaries in /opt/subversion and should automatically add it to your PATH.
Install MySQL Community Edition 5.1 32-bit. Why? Zenoss needs MySQL for storage of event data.

Ubuntu Prerequisites

These prerequisite instructions assume Ubuntu 9.04 32-bit Desktop Edition. Installing the server edition or one with different options may require additional prerequisites to be installed.

Install Subversion. Why? Working with the Zenoss product in source form requires Subversion to access the source repositories (it is also possible to build directly from a source tarball, but this is not discussed here).
sudo apt-get -y install subversion
Install MySQL. Why? Zenoss needs MySQL for storage of event data.
sudo apt-get -y install mysql-client mysql-server libmysqlclient15-dev
Install additional development environment tools. Why? Zenoss third-party dependencies require several binaries to be built from source.
sudo apt-get -y install patch make vim gcc g++ autoconf
Install SNMP support. Why? Zenoss requires SNMP libraries for monitoring, and having a local SNMP agent is useful for testing.sudo apt-get -y install libsnmp-base snmp snmpd
Install Liberation TrueType fonts. Why? Graphs generated by RRDtool will not contain the correct glyphs without this font package.sudo apt-get -y install ttf-liberation

Environment Configuration

Configuring Eclipse will require determining where you want to work with your Zenoss installation, and installing Eclipse plug-ins to provide the features required for Python and Subversion support.

Create your Zenoss root directory:mkdir $HOME/zenoss
Create and run a setup.sh script that will configure needed environment variables for zenoss. This script can be started from your log in profile if desired.cd $HOME/zenoss cat > zenoss-config.sh < ZENHOME=$HOME/zenoss/zenhome PYTHONPATH=$ZENHOME/lib/python PATH=\$ZENHOME/bin:\$PATH export ZENHOME PYTHONPATH PATH EOF chmod +x zenoss-config.sh . zenoss-config.sh mkdir $ZENHOME
Checkout the Zenoss source installation tree. Why? This tree is used to bootstrap the installation from the source repository and create a running Zenoss installation. We’ll need this before we modify it to fit the needs of our development environment.svn co http://dev.zenoss.org/svn/trunk/inst inst
Run the installation script to checkout, compile, and configure a zenoss environment. If you need to customize your MySQL configuration at all do not use the --no-prompt argument.cd inst ./install.sh --no-prompt
Modify the zensocket file to be setuid root. Why? Some of the Zenoss daemons use a privileged port and making the file owned by root and accessible by the Zenoss user allows the daemons to be run as a non-priviledged user but still use the privileged port.sudo chown root:`id -g` $ZENHOME/bin/zensocket sudo chmod 04750 $ZENHOME/bin/zensocket

At this point, you should have running zeo and zope processes and be able to log on to the local Zenoss instance. Your Zenoss root directory should look similar to the following:

Next…

In Part 2 of this series, we’ll download and configure the Eclipse IDE.

Share and Enjoy:

One Year at Zenoss

chadwick — Thu, 09 Apr 2009 02:13:31 +0000

Yesterday marked my one-year anniversary at Zenoss. When I decided to take the job at Zenoss I knew it would be a challenging position, and time has proven this assumption correct. The new (to me, anyway) technology, the remote work environment and the realities of a startup company with limited resources have all added their own challenges to the job.

I found the Zenoss product itself to be more mature than I had expected. While it still has plenty of rough edges, especially in the user interface, it is a rather feature-rich application with a very large user base in both the free community and paid enterprise versions. There is a tremendous amount of work left to do on the application, but with each version we see a significant step forward and an expanded user interest because of the changes.

When I came to Zenoss, my new co-workers and I brought with us a culture of using agile development methodologies and processes to the company. We expected, and were asked by company management, to implement those processes at Zenoss. Over the past year, we have seen the right things happen from this shift in engineering culture:

development time has become more predictable
planning and priority selection take a much more important role than ever before, and not just in engineering (in fact, most importantly not in engineering, but at the executive level)
quality assurance is now a fundamental part of the engineering process, not a luxury for later
code and design quality has improved dramatically as the engineering team begins to understand how much control they really do have over the process

There still remain many challenges, but these are the same ones I have seen with all teams that struggle with the discipline needed for agile methodologies. Effectively breaking down large development tasks into two-week iterations remains a hurdle; I suspect at a fundamental level many of the developers do not fully believe this is possible, which is a long, long discussion. There was also a reluctance, until recently, to fully engage with teammates on performing effective code and design reviews. These will both improve over time as the team matures.

When I first started at Zenoss there were 5 developers in Annapolis, 2 developers in Austin and me , usually working remote in Houston. Since then, the team dynamics have changed so that I and one other developer are remote most of the time, 3 developers are full-time in Austin and 1 in Annapolis. This team dynamic change has made the infrastructure needed to support remote development much more important, but at the same time it has made the remote developers somewhat more isolated since a bit more now happens only in one location. Since I travel to Austin frequently I can avoid the worst of this unavoidable trend, but it will likely always remain a challenge as long as we have remote employees.

On a personal level, working remotely full-time for this long has been much more difficult than I expected. I last did this much remote work in the mid-90s, and it was difficult to do then, as well. On the surface, I spend all of my work hours focused on some problem on the computer, but in reality much of my job is very social and without face time it starts to wear on me very quickly. Frequent visits to Austin help tremendously, but in reality this will likely always be an issue until I relocate.

I’ve encountered almost all of the traditional problems people have while working at home, but especially keeping focus and balancing the time spent between work and home life. Focus is purely a self-discipline issue, but it is remarkable how much of this is provided by the routine of commuting to an office. Maintaining a good work-life balance is also tremendously hard when working at home, unless you dedicate a specific place in the house for work. It took a few months to really learn this lesson but eventually I bought a second desk and made myself an office separate from my normal home office. Before then, I found it way too easy to bring my laptop into the living room and I’d continue to check e-mail and work on problems throughout the evening. Sometimes, I still do, but not at least it is not a mindless occurrence.

Zenoss is a very small company, and naturally we have very limited resources. Of course, today’s economic environment makes this true for almost every single company. But more specifically, I find ourselves working around what effectively is a lack of resources in our IT department. I often have to build my own virtual machines at home, or buy little widgets to help productivity because I am remote. At some level, I don’t mind this, since it’s nice being able to actually work somewhere you can do this, rather than being told you can’t.

After a year, I’m still not a huge fan of the Python programming language, but I’ve come to accept it and I can be productive with it. Working with it feels a whole lot like I stepped back in time 15 years, and saying that is a great way to start an argument, so I’ll just leave it at that.

One great surprise from the past year is that I became the guru for Zenoss’s Windows monitoring technology. We have a custom implementation of MS-RPC that is based upon the Samba 4 source tree. On top of this layer, we have provided our own implementations of the DCOM, WMI, and Windows registry interfaces so we can call these services directly on any Windows device. This is a great feature, as it allows us to communicate directly to Windows devices from our Linux based product, without requiring a Windows device to be running our software just to provide Windows-based connectivity, unlike some other systems management products.

Over the next few months I will be taking on more and more of the implementation of the new Zenoss user interface, which is a great project that should dramatically improve the user acceptance of quality of the application. It also gives me more time to work on user interfaces, which is always an area of software development I enjoy but rarely get a chance to do.

Share and Enjoy:

Secure Windows Monitoring with Zenoss

chadwick — Tue, 07 Apr 2009 23:50:06 +0000

Starting with version 2.3.x, Zenoss can monitor computers running Microsoft Windows with a variety of data collection protocols: SNMP, WMI over DCOM/MS-RPC and Perfmon over MS-RPC.

In Zenoss Core, the status of Windows services and the Windows Event Log are monitored using Windows Management Instrumentation (WMI) queries over the DCOM/MS-RPC protocol. In the implementation of MS-RPC that Zenoss is based upon, authentication credentials are sent to the remote server using the Windows Challenge / Response (NTLM) mechanism. Using this authentication mechanism, the actual password is never sent across the network, but rather the server produces a “challenge” value that the client must calculate using the password rather than sending it across the network.

NTLM authentication is the same mechanism that Windows devices themselves use for client/server communications, such as file sharing and remote administration.

Starting with version 2.3.x, Zenoss Enterprise gathers Perfmon data using the remote Windows registry API over the MS-RPC protocol. This technique is both more efficient and secure than the previous one. The same authentication mechanism used by Zenoss’s WMI library is used here, providing the same level of security.

Prior to version 2.3.x, Zenoss Enterprise used a different mechanism to collect Perfmon data from Windows devices. This mechanism used a utility known as winexe to remotely execute commands on the Windows device (in this case, the typeperf.exe Windows utility). Unfortunately, the winexe utility sends the username and password used for authentication across the network in clear text, providing a less than ideal configuration for security.

Zenoss users monitoring Windows devices should be running version 2.3.3 or newer for the best possible security when communicating with those devices.

Share and Enjoy:

Getting a Native Code Stack Trace from a Zenoss Daemon

chadwick — Thu, 19 Mar 2009 21:35:05 +0000

Zenoss uses the Python programming language for the vast majority of its code, and all of the daemons and commands that run are Python scripts. Several daemons also make use of native code (i.e. code written in languages like C or C++ that must be compiled into object files and organized into libraries) to perform functions such as remote Windows and SNMP connectivity.

Occasionally, one of these daemons crash in these native libraries, and not in the actual Python code. When this happens, the Python interpreter is unable to produce a relatively friendly stack trace that it would for pure Python code. For example, a crash in a Python script would produce something that looks familiar to most programmers:

Traceback (most recent call last):
File "test.py", line 5, in ?
z = y / x
ZeroDivisionError: integer division or modulo by zero

By contrast, if you have a crash inside of a native library you likely would not see anything more than Bus Error or a similar message, and often nothing at all — the daemon process will just exit. For example, here we have a dynamic library written in C with a single function: doit. This function will attempt to access a NULL pointer when called, which results in the following output:

$ python -i
Python 2.4.4 (#1, Feb 23 2009, 09:17:03) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> from ctypes.util import *
>>> lib = CDLL(find_library('test'))
>>> lib.doit()
Bus error

To get a stack trace from the native code, the Python interpreter must be run from the GNU debugger, or gdb. All of the Zenoss daemons share a common architecture in how they are started, so the process for running the daemon from within gdb will be similar no matter which daemon you use.

Determine your ZENHOME directory location by running echo $ZENHOME from the shell prompt. In the remainder of this example, we will assume it is set to /Users/cgibbons/zenoss
Pick the daemon you want to run. In this example, we will use zenwin — the daemon responsible for monitoring the state of services on Windows devices.

Look at the actual daemon script:

$ cat $ZENHOME/bin/zenwin
#! /usr/bin/env bash
#############################################################################
# This program is part of Zenoss Core, an open source monitoring platform.
# Copyright (C) 2007, Zenoss Inc.
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License version 2 as published by
# the Free Software Foundation.
#
# For complete information please visit: http://www.zenoss.com/oss/
#############################################################################
 
. $ZENHOME/bin/zenfunctions
 
PRGHOME=$ZENHOME/Products/ZenWin
PRGNAME=zenwin.py
CFGFILE=$CFGDIR/zenwin.conf
 
generic "$@"

Run gdb in the python interpreter:

$ gdb python
GNU gdb 6.3.50-20050815 (Apple version gdb-962) (Sat Jul 26 08:14:40 UTC 2008)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-apple-darwin"...Reading symbols for shared libraries .... done
 
(gdb)

Set the program arguments. Note how the zenwin script above is used to build the actual argument string:

(gdb) set args /Users/cgibbons/zenoss/Products/ZenWin/zenwin.py --configfile=/Users/cgibbons/zenoss/etc/zenwin.conf run -v10 -c

Finally, run the daemon process within the debugger:

(gdb) run
Starting program: /Users/cgibbons/zenoss/bin/python /Users/cgibbons/zenoss/Products/ZenWin/zenwin.py --configfile=/Users/cgibbons/zenoss/etc/zenwin.conf run -v10 -c

The daemon will then run as if it were started directly from the command-line. Any pdb trace statements will still be activated and you can use pdb commands as expected. But, once a native code crash is detected by the debugger, the gdb prompt will be provided and the gdb where command may be used to view the native code stack trace. For example, if we do this with our previous doit test, we’ll see this output:

>>> lib.doit()
 
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000000
0x93ebe457 in __vfprintf ()
(gdb) where
#0  0x93ebe457 in __vfprintf ()
#1  0x93ef2da7 in vfprintf_l ()
#2  0x93f17fbb in printf ()
#3  0x003b9ffe in doit () at test.c:6
#4  0x0039476d in .LCFI1 () at /Users/cgibbons/src/zenoss/trunk/inst/build/ctypes-1.0.1/source/libffi/src/x86/darwin.S:81
#5  0x00394701 in ffi_call (cif=0xbffff1a8, fn=0x3b9fe6 , rvalue=0xa0414584, avalue=0xbffff120) at /Users/cgibbons/src/zenoss/trunk/inst/build/ctypes-1.0.1/source/libffi/src/x86/ffi_darwin.c:249
#6  0x0038f21e in _CallProc (pProc=0x3b9fe6 , argtuple=0x15a030, flags=4097, argtypes=0x0, restype=0x21b600, checker=0x0) at source/callproc.c:665
#7  0x00389f02 in CFuncPtr_call (self=0x174880, inargs=0x15a030, kwds=0x0) at source/_ctypes.c:3357
#8  0x00007e12 in PyObject_Call (func=0x174880, arg=0x15a030, kw=0x0) at Objects/abstract.c:1795
#9  0x00080dcb in do_call [inlined] () at Python/ceval.c:3776
#10 0x00080dcb in PyEval_EvalFrame (f=0x209960) at Python/ceval.c:3591
#11 0x0008327f in PyEval_EvalCodeEx (co=0x1ac5e0, globals=0x173a50, locals=0x173a50, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2741
#12 0x00083547 in PyEval_EvalCode (co=0xbffff064, globals=0xbffff064, locals=0xbffff064) at Python/ceval.c:484
#13 0x000aae36 in PyRun_InteractiveOneFlags (fp=0xbffff064, filename=0xdb4a6 "", flags=0xbffff818) at Python/pythonrun.c:1287
#14 0x000aaf73 in PyRun_InteractiveLoopFlags (fp=0xa04175e0, filename=0xdb4a6 "", flags=0xbffff818) at Python/pythonrun.c:706
#15 0x000abe29 in PyRun_AnyFileExFlags (fp=0xa04175e0, filename=0xdb4a6 "", closeit=0, flags=0xbffff818) at Python/pythonrun.c:669
#16 0x000b5f8a in Py_Main (argc=0, argv=0xbffff8a4) at Modules/main.c:493
#17 0x00001d0b in _start ()
#18 0x00001c39 in start ()
(gdb)

If the native library was built with debugging symbols a nice programmer-friendly stack trace will be generated like in the above example. Here we can see exactly what line our doit function crashed at. Now the bug should be easy to find and fix, right?

Share and Enjoy:

new MacBook setup

chadwick — Wed, 18 Mar 2009 03:07:37 +0000

I bought another Mac today, a nice 2.4 GHz 13-inch unibody MacBook. I had planned on buying a 17-inch unibody MacBook Pro, and very nearly did, but luckily sanity won out and I remembered how much of a hassle it was to carry around those giant things, even if they are only “only” 6.6 lbs.

I had a 2.0 GHz 13-inch MacBook a couple of years ago when the first Intel-based models came out and I do remember the screen resolution, while not abundant, was more than adequate for browsing, e-mail and even development. And, the unibody model is only 4.5 lbs, so 2 lbs lighter will be a lot nicer to carry around. I also sprung for a spare battery to try and help get a little closer to the awesome battery life of the 17-inch model.

Of course, the obvious question is why buy another laptop when I’ve already got a nice 15-inch MacBook Pro that work provides? The answer there is easy: I don’t want to do anything personal, even development, on the work provided machine.

Now, on to the actual system setup, documented here for posterity.

After account creation, run software update and get all the latest updates installed first.
Create an applecare account with an easy, but secure, password. This way the Apple store geeks can have that account should there need to be any repair work done.
Change the battery lifetime display with Show -> Time.
Secure the screensaver by using System Preferences -> Security -> General and checking the Require password to wake this computer from sleep or screen saver option.
Disable the Front Row remote by using System Preferences -> Security -> General and checking the Disable remote control infrared receiver option.
Disable the Front Row keyboard shortcut by using System Preferences -> Keyboard & Mouse -> Keyboard Shortcuts and disabling the Hide and show Front Row shortcut.
Enable full keyboard shortcuts by checking the All controls option in the bottom of the same Keyboard Shortcuts screen.
Enable the Use secure virtual memory option in System Preferences -> Security -> General.
Encrypt my home directory with System Preferences -> Security -> FileVault.
Install Growl 1.1.4 from http://growl.info/
1. Install the GrowlSafari extra package.
2. Install the HardwareGrowler extra package.
  1. Drag HardwareGrowler.app to /Applications
  2. Disable the HardwareGrowler dock icon by following the instructions at http://growl.info/documentation/hardwaregrowler.php
  3. Add HardwareGrowler to the start at login list by using System Preferences -> Accounts -> Login Items and dragging HardwareGrowler to the list.
3. Enable Growl starting at login with System Preferences -> Growl and enabling the Start Growl at login option.
Remove unused printer drivers by deleting the appropriate folders in /Library/Printers folder (everything but Brother, hp and PPDs in my case).
Install the XcodeTools package from the Installation DVD’s Optional Installs directory.
Drag Xcode to the dock by going to /Developer/Applications and dragging the icon to the dock.
Add Activity Monitor to the dock by going to /Applications/Utilities and dragging the icon to the dock. Seondary-click on the icon and enable Open at Login.
Add Terminal to the dock by going to /Applications/Utilities and dragging the icon to the dock.
1. Change the default Terminal settings by starting Terminal.app, selecting Preferences (Cmd-,) and then changing the “new window with settings” to Pro.
2. Select the Pro scheme in the Settings tab and click default.
3. Choose the Window tab with the Pro scheme selected, click the Background color chooser and set the opacity level to 90%.
4. Change the window size to 80 columns and 36 rows.
Customize vim by creating ~/.vimrc with the following content:
:color elflord :syntax enable :set shiftwidth=4 :set expandtab :set autoindent :set cindent :set enc=utf-8 :set nu :set showmatch :set laststatus=2 :set nocompatible :set gfn=Monaco:h15:a
Enable color highlighting for ls by adding the following lines to /etc/bashrc:
alias ls='ls -CFG' alias dir='ls -FGlas'
Install the Safari 4 beta from http://www.apple.com/safari/download
Install Firefox 3 from http://getfirefox.com/ and drag it to the dock.
Install iStat pro from http://www.islayer.com/apps/istatpro/
Install MySQL 5.1 x86 community edition from http://dev.mysql.com/downloads/mysql/5.1.html be sure to install the StartupItem package as well as the preference pane.
Add MySQL to the shell profile by appending the following to /etc/bashrc:
export PATH=/usr/local/mysql/bin:$PATH
Install EverNote from http://www.evernote.com/
Install DropBox from http://www.getdropbox.com/
Install the Windows Media Components for QuickTime from http://www.microsoft.com/windows/windowsmedia/player/wmcomponents.mspx
Install Twitterrific from http://iconfactory.com/software/twitterrific
Disable automatic synchronization for iPhones and iPods since this won’t be the primary iTunes machine by going to iTunes Preferences and enabling Disable automatic syncing for iPhones and iPods on the Devices tab.
Install the iPhone SDK from http://developer.apple.com/
Party! Or maybe just nap.

Share and Enjoy:

Oddities in Gathering Windows Performance Data

chadwick — Wed, 18 Feb 2009 01:39:08 +0000

At Zenoss we do quite a bit of remote monitoring of computers running Windows. In the Enterprise edition of the product, we collect raw performance counter data using the conventional remote Windows Registry APIs.

We ran into an issue recently with a customer running Windows 2000 where the data from the remote server was being truncated prematurely. Since we implement our own remote API (so we can run natively on Linux and with Python, rather than requiring Windows), there was some immediately concern we ran into a low-level bug in our protocol implementation. Thanks to the release of the Windows Communications Protocols (MCPP) last year we have great detail on how our API layer should function.

Reviewing the MCPP in detail compared to our implementation showed no bugs against the specification, but I did notice some odd behavior. Normally when using the RegQueryValue API you specify a NULL buffer point and a zero-length buffer size so that the call will provide the actual size of the buffer needed. With this particular customer’s server I noticed that this behavior wasn’t behaving as documented in the MCPP.

An error code of ERROR_MORE_DATA was being returned. The MCPP says that when this value is returned the server will populate the size output variable with the actual size in bytes of the needed buffer. In this case, the size was always the same size as the input. After some experimentation I found that if I passed in approximately 64 Kbytes more data the call would finally succeed.

While quite odd behavior, this is actually the documented and expected state in the Win32 API documentation for RegQueryValueEx, but not in the MCPP. Specificially, when using the HKEY_PERFORMANCE_DATA key the ERROR_MORE_DATA behaves differently and the caller has more responsibility in guessing an appropriate buffer size.

The following pseudo-code shows the basic flow for how RegQueryValueEx should be used, either for locally or remote performance data access.

size = 65536 # starting size, probably computed from a previous registry call
params.in.data = params.out.data = buffer(size)
while 1:
    params.in.size = size
    params.out.size = 0
    dcerpc_winreg_QueryValue(params)
    if params.out.result == ERROR_MORE_DATA:
        size = size + 65536 # add another 64 Kbytes of data to the buffer
        params.in.data = params.out.data = buffer(size)
        continue
    break

After fixing that issue I was still left with one oddity. Let’s say, for example, it took 293,500 bytes of data before the RegQueryValueEx call was successful. And yet, the actual amount of returned data would only be 195,000 bytes, or something similar. This behavior seems quite different than on the other Windows operating systems we have tried so far.

This is the first time we’ve tried our data collection against a Windows 2000 server running Exchange locally. Windows 2000 has also been the source of several other key behavior differences in how performance data is returned, so my current speculation is how the server actually determines what data to be returned varies greatly between operating system versions. We normally query the performance counter registry for only a subset of values. It may well be that on Windows 2000 a buffer size large enough to retrieve all performance counters is required, even though once the call is complete it actually used quite a bit less.

Quirky, but another bug gone.

Share and Enjoy:

The Magic of a Quiet Fan

chadwick — Sat, 10 Jan 2009 16:31:06 +0000

A previous post discussed using an Intel D945GCLF2 Atom-based mainboard for a little mini server setup. It’s been working great, except for that damn fan for the memory controller. It was pretty noisy at first, but quickly degraded into a crunching worthless disaster.

The product review comments at newegg described this exact behavior so I wasn’t too surprised. My friend Matt replaced his with a $3 fan immediately and has been happy with it since.

I opted for a slightly more expensive fan that is supposed to run at less than 14 dBA – a Silenx Ixtrema Pro Series. It’s still a 40mm fan with the same depth but I can’t hear it run at all. Product reviews imply this fan won’t stay quiet after about 6 months, so we’ll see if that happens. Ideally I’d just replace the crappy heatsink with a larger one and maybe a heat pipe to the case.

Now the only noise in the system is the WD Raptor drive. I’ve noticed the drive runs at between 50 and 55 degrees C since the case is not actively cool, so that is on the hot side (WD’s MTBF testing was done at 50 degrees). It is way overkill for this box, so I may replace it with a WD Green Power drive, or just give up and do a small SSD.

But hey, I’ve got over a month of cool looking graphs out of Zenoss now, such as my router network traffic:

Zenoss Network Performance Graph

Share and Enjoy: