Branch-Based Development

At Zenoss we do all development, including bug fixes and small maintenance tasks, in a private development branch within the subversion repository. This allows us to independently work-on changes, check-them into the repository for safe-keeping, and then perform code reviews with team members without having to share files or using pastebin style tools (even though we do both at times).

1. Create a branch within your user’s sandbox. In this example, I’ve decided to name the sandbox new-widget to identify what I’m working on. If I were fixing a defect, I’d use the defect number from the defect tracking system. Create the branch by copying the trunk folder to the sandbox branch. In Subversion this is a low-overhead operation and doesn’t actually copy files.
svn copy http://dev.zenoss.org/svn/trunk http://dev.zenoss.org/svn/sandboxen/cgibbons/new-widget -m " * Copying trunk to sandbox branch."

2. Switch your working directory to use the new sandbox branch. You can do this either from the command-line or using Eclipse. From the command line, you’d do the following:
cd $HOME/zenoss/core
svn switch http://dev.zenoss.org/svn/sandboxen/cgibbons/new-widget
From within Eclipse, secondary-click on the core project and choose Switch to Another Branch/Tag/Revision… option from the Team menu. On the dialog that appears, enter in the sandbox URL. After switching, your Eclipse will show the new location next to the core project item.
   

Once your development environment has been switched, you can make changes and commit to the Subversion repository as desired. If you’re unsure if you are in the right branch, you can always use the svn info command to see which directory is being used.

Merging Branches

Once you have completed changes in a branch and have had them reviewed by a peer, it is time to merge them into trunk (or another branch, if using a maintenance release). Merging can be tricky, but a consistent process can make it much easier to handle.

1. Change your working directory to a checked-out and clean version of the branch you want to merge into. For example, I keep a $HOME/zenoss/clean-trunk directory that I never make changes to, except for merging.
2. Determine the base working revision of your working branch. There are a variety of ways to do this, but one of the best is to view the revision log graph within the Trac system directly. For example, for the branch discussed above we can browse to http://dev.zenoss.org/trac/log/sandboxen/cgibbons/new-widget/ and see that revision 15513 is the base.
3. Perform a dry-run on the merge to get a general idea of what the changes into the branch will be. You should see your expected changes, plus any conflicts from changes to the other branch while you have been working in the sandbox branch.
svn merge --dry-run --revision 15513:HEAD http://dev.zenoss.org/svn/sandboxen/cgibbons/new-widget .

4. If the merge results look satisfactory, rerun the command without the dry-run argument.
5. Look at the final merge results using svn status and svn diff, and once you’re ready, issue an svn commit.

Multiple Branches and Zenoss Configuration

As you switch between branches you will often render your Zenoss configuration useless.  Resetting your database after each branch switch is usually a good practice, and being able to quickly recreate any test data you may need makes this process less painful.

After switching a branch, my process is usually the following:

1. Shutdown zenoss and restart only zeo.
zenoss stop
zeoctl start
2. Run the zenwipe script from the inst source directory.
$HOME/zenoss/inst/zenwipe.sh --no-prompt
3. Run zenmigrate to install any database changes available within the current branch.

Depending upon the task at hand, I may install additional ZenPacks and add new devices through the command-line if those are needed. Helper scripts, such as install-windows.sh to install of the Windows ZenPacks and create several local test devices in the instance, are useful tools to have for your typical configurations.

Mac OS X Prerequisites

1. Install Eclipse Classic 3.5.x Mac Cocoa 32-bit.

Ubuntu Prerequisites

1. Install the Sun Java JDK. Why? Eclipse is a Java application and requires a full-fledged Java environment to run properly.
sudo apt-get -y install sun-java6-jdk

2. Install Eclipse Classic 3.5.x Linux 32-bit.

Eclipse Configuration

1. In part one, we created a Zenoss root project directory of $HOME/zenoss – use that directory, or the one you created, for the rest of these steps.
2. Launch Eclipse and configure it to use the Zenoss root project directory. Why? Eclipse needs a workspace directory to keep track of configuration settings for a group of related projects. By placing the workspace directory inside of the Zenoss root workspace, we can separate the requirements for a Zenoss workspace from other projects you may be using.
3. Install the PyDev plug-in for Eclipse. Why? PyDev provides Python language support to Eclipse.
   1. Go to the Help Menu and choose Install New Software.
   2. In the Available Software dialog, choose Add and enter in http://pydev.org/updates/ as the Location and close the dialog. Eclipse will return to the Available Software dialog.
   3. Software matching PyDev will appear in the dialog; pick the PyDev option but do not pick PyDev Mylyn Integration. Click Next and install the plug-in.

   

4. Install the Subclipse plug-in. Why? Subclipse allows you to work with the Subversion version control system directly from within Eclipse.
   1. In the Available Software dialog add the Subclipse plug-in update site:

      http://subclipse.tigris.org/update_1.6.x

   2. In the Available Software dialog, choose the Subclipse, Subversion Client Adapter, Subversion JavaHL Native Library Adapter and the Subversion Revision Graph items.
5. Go to the Window menu, choose Open Perspective and then SVN Repository Exploring (you may have to choose Other… to see this option).
6. Choose the New Repository Location button in the SVN Repositories panel. Add the Zenoss SVN site at http://dev.zenoss.org/svn
7. Open the SVN repository and select the trunk folder. Secondary-click the folder and choose the Checkout… option. Be sure to change the Depth option to be Only this item so that we don’t check out any of the sub-folders of trunk just yet (many of the folders within trunk are not needed for development, but we want to keep the directory structure). In the next dialog you will be asked to choose a Project Wizard. Open the Pydev tree item and select the Pydev Project option.
8. Create the project in the Pydev Project dialog:
   1. Use core as the project name and use the default location. This should create a core project at $HOME/zenoss/core.
   2. Make sure the Python project type is selected.
   3. Select 2.4 as the Python Grammar version.
   4. Uncheck the Create default ‘src’ folder... option.
   5. Click the Click here to configure an interpreter not listed… option in order to add the python interpreter built by the Zenoss installation scripts.
      1. In the Preferences Dialog, choose the Interpreter – Python item underneath Pydev and select the New… button to add a new Python interpreter.
      2. Browse to the $ZENHOME/bin directory and choose the python2.4 executable from that directory.
      3. Name the interpreter python-2.4, zenoss-python or some other variant.

      After the new Python interpreter has been added, return to the Pydev Project dialog and choose that interpreter from the list and then click Finish to create the project.

   6. Update the core folder from the command-line SVN client so you can selectively pull the sub-folders of the core project and not all of them:
cd $HOME/zenoss/core
svn update Products

   7. Move the Products directory the installation checked out and create a symbolic link to the one updated above. Why? This allows the Products source tree to be worked on from Eclipse and then also used by the Zenoss run-time.
cd $ZENHOME
mv Products Products.old
ln -s $HOME/zenoss/core/Products Products
   8. Return to Eclipse and choose the Refresh option from the File menu so that Eclipse notices the updated directory and builds necessary dependencies.
   9. Secondary-click on the core project folder in Eclipse and choose Properties. Choose the PyDev – PYTHONPATH item and add source folders so PyDev can reference the project from within itself.
      1. In the Source Folders tab choose the Add source folder button and pick the root core folder from the provided list.
      2. In the External Libraries tab choose the Add source folder button and choose the $ZENHOME/lib/python directory.

   At this point, your Eclipse project should allow you to navigate between dependencies within the Zenoss project. You can also simultaneously switch between using the Team feature within Eclipse to update and manage Subversion or do so using the command-line svn client.

   Next…

   In Part 3 of this series, we’ll discuss how to manage day-to-day activities such as creating sandbox branches for changes and dealing with multiple versions are done from within the same environment.

I develop Zenoss primarily on Mac OS X, with some work done on Ubuntu when necessary. I normally use Eclipse as my development editor, although I’ll often just use vim when doing quick changes or bug fixes. In either case, careful setup of the Zenoss environment is key to being able to productively work with both the new development version and with the shipping versions that require maintenance.

The environment you get by default when you check out the source version of Zenoss from the source repository, or from a source tarball, is not necessarily setup in the best way to develop productively using tools like Eclipse.

The rest of this post will document how both my Mac OS X and Ubuntu environments are initially configured so that a working source Zenoss installation is realized.

Mac OS X Prerequisites

These prerequisite instructions assume Mac OS X 10.5 Leopard; 10.6 Snow Leopard will not be able to compile Zenoss’s third-party dependencies, so an additional work-around is required for that platform until Zenoss moves to Python 2.6.

1. Install Xcode. Why? Installs the GNU C/C++ compiler and other command-line development tools needed to build dependencies used by Zenoss.
2. Install Universal Subversion 1.6.x from CollabNet Community.Why? Leopard only comes with Subversion 1.4.x. This version is not compatible with the Subclipse plug-in for Eclipse that will be used later. In order to be able to use both the command-line and Eclipse Subversion versions simultaneously on the same checked our source, the release of subversion should match. This installation will place the Subversion binaries in /opt/subversion and should automatically add it to your PATH.
3. Install MySQL Community Edition 5.1 32-bit. Why? Zenoss needs MySQL for storage of event data.

Ubuntu Prerequisites

These prerequisite instructions assume Ubuntu 9.04 32-bit Desktop Edition. Installing the server edition or one with different options may require additional prerequisites to be installed.

1. Install Subversion. Why? Working with the Zenoss product in source form requires Subversion to access the source repositories (it is also possible to build directly from a source tarball, but this is not discussed here).
   sudo apt-get -y install subversion

2. Install MySQL. Why? Zenoss needs MySQL for storage of event data.
   sudo apt-get -y install mysql-client mysql-server libmysqlclient15-dev

3. Install additional development environment tools. Why? Zenoss third-party dependencies require several binaries to be built from source.
   sudo apt-get -y install patch make vim gcc g++ autoconf
4. Install SNMP support. Why? Zenoss requires SNMP libraries for monitoring, and having a local SNMP agent is useful for testing.
sudo apt-get -y install libsnmp-base snmp snmpd
5. Install Liberation TrueType fonts. Why? Graphs generated by RRDtool will not contain the correct glyphs without this font package.
sudo apt-get -y install ttf-liberation

Environment Configuration

Configuring Eclipse will require determining where you want to work with your Zenoss installation, and installing Eclipse plug-ins to provide the features required for Python and Subversion support.

1. Create your Zenoss root directory:
mkdir $HOME/zenoss

2. Create and run a setup.sh script that will configure needed environment variables for zenoss. This script can be started from your log in profile if desired.
cd $HOME/zenoss
cat > zenoss-config.sh <<EOF
ZENHOME=$HOME/zenoss/zenhome
PYTHONPATH=$ZENHOME/lib/python
PATH=\$ZENHOME/bin:\$PATH
export ZENHOME PYTHONPATH PATH
EOF
chmod +x zenoss-config.sh
. zenoss-config.sh
mkdir $ZENHOME

3. Checkout the Zenoss source installation tree. Why? This tree is used to bootstrap the installation from the source repository and create a running Zenoss installation. We’ll need this before we modify it to fit the needs of our development environment.
svn co http://dev.zenoss.org/svn/trunk/inst inst

4. Run the installation script to checkout, compile, and configure a zenoss environment. If you need to customize your MySQL configuration at all do not use the --no-prompt argument.
cd inst
./install.sh --no-prompt

5. Modify the zensocket file to be setuid root. Why? Some of the Zenoss daemons use a privileged port and making the file owned by root and accessible by the Zenoss user allows the daemons to be run as a non-priviledged user but still use the privileged port.
sudo chown root:`id -g` $ZENHOME/bin/zensocket
sudo chmod 04750 $ZENHOME/bin/zensocket


At this point, you should have running zeo and zope processes and be able to log on to the local Zenoss instance. Your Zenoss root directory should look similar to the following:

Next…

In Part 2 of this series, we’ll download and configure the Eclipse IDE.

I had originally planned on using my Mac Pro and large monitor as my workstation and then just using the Mac OS X built-in Screen Saver tool to connect to my laptop as needed. After a few days I realized this wasn’t going to work that well, so I set about getting an actual workspace just for Zenoss activities.

Luckily I had an extra small computer cart from Anthro that I could use. I wound up using a spare Dell monitor and some newly purchased wireless input devices and then I was in business.

So far, this is working out well. It is making it easier to not be distracted by non-work computer stuff as well, since I don’t have everything personal on my work laptop.

On March 24th I finished my undergraduate degree. I had been working on it for four years non-stop, plus a couple of semesters earlier in life. Finishing was tremendously satisfying, of course, but more importantly it finally lets me have some free-time back. I have only had two weeks off a year for the past four and the lack of relaxation was starting to get to me. Thank goodness for my wife’s support and cycling or otherwise I would have gone insane.

The prime motivator for the undergraduate degree was a pathway to graduate school. That’s still in the cards, but not surprisingly I’d like to take a year or so off from school before returning. Luckily I’m willing to take more time with the graduate program than I was with the undergraduate so hopefully it won’t completely overwhelm my life when it is time.

The other major transition in the last week of March was leaving my job at BMC Software after being there for nearly 10 years. The product I had worked on for the previous 4 years had reached a maturity point to where the company decided to move most of the research and development offshore to India. My architect job remained in the US, but I decided to leave at the end of March when given the choice. I had expected to leave the company in 2008 anyway so the timing worked out well.

Finding my next job was quite an undertaking. Being at a career pinnacle I had a lot of choices and possibilities and I decided to explore as many as I could. In the end, I decided to pick between several job offers that gave me the most difference and challenge from what I had been working on before. Both Microsoft and Zenoss were about as different as I could get and still be working in the software industry. Both would require letting go of skills I had mastered in the previous parts of my career; both were tremendously different organizational from most places I had worked at before.

In the end, I chose Zenoss. A for-profit company using an open-source business model and Python technology sounded like a very appealing challenge. Not only that, but 4 other former co-workers that I highly respect were there, and working with them again made the choice even easier.