[Previous Page ] [Main Documentation Page ] [Next Page]

Hitchhiker's Guide to MidWatch

1. Overview

MidWatch is an automated build engine for NT platforms. It is designed with two important premises:
  1. The Microsoft Visual Studio’s .dsp files will govern the product being built.
  2. Everything required to build the product will be version controlled in Rational’s ClearCase.
Note: these design assumptions and the initial draft of this document occured long before any thought of open sourceing the project. MidWatch could be converted for any SCM use

The importance of the .dsp based build is that the NT developer working in his local environment (or "sandbox") will almost certainly be using the Visual Studio IDE. And since .dsp files as opposed to make files will govern his local builds, an engine should be constructed that uses these files. This way, the local sandboxes will be in synchronization with the master build machine.

At the time of this writing, MidWatch has matured to a complexity that is making it hard to extend some of the features that I dream about. It is the hope of the author that MidWatch be rewritten in C++.

1.1 Order of Operation

MidWatch begins first by running as a normal application in the user’s account. There is a timer object set to expire every 3 seconds. At each interval, the Scheduled Events Table contained in an Access database is checked for events to start.

1.1.1. The Scheduler

If an event in this table is scheduled for anytime before the current time, the Scheduler fires a build event. The Scheduler first reads information from the Product Launch Record (also a table in the database), and uses the PLR to find the rest of the required information in the Aengel Product Definition file (The .apd File) to build an internal Product Definition Record (PDR). The PLR is handed to the Engine and the build starts.

1.1.2. The Engine

The Engine is able to instantiate a Visual Studio IDE and using the order defined in the Project List table, loads a project, builds a project, and if there are no errors returned, loads the next project. The Engine’s first mission is to complete the assembly of the PDR, after receiving the PLR from the scheduler, the Engine calls the Software Configuration Management Controller (SCMC) to start a ClearCase snap shot view called the Construction Zone view (CZ). There are two build modes available, the first is a Target Labeled Build and the second is a Tip build. More detail is in 2.2 The Scheduler. Depending on the requested mode, the Engine uses a dynamic view (called the inspection view) to quickly change the config spec for the build, this view is used to initially load the .apd file and could also be used to make labels prior to the compilation phase of the build. The Engine uses the Project List to determine the order of the build. The Project list is loaded from a text file stored in ClearCase in a path defined in the .apd file. The Engine also reports to the webwriter the current status of the build.

2. User Interface

As indicated before, The UI is weaker than the developer is presently comfortable with. However, there is a UI, and it should be described. As MidWatch improves, this section should be updated, as the UI is the biggest weakness of MidWatch.

2.1. The Main Dialog

This is the main dialog of MidWatch, It consists mostly of a color coded log window and a pull down menu. Note the version number is in the title bar. The log is updated in real time. Sometimes the real time log updates can interfere with the Build Meister from reading the log. When the Build Meister wants to scroll up and the last line is hidden from view, the log will scroll down as soon as a new line is written to the log.

The button in the bottom right hand corner freezes the auto scroll, so that the Build Meister can concentrate on a particular portion of the log. When this toggle button is pushed in, new lines are still written to the log as they happen, but the log does not auto-scroll to the bottom. The Recover Busted Build command button opens a dialog with a combo box. The Build Meister selects the event number that was broken, and the build will restart from the project that was busted. The selection of the event number is not very user friendly, but it works for now. Note that the Build Meister should first have the problem resolved. If there was a compilation error in a particular file, and a new version of the file was not checked in and updated in the CZ View, the build will fail again. The Build Meister in making the corrections could move labels or what ever is appropriate to accomplish the fix.

The Clean Build toggle button causes the build to removed all derived object prior to compilation. There is also places in the .adp file that can describe certain directories to clean prior to the build commencing. If this button is pushed in, the build will be cleaned.

Each 24 hour period is stored in a new log file.

2.2. The Scheduler

Perhaps the most important of the dialogs is the scheduler. I also consider it the weakest part of MidWatch at the time of this writing. From here multiple build events can be scheduled and the re-occurrence of the event can be programmed.

2.2.1. The right pane is a list of the events that MidWatch is programmed to perform in order of the event time. An event can be selected for editing either by the right pane or by the data control on the bottom left.

2.2.2. The text box on the upper left, while not labeled is the date and time that the build is scheduled to start. This is editable by the Build Meister. The time to the right of the text box shows the system time and is updated every 250 ms. The Build Now command button set the date in the text box to either today’s date (if the hour of the build is prior to now) or yesterday’s date (if the hour of the build is later than now).

2.2.3. The event can be set for a Target Label Build or a Tip Build by the selection of the radio buttons in the event box. A Target Label Build will cause the config spec of the dynamic inspection view to be edited with the target label defined in the PLR. (see 3.2.9 TargetLabel) This view will then be used to drop an incremented build label defined in the PLR. (see 3.2.7 & 3.2.8) If set for a tip build, the config spec is unaltered, and the LATEST of the branch is built. In a tip build the Build Meister can optionaly cause MidWatch to Use and Increment a Build Number, and to Drop a label. Note that the label drop depends on the build number being incremented. These boxes are automatically checked in a Target Label Build. To use and increment a build number without dropping a label means that the .rc files will be stamped with version information prior to the compilation, but that no record of this version number will be applied to ClearCase.

This can speed up a build by a good half an hour or so, but there will be no labels in Clearcase to match the binary results with. When a code stream is far from a release date, and the discipline is loose, the purpose of the build is just to ensure the health of the build and provide QA with recent build to test. In this loose discipline, label drops do not provide a lot of value, as most of these build results have relatively small value and are quickly discarded when more recent builds are available. Once a code stream or product approaches a release date, it is typical to start dropping labels and move to the tight discipline of target labeled builds.

2.2.4. The text box in the upper right corner of the event box is a quick and dirty control used by the developer, there are three valid states for this text box, and it should be replaced with a group of radio buttons. It is not editable from this interface, but the meaning is:

The Build Meister can edit these values directly in the db.mdb (see 3.1.9 Status)

2.2.5. The Product combo box is used to select a particular product defined in tbl_ProductLaunchRecords. The current event can be switched to build any product for which there is a record in the table.

2.2.6. The Re-Occurs checkbox causes this event to be rescheduled at the conclusion of the event, merely select the days that you wish this to build. Once the Scheduler dialog is closed, the any pending event scheduled prior to the current time will fire, one at a time starting with the event scheduled furthest in the past.

2.3. Project Launch Record Dialog

Use this dialog rather than editing the data directly in the the db.mdb. There is no validation of the data, so you could edit the db.mbd table directly with the same result, but the dialog box make the editing of the config spec easier. Detailed information of these files is in the Tables and Data Files section. Refer to 3.2 The Product Launch Record

2.4. Table Parser Control

This requires some explanation. Firstly, you have read that MidWatch is controlled by tables in db.mdb. This includes the Project List which among other things controls the order of the build and which project to build for a particular product. This sort of information must be version controlled. The projects to build and their dependencies are as much a part of the code as the code itself. When developing MidWatch I could have simply checked the db.mdb into ClearCase anytime there was a change to the project list configuration, but it causes two problems. First, it is inefficient for ClearCase to store .mdb files. To ClearCase these are binary objects so the size of the delta of these files is equal to the file. Also, there is no way to do a difference between versions. The second problem is that I would store information into ClearCase that simply does not belong there. MidWatch would not be scalable at all. So, MidWatch is able to reduce an MS Access table into simple ASCII using MidWatch’s own format. The ASCII can be read by MidWatch and produce a table again. The ASCII can in fact be diffed by ClearCase and we have successfully merged these files! Ideally, every part of maintaining an MidWatch build should be handled through the UI, well, here is another thing that isn’t. Assume that it is your task to add a new project to the product you build with MidWatch. All that needs to be done is add a new record into tbl_ProjectList right? Well, that is sort of true except that the change has to be checked into ClearCase, or your new record will be wiped out when the next build starts and MidWatch dutifully reads the text file from ClearCase back into the table in the db.mdb.

Enter the Table Parser Control, (which is in the menu under Configure>Development Utilities>Table Parser because I never intended this to be a permanent part of the UI) From here you can select any of the tables in db.mdb and click the down arrow to write the table to text. The text file is located in MidWatch’s "home" directory. You still must checkout the current text file, copy the new version to the view, and check back in. Conversely, a text file read out of ClearCase can be copied to the "home" directory and be read into the db.mdb where it is more convenient to analyze, and or edit. The Renumber Build Order command button causes the build order field in the tbl_ProjectList to be renumbered in increments of 10 so that you could add a new project between two consecutively numbered projects. This is just like the old days of BASIC isn’t it? What fun, some things never change.

2.5. Making Tables From .dsp Files

There exists a utility to initally read a .dsw file into the tbl_ProjectList It is normally used only once, at the beginning of converting a project from a .dsw controlled build to a project list controlled build. The chief reason for the project list is that MidWatch can load individual projects and maintain a tighter feedback loop with the build. It would have been possible to perform the build using only a .dsw file, but there are additional fields available in the project list. The tbl_ProjectList generated from this function will still have to be stored in text format and checked into ClearCase. This was described in 2.4, the Table Parser Control.

This "control" is available from Configure> Development Utilities> Make tables from DSW. This will open a file browser, navigate to your .dsw file and select it. The tbl_ProjectList generated is not complete. This control simply automates the generation of the skeleton of a tbl_ProjectList. All information in a .dsw file is tranfered to the table, but there is information required in the table that is not found in the .dsw.

This will be (for the present) entered using Access 97 by hand.

Access 97 is of course becomming obsolete. It is likely that you no longer have it installed in your environment. If you convert db.mdb up to Access 2000, MidWatch will not be able to use it. However, you could create a blank .mdb with Access 2000, and then link to the tables in the db.mdb while it is still in Access 97. This will give you a bug free view into the tables and you can manipulate them with Access 2000.

[Previous Page ] [Main Documentation Page ] [Next Page]