Class-Based Management Of HP-UX Filesets

 Introduction

One of the most repetitive and boring jobs faced by a systems administrator is the loading of software onto a large number of client machines. HP-UX patches, which are distributed as individual "filesets" or "bundles" for Software Distributor (SDU) on HP-UX 10.X and Update on HP-UX 9.X, are an especially large chore due to the large numbers of filesets required per system (typically 60). While this document will concentrate mainly on resolving the patch tedium, the processes and some of the tools used are applicable to all HP-UX software packaged as filesets for the standard installation tools.

Defining An Automated Process For Managing Filesets and Patches

As with any repetitive process, the installation of HP-UX filesets onto client systems is a prime candidate for automation. The power and flexibility of UNIX and its software development tools, including shell scripts, allows for custom development of this type of solution. Before diving into the development process, it is important to completely understand the manual process used by system administrators for dealing with fileset installation. Automating a poorly understood or chaotic process only results in automated chaos.

I divide the patch process into three main areas: acquisition, distribution, and installation. Patches are usually obtained from HP in the form of a shar file, that is, a self-extracting ASCII shell script that contains the patch information. This shar file is expanded into the .text file, containing the description of the patch, and either a .updt file (9.X) or a .depot file (10.X). The shar file may be obtained from HP in a number of ways: EMAIL, FTP, or via the HP web page.

Once the patch information is extracted, it must be distributed to the client systems that are to install it. Patches and other filesets may be placed in a network distribution area: a depot (10.X) or a netdist area (9.X). Making the distribution areas available across multiple sites (replication) and keeping them current with a "master" distribution area are issues to be dealt with.

Once the filesets are available in the network distribution areas, the client systems must install them. This usually involves a manual, interactive "pull" operation performed by the system administrator at the client console. This particular approach is time-consuming and single-threaded (i.e. it can only be done on one client at a time.) Automating the client installation of filesets is very desirable.

Acquiring Patches And Filesets From Hewlett-Packard

Obtaining operating system (OS) and application filesets is fairly easy as long as the customer has support. In this case, the filesets are distributed on a CD-ROM, which may be unloaded into the network distribution areas. Patches, however, are another matter.

The main problem for system administrators is selecting the appropriate patches for their environment from the large number available. This involves manually searching the HP patch lists for potentially useful patches. There are a number of different approaches to deciding what patches to install, ranging from conservative: "install patches only for encountered problems", to liberal: "install patches for everything. " Most customers select an approach that is somewhere in the middle of the two.

Automating Distribution Of Filesets And Patches

Once the patches are obtained from HP, they are typically sorted based on the architecture (700 or 800) that they are applied to. There is no real reason to use separate distribution areas for the different architectures, but it can help save some confusion. Some patches are generated for both architectures, others are architecture specific. The patch filesets are then loaded into a network distribution area using updist (9.X) or swcopy (10.X).

 Automating Installation Of HP-UX Filesets

Loading patches to a client system from a network distribution area that is across a wide area network (WAN) link can be a slow process. A better solution is to have a "local" or "nearly-local" distribution area to use that is not located on the "other side" of the link. Allowing automated propagation of distribution areas from a set of master distribution areas can allow efficient access to local distribution areas by client systems installing the filesets.

The HP-UX 9.X fileset installation tools were the predecessors to the HP-UX 10.X installation tools, and as one might expect, the 10.X tools offer more features and "intelligence" than their younger cousins. When installing patches on 9.X, the patches must be properly sequenced, since there is no dependency information kept in the patch filesets. There is a special "patch bundle" installer for 9.X, but this requires special bundles be created (which may be obtained from the response center or on "extension media" CD-ROMs).

The single most inconvenient aspect of installing patches on HP-UX 9.X occurs if the patch reboots the system. At this point, the administrator must wait for the reboot, log in, and run the installation tool again, repeating the steps over and over. This process is necessary because selecting a large number of patches from a network distribution area at one time does not guarantee proper installation order for the patches.

Installing patches on 10.X with the SDU tools is much nicer, since the patch dependency information is contained in the filesets. This means that selecting a large number of patches at one sitting will install the patches in the proper order. There are still circumstances when sequencing the installation of the patches may be necessary, but they appear to be rare. Patch bundles are still available for 10.X, but no special tool is needed for their installation.

Design Goals For An Automated Fileset Process

Based on the previous discussion, a set of design goals for an automated patch (fileset) installation tool set would be:

• Provide a single point of administration for filesets within a work-group.
   
• Support both HP-UX 9.X and 10.X installation tools, as well as multi-vendor installation tools.
   
• Allow proper sequencing of fileset installation.
   
• Flexible location of network distribution areas.
   
• Traceability of installation actions on the client systems.
   
• Eliminate need for interaction with administrator during installations.

Flow Diagram For Automated Patch Acquisition

A flow diagram of the automated patch acquisition phase includes an automated acquisition script which uses patch catalogs and user-supplied keywords to select patches from the HP distribution point. Once the patches are acquired, the shar files are placed in a central work area. This automated script does not exist yet, but the patch catalog filters that will eventually drive it do. This does not affect later stages of the automated process, all they care about is that the shar file ends up in the work area.

 Flow Diagram For Automated Fileset Distribution And Replication

Once the patch shar files are placed in the work area, they are operated on by a set of scripts. These scripts extract the patch files (.text, .updt, or .depot), sort the patches by architecture, and copy them to architecture specific network distribution areas. The patches are now available for installation on client machines. At this point, both standard OS and application filesets and patches are on equal footing: they are both in network distribution areas, ready for any client system that may need them.

Once the network distribution areas exist, an automated script is available to propagate filesets from distribution area to distribution area. This script obtains an index of the remote (master) distribution area and compares it to the local (slave) distribution area's contents. Any filesets present in the master distribution area that are not in the slave distribution area are automatically copied. This script may be run on a regular basis by entering it in the distribution system's cron table.

 Class-Based Fileset Management

The remaining task is to decide which patches belong on which systems and to perform the installation. This process is based on dividing client systems into "classes" that are based on differences in the client's software configuration. Sets of patches may be associated with each unique class and an automated installation script will select and install all client patches based on that system's class membership(s).

Automated Fileset Installation Process

The automated installation script reads a centralized class database (a flat, ASCII file) to find an entry for the client hostname that is running the installation script. The database entry contains a list of class names associated with the client machine. Each class name is used to generate a "fileset list file" that contains the filesets to install from the network distribution area for that class. Both the class database and the fileset list files are located in a well-known, central location which is accessible by all client systems. Alternately, the class database and fileset list files may be copied and installed on each client.

As an example, assume that the client system was an HP 9000 Series 700 running HP-UX 10.01. The database entry for this client shows that it belongs to two classes: PATCHES and USERAPPS.

gandolf                  PATCHES USERAPPS

frodo                     PATCHES USERAPPS CADAPPS

These class names are used to construct two fileset list files, using a naming convention that contains the OS revision and architecture for the client system: FILESETS-1001-PATCHES-700 and FILESETS-1001-USERAPPS-700. These two fileset list files contain the filesets that need to be installed for those classes. The naming convention for a fileset list file is:

FILESETS-<OS_REVISION>-< CLASS_NAME>-<ARCHITECTURE>

In addition to the filesets to be installed, the fileset list files contain control information used by the client installation scripts. This information includes directives that control the location of the network distribution areas, error handing behavior, and the name of the installation tool to use. The fileset names in the fileset list files support the full naming conventions used by update and swinstall:

bundle[.product[.subproduct[.fileset]][,version].

The class-based fileset installation tool, load_filesets, uses the fileset list files to determine the order that the filesets are loaded (unless the directive .SELECT_ALL is present in the fileset list file.) Filesets in the fileset list file that are already installed on the system are ignored (unless the .REINSTALL) directive is encountered. This allows the installation database maintained by update (in the /system directory) and swinstall (in the /var/adm/sw/product directory) to determine whether or not to install a particular fileset.

Installing the load_filesets script in the system start-up process allows the script to be run every time the system reboots. And, by checking the filesets for prior installation, the load_filesets script can maintain proper order across system reboots. On an HP-UX 9.X system, this means that the filesets or patches in the fileset list files are installed in order even if a patch causes the system to reboot, eliminating the need for an administrator to be present.

Running the installation script after reboot, with the class database and fileset list files accessed via NFS in a central location allows single point administration of multiple client systems. The fileset list files are checked at reboot time only if the fileset list file is newer than 24 hours old, so that minimal boot-time impact occurs if there are no filesets to install. In addition to these behaviors, the script is extensible for multiple operating system types and installation tools.

Flow Diagram For The Automated Fileset Installation Tool

A block diagram of the load_filesets tool environment depicts two main entities, the patch server and the patch client. The installation script is the only portion of the system that is normally resident on the client machine (the script and the associated databases may be installed locally on the client, but the advantage of single point administration is lost.) The HOST-CLASSES database, the fileset list files, and usually the patch depots, all reside on the patch server. The database and fileset list files are accessed via NFS from the client machine.

 An Automated Patch Management Tool Kit

In our previous discussion, we covered an automated patch/filesets management process, class-based methods for managing filesets, and design goals and flow diagrams for an automated process. In the case of patches, there are extra qualification and handling steps that are not necessary for product filesets (unpacking, sorting, etc.) in order to get them into the network distribution areas. In the next section, we will describe the structure and components of an automated toolkit that addresses the three main areas of our defined process for both patches and product filesets.

Automated Patch/Fileset Tool kit Structure

All tools and data in the toolkit are relocated relative to a "Patch Base Directory" that may be placed anywhere in the file system hierarchy. The toolkit should be placed in a location that contains sufficient disk space for both the toolkit and any network distribution areas that may be located within the toolkit directories.

The load_filesets script allows the addition of operating system specific functions to the startup and shutdown process. To do this, the "eval" shell command is used to generate functions that may be auto-loaded, using variable names to generate the autoloaded function names. For example, if the variable OS_SUFFIX contained "HP-UX" then the command line "eval 'Startup'${OS_SUFFIX}" would generate and execute a shell command "StartupHP-UX". This type of behavior is supplied prior to script execution and at the termination of execution.

Code Fragments From Automated Patch Toolkit

To provide tool extensibility, the load_filesets script uses the shell "eval" command to generate function names for running the tool, checking for fileset installation, and handling tool error returns. For example, if the variable TOOL_NAME contained "Swinstall" then the "eval 'RETURN=${ run'${TOOL_NAME}'Tool )'" command would generate and execute a shell command of the form "RETURN=$( runSwinstallTool ) ". The routine that checks the tool return value is run in a similar fashion

Each separate script in the automated toolkit needs a common set of data definitions. Either the data can be defined in each individual script (a support nightmare) or there is a need for a central "database" of definitions. The centralized approach is what was selected.

While a central location for the toolkit configuration database is desirable, choosing a flexible way of locating the information becomes an issue. The directory from which the script was run is available from the shell as the ${0} variable, so locating the common data in that directory is a natural choice. This, coupled with the shell's ability to "source" a file (read it in and execute it as a set of commands or data definitions) allowed implementing the desired functionality.

Automated Patch Toolkit Summary

The author continues to work on the automated patch/fileset toolkit as time allows. Current plans include adding a graphical user interface and further automating the patch acquisition phase of the process. Feedback and requests for added functionality are welcome.