1 System Monitor for Network of Workstations

System Monitor for Network of WorkstationsIsaac Cheng and Amy HsuComputer Science Division, University of California, Berkeley **E-mail: {isaacc, ashsu}@ This work is done as a term project for CS 262, taught by Professor Eric Brewer, Fall 1996.An experimental information management system is developed to monitor system data for a network of workstations, a heterogeneous computing environment consisting of more than a hundred machines. The information system is designed to help system administration and research activities by managing system configu-rations and keeping track of system load his-tory. It uses Informix database to store all the data and provide general SQL access. We integrated the information system with the existing system monitor to support system load visualization.1. IntroductionThis paper describes a prototype of an information management system developed to monitor system data for a network of workstations. The network of workstations (NOW) is a heterogeneous computing environment consisting of more than a hundred machines, and our information system is designed to help both system administration and research activities in NOW. Having interviewed three researchers, a system administrator, and a programmer, we set up the following goals to satisfy some of the information needs of the NOW community:• Manage the inventory and system configurations• Keep track of system load history •Support automated diagnosisTo support the first goal, we use a database to organize and store information like the number and speeds of processors, amount of memory, devices, operating system versions and patches, etc. To support the second goal, we integrated our prototype with the existing system monitor, Netwreak, to allow users to visualize load history of both individual hosts and clusters. Due to severe time constraint, our current implementation was unable to achieve the last goal. However, we believe that to support simple diagnosis like detection of load balancing failure, one can extend our prototype by writing a simple Perl script without any major change in the information infrastructure.The following two sections this paper describe the design and implementation of the information system. The “Usage Examples”section contains several examples to show what the information system currently can do. The next two sections of the paper talk about related works and possible future improvements.Finally, we will sum up what we have achieved in this project. The appendix of this paper is designed to be a brief technical reference of our information system for system administrators.2. MotivationThe existing NOW system monitor can only display current usage information. And once the data is being displayed, it is thrown away that there is no way of retrieving any load history.By having a database as the data repository, not only can we store the past load information, but we can also query the data. For example, we can query the database for a set of data from which the system monitor can produce a time series graph that would help both thevisualization and analysis of system load. Having a database also opens an avenue for automatic system diagnosis and automated problem fixing. The database can perform analysis on the system data that it had and execute some code to automatically fix the problems it identifies.People often ask questions such as which machine has what and how many workstations have a certain configuration. These questions cannot be answered easily if there is no organized bookkeeping of the system configurations. By having a database that will store all system configuration information for each machine, those questions can simply be answered by sending queries to the database. It will also simplify system administrators’ job of keeping track the equipment.3. DesignAs mentioned in the introduction, our information system is designed to perform two major tasks: manage system configurations and keep track of load history. To collect system configuration data, we wrote a Perl script that runs system commands on each of the machines in NOW. To collect load information, we wrote a Perl package that receives data from Netwreak. When a user requests Netwreak to plot some load data against time, the Perl package sends the appropriate time series to Netwreak. We store both the system configurations and the load history into an Informix database server. We will describe the software architecture in the implementation section. In the rest of this section, we will discuss some of the decisions that we made in the design of our information system.3.1 Database EngineA major design decision is the use of a commercial relational database engine instead of simpler choices, such as ASCII flat files or hash tables in Perl. The main reason for this choice is that we need to support ad hoc queries. When we interviewed our prospective users, it seemed to be very difficult for them to say in advance what kind of queries they would like to perform on our information system. A full-blown relational database engine with a general SQL interface provides the flexibility that they need. Hash tables in Perl require the programmer to hard-code what the search keys are. Using Perl hash tables, it would be hard to create secondary indices, and it would be rather inconvenient to perform joins among tables, which is required by many ad hoc queries. For example, finding the number of processors in each machine requires joining the machine table with the processor table in the database (see “Usage Examples” for more details). A flat-file database is slow for searching especially when we have large amount of data. It does not satisfy the long-term goals of the NOW community. 3.2 Programming LanguageAnother design decision is the choice of programming language. We use Perl for a number of reasons. First, Perl is a good for rapid prototyping. Compared with C, Perl generally allows programmers to say less and tell more. Second, Perl is extremely portable across different platforms. This is a significant advantage for a highly heterogeneous environment. Third, the existing monitor, Netwreak, was written in Perl/Tk, and using Perl made our integration simpler. If Netwreak was written as a Java applet, Java would be a better choice for our information system.3.3 Logical Database DesignA significant portion of the design of the information system is the logical database design. Our database schema is based on the relational data model. Our user’s needs do not seem to require any object-oriented extension to the database. We ensure that the schema is normalized (at least to the third level) by designing it using the popular entity-relationship (E-R) approach. Figure 1 is a simplified E-R diagram with all attributes and unnamed relationships omitted.The functions of most entities are obvious from their names. The entity DynamicData is used to temporarily store system load data as they arrive. As they are quantized into regular time slices, they are moved to AverageDynamicData,which is used for generating time series.In both DynamicData and AverageDynamicData, the data records are self-describing. For example, a record might say: “I am talking about the CPU utilization of the host u0 at noon on 11/7/1996, and my value is 3.0.”Since there is no dedicated field for each type of load data, the information system can be easily extended to include any type of system load,such as network traffic, without making any change to the database schema.4. Implementation4.1 Collection of System Configuration DataThe collection of system configuration data is done by a Perl script. The script runs system commands (sysinfo, psrinfo, and installpatch) on each host. We use system commands instead of directly interacting with the underlying network protocol for two reasons. First, our approach is much more portable across different network protocols. Second, without having to deal with the complexity of the underlying network protocol, the approach enhances rapidprototyping.Figure 1: A simplified entity-relationship diagramCurrently, the device information is automatically detected by sysinfo. While the automatic detection algorithm is not perfectly reliable, it is more cost effective than employing human operators keying in the information for every device installed on each host. Yet, the device information serves at least as a good approximation of reality.The current implementation of the data collection script resides on a single host, runs remote shells on each machine sequentially, and collects the data into flat files. We manually loaded the data into our Informix server. However, we believe that the data collection can be parallelized so that each host runs the system commands concurrently, and the script can directly interact with Informix in the same way that we implemented the collection of system load data (see the next section). It is also possible to change the script to run it as a corn job and automatically send e-mail to system administrators to notify them of any change in system configuration. The inelegance of the current implementation was merely due to our lack of experience with Informix.4.2 Collection of System Load DataThe collection of system load data is done by a Perl package that works with the existing system monitor, Netwreak. It receives data asynchronously from Netwreak, and send back a time series to Netwreak when it needs to display load graph to the user. The user only sees the user-interface of Netwreak, and the database manages the load history behind the scene. Figure 2 illustrates the software architecture.Netwreak spawns GLUnix jobs on each host, and it displays the load data on the screen as usual. Once a piece of data is displayed, it sends to the database, which timestamps it and stores it in the DynamicData table. There is a daemon process running in the background that read data from that table, quantize and average them, and store them into another table. Once the data are process, they are removed from the DynamicData table. When a user requests a plot from Netwreak, the database retrieves a time series for the appropriate time interval, host, and type of load data for plot. The granularity or the length of each time slice can be adjusted by modifying a text file, namedtimeSlice. Currently, users can onlyrequest plots for the load data of thecurrent day. However, the systemcan be enhanced to support plots inthe further past by changing thefunction convertTimeForInfomix inupdataDynamicData.Since Informix currently does not support SQL embedding in Perl,we worked around it using Perlbackticks to invoke dbaccess, whichis not an optimized programminginterface. This introduces the extraoverhead of spawning a temporaryprocess per SQL access.5. Usage ExamplesIn the previous sections, we have seen how the informationsystem was designed and built. In this section, we will see what we can do with it. It will start by showing a few examples of how one might request system configuration information from the database. Afterwards,we Figure 2: Collection of system load datawill see a couple examples of how one might visualize system load over a period of time.5.1 System Configuration QueriesA simple way to construct queries to find out system configuration information is to create a text file that contains an SQL query, and then invoke dbaccess with the file as the second argument. The command dbaccess is an SQL interface to Informix. The first argument is the name of the database, in our case, “monitor.”The following shows an example where a user logged on to the Informix account on NOW to query about processor information.One can query memory information the same way.The user can also ask for device information. Note that if one does not know the exact name of the device, as in the following example, one can perform a simple pattern matching using the “like” operator in SQL.operating systems.5.2 System LoadVisualizationUsers obtain plots of system load history via the graphical user-interface of Netwreak.Clicking a the icon of a host with the right mouse button pops up a menu. By choosing “Plot,” a dialog box that allows the user toadjust various parameters of the graph. After clicking “Ok,” the user closes the dialog box and creates a window that contains the plot he/she requested (Figure 3).If the user right-click a cluster icon instead of a host icon and follow the above steps, he/she will obtain a plot of the aggregated system load of the cluster instead of individual hosts (Figure 4).There are other ways to obtain the same plots. For example, a user can right-click the background of Netwreak and select “Plot.”Then, he/she can manually type in the name of any host or cluster depending what he/she wants to visualize.6. Related WorkAnderson, Goto, and Patterson [1]developed a system diagnostic console for the Networks of Workstations (NOW) to gather systemusage datasuch as load, CPU, and memory on a single or a cluster of machines.Currently, the data is being discarded once it is being collected and displayed to the diagnosticconsole. We extended the functionality of the NOW system monitor by setting up a database which will store previous load information into the database so that the data can be queried for future reference.UNIX Host Administration(UHA) system [2]also implemented aFigure 3: Load graph of a hostdata model in a database that tracks hardware and software configuration, users, and accounts. The system is developed to simplify common administrative tasks and improve the efficiency and integrity of processes that track users, accounts, and equipment in a scientific computing environment. But although UHA also stores system configuration information into the database, it does not keep any load information in its database.Walters [3] also presents a solution of keeping track of system configuration with a method which allows the workstations themselves to log their current hardware configuration with central host every time they boot. The method is implemented in Perl using sysinfo to extract information in a platform independent manner and using UNIX syslog to provide a simple client-server database capability.7. Future Works7.1 Tracking Network TrafficThe design of the information system has the flexibility to allow programmers to add additional types of load data without making any change to the database schema. A useful extension of this kind is to add network traffic as a new type of load data. All one needs to do is to add a few lines of Perl code into Netwreak.7.2 Updating System Configuration DataCurrently, our program collects system configuration through a single point and stores all information into the database at once. Although under this model, the system configuration cannot be updated, we can easily modify the Perl script to collect and update system configuration data in a cron job for each machine locally. By running the modified script periodically, information in the database can be updated to match the new configuration. And whenever the database detects a change in configuration for a machine, we can have the program generating an e-mail message which warns the system administrator about the change. Then later if the machine crashes, it may help the system administrator to determine the cause of the crash.7.3 Diagnostic SupportThe current database schema allows one to support detection of load-balancing failure by writing a Perl script that queries the database on load data and detects pathological cases where only a few machines in a large cluster have exceptionally heavy load. The database schema can be extended to store the uptime of each host, and one can write a Perl script that sends system administrators e-mail when some hosts are down.7.4 EfficiencyThe current prototype is rather slow for two major reasons:• There is no secondary index in the database.• Everything is written in Perl and communicates with Informix using a work-around interface.One can create secondary indices to speed up data retrieval. However, secondary indices may slow down the process of adding and deleting records. For frequently referenced fields in relatively static tables, such as the hostName field in table Machine, building a secondary index may be the right choice. Judicious creation of indices can sometimes improve performance by two orders of magnitude, but it usually requires some experimentation.Rewriting Perl code in C usually gains some efficiency. In our case, it would have the additional benefit that Informix supports an ESQL API for C but not Perl. However, C is a little harder to port than Perl as porting C code to a new platform requires recompilation.7.5 User InterfaceThe current user-interface for querying system configuration is SQL. Although it is a popular standard, it requires the users to know the moderately complex syntax of the query language. There are two possible improvements. A possible way is to build a graphical user-interface that looks likePowerBuilder. Another way is to use a natural-language query interface, such as English Wizard.7.6 Improve AvailabilityAt the moment, our database suffers from single-point failure since all data is currently stored on one server. To solve this problem, we can setup the Informix backup server which will continue to run in the case when master server goes down. By using the built-in Informix backup server, the problem of concurrency is taken care of automatically.Another alternative is to have multiple Informix servers storing multiple copies of data.A possible hierarchical structure is for each machine to have its own server storing its own data. Then the data of several lower level servers will be combined and stored in a single higher level server. This way, the data is being replicated across the different level. Under this model, the system can tolerate not only one but several simultaneous failures. But concurrency can be a problem.7.7 Saving Disk SpaceRight now, the load data will be averaged once for a certain period of time, and then that piece of data will reside in the database permanently. But usually, people care less about the data as time goes on, and so it may be wise to trade finer granularity of data with more space preservation in the database. And this can be done by quantizing the load data at different interval with different resolution so the older data will be stored with coarser granularity. 8. ConclusionWe have developed a prototype of an information management system to monitor system data for a network of workstations. It manages an inventory and system configurations of computers, and keeps track of system load history. It supports general SQL interface for querying system configuration information. Finally, we integrated our prototype with the existing monitor system for plotting system loads.9. AcknowledgmentsWe would like to thank Albert Goto for explaining Netwreak to us and spending long hours debugging with us during the integration of our information system with Netwreak. We appreciate the effort by Kim Keeton for installing an Informix server on NOW. We are grateful to Eric Fraser for telling us the important system commands, keeping the database server running, and letting us know the needs of system administrators. Many of our initial project ideas were due to Professor David Culler and Eric Anderson. Finally, we want to thank Douglas Ghormley for telling us the information needs of the GLUnix research group.10. References[1] E. Anderson, A. Goto, D. Patterson, "A System Diagnostic Console for Networks of Computers." LISA 1996.[2] G. Thomas, et. al., "UNIX Host Administration in a Heterogeneous Distributed Computing Environment." LISA 1996.[3] R. Walters, "Tracking Hardware Configurations in a Heterogeneous Network with syslogd." LISA 1995.11. Appendix11.1 Database Schema (Logical Structure)The fields and field types of each table can be looked up in the following SQL statements.11.2 Getting information about tables and columns in the databaseTo find out all the tables and columns in the monitor database, go to /usr/now/informix/now_database and type command allTables . allTables will list all the columns within the database along with the corresponding table name by querying the database with an SQL command.11.3 Adding and deleteing column from an existing tableTo add or delete columns from a table after its creation, use the SQL alter table command.Adding a new column:alter table "table name" add "column name" "type of the new column";e.g., alter table OperatingSystem add releaseLevel char(15);Updating the table:update"table name" set"column name" = "expression" (where "statement");e.g., update Machine set osversion = '5.5'; This sets all the osversion columns in table Machine to 5.5.e.g., update Machine set osversion = '5.5' where hostName = ‘’;This only sets the osversion columns to 5.5 for .Deleting a column:alter table "table name" drop "column name";e.g., alter table OperatingSystem drop releaseLevel;。