An Architecture for Building Scalable, Web-based Management Services

An Architecture for Building Scalable, Web-basedManagement ServicesNikolaos AnerousisAT&T Labs Research180 Park Avenue, Bldg. 103Florham Park, NJ 07932-0971tel: (973) 360-8767, fax: (973) 360-8871e-mail: nikos@/~nikos/AbstractWe present the architecture of Marvel,a distributed computing environment for building scal-able management services using intelligent agents and the world-wide web.Marvel is basedon an information model that generates computed views of management information and adistributed computing model that makes these views available to a variety of client applica-puted views consist of monitoring,control and event views of information collect-ed from network elements and subsequently aggregated using a series of spatial and temporalfilters.Marvel does not replace existing element management agents but rather builds on topof them a hierarchy of servers that generate computed views and present them to client appli-cations in a number of formats,including Java-enriched web pages.It uses a distributed per-sistent store to reduce the cost associated with centralized network management systems andmobile agent technology to a)support thin clients by uploading the necessary code to accessMarvel services and b)extend its functionality dynamically by downloading code that incor-porates new objects and services.A prototype implementation in Java is presented togetherwith results from itsfirst application on a residential broadband access system using cablemodems.Keywords:Network management,web-based management,distributed computing,information aggregation, Marvel.1.IntroductionIn the recent years,one can observe a trend in the network management market that shifts the em-phasis from the underlying technology that the network management system employs to the func-tionality that the system provides.Finally people have come to agree that the choice of a network management protocol(e.g.SNMP,CMIP or other)is only a minor detail in the overall management framework.Rather,emphasis is now given on the system’s ease of integration with other compo-nents,expandability,availability,ease of use(web-based management systems are increasing steadily their presence),and reduced maintenance costs(including the time to train people,install and configure software,etc.).A common pitfall of most management applications today is that they still operate at a very low level,many times exposing the details of communication with network elements using a low level management protocol.1In addition,networks are continuously shifting the traditional cost structure from hardware to sup-port.Given this shift in investment,the emphasis in network management has moved from manag-ing machines to managing services.Service-level management is necessary for providing high levels of service for mission-critical applications.Service management requires the definition of higher-level management services that monitor control the coordination of a large number of net-work elements.This paper explores how the combination of web and distributed computing tech-nologies with an information model capable of expressing complex relationships between network elements can form the basis for designing and deploying complex management services.1.1The Emergence of Web TechnologiesJava and the world wide web have received significant attention recently as a means of providing low cost and easy to use management services.Customer network management is an example of an application that has benefited significantly from the web-based management model.Tradition-ally,customer network management services have required the installation at the customer pre-mises of high-end network management applications such as HP Openview to access fairly low level management information using SNMP or logfiles that are copied and examined off-line.By incorporating this information into web pages,service providers have a more cost effective way of providing management services to their customers.Moreover,customers have come to value the easy-to-use interface and the absence of a need to maintain a separate service management system. Another very successful application of web technology has been element management.An increas-ing amount of networking equipment is shipped today with an integrated web server that allows to monitor and control the equipment from a web browser.In general,web technology has a number of advantages that make it attractive for network man-agement applications:•The user of the management services is no longer required to install and configure expensive management clients.Rather,the responsibility for converting managed objects and services toa graphical form lies with the provider of these services.•Services can be accessed by low cost client applications such as web browsers,that are widely available,and well understood by a large number of users.These clients are agnostic with re-spect to the structure of management information and nature of management services provid-ed by the web-based system;all the necessary code and state are transferred from the provider of the management services to the client.•The information provided in a web-based form is usually at a much higher level compared to traditional management services such as the ones provided by SNMP.For example,perfor-mance information can be formatted into easy-to-understand graphs rather than provided in the form of a logfile that is processed off line.Similarly,control interfaces can be tailored fora particular application by providing a friendlier user interface,an integrated help system,etc.In reality,it is the provider of the management services that determines the interface that the user sees.This has many advantages:it prevents users from misinterpreting management in-formation(the provider ensures that all information is presented in a user-friendly and under-standable form),and allows for bundling of other services(such as provisioning,customer care, etc.).2•Web-based information can be made to represent an up-to-date view of network state informa-tion, thereby improving the interactivity of the user with the network management system. Web technology requires a provider to define both the level of abstraction at which management information is presented and the graphical user interface to interact with these services.By hiding the details of the underlying management technology under an easy to use and access interface,net-work management becomes a useful commodity rather than a privilege of the network operator. Today,the bundling of management capabilities with a service is becoming a product differentiator of increasing importance.The importance of web technologies in the future is justified by the speed with which vendors of network management products are supplementing web-based interfaces.When enhanced with a web-based interface,many conventional management applications allow network managers to ac-cess a managed system from low-end clients and troubleshoot problems while at home or travel-ling.Further,many routine data mining and performance reporting tasks that use charts and tables can be better reproduced in a web-based form.On the other hand,web technology has its limitations and cannot be used yet to replace every man-agement application,particularly the high-end network operation centers(NOCs).There are sev-eral reasons contributing to this fact:1.Management systems for large networks require a large display area to project maps of differ-ent logical views of the network.Even if the web-based interface is replicated,there is little support in current web technologies for coordinating individual displays with each other.2.Web interfaces require the client to retrieve large amounts of code and state information froma server.As a result,the rate at which information can be consumed by the client’s display islimited compared to the capabilities of a traditional centralized network management system.3.Web-based interfaces that are limited to HTTP interactions have a very restricted control inter-face with the server (hypertext links, post operations, CGI scripts, etc.)4.Java-enriched interfaces are affected by the performance problems of Java and the reliabilityof its graphical user interface.5.The Java security model does not allow clients(applets)to use some local resources(e.g.thefile system,network connections to other servers,etc.),and as a result significant bandwidth and computing resources are wasted in transferring information from and to the server.6.Web interfaces require that an additional layer of processing be introduced in the managementarchitecture to convert management information into a web-based form.This may impact the freshness of information and the response time of the management system to control actions.7.Finally,the power behind web-based management can sometimes be its most significant limi-tation:since management services and the way management information is presented at the client are defined at the server,there is no capability for the client to further process this infor-mation to create,for example,more detailed management views or correlate the information with other prior knowledge.1.2Information ModelsScalability is a major concern in the design and deployment of a network management system. There are no widely established methods today for dealing with large numbers of network ele-3ments.Managing large enterprise networks requires powerful abstractions that capture the essen-tials of the state of the network rather than the details.Most approaches for reducing state and event information in commercially available network management(NM)platforms are ad-hoc and usu-ally customized for a particular management problem or network.As networks grow larger and in-tegrate an ever increasing number of components and services,the existence of a scalable network management architecture becomes critical.We believe that the key behind offering truly scalable management services will be a new management information model that expresses naturally the most common forms of aggregating network state.This model must further be application-inde-pendent, to ensure that it can be applied to a variety of management problems.The new information model will allow managers to“customize”the way that the network appears to them for monitoring and control purposes,and will be key element behind management systems that scale as networks grow in size.The model that we propose provides high level constructs to hide elements behind groups,define managed objects that represent computed views of manage-ment information and provide higher level monitoring and control primitives,that allow the man-ager to interact with the network management system at the service level as opposed to the element level that has been the common practice so far.1.3MarvelThe Marvel project at AT&T Labs Research is an attempt to investigate the limitations of web tech-nology in network management and propose an architecture that can both scale and rival conven-tional management systems in terms of performance and expandability.Marvel(which stands for Management Aggregation and Visualization Environment)is trying to address the following is-sues:•Expand the web-based views of management information well beyond customer network management and network element management applications.Marvel uses a framework for in-formation aggregation that allows the dynamic construction of arbitrary views of management information that can also be beneficial to network operators.•Propose a distributed computing model for accessing management information that can be easily incorporated into web clients.Standards such as CORBA and Java remote method invo-cation(RMI)[SUN97]are now widely accepted for network management applications.Mar-vel uses a distributed computing environment to reduce the cost related to deploying centralized hardware and software and at the same time hide all details of this distribution from its users.•Use concepts from intelligent agent and mobile code technology to enhance the functionality of the web-based interface and improve the availability and maintainability of management services.Marvel is not limited to the web-based interaction model.Rather,any type of client applications can be built around its distributed computing services,from traditional applications using their own graphical interface,to server-driven management interfaces.The work presented here,however, emphasizes the features of Marvel that are central to providing web-based management services. This paper is organized as follows:Sections2and3present the architecture of Marvel;Section4 describes the current implementation work.Experiences from using the Marvel system on a pro-45duction network can be found in Section 5.Section 6presents related work in the field,and Section 7 our conclusions and directions for further study.rmation ModelManagement information in a large network today is usually distributed between the MIBs of net-work elements and,as a consequence,represents small aspects of the configuration or operation of those elements rather than of the network as a whole.Management applications today require ac-cess to a much higher level of management information and services.Our framework proposes an object-oriented information model where the value of an object’s attribute can be defined as an ar-bitrary computation over other attribute values.The latter can be information residing inside ele-ment management agents or other computed attributes.The emphasis of our model is in providing a technology-independent specification framework in which these computations can be ing this model,the network manager can define new managed objects that represent computed views of management puted views can represent a summary of lower level con-figuration and performance information,or a more detailed view of a particular management pa-rameter.Objects representing computed views of management information can be regarded as implement-ing a “middleware management services”layer (Figure 1).This layer extracts information from managed elements using a standards-based management protocol,processes this information ac-cording to the specification of the computed view and makes it available to management applica-tions through a distributed computing environment.Objects within the management middleware layer can follow the SNMP or OSI structure for management information (in which case they are accessed using the corresponding management protocol),or a proprietary format that exports man-agement services to a legacy distributed computing environment such as CORBA or Java.Our model specifies only the way that an attribute value is computed from a set of components and,for this reason,it can be used as an extension to standards-based models for structuring management information such as SNMP SMI and GDMO [ISO91].However,we believe that it fits better in a distributed computing environment such as CORBA [OMG93],since the notion of computedElement Management Informationobjects representing computed views Management ApplicationsSNMP CMIPDMI Distributed Computing EnvironmentFigure 1:3-level architecture for generating computed views of management informationManagement Middlewareviews for network management is closely related to the notion of higher level management services that can be implemented more efficiently in this framework.2.1Computed Views of Management InformationComputed views are constructed by applying an aggregation process to management information collected from network elements.Every computed view in our framework is stored in an object and has one or more of the following components:1. A monitoring, view, which represents a higher-level view of network state,2.a control view,which represents a control interface to higher-level network management ser-vices,3.an event view,which represents notifications that are generated by the object following the oc-currence of a series of other (elementary) events.The object’s attributes represent the network state that corresponds to the monitoring and control views of the object.In order to define a new view,the network manager specifies an aggregation rule with which every attribute value is computed.The rule can be specified declaratively,in which case a description of the aggregation is provided in a structured language,or explicitly,in which case the manager provides a piece of code that will be executed to compute the attribute’s value. Aggregations can be spatial or temporal.2.1.1Spatial aggregationsSpatial aggregations attempt to reduce management information distributed between a number of network elements by presenting information about groups of elements.By defining the appropriate groups,the manager can obtain a high-level view of network configuration,performance,etc.For example,the ingress traffic to a network region can be computed by processing traffic information collected from switches at the border of the region.In order to describe spatial aggregations efficiently,network elements(NEs)must be organized into ers can dynamically define groups based on any factor that makes sense.For example, groups can be formed according to geographical criteria(location)-a group of all NEs in a build-ing,campus,state,etc.,or functionality(a group of all ATM switches),or some combination such as all ATM switches in New York City area.Groups can be defined easily through a network con-figuration database.In this case,group membership can be determined by identifying the network elements whose properties match a query predicate.The directory enabled networks(DEN)pro-posal is an effort to standardize the schemata for such a configuration database.The DEN specifi-cation also identifies the need to model network services and groups to support more complex management functions.It is easy to generalize and define groups with other groups as members.Thus,the network group-ing hierarchy can be represented by a tree whose leaves are the network elements.Groups are not necessarily disjoint.Every group is characterized by a level indicator that corresponds to the depth of the tree where the particular group belongs.Level0is reserved for the leaves of a hierarchy which(from a network management standpoint)represent element management agents(EMA).A group at thefirst level corresponds to a set of EMAs.A group at the second level may contain groups of level one and perhaps one or more EMAs.In general,a group at level n is allowed to6contain groups of any level lower than n,i.e.,level n-1,n-2,...,level0inclusive.Sometimes it is convenient to refer to a group of level0which,however,really implies an element management agent.Figure2presents an example where network elements arefirst organized into groups G1,G2andG5G4G2G3G1Figure 2:Example for grouping network elementsG3that compose thefirst level of aggregation.Group G4is defined as the union of G1and G2,and similarly G5as the union of G2and G3.Once the group hierarchy has been specified,the manager can define higher-level management views and services by referring to groups rather than network elements.As the simplest example,assume that the attribute ErrorCount is defined on some EMAs,representing the number of unrecognized packets arriving at the corresponding network el-ement.A new attribute ErrorCount can be defined in a managed object to count the total number of errored packets received in the group G1.The latter can be computed by summing the Error-Count variables retrieved from every member of G1.In general in order to compute the value of an attribute that refers to a group,the group must be resolved to a set of agents that contain the required information.The resolution is performed recur-sively in a number of steps equal to the order of the group.Once the information components have been identified,their values are retrieved and a new value for the attribute is computed.This pro-cedure is described in detail in Section2.2.Similarly,control operations on an attribute represent-ing a group of level n are mapped to a series of(elementary)control operations on the components of the group.2.1.2Temporal aggregationsTemporal aggregations expose the time-varying properties of management information.In order to compute a temporal aggregation,information is collected periodically and provided as input to an aggregation function.This procedure can be used,for example,to form a time series of various granularities(minutes,hours,etc.)or,to provide an autocorrelation or cross-correlation measure-ment.2.2Declarative specification of attributesAs mentioned previously,attributes may be specified declaratively.In this case,the attribute is as-sociated with a list of groups that determine the locations(management agents)that contain the in-formation necessary to calculate the attribute’s value.This list of groups is further expanded into a78list of objects within these agents.When the appropriate attribute value from each one of these ob-jects is retrieved,a filter function is applied on the entire set to calculate the final value.The filter function operates on the collected attribute values and stores the result as the current value of the attribute.For example,the operation SUM adds all the retrieved values and stores the result as the new value of the attribute.The operation NULL stores all the retrieved values in an array indexed by each retrieved attribute.More complex filter functions may,for example,compute statistical properties such as the mean and standard deviation of a distributed data set,extract topological in-formation to generate a topology map, etc.More formally, every attribute can be expressed by the following formula:,(EQ 1)where f is the filter function,G i is a group,a i is the component attribute’s name and o i is an object selection predicate.The latter is used to select the managed objects (MOs)within the group from which information will be collected.Depending on the underlying management protocol,the ob-ject and attribute selection predicates may be specified in a different syntax to comply with the par-ticular structure of management information (e.g.,SNMP or CMIP)within an element management agent.Note that an attribute need not be computed exclusively from components of the same type.Figure 3demonstrates an example of the attribute value computation procedure.In this example,the attribute value V is computed from information components in groups G 1and G 2.First,G 1and G 2are resolved into a list of management agents.For each agent in group G 1,the object selection predicate o 1identifies the managed objects that contain the required information.From each suchV f G 1o 1a ,1,()G 2o 2a ,2,()...G n o n a ,n ,(),,,{}=EMA or Marvel server MOs attr1, val1attr2, val2...attrn, valn Managed ObjectG1G2Attribute value collectionobjID attr_name attr_valV =f {(G 1,o 1,a 1), (G 2,o 2,a 2)}Aggregated Expression:apply filter function Figure 3:Computing an aggregated attribute valueobject,we obtain only the values of attributes that correspond to the attribute selection predicate a1.The group G2is processed similarly.The result of the collection process from all the agents of G1and G2is stored in a temporary table that contains the origin of every attribute,its type and its value. The table is then used as input to the filter function which calculates the new value V. Filter functions can be specified by reference,in which case the required code to process the col-lected attribute values is loaded dynamically from an external function directory.The benefit of this approach is thatfilter functions need not be integrated with the server,which allows the definition of new functions or the improvement of existing ones without disrupting the operation of the man-agement system.Temporal aggregations of an attribute are accomplished by using specialfilter functions.For ex-ample,a sliding windowfilter can store a collected attribute value as a time-series.It is also possi-ble to define new attributes usingfilter functions that operate on the stored time-series such as delta functions, cross-correlation functions, etc.For a settable attribute,there exists a mapping function that describes how a value set by the man-ager will be propagated to the underlying components.The simplest mapping function is the one that distributes the same value to all of its component attributes.It can be used for control opera-tions that require setting the same value to a group of devices,such asfirmware upgrades,enabling or disabling features,etc.More complex mapping functions may use additional information from the environment to determine the exact value that every component attribute must receive.For ex-ample,in an ATM virtual private network(VPN),the manager may decide to increase its total al-located capacity by10Mbps.The mapping function in this case will determine the details of how the new capacity will be distributed between individual virtual paths.A refresh policy specifies how an attribute value is computed:A synchronous policy implies that the value is computed dynamically upon a get operation on the attribute.In the asynchronous case, an internal or external condition triggers the computation and storage of the value.The trigger con-dition can be a time interval,in which case the value is computed periodically by“pulling”infor-mation from the component objects.It is also possible to link the computation of an attribute’s value with the occurrence of an event.For example,an event could be an indication that one of the component attributes has changed its value.An eager policy would recompute the attribute’s value each time that any of its components change.The choice of the update condition must be made with great care:Infrequent updates introduce the danger that the computed information is out of date. On the other hand,an eager policy may trigger very frequent computations of an attribute’s value, some of which may not even be necessary(if the value is accessed at slower time scales).The man-ager sets the update condition taking into consideration the sensitivity of management applications that use this information with regard to its accuracy and the complexity involved in computing its value.2.3Accessing Element Management Agents through SNMP or CMIPIn many cases,a computed view is generated by collecting information from element management agents(EMAs)using a standards-based network management protocol such as SNMP or CMIP. When attributes are specified declaratively,additional information is usually required to identify which MIB objects inside the EMAs should be involved in the computation.For this reason,we provide mappings for both the SNMP and CMIP domains. The following rules apply:9First,group definitions must always be resolved into a set of application-layer addresses and other parameters of EMAs that represent the members of the group.In the SNMP domain,for example, every EMA is represented by the combination of an IP address,a port number and security infor-mation such as community strings that may be required.In the OSI domain,every CMIP agent must be described by an application layer address to which an association can be established,and additional security information.For every SNMP agent within a group,the object selection predicate is a set of object identifiers (OIDs).If the OID corresponds to a column of a table,the entire column is retrieved using a series of get-next operations.The OSI framework requires additional parameters to specify a target attribute,namely the object class,object instance,scope,filter and attribute name.Again,the group G refers to a set of CMIP agents.The object selection predicate contains the objectClass and objectInstance parameters to specify the class and distinguishing name of a base object.In addition,a scoping andfiltering pred-icate further identifies a subtree of managed objects in the OSI containment tree rooted at the base object.The attribute selection predicate identifies the attributes within the selected objects whose value will be retrieved.2.4Event ViewsTraditionally,eventfiltering and generation of higher level(composite)events has been done by a separate centralized event processing engine(usually an expert system).There are many advantag-es in decentralizing the event filtering process:•A higher degree of integration is achieved between the computed view and the events associ-ated with the view.Most management systems today store events in a single event database.Now,a computed view can store all events associated with the view inside the corresponding object and provide a customized event browsing/monitoring tool as part of its monitoring in-terface.•A distributed architecture for eventfiltering and generation can be implemented at significant-ly less cost compared to configuring and installing centralized event filtering systems.There has been a significant amount of work in thefield of composite event generation[NYG95, HAY96,MAN97].Our event architecture uses the services of READY,an event processing and no-tification system[GRU97]to generate composite events.This section describes the overall model for producing and consuming events in our environment and the structure of event objects.We distinguish between4entities:Event producers,consumers,clients and servers.Every view ob-ject can assume any or all of the above roles.Event producers are the sources where events are gen-erated.These can be element management agents,or other view objects that generate composite events.An event consumer is an entity that has subscribed to receive events from a producer.An event server consumes events,processes them and stores the result in a local database.Finally,an event client contacts an event server and performs operations on the stored event information.For example,event servers can be simple event log objects and clients browsing tools that examine these logs.Events are defined using an object-oriented model.The EventRoot class contains the attributes that are common to all event classes.Further specialization from this class allows the definition of new10。