HP System Fault Management White Paper

System Fault Management White Paper HP Part Number:5992-5243Published:June2008Edition:1.0Legal Notices©Copyright2008Hewlett-Packard Company,L.P.Confidential Computer Software.Valid license from HP required for possession,use or copying.Consistent with FAR12.11and12.212,Commercial Computer Software,Computer Software Documentation,and Technical Data for Commercial Items are licensed to the ernment under vendor’s standard commercial license.The information contained herein is subject to change without notice.The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services.Nothing herein shall be construed as constituting an additional warrenty.HP shall not be liable for technical or editorial errors or omissions contained herein.UNIX is a registered trademark of The Open Group.Table of ContentsExecutive Summary (5)Intended Audience (5)Scope of Hardware Diagnostics (5)Traditional Diagnostic Solutions (5)Overview (6)Integrating SFM with HP Service Essentials Remote Support Pack (8)Follow-the-Red Strategy (8)Advantages of SFM (9)Use Cases (10)Viewing Errors on a Monitored System (10)Receiving Notification of Errors (10)Viewing System Health Status (11)Defining WBEM Indications Criteria (12)Viewing Error Metadata (13)Troubleshooting Hardware (13)Viewing Event Logs (14)For More Information (14)Glossary (15)Table of Contents3List of Figures1CMS and Local System Display (7)2SFM with HP Service Essentials Remote Support Pack (8)3Follow-the-Red (9)4Event List (10)5Configuring Event Destination (11)6Health Status (12)7Subscription Administration (12)8Error Metadata (13)9Event Details (14)4List of FiguresExecutive SummaryThis white paper discusses the latest hardware diagnostics product on the HP-UX operatingsystem,System Fault Management(SFM).It describes the SFM features and benefits.The white paper also describes how to use SFM to obtain information about server hardware,troubleshoot faulty hardware devices,and perform server manageability tasks.Intended AudienceThis white paper is intended for system administrators,HP support personnel,and HP fieldengineers who require information about server hardware inventory and other details totroubleshoot faulty devices.Readers of this white paper must be familiar with the HP-UX system administration.Scope of Hardware DiagnosticsThe administration and maintenance of servers is becoming increasingly complex,expensive,and time-consuming owing to the growing need to consolidate IT hardware resources acrossbusinesses.Also,in the event of a system failure,key user operations receive a setback asapplications,tools,and databases running on the servers are affected.Consequently,huge losses are incurred.Therefore,it is imperative that servers operate continuously,in a reliable manner, because they run critical applications and serve multiple needs.Hardware diagnostic tools ensure that the system is functioning normally.They provide critical information in the event of a system failure,and retrieve details about the system components.Also,operating conditions can lead to a device failure or affect the system performance.Therefore,it is essential to monitor the server components and notify the user of any occurrence of any abnormalities.Hardware diagnostic tools monitor server hardware and notify the occurrence of any abnormal behavior with a monitored component,thereby avoiding a potential system breakdown.If adevice breaks down,hardware diagnostic tools narrows the search to the faulty component.Another key feature of hardware diagnostic tools is its ability to retrieve information related to the hardware inventory,such as serial number and part number of processors and memorymodules.Obtaining information such as the processor capacity and memory module path can be complex and time-consuming.However,using a tool,it is both simple and quick to obtainsuch details.Traditional Diagnostic SolutionsHP has been delivering a diagnostics product,called Online Diagnostics,on HP-UX-basedsystems.The Online Diagnostics product consists of EMS Hardware Monitors and the Support Tools Manager(STM).EMS Hardware Monitors comprise a group of monitors.They monitorvarious hardware components on a supported HP9000or an HP Integrity®server.When amonitored component does not function normally,an event is generated and reported to theuser through configured channels.STM manages a set of support tools the user can use to verify and troubleshoot system hardware.It provides information about system configuration and provides an interface to run hardware validation tests.However,system manageability needs are constantly evolving.System manageability has spread to newer dimensions,where tools must support a wide range of products,adhere to industrystandards,co-exist,and must be easy to manage.System Fault Management(SFM)delivers the following characteristics that make it a viablesolution to meet the current industry requirements:•supports a wide range of servers,disks,and firmware,•provides critical,and granular details to help troubleshoot faulty devices,Executive Summary5•co-exists with other industry-standard solutions,•enables simplified user managementThe following sections discuss SFM in detail.OverviewSystem Fault Management(SFM)is a collection of tools that are used to monitor the health ofHP servers running the HP-UX operating system.SFM includes providers that:•monitor hardware components on a server•predict a hardware fault•notify users of the fault•enable the system to automatically recover from the faultThe SFM providers retrieve information about a system's hardware devices(such as CPU,memory,power supply,and cooling devices).These providers support the Cell Online Addition and Replacement(OL*)feature.If new cells are added or if existing cells are removed from the partition,these providers automatically detect the changes in configuration.SFM enables users to perform a host of administrative tasks related to the events generated by the providers.It integrates with HP Service Essentials Remote Support Pack,which facilitatesthe analysis of errors that occur on a system,and helps troubleshoot those errors.SFM enables a user to browse and customize an online dictionary of error metadata,including the probable cause and recommended actions that help resolve system faults.SFM is based on the Web-Based Enterprise Management(WBEM)standards.WBEM is a set of management and internet standard technologies that was developed by the DistributedManagement Task Force(DMTF)to unify the management of enterprise computing environments.Adhering to the WBEM standards enables SFM to function with similar products that comply with WBEM.SFM integrates with HP System Management Homepage(HP SMH)management application.The functionalities of SFM can be used on a local system through the HP SMH user interface.In addition,the functionalities of SFM can be used on the Central Management Server(CMS)through the HP Systems Insight Manager(HP SIM),regardless of the supported operating system HPSIM is running on.Figure1illustrates the way details provided by SFM are displayed on HP SMH and HP SIM. 6Figure1CMS and Local System DisplayThe SFM product includes the following components:•SFM Providers•EVWEB•Error Management Technology(EMT)A user can use each of these components through the interface of a supported management application,such as HP SMH and HP SIM.SFM providers are components of SFM that retrieve information about the inventory on a system and the events that occur on the hardware resources.SFM providers are of two types:instance providers and indication providers.Each instance and indication provider monitors a server hardware device.When a user requests for information,an SFM instance provider dynamically queries the monitored device for property information,such as the type of memory module, processor speed,and serial number,and reports the information to the Common Information Model Object Manager(CIMOM).Subsequently,using the HP SMH interface,a user can view the property information in the property pages.An SFM indication provider generates WBEM indications if it detects any abnormal activity on the monitored device.These WBEM indications provide details about the detected activity,such as its summary,the probably cause,and the corresponding action recommended.These details help diagnose faults,if any,and take the required corrective actions,thereby avoiding a system failure.The WBEM indications are reported to the CIMOM,and subsequently,the user is notified of these occurrences through configured channels,such as email and syslog entries.The indication providers are analogous to the EMS Hardware Monitors in Online Diagnostics.However,the WBEM indications contain additional details than what is present in the EMS event details. EVWEB is an SFM component that enables a user to administer and view indications that have occurred on the system.A user can subscribe to these WBEM indications based on certain criteria, and also specify the destination where the indications must be delivered.EMT is an SFM component that enables the user to view most of the errors that can occur on a system running the HP-UX11i v3operating system.EMT also enables a user to perform certain administrative tasks related to the error metadata.Overview7Integrating SFM with HP Service Essentials Remote Support Pack SFM integrates with other diagnostic tools,such as the HP Service Essentials Remote SupportPack.HP Service Essentials Remote Support Pack analyzes WBEM indications generated by the SFM providers,generate problem reports,and notify the user of any potential problems.Figure2illustrates the integration of SFM with HP Service Essentials Remote Support Pack.Figure2SFM with HP Service Essentials Remote Support PackFollow-the-Red StrategySFM applies the follow-the-red strategy to identify the origin of a problem that has occurred ona device.When a problem is detected with a device,a red sign-indicating the problem-isdisplayed against the device.The red sign can be viewed on HP SMH and HP SIM.A user can click the system name associated with the sign to view details such as the device on which the fault has occurred.Subsequently,the user can click the device to view which instance of thedevice has failed.Clicking the instance that has a fault provides further details about the error, such as the probable cause and the recommended action.The defective device can be rectified based on the instructions provided.Figure3illustrates how the follow-the-red strategy works.8Figure3Follow-the-RedA demonstration of the follow-the-red strategy is available at:/hpsdp/index.jsp?auto=1&ib=5009160&category_id=5009304&demo_id=5038469 Advantages of SFMSFM offers the following advantages:•Displays information on standards-compliant graphical and command-line system management applications,such as HP SIM and HP SMH.•Operates within the WBEM environment.•Supports the Central Management Server(CMS)running on HP-UX,Linux®,or Windows®.•Enables the user to view and administer WBEM indications.•Provides the same features and benefits as those found in the EMS Hardware Monitors.•Enables the follow-the-red strategy that facilitates easy and quick identification of errors.Advantages of SFM9Use CasesThis section describes various use cases of SFM.They are as follows:•Viewing Errors on a Monitored System•Receiving Notification of Errors•Viewing System Health Status•Defining WBEM Indications Criteria•Viewing Error Metadata•Troubleshooting Hardware•Viewing Event LogsViewing Errors on a Monitored SystemThis use case illustrates how events generated by SFM are displayed on HP SMH and HP SIM.Figure4is a sample output of the event list on HP SMH.Figure4Event ListReceiving Notification of ErrorsThis use case illustrates how events generated by SFM can be delivered by email or logged inthe Event Archive.Figure5is a sample output of how notification of errors can be configured.10Figure5Configuring Event DestinationViewing System Health StatusSFM provides the health status of the system as well as the health status of each of the monitored components.It also provides details of the hardware inventory on the system.Figure6shows a sample output of the health status on HP SMH.Use Cases11Figure6Health StatusDefining WBEM Indications CriteriaSFM enables the user to subscribe to indications that suit the user’s needs.It also enables the user to specify the destination of the indications.Figure7shows a sample output of the subscription administration tasks.Figure7Subscription Administration12Viewing Error MetadataSFM enables the user to search and view most of the errors that can occur on the HP-UX11i v3 operating system.Figure8is a sample of the error metadata on a system.Figure8Error MetadataTroubleshooting HardwareThe faulty hardware can be rectified using the details displayed in the WBEM indication.Figure9shows a sample output of the details of a WBEM indication.Use Cases13Figure9Event DetailsViewing Event LogsSFM enables the user to receive event logs in the textual format.These logs contain details of all the events that are generated.Following is a sample excerpt of an event text log:EvArchNo Severity Event # Event Category Archive Time Summary========= =========== ============== ============== ============== ============= =13 Minor 100101 System Inte... 2008-06-04 15: A platform ...12 Information 7 System Hard... 2008-06-02 11: The Diagnos...Summary:A Corrected Platform Error was reported by PCI bus.Description of Error:A platform error was corrected by the firmware/hardware. The erroroccurred on the Host Bridge Adapter (Elroy). The PCI card connected to this adapter is at slot (unknown3).More information is available in the Event Details section of this event.Probable Cause Description: {The Diagnostic mode is switched from EMS to SFM. SysFaultMgmt will now monitor the devices & EMS hardware monitors will be shutdown.}For More InformationTo learn more about SFM,see the following documents available at:/en/diag •SFM Administrator’s and User’s Guide•SFM Release Notes•SFM Provider Data Sheets•SFM Tables of Versions•SFM Patch Descriptions14GlossaryA-CCentralManagementServer(CMS)The server monitoring the client systems in the network using SFM.Common Information Model Object Manager (CIMOM)The component of WBEM that manages the interaction between the providers and other modules in the WBEM environmentDDistributed Management Task Force (DMTF)An industry organization involved in the development,adoption,and interoperability of management standards and initiatives for enterprise and Internet environments.E-GError Management Technology (EMT)A user component that enables the user to obtain information about errors that can be generated on the system.Event MonitoringService(EMS)A command-line tool used to create extracts and encrypt them.EVWEB A user component that enables administering and viewing WBEM indications that are generated on the system.H-RHP ServiceEssentials RemoteSupport PackA new generation remote support product.It is integrated with HP SIM.HP SystemManagementHomepage(HPSMH)A management application installed on the local system.SFM is integrated with HP SMH.HP Systems Insight Manager (HP SIM)A management application installed on the CMS.It enables remote management of systems in the network.S-VSupport ToolsManager(STM)Manages a set of support tools the user can use to verify and troubleshoot system hardware.System FaultManagement(SFM)A collection of tools that are used to monitor the health of HP servers.15W-ZWBEM (Web-Based Enterprise Management)A set of management and internet standard technologies to unify the management of enterprise computing environments.16Glossary。