Borcade FC SWITCH Troubleshooting

IntroductionA SAN is a complex system that can consist of multiple switches, hosts, storage devices, routers, and hubs. A SAN can also be as simple as a single switch with attached storage and hosts. A breakdown of the individual components yields a range of subcomponents, from simple subcomponents, such as cables, to complex subcomponents, such as switches. At a macro level, the fabric itself is considered a component that might require troubleshooting. Switches are logically positioned in the middle of the network between hosts and storage, and have visibility to both storage and hosts. This visibility into both sides of the storage network enables you to use switches to determine the cause of any malfunction in the SAN. This chapter presents a structured process for identifying marginal or faulty SAN components by helping you figure out where to start and then to methodically home in on the problem. Specific areas of focus include troubleshooting the following symptoms and SAN components:▪▪Fabric▪▪“Missing” devices▪▪Marginal links▪▪Input/Output (I/O) interruptionsThe context of your problem influences how to interpret the data output by the variety of commands available in Fabric OS. For example, focus on the port state information for switchShow output when you are troubleshooting a port issue, and the switch status information from the same command when investigating a fabric issue.We will cover the details of how to troubleshoot using Fabric OS commands such as switchShow, errShow, portStatsShow, and other commands. Understanding host behavior and interpreting host information is also an important part of the troubleshooting process we discuss in this chapter.The Troubleshooting Approach: The SAN Is a Virtual CableWhen first approaching troubleshooting, think of the SAN as a virtual cable. Storage traditionally involved connecting a Small Computer Systems Interface (SCSI) disk via a SCSI cable to a host; with this scenario, you focus on four_components: the storage, the Host Bus Adapter (HBA), the host’s OS, and the cable/terminator. Troubleshooting a SAN is more challenging, but still has many things in common with the traditional storage troubleshooting process.To the operating system, the SAN provides a link to a disk, just as a traditional SCSI connection would.You can apply the same “tried-and-true” process of elimination used to trouble shoot a direct-attach SCSI issue or Ethernet network issue to SAN trouble shooting. At a macro level, if you consider the SAN a virtual cable, the issue can reside in three possible areas: the host, the “cable,” or the storage. Troubleshooting can work like a binary search when you start investigating these areas. Start in the middle and determine whether you are “above” or “below” the problem, and then keep dividing the suspect path until you resolve the problem.When troubleshooting with a simple single-switch configuration, a single host, and a single storage device, you need to focus on the HBA, the Gigabit Interface Converter (GBIC), the host’s OS, the cable, the switch, and the storage. Brocade fabrics run a single-image distributed operating system known as Fabric OS. Fabric OS delivers functionality such as Name Server, Registered State Change Notification (RSCN), Zoning, and security. These functions are part of the SAN and are also variables in the troubleshooting equation. A large SAN can consist of dozens of switches and is capable of growing to thousands of ports. Knowing where in the SAN to initiate troubleshooting can be daunting. The next section uses a typical SAN troubleshooting scenario—a host unable to “see” its disks—to illustrate the method of resolving the problem by treating the SAN as a virtual cable and working with a process of elimination.A Typical Scenario: “I Cannot See My Disks”We provide the scenario described in this section to introduce the troubleshooting process and to establish a framework with which you are familiar. Some terms, commands, and concepts may seem foreign. This is okay. We address everything discussed in this section in greater detail later in the chapter.When a host cannot see its disks, one thing to check is whether that device is logically connected to the switch by reviewing the output from the switchShow command. If the device is not logically connected (that is, it does not show up as an Nx_Port), you need to focus on the port initialization. Notice that port 15 in Figure 8.1 indicates a logically connected device, as this port is connected as an F_Port. Port 14 is an example of an unsuccessful device connection, as the device connected to port 14 is connected as a G_Port. A G_Port indicates an incomplete connection to the fabric. Initially knowing that the missing device is not logically connected eliminates the host and everything on that side of the data path from the suspect list, as depicted in Figure 8.2. This includes all aspects of the host’s OS, the HBA driver settings and binaries, the HBA Basic Input Output System (BIOS) settings, the HBA GBIC, the cable going from the switch to the host, the GBIC on the switch side of that cable, and all switchsettings related to the host. That is quite a lot for one command! If the missing device is logically connected to the switch, you need to check to see if the device is present in the Simple Name Server (SNS).Figure 8.1 Example of a Successful and Unsuccessful Device Connectioncore2:admin> switchshowswitchName:core2switchType:2.4switchState:OnlineswitchRole:SubordinateswitchDomain:5switchId:fffc05switchWwn:10:00:00:60:69:10:9b:5bswitchBeacon:OFFport 0: swOnlineE-Port10:00:00:60:69:11:f9:f7 "edge1"(upstream)port1: swOnlineE-Port10:00:00:60:69:10:9b:52 "edge2"port2: swOnlineE-Port10:00:00:60:69:11:f9:f7 "edge1"port3: swOnlineE-Port10:00:00:60:69:10:9b:52 "edge2"port4: swOnlineE-Port10:00:00:60:69:12:f9:8c "edge3"port5: swOnlineE-Port10:00:00:60:69:12:f9:8c "edge3"port6: —No_Moduleport7: —No_Moduleport8: —No_Moduleport9: —No_Moduleport 10: —No_Moduleport 11: idOnlineE-Port10:00:00:60:69:12:f9:8c "edge3"port 12: —No_Moduleport 13: —No_Moduleport 14: cuOnlineG-Port //incomplete fabric connectionport 15: idOnlineF-Port50:06:04:82:bc:01:9a:0cFigure 8.2 The SAN Virtual CableThe SNS is a directory service provided by the fabric. Initiators query the Name Server much in the same way you would query a telephone directory looking for a particular person or service. If a device is not in the Name Server,it is essentially invisible to other devices in the fabric. When a device connects to the fabric, that device will register itself with the Name Server. This is similar to the situation in which you change neighborhoods and have your name listed in the new telephone directory. When an initiator, which is normally an HBA, enters the fabric, it queries the Name Server to identify all accessible devices and obtain the addresses of these devices, just like you might scan your telephone directory for a name. Some targets also will query the Name Server. Then the initiator starts the process of establishing a connection with those devices for which the Name Server provides addresses.Check the Name Server for the presence of your missing device by issuing the nsShow command on the switch to which the device is attached (see the sample output in Figure 8.3). This will list all of the nodes connected to that switch, allowing you to determine if a particular node is accessible on the network. An alternate method is to check the Name Server list in the WEB TOOLS Graphical User Interface (GUI) on any switch in the fabric, as it contains a consolidated list of all devices in the fabric. Note that we started the process in the middle of the virtual SAN cable, which is the fabric. This is the process we described earlier as being like a binary search algorithm. You start in the middle half of the data path, figure out if you are “above” the problem or “below”it and keep dividing the suspect path in half until you identify the problem.Figure 8.3nsShow Sample Outputore2:admin> nsshowThe Local Name Server has 9 entries {Type PidCOSPortNameNodeNameTTL(sec)*N021a00;2,3;20:00:00:e0:69:f0:07:c6;10:00:00:e0:69:f0:07:c6; 895Fabric Port Name: 20:0a:00:60:69:10:8d:fdNL051edc;3;21:00:00:20:37:d9:77:96;20:00:00:20:37:d9:77:96; naFC4s: FCP [SEAGATE ST318304FC0005]Fabric Port Name: 20:0e:00:60:69:10:9b:5bNL051ee0;3;21:00:00:20:37:d9:73:0f;20:00:00:20:37:d9:73:0f; naFC4s: FCP [SEAGATE ST318304FC0005]Fabric Port Name: 20:0e:00:60:69:10:9b:5bNL051ee1;3;21:00:00:20:37:d9:76:b3;20:00:00:20:37:d9:76:b3; naFC4s: FCP [SEAGATE ST318304FC0005]Fabric Port Name: 20:0e:00:60:69:10:9b:5bNL051ee2;3;21:00:00:20:37:d9:77:5a;20:00:00:20:37:d9:77:5a; naFC4s: FCP [SEAGATE ST318304FC0005]Fabric Port Name: 20:0e:00:60:69:10:9b:5bNL051ee4;3;21:00:00:20:37:d9:74:d7;20:00:00:20:37:d9:74:d7; naFC4s: FCP [SEAGATE ST318304FC0005]Fabric Port Name: 20:0e:00:60:69:10:9b:5bNL051ee8;3;21:00:00:20:37:d9:6f:eb;20:00:00:20:37:d9:6f:eb; naFC4s: FCP [SEAGATE ST318304FC0005]Fabric Port Name: 20:0e:00:60:69:10:9b:5bNL051eef;3;21:00:00:20:37:d9:77:45;20:00:00:20:37:d9:77:45; naFC4s: FCP [SEAGATE ST318304FC0005]Fabric Port Name: 20:0e:00:60:69:10:9b:5bN051f00;2,3;50:06:04:82:bc:01:9a:0c;50:06:04:82:bc:01:9a:0c; naFC4s: FCP [EMCSYMMETRIX5267]Fabric Port Name: 20:0f:00:60:69:10:9b:5b}At this point, if the device is not present in the Name Server, you have narrowed your search along the virtual SAN cable to the Name Server interface between the storage. The missing device process defined in this section is summarized in flowchart form in Figure 8.4. Note that Figure 8.4 is an excerpt from the complete missing-device troubleshooting process, which is shown in Figure 8.25. Remember that we will go deeper into this missing-device troubleshooting process and flowchart later in the chapter.Where to Start and What Data to GatherAs stated in the previous section, SAN troubleshooting should begin in the center of the SAN and proceed outward. Once you know where to start troubleshooting, the next question is how to proceed. Start the troubleshooting process by gathering a preliminary set of data, and then analyze this data to identify where the problem resides: the host, the fabric, or the storage. Then gather additional data from the appropriate area and home in on the cause of the problem. _A plethora of data is available from the switches, hosts, and storage. Knowing what data to look at and when to look at it is fundamental to the SAN _trouble¬shooting process.Take a Snapshot: Describe the Problem and Gather InformationStart with a general description of the problem and identify as much supporting detail as possible. At the very least, this should include a statement about what the ―bad‖ behavior is, and a statement about what you are doing or have done to expose this behavior. Note that this is not the same as describing what you have done that causes the behavior. You might be doing something correctly, like plugging in a disk array and adding it to a zone, yet it might affect something else in the fabric if there is an underlying problem that is exposed whenever a zone change occurs.For example, an HBA responding incorrectly to an RSCN could fail when the new zone configuration is enabled. An RSCN is a fabric service for which an edge device optionally registers. When a device registers for an RSCN, it is asking the fabric to send that device a notice anytime something in the fabric changes. For example, when a new device is added to the fabric, any devices that registered for RSCNs will receive a notice. The registered device receiving the RSCN then checks the Name Server to see what has changed and takes appropriate action. For example, if the registered device is a host and a new disk drive is added to the SAN, the host might create the necessary device operating system structures so the new device is accessible to the user.This information will help you with the problem resolution, and might be necessary if you need to contact Brocade or any Brocade-authorized support channel. Some examples of a general problem description include:When I enable a switch zoning♣♣ configuration with cfgEnable, storage devices are no longer accessible to the host.There are frequent pauses in I/O when I copy large♣♣ files between arrays.My edge device sometimes connects as♣♣ an N_Port, and other times it connects as a Node Loop (NL)_Port when I power it up.The fabric segments and the following error message is♣♣ logged (provide error message in your description). It does this under normal operation, even when I do not touch any device on the SAN.Include the answers to the following questions with your problem description:♣♣ Can you recreate the problem on demand? If so, how? (Go into detail.)Is the problem intermittent? If so, how♣♣ frequently does it occur?Has anything at all changed♣♣ recently on the fabric? If so, what? (Provide a complete list.)♣♣ Is the problem localized or fabric-wide? For example, is the problem happening with other devices in the fabric, or just locally with a single device attached to the switch?Is this an initial install and the♣♣ device was never working, or was the device working and now it has stopped working?Other information to record:If there are any♣♣ error messages, include them with the problem_description.♣♣ Firmware and driver versions for the affected HBA and storage devices.♣ Firmware and operating system versions for affected hosts andall♣_fabric switches.External switch information, such as LED♣♣ state.External HBA and port information, such as LED♣♣ state.A diagram of the SAN configuration.♣♣♣♣ If long-distance links are present, include information about the length and quality of the lines, and the mechanism being used to achieve the distance (for example, ―The line is 10 km long, and we are using Long Wavelength [LWL] GBICs,‖ or ―It is 80 km long, and we are using a Dense Wave Division Multiplexor [DWDM] and the Extended Fabrics product‖).Finally, gather supportShow information from the switches. The supportShow command is a switch command used to gather information about the switch and the fabric; it can provide valuable clues about what is happening in your switch network. It is like a macro in that it executes a long list of switch commands, which Brocade identifies as important for the troubleshooting process. Note that the commands that supportShow executes vary between Fabric OS releases. The v2.4.1 supportShow command executes the following switch commands:♣♣ versionuptime♣♣tempShow♣♣♣ psShow♣licenseShow♣♣♣♣ diagShowerrDump♣♣switchShow♣♣♣ portFlagsShow♣portErrShow♣♣♣♣ mqShowportSemShow♣♣♣♣ portShowportRegShow♣♣♣♣ portRouteShowfabricShow♣♣♣♣ topologyShowqlShow♣♣faShow♣♣♣ portCfgLport♣nsShow♣♣♣♣ nsAllShowcfgShow♣♣configShow♣♣♣ faultShow♣traceShow♣♣♣♣ portLogDumpOne benefit of supportShow is that you do not have to repeatedly retrieve various types of data, since most of the data you need is available from supportShow in one place. As this command rapidly streams in a telnet window, capture mode should be turned on prior to executing the command so that it can be captured to a text file for later review.NOTEIt is important to execute the supportShow command at the time the problem is occurring, rather than waiting until the fabric is functioning normally.Due to the large volume of data created by supportShow, you might choose to gather the supportShow data once and then selectively issue a subset of its commands as part of your troubleshooting process.Troubleshooting ToolsMany tools are available to the SAN troubleshooter. Many of these tools are switch commands. Other tools involve viewing the switch LEDs, host information such as Solaris’ /var/adm/messages file, Fibre Channel analyzers, and diagnostics available on many storage arrays. Rarely is it possible to use a single tool to successfully troubleshoot a problem. It is more common to use several tools to attain a successful resolution of a problem.Using the Switch LEDsA significant amount of information can be gathered just by looking at the switch LEDs. At a rudimentary level, it is possible to identify that a device _has faulted or is not yet online by looking for a ―fast yellow.‖ If the switch is located in another room, you can get a visual real-time LED status using the WEB TOOLS interface. Fast flickering green lights are a sign of a healthy SAN. By physically observing the switches that comprise a SAN, it is possible to detect patterns and identify a marginal or faulty component. For example, if you have a situation in which you are trying to identify a device that is repeatedly toggling online and offline, you can use the switch LEDs.While observing a functional fabric, you can easily identify a potentially disruptive device by scanning for a port that goes offline (no LED light), sends light (steady yellow), comes online (steady green), and then cycles through the same steps—blank, yellow, green. You also want to look for correlations or patterns, such as one device going offline followed by a group of devices going offline and back online again. This situation is common in QuickLoop configurations when the first device going offline is sending a Loop Initialization Primative (LIP), which then causes the other devices to LIP.How to Identify a Healthy SAN Using the LEDsA settled and healthy fabric should have solid green or fast flickering green lights. A solid green light indicates an active link, while a fast flickering green light indicatesI/O activity.How to Identify a SAN Problem Using the LEDsA yellow light or blinking yellow light indicates a problem with your SAN. An LED that transitions from yellow to green, however, is not a problem. A powered-off edge device, or edge device that is not yet online, might cause the switch LEDs to blink yellow.Another helpful use for the LEDs is for fabric ―bring up.‖ When bringing up a fabric, one sign to look for that indicates a fabric has reached convergence are steady green lights. When the fabric is coming up, the Inter-Switch Links (ISLs) go through initialization, which appear to the observer as flickering green and yellow lights prior to the fabric fully converging. Once the fabric is converged, the lights go to a steady green. Then, as I/O in the fabric begins, you will see flickering green lights on the ISL ports and the edge device ports.A slowly flashing switch power LED indicates that the switch failed the Power-On Self-Test (POST) and is not able to come online. Refer to the associated switch manual for the location of the power LED. Table 8.1 lists the port LEDs and their definitions (you can also find this table in the Brocade SilkWorm 2800 Hardware Reference Manual).Table 8.1 Front Panel LED Port IndicatorsPorts LED DefinitionNo light showing No light or signal carrier (no module, no cable) for media interfaceSteady yellow Receiving light or signal carrier, but not yet onlineSlow yellow (flashes two seconds) Disabled (result of diagnostics, switchDisable, or portDisable command)Fast yellow (flashes a half second) Error, fault with portSteady green Online (connected with external device over cable)Slow green (flashes two seconds) Online, but segmented (loopback cable or incompatible fabric parameters)Fast green (flashes a half second) Internal loopback (diagnostic)Flickering green Online and frames flowing through portSwitch DiagnosticsA robust set of switch diagnostics is available so you can validate the operational level of a SilkWorm switch. Several of these diagnostics, such as portLoopbackTest, are also helpful in the troubleshooting process. For example, if you suspect a bad GBIC or switch port, you can use portLoopbackTest to confirm your suspicion. Using portLoopbackTest for troubleshooting is discussed in the section ―Troubleshooting Marginal Links‖ later in the chapter. The supportShow diagnostic command in particular, discussed in detail later in this chapter, is very helpful to the troubleshooting process. The Brocade Fabric OS manuals provide detailed description regarding the usage of diagnostic commands. To see what diagnostic commands are available online, issue the command diagHelp at the switch prompt. The following list of diagnostic commands is available in the V2.4.1 Fabric OS:ramTest System DRAM diagnostic♣♣♣♣ portRegTest Port register diagnostic♣♣ centralMemoryTest Central memory diagnosticcmiTest CMI♣♣ bus connection diagnosticcamTest QuickLoop CAM♣♣ diagnosticportLoopbackTest Port internal loopback♣♣ diagnosticsramRetentionTest SRAM Data Retention♣♣ diagnosticcmemRetentionTest Central Mem Data Retention♣♣ diagnosticcrossPortTest Cross-connected port♣♣ diagnosticspinSilk Cross-connected line-speed♣♣ exerciserdiagClearError Clear diag error on specified♣♣ portdiagDisablePost Disable Power-On-Self-Test♣♣♣ diagEnablePost Enable Power-On-Self-Test♣♣♣ setGbicMode Enable tests only on ports with GBICs♣♣ setSplbMode Enable 0=Dual, 1=Single port LB mode♣♣ supportShow Print version, error, portLog, etc.♣♣ diagShow Print diagnostic status information♣♣ parityCheck Dram Parity 0=Disabled, 1=Enable♣♣ spinFab ISL link diagnosticloopPortTest L_Port cable♣♣ loopback diagnosticHelpful CommandsWith dozens of switch commands at your disposal, it can be difficult to determine which command to use in a given situation. An annotated list of helpful commands follows in this section, with additional commands highlighted as they relate to specific issues discussed in following sections. This list of commands is a starting point for gathering data and initiating your troubleshooting process. While the information generated by these commands is also available in supportShow, you will want to use individual commands as you advance through the troubleshooting process. SupportShow creates a significant amount of data and is helpful when you want to perform the original snapshot of the configuration and environment (to report a problem to your switch supplier), or you are not sure what data to capture. NOTEAlthough the switch commands are shown with various capitalization as originally coded in Fabric OS, the commands are no longer case-sensitive and can be entered with all lowercase if desired.Entering the command help at the switch prompt generates a list of commands available to the user as shown in Figure 8.5. Entering the command help<command> generates a help page (similar to UNIX man pages) for that specific command. Many commands differ by the extension show or dump (for example, errShow and errDump). The difference is that show commands require you to type a return between entries, while the dump commands stream data to the screen without any pauses. Dump commands are used when you have a facility for loggingcommand output to a file. It might be necessary to execute commands on more than one switch in the fabric, especially if the location of the problem is unclear.As of Fabric OS 2.4.1, there is no time synchronization among the switches, which can make troubleshooting a challenge if the clocks between the switches are skewed. Before you begin troubleshooting your fabric, you should make a note of any time skew so that you can compensate for it when reading command outputs. You should also make an effort to keep switch clocks set correctly during normal operation to avoid this problem.Figure 8.5 Use the help Command to See What Commands Are Available or Type the help Command for Help About a Specific Commanddev172:admin> helpagtcfgSet Set SNMP agent configurationagtcfgShow Print SNMP agent configurationagtcfgDefault Reset SNMP agent to factory default...qlHelp Print quick loop help inforouteHelp Print routing help infotrackChangesHelp Print Track Changes help infozoneHelp Print zoning help infodev172:admin> help errShowNAMEerrShow - display the error logSYNOPSISerrShowAVAILABILITYall usersDESCRIPTIONThis command displays the error log, prompting the user to typereturn between each log entry. It is identical to errDump, except...SEE ALSOerrDump, uptimeThe errShow CommandThe errShow command provides a listing of up to 64 logged errors and is helpful for identifying where a problem might reside. It sends messages to the console and to the error log. Note that the error log is cleared after a reboot or power cycle; if you want to maintain error logs that persist after reboots or power cycles, consider using the syslog facilities of the switch to log errors to persistent storage. See syslogdIpAdd, syslogdIpRemove, and syslogdIpShow for further detail on how to set up persistent logging.When examining errShow data, which can be quite wordy, look for trends or patterns. For example, look for an excessive number of errors associated with a specific port. In addition, watch for high error-count values, which indicate a repeated error that has been logged many times. Logging error counts limits errors that occur multiple times from consuming the space provided for the error log. It is important to note that with every error, a severity level is associated. A warning (error level 3) is just that—a warning. An error (error level 2) or critical (error level 1) message is more severe and requires further attention.An excerpt from the errShow help entry is provided in Figure 8.6. Please refer to the help page or the Fabric OS manual for details on interpreting Diag Err#, as the list of codes is lengthy. A Diag Err# usually indicates a problem with hardware, so contact your switch supplier for further assistance.In addition to software errors, errShow logs environmental issues, such as over-temperature conditions, and equipment issues such as fan failures or power supply failures. A detailed list of error messages, descriptions, probable causes, and actions is maintained in the Fabric OS Reference Manual Version 2.4 (Publication Number 53-0001569-01).Figure 8.6 Excerpt from the errShow help EntryEach entry in the log has the same format:Error Number——————taskId (taskName): Time Stamp (count)Error Type, Error Level, Error MessageDiag Err#Error Number Starting from one. If there are more error thanthe size of the log, only the most recent errorsare shown.Task Id & Name The ID and name of the task recording the error.Time Stamp The date and time of the first occurrence ofthe error.Error Count For errors that occur multiple times, the repeatcount is shown in parenthesis. The maximum countis 999.Error Type An uppercase string showing the firmware moduleand error type. The switch manual contains adetailed explanation of each error type.Error Level 0 panic (the switch reboots)1 critical2 error3 warning4 information5 debugError Message Additional information about the error.Figure 8.7 is an example of an errShow message. The fabric is segmented, meaning that the switch that generated this message is logically disconnected from the SAN, and any devices in the SAN that are not directly connected to this switch are inaccessible to this switch. Moreover, any devices located on this switch are unable to access other devices in the fabric. The error level is a warning (3). The task ID (0x10e2b7f0) can be cross-referenced by issuing the telnet command ―i‖ to obtain additional information on the task in question. The Task Name is self-explanatory, and interpreting it is somewhat intuitive. For example, tTransmit is the transmit task. The Task Name can be helpful in identifying the nature of the problem. Finally, the error message indicates that there is a discrepancy between the zone information contained on this switch and the zone information contained in the rest of the fabric. When the switch tried to join _the fabric with this conflicting information, the join request was denied; hence, the segmented fabric. The message even identifies the zone that is causing the conflict; in this case, it is the ―red‖ zone. This zone should be checked and compared to the rest of the fabric, and if the zone information is different, either correct or delete it.Figure 8.7 errShow ExampleThe portErrShow CommandThe portErrShow command is an effective command for troubleshooting marginal ports. This command provides an error summary for all ports associated with the switch and provides a status of all ports from a link integrity perspective. The key to interpreting the statistics is looking for a very high number of errors relative to the frames transmitted and frames received. For example if 2,000,000 frames have been received and only three Cyclic Redundancy Check (CRC) errors have been logged, the CRC errors relative to the frames received is a very low ratio and the。