Mellanox -ib驱动安装手册linux
Mellanox CS7510智能交换机说明书
©2018 Mellanox Technologies. All rights reserved.†For illustration only. Actual products may vary.Mellanox provides the world’s first smart switch, enabling in-network computing through the Co-Design Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)™ technology. The CS7510 system provides the highest performing fabric solution in a 16U form factor by delivering 64Tb/s of full bi-directional bandwidth with 400ns port latency.SCALING-OUT DATA CENTERS WITH EXTENDED DATA RATE (EDR) INFINIBANDFaster servers based on PCIe 3.0, combined with high-performance storage and applications that use increasingly complex computations, are causing data bandwidth requirements to spiral upward. As servers are deployed with next generation processors, High-Performance Computing (HPC) environments and Enterprise Data Centers (EDC) will need every last bit of bandwidth delivered with Mellanox’s next generation of EDR InfiniBand high-speed smart switches.SUSTAINED NETWORK PERFORMANCEBuilt with Mellanox’s latest Switch-IB™ InfiniBand switch devices, the CS7510 provides up to 324 100Gb/s full bi-directional bandwidth per port. The CS7510 modular chassis switch provide an excellent price-performance ratio for medium to extremely large size clusters, along with the reliability and manageability expected from a director-class switch.CS7510 is the world’s first smart network switch, designed to enable in-network computing through the Co-Design Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) technology. The Co-Design architecture enables the usage of all active data center devices to accelerate the communications frameworks, resulting in order of magnitude applications performance improvements.WORLD-CLASS DESIGNCS7510 is an elegant director switch designed for performance, serviceability, energy savings and high-availability. The CS7510 comes with highly efficient, 80 gold+ and energy star certified AC power supplies.The leaf, spine blades and management modules, as well as the power supplies and fan units, are all hot-swappable to help eliminate down time.COLLECTIVE COMMUNICATION ACCELERATIONCollective is a term used to describe communication patterns in which all members of a group of communication endpoints participate.324-Port EDR 100Gb/s InfiniBand Smart Director SwitchCS7510 InfiniBand SwitchPRODUCT BRIEFSWITCH SYSTEM†350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: © Copyright 2018. Mellanox Technologies. All rights reserved.Mellanox, Mellanox logo and MLNX-OS are registered trademarks of Mellanox Technologies, Ltd. Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP), Switch-IB, UFM, and Unified Fabric Manager are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners.Mellanox CS7520 InfiniBand Switchpage 2Collectives have implications on overall application performance and scale. CS7510 introduces the Co-Design SHARP technology, which enables the switch to manage collective communications using embedded hardware. Switch-IB 2 improves the performance of selected collective operations by processing the data as it traverses the network, eliminating the need to send data multiple times between endpoints. This decreases the amount of data traversing the network and frees up CPU resources for computation rather than using them to process communication.MANAGEMENTThe CS7510, dual-core x86 CPU, comes with an onboard subnetmanager, enabling simple, out-of-the-box fabric bring-up for up to 2048 nodes. CS7510 switch runs the same MLNX-OS ® software package as Mellanox FDR products to deliver complete chassis management, to manage the firmware, power supplies, fans and ports.Mellanox CS7510–16U modular chassis–36 QSFP28 EDR 100Gb/s InfiniBand ports per dual IC leaf bladeSwitch Specifications–Compliant with IBTA 1.21 and 1.3 –9 virtual lanes:8 data + 1 management –256 to 4Kbyte MTU–4x48K entry linear forwarding databaseManagement Ports–DHCP–Familiar Industry Standard CLI–Management over IPv6 –Management IP –SNMP v1,v2,v3 –Web UIFabric Management–On-board Subnet Manager supporting fabrics of up to 2048 nodes–Unified Fabric Manager™ (UFM™) AgentConnectors and Cabling–QSFP28 connectors–Passive copper or active fiber cables –Optical modulesIndicators–Per port status LED Link, Activity –System status LEDs: System, fans, power supplies –Port Error LED –Unit ID LEDPhysical Characteristics–Dimensions:28’’H x 17.64’’W x 30.3’’D –Weight:Fully populated 275kg (606lb)Power Supply–Hot swappable with N+N redundancy–Input range: 180-265VAC –Frequency:47-63Hz, single phase ACCooling–Hot-swappable fan trays –Front-to-rear air flow –Auto-heat sensing fansPower Consumption–Typical power consumption (fully populated):–Passive cable: 4939W –Active cable: 6543WFEATURESSafety–CB –cTUVus –CE –CUEMC (Emissions)–CE –FCC –VCCI –ICES –RCMOperating Conditions–Operating 0ºC to 40ºC–Non-Operating -40ºC to 70ºC–Humidity: Operating 10% to 85%, non-condensing–Altitude: Operating -60 to 3200mAcoustic–ISO 7779 –ETS 300 753Others–RoHS compliant –1-year warrantyCOMPLIANCETable 1 - CS7510 Series Part Numbers and Descriptions53363PB Rev 2.0。
IB卡使用说明
Mellanox infiniband 卡使用说明：非SR-IOV安装驱动后使用service openibd start来启动IB卡，使用chkconfig openibd on来加入启动。
请不要删除驱动安装包，中间还有几个应用程序需要使用。
IB卡默认支持IPoIB模式，直接在/etc/sysconfig/netwok-scripts/ifcfg-ibx里修改即可(注意#掉MAC地址，IB的MAC地址太长无法识别)，这是redhat系统。
Ubuntu中直接在/etc/netwok/interfaces里输入ib信息即可，配置等同网卡。
connetcx.port_config 改接口类型（未SR-IOV之前）cat /etc/infiniband/openib.conf中的SET_IPOIB_CM=auto 改成connetced才能对插（无IB交换机）互联（单边改就可以），改完service openibd restart，默认值（auto）是datagram IB卡的sr-iov确认bios里打开了VT-X VT-D 及SR-IOV（一般都在CPU设置里，几个选项在一起，有时SR-IOV会在PCI选项里）Vi /boot/grub/grub.conf在kernel后面加上intel_iommu=on重启后生效，可以cat /proc/cmdline确认是否已经生效输入mst start或service mst start打开mst，使用chkconfig mst on将mst加入启动，然后用mst status来检查状态：一般情况下提供了两种方式来打开IB卡对SR-IOV的支持，我们选择最简单的方式，使用驱动程序中带的mlnx_fw_updater.pl来操作，具体为./mlnx_fw_updater.pl –enable-sriov即可。
使用mstflint –d PCI号 dc来检查是否已经打开sr-iov检查[HCA]项目。
mellanox infiniband技术原理
Mellanox Infiniband是一种高速计算机网络技术，主要用于数据中心的互连。
其设计目的是为了提高数据传输的速度和效率，降低延迟，并提供更高的带宽。
Infiniband技术使用点对点通信，可以在服务器、存储设备和网络设备之间提供高带宽、低延迟的数据传输。
Infiniband技术的工作原理如下：
1. 基于通道化：Infiniband使用通道化技术，将数据传输划分为多个通道，每个通道可以独立进行数据传输。
这种方式可以提高带宽利用率，降低冲突和延迟。
2. 基于信用机制：Infiniband使用信用机制来管理数据传输。
发送方在发送数据之前，先发送一个请求，接收方收到请求后，返回一个信用。
发送方在收到信用后，才开始发送数据。
这种方式可以确保数据传输的顺序性和可靠性。
3. 具有流量控制：Infiniband技术具有流量控制功能，可以根据网络状况和接收方的能力来调整数据传输的速度和大小，避免网络拥塞和数据丢失。
4. 支持虚拟化：Infiniband支持虚拟化技术，可以在不同的虚拟机和应用程序之间提供高效的数据传输。
总之，Mellanox Infiniband技术通过通道化、信用机制、流量控制和支持虚拟化等技术，提供了高性能、高可靠性和高带宽的网络连接，适用于数据中心和云计算环境。
Mellanox SwitchX Firmware Release Notes说明书
Mellanox SwitchX® FirmwareRelease NotesRev 9.4.2000Last Modified: November 29, 20169.4.2000Table of ContentsTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Chapter 1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.1 Supported Systems in 9.4.2000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2 Supported Cables and Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.3 Firmware Upgrade. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 PRM Revision Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Chapter 2 Changes and Major New Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1 Changes in Rev 9.4.2000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Changes in Rev 9.4.1000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3 Changes in Rev 9.3.8000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4 Changes in Rev 9.3.6000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.5 Changes in Rev 9.3.4000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.6 Changes in Rev 9.3.2000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.7 Changes in Rev 9.3.1200 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.8 Changes in Rev 9.3.0000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.9 Changes in Rev 9.2.8000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.10 Changes in Rev 9.2.6100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.11 Changes in Rev 9.2.4002 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.12 Changes in Rev 9.2.3000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.13 Changes in Rev 9.2.0000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.14 Changes in Rev 9.1.7000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.15 Changes in Rev 9.1.5000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.16 Changes in Rev 9.1.3000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.17 Changes in Rev 9.1.2000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.18 New Features and Changes in Rev 9.1.0000 . . . . . . . . . . . . . . . . . . . . . . 8 Chapter 3 Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Chapter 4 Bug Fixes History. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101OverviewThese are the release notes for the SwitchX® and SwitchX-2 firmware, 9.4.2000. This firmware complements the SwitchX silicon architecture with a set of advanced features, allowing easy and remote management of the switch.1.1Supported Systems in 9.4.2000This firmware supports the devices and protocols listed in Table 1. For the most updated list of switches supported, visit the firmware download pages via .1.2Supported Cables and ModulesPlease refer to the interconnect overview page on the Mellanox website:/page/cables?mtag=cable_overview1.3Firmware UpgradeFirmware upgrade may be performed directly from any previous version to this version. To upgrade firmware, please refer to the Mellanox Firmware Tools (MFT) package at:/page/management_toolsFirmware 9.4.2000 is compatible with MFT tools v2.7.1 and above. Using older ver-sions of MFT tools causes failure during image creation.Table 1 - Supported PSIDs1.4PRM Revision CompatibilityFirmware 9.4.2000 complies with the following programmer’s reference manual:•Switches Programmer's Reference Manual (PRM), Rev 1.40 or later2Changes and Major New Features2.1Changes in Rev 9.4.2000•N/A2.2Changes in Rev 9.4.1000•Added system MKey support2.3Changes in Rev 9.3.8000•Support FORE EMAD query operation according to section 2.17.5 of the SwitchX® PRM rev 1.05•Added support to read UCD sensor and V in sensing according to section 2.8.5 of the SwitchX® PRM rev 1.05•Added AccessRegisterGMP support for MSPS and fix MSPS firmware configuration2.4Changes in Rev 9.3.6000•Added support for bandwidth monitoring according to Performance histogram in Vendor Specific MAD spec2.5Changes in Rev 9.3.4000•Added support for fan-out-of-range event (FORE) according to PRM section “FORE – Fan Out of Range Event”•Added LLR max retransmission rate as specified in Vendor Specific MAD V1.1, Table 110 – PortLLRStatistics MAD Description2.6Changes in Rev 9.3.2000•Lowered the externally managed switches default fan speed from 80% to 60%•Added congestion control attribute “PORT_VL_XMIT_TIME_CONG” according to3.23 CounterGroupInfo in Vendor Specific MADs version 1.1•Added support for adaptive routing per L4 transport layer2.7Changes in Rev 9.3.1200•Added GA support for MFA1A00-Exxx EDR 100Gb/s AOCs over SwitchX® based systems2.8Changes in Rev 9.3.0000•Added support for SX6710 and SX1710 switch systems•Added support for timestamp in mirrored packages•Added support for MCP1600-E001/MCP1600-E002 EDR cables•Added support for FDR 2.5m (MC2207128-0A2)•Added power optimizations when ports are disabled•Added TuneX support•Bug fixes2.10Changes in Rev 9.2.6100•Added LLR capability indication as part of extended port info MAD according to Ven-dor Specific MADs document version 1.1•3rd party FDR cables are allowed to raise link at FDR (56Gb/s)•Added support for fan and power supply monitor MAD according to section 2.12 in Vendor Specific MADs document version 1.1 and section 2.17 in Mellanox PRM ver-sion 1.0•Improved FDR systems link-up time after reboot2.11Changes in Rev 9.2.4002•Added support for 12-port SX6005 switch system•Added ability to Set Node Description according to errata proposal #9109•Updated SMP and GMP CapabilityMask as appear in VS Mad Specification document version 0.56•Added support for temperature sensor MAD according to section 2.12 in Vendor Spe-cific MADs document version 1.1 and section 2.17 in Mellanox PRM version 1.0•Added support for voltage sensor MAD•Added a port disable power-saving feature•Added support for reading VPD from flash (supported with systems coming out of fac-tory with FW 9.2.4002 or higher only)•Added support for alternate source of the following fiber cable P/Ns: MC2207310-XXX-F-A2, MC2210310-XXX-F-A2, MC2206310-XXX-F-A2 (See PCN MLNX-15-838)•Added support for the MC2207312-XXX 56Gb/s active fiber cables.•Added support for MC2207310-XXX-F-A2, MC2207310-XXX-F-A2, MC2206310-XXX-F-A2 active fiber cables•Added support for MC2207411-SR4 56Gb/s active optical module2.12Changes in Rev 9.2.3000•Improved FDR and FDR10 link stability•Port mirroring bug fixes•Added best-effort mirroring capability•Fixing FDR10 switch link issue discovered in 9.2.0000•Added support for SwitchX®-2 SX6015•Added support for Congestion Control Class•Added support for InfiniBand multi-swid•Enabled MultiCastFDBTop=0xBFFF to discard MC traffic•Added a new INI field for the NodeInfo and DeviceID parameters2.14Changes in Rev 9.1.7000•Enabled LLR by default•Added Mellanox Proprietary Link negotiation (MLPN) support•Added support for MC2207312-XXX FDR optical cables up to 300m•Added support for SwitchX®-2 based FDR and FDR10 edge switch systems•Improved Eye Opener Machine2.15Changes in Rev 9.1.5000•Added InfiniBand Port Mirroring support•Reduced port PLLs jitter•Enabled Gradient Decent (GDS) in MSX6025T and MSX6025F systems•Added optimize SL2VL support•Improved link negotiation process2.16Changes in Rev 9.1.3000•Link negotiation protocols improvements2.17Changes in Rev 9.1.2000•Added PKEY enforcement on external ports2.18New Features and Changes in Rev 9.1.0000•InfiniBand switch supported speed:•FDR v1.3•FDR10 - Mellanox Technologies propriety•QDR/DDR/SDR v1.2.1•36 FDR (56Gb/s) ports in a 1U switch•FDR10 supports 20% more bandwidth over QDR using the same cables/connectors3Known IssuesTable2 describes known issues in this firmware release and possible workarounds.Table 2 - Known Issues4Bug Fixes HistoryTable3 lists the bugs fixed in this release.Table 3 - Bug Fixes History。
Mellanox 红帽企业级 Linux（RHEL）7.5 驱动用户手册说明书
Red Hat Enterprise Linux (RHEL) 7.5 DriverUser ManualMellanox Technologies350 Oakmead Parkway Suite 100Sunnyvale, CA 94085U.S.A.Tel: (408) 970-3400Fax: (408) 970-3403© Copyright 2016. Mellanox Technologies Ltd. All Rights Reserved.Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®,CoolBox®, CORE-Direct®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®,GPUDirect®, InfiniHost®, InfiniBridge®, InfiniScale®, Kotura®, Kotura logo, Mellanox CloudRack®, MellanoxCloudXMellanox®, Mellanox Federal Systems®, Mellanox HostDirect®, Mellanox Multi-Host®, Mellanox OpenEthernet®, Mellanox OpenCloud®, Mellanox OpenCloud Logo®, Mellanox PeerDirect®, Mellanox ScalableHPC®,Mellanox StorageX®, Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox Virtual ModularSwitch®, MetroDX®, MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, Open Ethernet logo, PhyX®, PlatformX®,PSIPHY®, SiPhy®, StoreX®, SwitchX®, Tilera®, Tilera logo, TestX®, TuneX®, The Generation of Open Ethernetlogo, UFM®, Unbreakable Link®, Virtual Protocol Interconnect®, Voltaire® and Voltaire logo are registeredtrademarks of Mellanox Technologies, Ltd.All other trademarks are property of their respective owners.For the most updated list of Mellanox trademarks, visit /page/trademarksNOTE:THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT (S)”) AND ITS RELATEDDOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS -IS” WITH ALL FAULTS OF ANYKIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USETHE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENTHAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S)AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOTGUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANYEXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OFMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. INNO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOTLIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISINGIN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OFTHE POSSIBILITY OF SUCH DAMAGE.Table of ContentsDocument Revision History (5)1Firmware Burning (6)2Port Type Management (7)3Modules Loading and Unloading (8)4Important Packages and Their Installation (9)5SR-IOV Configuration (10)5.1Setting up SR-IOV in ConnectX-3/ConnectX-3 Pro (10)6Default RoCE Mode Setting (12)7PXE over InfiniBand Installation (13)Table 1: Document Revision History (5)Table 1: Document Revision History1 Firmware Burning1.Check the device’s PCI address.lspci | grep MellanoxExample:00:06.0 Infiniband controller: Mellanox Technologies MT27520 Family[ConnectX-3 Pro]2.Identify the adapter card's PSID.# mstflint -d 81:00.0 qImage type: FS2FW Version: 2.36.5000FW Release Date: 26.1.2016Rom Info: type=PXE version=3.4.718 devid=4103Device ID: 4103Description: Node Port1 Port2Sys imageGUIDs: e41d2d0300b3f590 e41d2d0300b3f591 e41d2d0300b3f592e41d2d0300b3f593MACs: e41d2db3f591 e41d2db3f592VSD:PSID: MT_10901110193.Download the firmware BIN file from the Mellanox website that matches your card'sPSID: → Support/Education→Support Downloader4.Burn the firmware.# mstflint -d <lspci-device-id> -i <image-file> b5.Reboot your machine after the firmware burning is completed.2 Port Type ManagementConnectX®-3/ConnectX®-3 Pro/ConnectX®-4/ConnectX®-4 Lx/ConnectX®-5/ConnectX®-5 Ex ports can be individually configured to work as InfiniBand or Ethernetports. By default both ConnectX®-4 VPI ports are initialized as InfiniBand ports. If youwish to change the port type use the mstconfig after the driver is loaded.1.Install mstflint tools.yum install mstflint2.Check the device’s PCI address.lspci | grep MellanoxExample:00:06.0 Infiniband controller: Mellanox Technologies MT27520 Family[ConnectX-3 Pro]e mstconfig to change the link type as desired IB – for InfiniBand, ETH – for Ethernet.mstconfig –d <device pci> s LINK_TYPE_P1/2=<ETH|IB|VPI>Example:# mstconfig -d 00:06.0 s LINK_TYPE_P1=ETHDevice #1:----------Device type: ConnectX3ProPCI device: 00:06.0Configurations: Current NewLINK_TYPE_P1 IB(1) ETH(2)Apply new Configuration? ? (y/n) [n] : yApplying... Done!-I- Please reboot machine to load new configurations.4.Reboot your machine.3 Modules Loading and UnloadingMellanox modules for ConnectX®-2/ConnectX®-3/ConnectX®-3 Pro are:∙mlx4_en, mlx4_core, mlx4_ibMellanox modules for ConnectX®-4/ConnectX®-4 Lx/ ConnectX®-5/ ConnectX®-5 Exare:∙mlx5_core, mlx5_ibIn order to unload the driver, you need to first unload mlx*_en/ mlx*_ib and then themlx*_core module.To load and unload the modules, use the commands below:∙Loading the driver: modprobe <module name># modprobe mlx5_ib∙Unloading the driver: modprobe –r <module name># modprobe –r mlx5_ib4 Important Packages and Their Installationrdma-corerdma-core RDMA core userspace libraries and daemonslibibmad: Low layer InfiniBand diagnostic and management programslibibmad OpenFabrics Alliance InfiniBand MAD libraryopensm: InfiniBand Subnet Manageropensm-libs Libraries used by OpenSM and included utilitiesopensm OpenIB InfiniBand Subnet Manager and management utilitiesIbutils: OpenIB Mellanox InfiniBand Diagnostic Toolsibutils-libs Shared libraries used by ibutils binariesibutils OpenIB Mellanox InfiniBand Diagnostic Toolsinfiniband-diags: OpenFabrics Alliance InfiniBand Diagnostic Toolsinfiniband-diags OpenFabrics Alliance InfiniBand Diagnostic Toolsperftest: IB Performance testsperftest IB Performance Testsmstflint: Mellanox Firmware Burning and Diagnostics Toolsmstflint Mellanox firmware burning toolTo install the packages above run:# yum install libibverbs librdmacm libibcm libibmad libibumad libmlx4libmlx5 opensm ibutils infiniband-diags srptools perftest mstflint rdmacm-utils ibverbs-utils librdmacm-utils -y5 SR-IOV Configuration5.1 Setting up SR-IOV in ConnectX-3/ConnectX-3 Pro1.Install the mstflint tools.# yum install mstflint2.Check the device’s PCI.# lspci | grep MellanoxExample:00:06.0 Infiniband controller: Mellanox Technologies MT27520 Family[ConnectX-3 Pro]3.Check if SR-IOV is enabled in the firmware.mstconfig -d <device pci> qExample:# mstconfig -d 00:06.0 qDevice #1:----------Device type: ConnectX3ProPCI device: 00:06.0Configurations: CurrentSRIOV_EN True(1)NUM_OF_VFS 8LINK_TYPE_P1 ETH(2)LINK_TYPE_P2 IB(1)LOG_BAR_SIZE 3BOOT_PKEY_P1 0BOOT_PKEY_P2 0BOOT_OPTION_ROM_EN_P1 True(1)BOOT_VLAN_EN_P1 False(0)BOOT_RETRY_CNT_P1 0LEGACY_BOOT_PROTOCOL_P1 PXE(1)BOOT_VLAN_P1 1BOOT_OPTION_ROM_EN_P2 True(1)BOOT_VLAN_EN_P2 False(0)BOOT_RETRY_CNT_P2 0LEGACY_BOOT_PROTOCOL_P2 PXE(1)BOOT_VLAN_P2 1IP_VER_P1 IPv4(0)IP_VER_P2 IPv4(04.Enable SR-IOV:mstconfig –d <device pci> s SRIOV_EN=<False|True>5.Configure the needed number of VFsmstconfig –d <device pci> s NUM_OF_VFS=<NUM>NOTE: This file will be generated only if IOMMU is set in the grub.conf file (byadding “intel_iommu=on” to /boo t/grub/grub.conf file).6.[mlx4 devices only] Create/Edit the file /etc/modprobe.d/mlx4.conf:options mlx4_core num_vfs=[needed num of VFs] port_type_array=[1/2 for IB/ETH],[ 1/2 for IB/ETH]Example:options mlx4_core num_vfs=8 port_type_array=1,17.[mlx5 devices only] Write to the sysfs file the number of needed VFs.echo [num_vfs] > sys/class/net/ib2/device/sriov_numvfsExample:# echo 8 > /sys/class/net/ib2/device/sriov_numvfs8.Reboot the driver.9.Load the driver and verify that the VFs were created.# lspci | grep mellanoxExample:00:06.0 Network controller: Mellanox Technologies MT27520 Family[ConnectX-3 Pro]00:06.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.7 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.0 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]For further information, refer to section Setting Up SR-IOV MLNX_OFED User Manual.6 Default RoCE Mode Setting1.Mount the configfs file.# mount -t configfs none /sys/kernel/config2.Create a directory for the mlx4/mlx5 device.# mkdir -p /sys/kernel/config/rdma_cm/mlx4_0/3.Validate what is the used RoCE mode in the default_roce_mode configfs file.# cat /sys/kernel/config/rdma_cm/mlx4_0/ports/1/default_roce_modeIB/RoCE v14.Change the default RoCE mode,∙For RoCE v1: IB/RoCE v1∙For RoCE v2: RoCE v2# echo "RoCE v2" >/sys/kernel/config/rdma_cm/mlx4_0/ports/1/default_roce_mode# cat /sys/kernel/config/rdma_cm/mlx4_0/ports/1/default_roce_modeRoCE v2# echo "IB/RoCE v1" >/sys/kernel/config/rdma_cm/mlx4_0/ports/1/default_roce_mode# cat /sys/kernel/config/rdma_cm/mlx4_0/ports/1/default_roce_modeIB/RoCE v17 PXE over InfiniBand InstallationPXE over InfiniBand infrastructure has additional parameter in the Boot Loader file forloading the necessary modules and interfaces and for allowing sufficient time to get the link.To install RHEL from PXE using the IPoIB interfaces, add the following parameters to theBoot Loader file, located in the var/lib/tftpboot/pxelinux.cfg directory, at thePXE server:bootdev=ib0 ksdevice=ib0 net.ifnames=0 biosdevname=0 rd.neednet=1rd.bootif=0 rd.driver.pre=mlx5_ib,mlx4_ib,ib_ipoib ip=ib0:dhcp.dhcp.retry=10 .timeout.iflink=60 .timeout.ifup=80.timeout.carrier=80Example:default RH7.5prompt 1timeout 600label RH7.5kernelappend bootdev=ib0 ksdevice=ib0 net.ifnames=0 biosdevname=0 rd.neednet=1rd.bootif=0 rd.driver.pre=mlx5_ib,mlx4_ib,ib_ipoib ip=ib0:dhcp.dhcp.retry=10 .timeout.iflink=60 .timeout.ifup=80.timeout.carrier=80。
Mellanox 光纤网卡驱动Driver for Linux的安装
\
英文原厂文档
/related-docs/prod_software/Mellanox_EN_for_Linux_Release_Notes_v3_4-1_0_0_3.pdf
Mellanox ConnectX®-4
Mellanox ConnectX®-4 Lx
Mellanox ConnectX®-3/ConnectX®-3 Pro
Mellanox ConnectX®-2
Mellanox PCI Express 2.0
Mellanox PCI Express 3.0
2
2.1下载驱动的地址比较难找
Mellanox EN Driver for Linux的安装
编者按：Redhat默认支持博科和intel的网卡驱动，对于其他厂商的光纤网卡，或者统后，网卡不能识别工作，安装厂家驱动后正常工作的情况
1
Mellanox EN Driver for Linux支持的网络光纤卡系列有大致以下几种：
2.2Redhat 7系列安装比较麻烦
3
1、在redhat/centos/Oracle linux 6等其他linux上安装驱动
2、在redhat/centos/Oracle linux等其他linux 7上安装驱动
(原程序不支持redhat7，需要重新编译添加支持)
3、驱动下载地址
第一部分
redhat 6.X(在6.6测试过)
yum install kernel-devel-3.10.0-229.el7.x86_64 redhat-rpm-config
---安装网卡驱动
[root@localhost /itsm/mlnx-en-3.4-1.0.0.3-ol7.1-x86_64 ]#./install --add-kernel-support
Mellanox 直接连接铜（DAC）被动电缆组件说明书
Mellanox® direct attach copper (DAC) passive cable assemblies are high speed, cost-effective alternativesto fiber optics in InfiniBand EDR 100Gb/s applications.Based on the QSFP28 electrical connector, 100G EDR DAC cables are configured as 4 channels of 25G NRZ.Each QSFP28 connector includes an EEPROM providing product information which can be read by the hostsystem.Mellanox’s unique quality, passive, copper cable solutions provide power-efficient connectivity for shortdistance interconnects. They enable higher port bandwidth, density and configurability at a low cost andreduced power requirement in the data centers.Rigorous cable production testing ensures best out-of-the-box installation experience, performance anddurability.T able 1 - Absolute Maximum RatingsT able 2 - Operational SpecificationsT able 3 - Electrical SpecificationsMCP1600-ExxxINTERCONNECTPRODUCT BRIEF†100Gb/s QSFP28 Direct Attach Copper Cable©2018 Mellanox Technologies. All rights reserved.†For illustration only. Actual products may vary.Table 4 - Cable Mechanical SpecificationsTFigure 1.Cable Length Definition©2018 Mellanox Technologies. All rights reserved.350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: © Copyright 2018. Mellanox Technologies. All rights reserved.Mellanox and Mellanox logo are registered trademarks of Mellanox Technologies, Ltd.LinkX is a trademark of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners.Warranty InformationMellanox LinkX direct attach copper cables include a 1-year limited hardware warranty, which covers parts repair or replacement.Mechanical Schematics53311PBRev 1.72。
Mellanox Quantum QM8700 InfiniBand交换机产品说明说明书
©2020 Mellanox Technologies. All rights reserved.†For illustration only. Actual products may vary.© Copyright 2020. Mellanox, Mellanox logo, Connect-X, MLNX-OS, and UFM are registered trademarks of Mellanox Technologies, Ltd. Mellanox Quantum and Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners.page 2Mellanox QM8700 InfiniBand Switch 350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: Table 1 - Part Numbers and Descriptions53779PB Rev 2.2Note : All tall-bracket adapters are shipped with the tall bracket mounted and a short bracket as accessory.Support : For information about our support packages, please contact your Mellanox Technologies sales representative or visit our Support Index page .HDR100QM8700 together with the Mellanox ConnectX ®-6 adapter card support HDR100. By utilizing two pairs of two lanes per port, theQM8700 can support up to 80 ports of 100G to create the densest TOR switch available in the market. This is a perfect solution for double dense racks with more than 40 servers per rack and also helps small-medium deployments with the need to scale to 3-level fat-tree, to lower power, latency and space.MANAGEMENTThe QM8700’s x86 ComEx Broadwell CPU comes with an on-board subnet manager, enabling simple, out-of-the-box bring-up for up to 2K nodes in the fabric. Running the MLNX-OS ® software package, it delivers full chassis management through CLI, WebUI, SNMP or JSON interfaces.QM8700 also incorporates Mellanox’s Unified Fabric Manager (UFM ®) software for managing scale-out, InfiniBand, computing environments to enable efficient provisioning, health indications and monitoring of the cluster. UFM ® ensures that the fabric is up and running at maximum performance at all times.Safety –CB –cTUVus –CE –CUEMC (Emissions) –CE –FCC –VCCI –ICES –RCMOperating Conditions –Temperature:• Operating 0ºC to 40ºC• Non-operating -40ºC to 70ºC –Humidity:• Operating 10% to 85% non-condensing• Non-operating 10% to 90% non-condensing–Altitude: Up to 3200m Acoustic –ISO 7779 –ETS 300 753Others–RoHS compliant –Rack-mountable, 1U –1-year warrantyCOMPLIANCEMellanox QM8700–19’’ rack mountable 1U chassis –40 QSFP56 non-blocking ports with aggregate data throughput up to 16Tb/s (HDR)Switch Specifications–Compliant with IBTA 1.21 and 1.3 –9 virtual lanes: 8 data + 1 management–256 to 4Kbyte MTU –Adaptive Routing –Congestion control –Port Mirroring–VL2VL mapping–4X48K entry linear forwarding databaseManagement Ports–100/1000 RJ45 Ethernet port –RS232 console port –USB port –DHCP–Industry standard CLI –Management over IPv6 –Management IP –SNMP v1,v2,v3 –WebUIFabric Management–On-board Subnet Manager (SM) supporting fabrics of up to 2K nodes –Unified Fabric Manager (UFM) agentConnectors and Cabling–QSFP56 connectors–Passive copper or active fiber cables–Optical modulesIndicators–Per port status LED Link, Activity–System LEDs: System, fans, powersupplies–Unit ID LEDPower Supply–Dual redundant slots –Hot plug operation –Input range:100-127VAC, 200-240VAC –Frequency: 50-60Hz, single phase AC, 4.5A, 2.9A Cooling –Front-to-rear or rear-to-front cooling option–Hot-swappable fan unitPower Consumption –Contact Mellanox Sales FEATURES。
Mellanox 应用识别引用文件说明书
Reference GuideTable of ContentsChapter 1. Introduction (1)Chapter 2. System Design (2)Chapter 3. Application Architecture (5)Chapter 4. Configuration Flow (6)Chapter 5. Running Application (8)Chapter 6. References (10)Chapter 1.IntroductionApplication Recognition (AR) allows identifying applications that are in use on a monitored networking node.AR enables the security administrator to generate consolidated reports that show usage patterns from the application perspective. AR is also used as a corner stone of many security applications such as L7-based firewalls.Due to the massive growth in the number of applications that communicate over Layer 7 (HTTP), effective monitoring of network activity requires looking deeper into Layer 7 traffic so individual applications can be identified. Different applications may require different levels of security and service.This document describes how to build AR using the deep packet inspection (DPI) engine, which leverages NVIDIA® BlueField®-2 DPU capabilities such as regular expression (RXP) acceleration engine, hardware-based connection tracking, and more.Chapter 2.System DesignThe AR application is designed to run as "bump-on-the-wire" on the BlueField-2 instance, it intercepts the traffic coming from the wire, and passes it to the Physical Function (PF) representor connected to the host.System DesignSystem DesignChapter 3.Application ArchitectureAR runs on top of Data Plan Development Kit (DPDK) based Stateful Flow Tracking (SFT) to identify the flow that each packet belongs to, then uses DPI to process L7 classification.1.Signatures are compiled by DPI compiler and then loaded to DPI engine2.Ingress traffic is identified using Connection Tracking module3.Traffic is scanned against DPI engine compiled signature DB4.Post processing is performed for match decision5.Matched flows are identified and actions can be executed (Allow/Deny)Chapter 4.Configuration Flow1.DPDK initializationdpdk_init(&argc, &argv, &nb_queues);2.AR initializationar_init(argc, argv, cdo_filename, csv_filename);a).Initialize NetFlow using default configuration /etc/doca_netflow.conf.b).Initialize signature database.3.Stateful Flow Table (SFT) and Port initializationflow_offload_init(nb_queues);a).SFT initialization.b).Mempool allocation.c).Port initialization.4.DPI initializationdpi_ctx = doca_dpi_init(&doca_dpi_config, &err);a).Configure RegEx engine.b).Configure DPI queues.5.Load compiled signatures to RegEx engine.doca_dpi_load_signatures(dpi_ctx, ar_config.cdo_filename);6.Configure DPI packet processing.dpi_worker_lcores_run(nb_queues, CLIENT_ID, ar_worker_attr);a).Configure DPI enqueue packets.b).Send jobs to RegEx engine.c).Configure DPI dequeue packets.7.Send statistics and write database.sig_database_write_to_csv(ar_config.csv_filename);send_netflow();a).Send statistics to the collector.b).Write CSV file with signatures statistics.8.AR destroyar_destroy(cmdline_thread, ar_config);‣Clear thread9.DPI destroydoca_dpi_destroy(dpi_ctx);Configuration Flow ‣Free DPI resourcesChapter 5.Running Application‣PrerequisitesPlease refer to the DOCA Installation Guide for details on how to install BlueField related software.‣BlueField software image 3.6‣DOCA software package 1.0‣To build the application:1.Prepare the environment variables (DPDK is installed under /opt/mellanox). Run:‣For Ubuntu:export LD_LIBRARY_PATH=/opt/mellanox/dpdk/lib/aarch64-linux-gnu/‣For CentOS:export LD_LIBRARY_PATH=/opt/mellanox/dpdk/lib642.The application recognition example is installed as part of the doca-dpi-lib package,the binary is located under/opt/mellanox/doca/examples/ar/bin/doca_app_rec.To re-build the application recognition sample, run:‣For Ubuntu:export PKG_CONFIG_PATH=/opt/mellanox/dpdk/lib/aarch64-linux-gnu/pkgconfig/cd /opt/mellanox/doca/examples/ar/srcmeson /tmp/buildninja -C /tmp/build‣For CentOS:export PKG_CONFIG_PATH=/opt/mellanox/dpdk/lib64/pkgconfig/cd /opt/mellanox/doca/examples/ar/srcmeson /tmp/buildninja -C /tmp/builddoca_app_rec will be created under build/app.3.The application recognition example is a DPDK application. Therefore, the user isrequired to provide DPDK flags, and allocate huge pages. Run:echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages‣To run the application:doca_app_rec [flags] -- --cdo [cdo_file] -output_csv [output_csv_file] -print_matchFor example:Running Application/opt/mellanox/doca/examples/ar/bin/doca_app_rec -a0000:03:00.0,class=regex:eth,representor=[65535],sft_en=0 -- --c /usr/etc/doca/ dpi/ar/ar.cdo –o ar_stats.csv –pNote: The flag -a 0000:03:00.0,class=regex:eth,representor=[65535],sft_en=0is a must for proper usage of the application. Modifying this flag will result unexpectedbehavior as only 2 ports (an uplink and a representor) are supported.To print the output when the DPI engine finds a match, use --print_match.For additional information about the available flags for DPDK use -h before the -separator. For the application, use -h after the --.The application will periodically dump a .csv file with the recognition results containing statistics about the recognized apps in the format SIG_ID, APP_NAME, MATCHING_FIDS, and DROP.As per the example above, a file called ar_stats.csv will be created.Additional features can be triggered by using the shell interaction. This allows blocking and unblocking specific signature IDs using the following commands:‣block <sig_id>‣unblock <sig_id>The TAB key allows autocompletion while the quit command terminates the application. Application flags:‣-c or --cdo <path> – path to CDO file compiled from a valid PDD‣-o or --output_csv <path> – path to the output of the CSV file‣-p or --print_match – prints FID when matched in DPI engine‣-i or --interactive – adds interactive mode for blocking signatures‣-n or -netflow – exports data from BlueField to remote NetFlow collector NetFlow collector UI example:Chapter 6.References ‣/opt/mellanox/doca/examples/ar/src/ar.cNoticeThis document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation nor any of its direct or indirect subsidiaries (collectively: “NVIDIA”) make no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assume no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality.NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete.NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document.NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at customer’s own risk. NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to evaluate and determine the applicability of any information contained in this document, ensure the product is suitable and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product designs.No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual property right under this document. Information published by NVIDIA regarding third-party products or services does not constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property rights of the third party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA.Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all associated conditions, limitations, and notices.THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be limited in accordance with the Terms of Sale for the product.TrademarksNVIDIA, the NVIDIA logo, and Mellanox are trademarks and/or registered trademarks of Mellanox Technologies Ltd. and/or NVIDIA Corporation in the U.S. and in other countries. Other company and product names may be trademarks of the respective companies with which they are associated.Copyright© 2021 NVIDIA Corporation. All rights reserved.NVIDIA Corporation | 2788 San Tomas Expressway, Santa Clara, CA 95051。
Mellanox Ethernet 网络设备用户手册说明书
SOLUTION BRIEFKEY BUSINESS BENEFITSEXECUTIVE SUMMARYAnalytic tools such as Spark, Presto and Hive are transforming how enterprises interact with and derive value from their data. Designed to be in memory, these computing and analytical frameworks process volumes of data 100x faster than Hadoop Map/Reduce and HDFS - transforming batch processing tasks into real-time analysis. These advancements have created new business models while accelerating the process of digital transformation for existing enterprises.A critical component in this revolution is the performance of the networking and storage infrastructure that is deployed in support of these modern computing applications. Considering the volumes of data that must be ingested, stored, and analyzed, it quickly becomes evident that the storage architecture must be both highly performant and massively scalable.This solution brief outlines how the promise of in-memory computing can be delivered using high-speed Mellanox Ethernet infrastructure and MinIO’s ultra-high performance object storage solution.IN MEMORY COMPUTINGWith data constantly flowing from multiple sources - logfiles, time series data, vehicles,sensors, and instruments – the compute infrastructure must constantly improve to analyze data in real time. In-memory computing applications, which load data into the memory of a cluster of servers thereby enabling parallel processing, are achieving speeds up to 100x faster than traditional Hadoop clusters that use MapReduce to analyze and HDFS to store data.Although Hadoop was critical to helping enterprises understand the art of the possible in big data analytics, other applications such as Spark, Presto, Hive, H2O.ai, and Kafka have proven to be more effective and efficient tools for analyzing data. The reality of running large Hadoop clusters is one of immense complexity, requiring expensive administrators and a highly inefficient aggregation of compute and storage. This has driven the adoption of tools like SparkDelivering In-memory Computing Using Mellanox Ethernet Infrastructure and MinIO’s Object Storage SolutionMinIO and Mellanox: Better TogetherHigh performance object storage requires the right server and networking components. With industryleading performance combined with the best innovation to accelerate data infrastructure Mellanox provides the networking foundation needed to connect in-memory computing applications with MinIO high performance object storage. Together, they allow in-memory compute applications to access and process large amounts of data to provide high speed business insights.Simple to Deploy, Simpler to ManageMinIO can be installed and configured within minutes simply by downloading a single binary and executing it. The amount of configuration options and variations has been kept to a minimum resulting in near-zero system administration tasks and few paths to failures. Upgrading MinIO is done with a single command which is non-disruptive and incurs zero downtime.MinIO is distributed under the terms of the Apache* License Version 2.0 and is actively developed on Github. MinIO’s development community starts with the MinIO engineering team and includes all of the 4,500 members of MinIO’s Slack Workspace. Since 2015 MinIO has gathered over 16K stars on Github making it one of the top 25 Golang* projects based on a number of stars.which are simpler to use and take advantage of the massive benefits afforded by disaggregating storage and compute. These solutions, based on low cost, memory dense compute nodes allow developers to move analytic workloads into memory where they execute faster, thereby enabling a new class of real time, analytical use cases.These modern applications are built using cloud-native technologies and,in turn, use cloud-native storage. The emerging standard for both the public and private cloud, object storage is prized for its near infinite scalability and simplicity - storing data in its native format while offering many of the same features as block or file. By pairing object storage with high speed, high bandwidth networking and robust compute enterprises can achieve remarkable price/performance results.DISAGGREGATE COMPUTE AND STORAGE Designed in an era of slow 1GbE networks, Hadoop (MapReduce and HDFS) achieved its performance by moving compute tasks closer to the data. A Hadoop cluster often consists of many 100s or 1000s of server nodes that combine both compute and storage.The YARN scheduler first identifies where the data resides, then distributes the jobs to the specific HDFS nodes. This architecture can deliver performance, but at a high price - measured in low compute utilization, costs to manage, and costs associated with its complexity at scale. Also, in practice, enterprises don’t experience high levels of data locality with the results being suboptimal performance.Due to improvements in storage and interconnect technologies speeds it has become possible to send and receive data remotely at high speeds with little (less than 1 microsecond) to no latency difference than if the storage were local to the compute.As a result, it is now possible to separate storage from the compute with no performance penalty. Data analysis is still possible in near real time because the interconnect between the storage and the compute is fast enough to support such demands.By combining dense compute nodes, large amounts of RAM, ultra-highspeed networks and fast object storage, enterprises are able to disaggregate storage from compute creating the flexibility to upgrade, replace, or add individual resources independently. This also allows for better planning for future growth as compute and storage can be added independently and when necessary, improving utilization and budget control.Multiple processing clusters can now share high performance object storage so that different types of processing, such as advanced queries, AI model training, and streaming data analysis, can run on their own independent clusters while sharing the same data stored on the object storage. The result is superior performance and vastly improved economics.HIGH PERFORMANCE OBJECT STORAGEWith in-memory computing, it is now possible to process volumes of data much faster than with Hadoop Map/Reduce and HDFS. Supporting these applications requires a modern data infrastructure with a storage foundation that is able to provide both the performance required by these applications and the scalability to handle the immense volume of data created by the modern enterprise.Building large clusters of storage is best done by combining simple building blocks together, an approach proven out by the hyper-scalers. By joining one cluster with many other clusters, MinIO can grow to provide a single, planet-wide global namespace. MinIO’s object storage server has a wide rangeof optimized, enterprise-grade features including erasure code and bitrot protection for data integrity, identity management, access management, WORM and encryption for data security and continuous replication and lamba compute for dynamic, distributed data.MinIO object storage is the only solution that provides throughput rates over 100GB/sec and scales easily to store 1000s of Petabytes of data under a single namespace. MinIO runs Spark queries faster, captures streaming data more effectively, and shortens the time needed to test, train and deploy AI algorithms.LATENCY AND THROUGHPUTIndustry-leading performance and IT efficiency combined with the best of open innovation assist in accelerating big data analytics workloads which require intensive processing. The Mellanox ConnectX® adapters reduce the CPU overhead through advanced hardware-based stateless offloads and flow steering engines. This allows big data applications utilizing TCP or UDP over IP transport to achieve the highest throughput, allowing completion of heavier analytic workloads in less time for big data clusters so organizations can unlock and efficiently scale data-driven insights while increasing application densities for their business.Mellanox Spectrum® Open Ethernet switches feature consistently low latency and can support a variety of non-blocking, lossless fabric designs while delivering data at line-rate speeds. Spectrum switches can be deployed in a modern spine-leaf topology to efficiently and easily scalefor future needs. Spectrum also delivers packet processing without buffer fairness concerns. The single shared buffer in Mellanox switches eliminates the need to manage port mapping and greatly simplifies deployment. In an© Copyright 2019. Mellanox, Mellanox logo, and ConnectX are registered trademarks of Mellanox Technologies, Ltd. Mellanox Onyx is a trademark of Mellanox Technologies, Ltd. All other trade-marks are property of their respective owners350 Oakmead Parkway, Suite 100 Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: MLNX-423558315-99349object storage environment, fluid resource pools will greatly benefit from fair load balancing. As a result, Mellanox switches are able to deliver optimal and predictable network performance for data analytics workloads.The Mellanox 25, 50 or 100G Ethernet adapters along with Spectrum switches results in an industry leading end-to-end, high bandwidth, low latency Ethernet fabric. The combination of in-memory processing for applications and high-performance object storage from MinIO along with reduced latency and throughput improvements made possible by Mellanox interconnects creates a modern data center infrastructure that provides a simple yet highly performant and scalable foundation for AI, ML, and Big Data workloads.CONCLUSIONAdvanced applications that use in-memory computing, such as Spark, Presto and Hive, are revealing business opportunities to act in real-time on information pulled from large volumes of data. These applications are cloud native, which means they are designed to run on the computing resources in the cloud, a place where Hadoop HDFS is being replaced in favor of using data infrastructures that disaggregates storage from compute. These applications now use object storage as the primary storage vehicle whether running in the cloud or on- premises.Employing Mellanox networking and MinIO object storage allows enterprises to disaggregate compute from storage achieving both performance and scalability. By connecting dense processing nodes to MinIO object storage nodes with high performance Mellanox networking enterprises can deploy object storage solutions that can provide throughput rates over 100GB/sec and scales easily to store 1000s of Petabytes of data under a singlenamespace. The joint solution allows queries to run faster, capture streaming data more effectively, and shortens the time needed to test, train and deploy AI algorithms, effectively replacing existing Hadoop clusters with a data infrastructure solution, based on in-memory computing, that consumes a smaller data center footprint yet provides significantly more performance.WANT TO LEARN MORE?Click the link below to learn more about object storage from MinIO VAST: https://min.io/Follow the link below to learn more about Mellanox end-to-end Ethernet storage fabric:/ethernet-storage-fabric/。
Mellanox MLNX-OS 发行说明书：IBM SX90Y3452 软件版本 3.4.0000
Mellanox MLNX-OS® Release Notes for IBM SX90Y3452Software Ver. 3.4.0000Table of ContentsChapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Chapter 2 Supported Platforms, Firmware, Cables and Licenses . . . . . . . . . . . . . . . . . 42.1 Supported Switch Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2 Supported CPU Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.3 Supported Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.4 Supported Mezzanine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.5 Supported CPLD Version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.6 Supported Software Licenses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.7 Upgrade From Previous Releases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.8 Supported Cables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Chapter 3 Changes and New Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Chapter 4 Known Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.1 General Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.2 InfiniBand Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Chapter 5 Bug Fixes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Chapter 6 Submitting a Service Request. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131IntroductionThis document is the Mellanox MLNX-OS® Release Notes for IBM SX90Y3452.MLNX-OS is a comprehensive management software solution that provides optimal perfor-mance for cluster computing, enterprise data centers, and cloud computing over MellanoxSwitchX® family. The fabric management capabilities ensure the highest fabric performancewhile the chassis management ensures the longest switch up time.The MLNX-OS documentation package includes the following documents:•User Manual – provides general information about the scope, organization and com-mand line interface of MLNX-OS as well as basic configuration examples•Release Notes – provides information on the supported platforms, changes and new fea-tures, and reports on software known issues as well as bug fixes2Supported Platforms, Firmware, Cables and Licenses 2.1Supported Switch SystemsTable 1 - Supported Switch Systems2.2Supported CPU Architecture•PPC 460 CPU families2.3Supported Firmware•SwitchX® firmware version 9.2.9190•SwitchX-2 firmware version 9.2.9190•ConnectX®-2 firmware version 2.9.1200 and higher•ConnectX-3 firmware version 2.30.8000 and higher2.4Supported Mezzanine•ConnectX-2, Mezzanine P/N 90Y3460 (MalayaP), 2.9.1316 and higher•ConnectX-2, Mezzanine P/N 90Y3480 (MalayaP-Net), 2.9.1318 and higher•ConnectX-3, Mezzanine P/N 90Y3488 (Merlin), 2.32.5100 and higher•ConnectX-3, Mezzanine P/N 90Y3484 (Nevada), 2.32.5100 and higher•ConnectX-3, Mezzanine P/N 90Y3456 (MalayaX), 2.32.5100 and higher•ConnectX-3, Mezzanine P/N 90Y3468 (MalayaX-Net), 2.32.5100 and higher2.5Supported CPLD Version• 1.0.182.6Supported Software LicensesFor the software licenses supported with MLNX-OS® software please refer to the “Licenses” section of the “Getting Started” chapter of the Mellanox MLNX-OS User Manual .2.7Upgrade From Previous ReleasesOlder versions of MLNX-OS may require upgrading to one or more intermediate versions prior to upgrading to the latest. Missing an intermediate step may lead to errors. Please refer to Table 2to identify the correct upgrade order.Table 2 - Supported Software Upgrades for SX90Y3452For upgrade instructions refer to the section “Upgrading MLNX-OS Software” in Mellanox MLNX-OS User Manual .2.8Supported CablesFor a list of the Mellanox supported cables please visit the LinkX™ Cables and Transceivers page of the Mellanox Website at /page/cables?mtag=cable_overview.When using Mellanox AOC cables longer than 50m use one VL to achieve full wirespeed.3Changes and New Features Table 3 - IBM SX90Y3452 Changes and New Features4Known IssuesThe following table describes MLNX-OS® known issues in this software release and possible workarounds.4.1General Known IssuesTable 4 - General Known Issues (Sheet 1 of 4)Table 4 - General Known Issues (Sheet 2 of 4)4.2InfiniBand Known IssuesTable 5 - InfiniBand Known Issues5Bug FixesThe following table describes MLNX-OS® bug fixes in this software release.Table 6 - Bug Fixes6Submitting a Service RequestThe Mellanox® Support Center is at your service for any issues. You may access the Warranty Service through the Web Request Form by using the following link:/content/pages.php?pg=support_index.。
Mellanox InfiniBand路由器SB7780和SB7880的产品介绍说明书
©2020 Mellanox Technologies. All rights reserved.†For illustration only. Actual products may vary.SB7780 and SB7880 InfiniBand routers enable new levels of subnets isolation and compute-to-storage connectivity, critical to large-scale and diverse data-centers.Scaling-Out Data Centers with EDR 100G InfiniBandHigh Performance Computing (HPC), Artificial Intelligence (AI), and Data-Intensive and Cloud infrastructures all leverage InfiniBand’s high data throughput, extremely low latency, and smart In-Network Computing acceleration engines to deliver world-leading application performance and scalability, while reducing operational costs and infrastructure complexity. Mellanox’s innovative In-Networking-based Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)™ technology enables the acceleration of communications frameworks using embedded hardware, resulting in order of magnitude application and performance improvements.In cases where the separation between InfiniBand subnets and keeping connectivity to a central InfiniBand Storage are required, InfiniBand Routers are the ideal solution.Sustained Network PerformanceSB7780 / SB7880 InfiniBand routers are based on Switch-IB ® / Switch-IB ® 2 ASICs, respectively. They offer fully-flexible 36 EDR 100Gb/s ports that can be split among several different subnets. InfinBand routers bring two major enhancements to the Mellanox InfiniBand portfolio:•Increase resiliency by s plitting the data center’s network into several subnets; each subnet runs itsown subnet manager, effectively isolating each subnet from the others’ availability or instability •Enable multiple compute to storage connections while separately isolating login access to each compute subnetWorld-Class DesignSB7780 / SB7880 InfiniBand routers are elegantly designed for performance, serviceability, energy savings and high-availability. They come equipped with two highly efficient, 80 gold+ and energy star certified, power supplies.Their best-in-class design enables the EDR IB routers to support low power consumption, with ATIS weighted power consumption as low as 122W for a fully populated system. This means more power reduction if not all ports are used or fully utilized.SB7780 / SB7880 also have redundant power supplies (1+1) and their fans draw (N+1) both with air shutters for achieving maximal thermal protection.36-port Non-blocking Managed EDR 100Gb/s InfiniBand RoutersInfiniBand EDR 100Gb/s RoutersPRODUCT BRIEFSWITCH SYSTEM †© Copyright 2020. Mellanox Technologies. All rights reserved. Mellanox, Mellanox logo, MLNX-OS , Switch-IB, and UFM are registered trademarks of Mellanox Technologies, Ltd. Switch-IB, UFM and Unified Fabric Manager are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners.350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: Mellanox SB7700 and SB7880 InfiniBand Routerspage 2ManagementThe InfiniBand router’s dual-core x86 CPU runs MLNX-OS ® software package, which delivers complete chassis management of the router’s firmware, power supplies, fans and ports. The router can also becoupled with Mellanox’s Unified Fabric Manager (UFM ®) platforms to manage scale-out InfiniBand computing environments.Revolutionizing data center management, the UFM family of products combine enhanced, real-time network telemetry with AI-poweredcyber intelligence and analytics. The UFM solutions minimize downtime by enabling system admins to quickly detect and respond to potential security threats and operational issues, and predict upcoming failures.Mellanox SB7780 / SB7880–19’’ rack mountable 1U chassis –36 QSFP28 non-blocking ports with aggregate data throughput up to 7Tb/s (EDR)Router Specifications–Compliant with IBTA 1.21 and 1.3 –9 virtual lanes: 8 data + 1 management–256 to 4Kbyte MTU–8X 48K entry linear forwarding databaseManagement Ports–100/1000Mb/s Ethernet ports –RS232 port over DB9–USB port –DHCP–Familiar Industry Standard CLI –Management over IPv6 –Management IP –SNMP v1,v2,v3 –Web UIFabric Management–Unified Fabric Manager (UFM™) AgentConnectors and Cabling–QSFP28 connectors–Passive copper or active fiber cables –Optical modulesIndicators–Per port status LED Link, Activity –System status LEDs: System, fans, power supplies –Port Error LED –Unit ID LEDPhysical Characteristics–Dimensions: 1.7’’ (43.6 mm) H x 16.85’’ (428mm) W x 27’’ (685.8mm) D –Weight: 11kg (24.2lb)Power Supply–Dual redundant slots –Hot plug operation –Input range:100-127 VAC, 200-240VAC–Frequency: 50-60Hz, single phase AC, 4.5A, 2.9ACooling–Front-to-rear or rear-to-front cooling option–Hot-swappable fan unitPower Consumption–Typical Power with Passive Cables (ATIS): 136WFEATURESSafety–CB –cTUVus –CE –CUEMC (Emissions)–CE –FCC–VCCI –ICES –RCMOperating Conditions–Temperature:–Operating 0ºC to 45ºC–Non-operating -40ºC to 70ºC–Humidity:–Operating 10% to 85% non-condensing–Non-operating 10% to 90% non-condensing–Altitude: Operating -60m to 3200mAcoustic–ISO 7779 –ETS 300 753Others–RoHS compliant –Rack-mountable, 1U –1-year warrantyCOMPLIANCETable 1 - Part Numbers and Descriptions*P 2C is connector side outlet; C 2P is connector side inlet.15-7858PB Rev 1.3。
Mellanox OFED 安装指南说明书
INSTALLING UBUNTU OS FOR MELLANOX OPENFABRICS ENTERPRISE LINUX DISTRIBUTION ON HPE INFINIBAND EDR/ETHERNET 100G ADAPTERSInstallation guideCONTENTSIntroduction (3)About this document (3)System requirements (3)Installation process overview (3)Verify network adapter is installed (3)Download the Mellanox drivers (4)Install the OFED installation package (4)Post-installation actions (8)Additional links (11)INTRODUCTIONHPE and Mellanox® have aligned to enable end-to-end connectivity solutions for hyperconverged infrastructures. By leveraging the highest throughputs and lowest latency that Mellanox intelligent interconnect solutions provide, HPE customers can take advantage of data delivery speeds of up to 100Gb/s InfiniBand or 10/25/40/50/56/100Gb/s Ethernet. These speeds enable customers to run high-performance workloads such as real-time response and extreme computing.About this documentThis document is intended for users familiar with Linux® environments. This guide covers the steps for installing the Mellanox Linux OpenFabrics Enterprise Distribution (OFED), the software stack used to manage InfiniBand and Ethernet network adapter cards, specifically using the Mellanox ConnectX®-5 platform for HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter 872726-B21. IMPORTANTSuperuser privileges are required for installation. Use the superuser (root) or use the sudo prefix if the sudo package is enabled.SYSTEM REQUIREMENTSThe following table lists the system requirements to install the Mellanox InfiniBand drivers.TABLE 1. Installation requirementsRequirementOperating system Ubuntu 18.04NIC HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter 872726-B21INSTALLATION PROCESS OVERVIEWThe high-level steps for installing the drivers are:1.Verify the network adapter is present.2.Download the Mellanox drivers (OFED file).3.Install the OFED installation package.4.Perform the post-installation actions.5.Update the firmware.Verify network adapter is installedIssue the following command to verify the network adapter is installed:user@ubuntu-gpu1:~$ lspci -v | grep MellanoxSample output:user@ubuntu-gpu1:~$ lspci -v | grep Mellanox86:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]86:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]Download the Mellanox driversThe Mellanox drivers are available at: https:///page/products_dyn?product_family=26&mtag=linux_sw_driversFIGURE 1. MLNX_ OFED Download Center screenSelect the following options from the Mellanox Download Center:•Version: 4.7-1.0.0.1•OS Distribution: Ubuntu•OS Distribution Version: Ubuntu 18.04•Architecture: As applicable•Download: tgz: MLNX_OFED_LINUX-4.7-1.0.0.1-ubuntu18.04-x86_64.tgzInstall the OFED installation packageFollow these steps to install the OFED package.Installation of the OFED package is executed by running the installation script mlnxofedinstall. Refer to the MLNX_OFED Documentation Rev 4.7-1.0.0.1 for other installation methods.1.Copy the downloaded OFED file to the /tmp directory.user@ubuntu-gpu1: cp MLNX_OFED_LINUX-4.7-1.0.0.1-ubuntu18.04-x86_64$ /tmp2.Issue the following commands to extract the .tar file.user@ubuntu-gpu1:~$ cd /tmpuser@ubuntu-gpu1:/tmp$ sudo tar -xzf ./MLNX_OFED_LINUX-4.7-1.0.0.1-ubuntu18.04-x86_64.tgz3.Run the installation script (mlnxofedinstall) by following these commands:user@ubuntu-gpu1:/tmp$ cd MLNX_OFED_LINUX-4.7-1.0.0.1-ubuntu18.04-x86_64/user@ubuntu-gpu1:/tmp/MLNX_OFED_LINUX-4.7-1.0.0.1-ubuntu18.04-x86_64$ sudo ./mlnxofedinstall Sample output:Logs dir: /tmp/MLNX_OFED_LINUX.3524.logsGeneral log file: /tmp/MLNX_OFED_LINUX.3524.logs/general.logThe following list shows the MLNX_OFED_LINUX packages that you have chosen (some might have been added by the installer due to package dependencies):ofed-scriptsmlnx-ofed-kernel-utilsmlnx-ofed-kernel-dkmsrshim-dkmsiser-dkmssrp-dkmslibibverbs1ibverbs-utilslibibverbs-devlibibverbs1-dbglibmlx4-1libmlx4-devlibmlx4-1-dbglibmlx5-1libmlx5-devlibmlx5-1-dbglibrxe-1librxe-devlibrxe-1-dbglibibumadlibibumad-staticlibibumad-develibacmibacm-devlibrdmacm1librdmacm-utilslibrdmacm-devmstflintibdumplibibmadlibibmad-staticlibibmad-devellibopensmopensmopensm-doclibopensm-develinfiniband-diagsinfiniband-diags-compatmftkernel-mft-dkmslibibcm1libibcm-devperftestibutils2libibdm1cc-mgrar-mgrdump-pribsimibsim-docknem-dkmsknemmxmucxsharphcollopenmpimpitestslibdapl2dapl2-utilslibdapl-devsrptoolsmlnx-ethtoolmlnx-iproute2This program will install the MLNX_OFED_LINUX package on your machine. Note that all other Mellanox, OEM, OFED, RDMA, or Distribution IB packages will be removed. Those packages are removed due to conflicts with MLNX_OFED_LINUX. Do not reinstall them. Do you want to continue?[y/N]:yChecking SW Requirements...One or more required packages for installing MLNX_OFED_LINUX are missing.Attempting to install the following missing packages:libgfortran3 flex swig chrpath tk debhelper automake quilt gfortran libnl-route-3-200 m4 libltdl-dev dpatch bison graphviz autoconf autotools-dev tclRemoving old packages...Installing new packagesInstalling ofed-scripts-4.7...Installing mlnx-ofed-kernel-utils-4.7...Installing mlnx-ofed-kernel-dkms-4.7...Installing rshim-dkms-1.8...Installing iser-dkms-4.7...Installing srp-dkms-4.7...Installing libibverbs1-41mlnx1...Installing ibverbs-utils-41mlnx1...Installing libibverbs-dev-41mlnx1...Installing libibverbs1-dbg-41mlnx1...Installing libmlx4-1-41mlnx1...Installing libmlx4-dev-41mlnx1...Installing libmlx4-1-dbg-41mlnx1...Installing libmlx5-1-41mlnx1...Installing libmlx5-dev-41mlnx1...Installing libmlx5-1-dbg-41mlnx1...Installing librxe-1-41mlnx1...Installing librxe-dev-41mlnx1...Installing librxe-1-dbg-41mlnx1...Installing libibumad-43.1.1.MLNX20190905.1080879...Installing libibumad-static-43.1.1.MLNX20190905.1080879...Installing libibumad-devel-43.1.1.MLNX20190905.1080879...Installing ibacm-41mlnx1...Installing ibacm-dev-41mlnx1...Installing librdmacm1-41mlnx1...Installing librdmacm-utils-41mlnx1...Installing librdmacm-dev-41mlnx1...Installing mstflint-4.13.0...Installing ibdump-5.0.0...Installing libibmad-5.4.0.MLNX20190423.1d917ae...Installing libibmad-static-5.4.0.MLNX20190423.1d917ae...Installing libibmad-devel-5.4.0.MLNX20190423.1d917ae...Installing libopensm-5.5.0.MLNX20190923.1c78385...Installing opensm-5.5.0.MLNX20190923.1c78385...Installing opensm-doc-5.5.0.MLNX20190923.1c78385...Installing libopensm-devel-5.5.0.MLNX20190923.1c78385...Installing infiniband-diags-5.4.0.MLNX20190908.5f40e4f...Installing infiniband-diags-compat-5.4.0.MLNX20190908.5f40e4f...Installing mft-4.13.0...Installing kernel-mft-dkms-4.13.0...Installing libibcm1-41mlnx1...Installing libibcm-dev-41mlnx1...Installing perftest-4.4...Installing ibutils2-2.1.1...Installing libibdm1-1.5.7.1...Installing cc-mgr-1.0...Installing ar-mgr-1.0...Installing dump-pr-1.0...Installing ibsim-0.7mlnx1...Installing ibsim-doc-0.7mlnx1...Installing knem-dkms-1.1.3.90mlnx1...Installing knem-1.1.3.90mlnx1...Installing mxm-3.7.3112...Installing ucx-1.7.0...Installing sharp-2.0.0.MLNX20190922.a9ebf22...Installing hcoll-4.4.2938...Installing openmpi-4.0.2rc3...Installing mpitests-3.2.20...Installing libdapl2-2.1.10mlnx...Installing dapl2-utils-2.1.10mlnx...Installing libdapl-dev-2.1.10mlnx...Installing srptools-41mlnx1...Installing mlnx-ethtool-5.1...Installing mlnx-iproute2-5.2.0...Selecting previously unselected package mlnx-fw-updater.(Reading database ... 148359 files and directories currently installed.)Preparing to unpack .../mlnx-fw-updater_4.7-1.0.0.1_amd64.deb ...Unpacking mlnx-fw-updater (4.7-1.0.0.1) ...Setting up mlnx-fw-updater (4.7-1.0.0.1) ...Added 'RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.confAttempting to perform Firmware update...The firmware for this device is not distributed inside Mellanox driver: 86:00.0 (PSID: HPE0000000009) To obtain firmware for this device, please contact your HW vendor.Failed to update Firmware.See /tmp/MLNX_OFED_LINUX.3524.logs/fw_update.logDevice (86:00.0):86:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]Link Width: x16PCI Link Speed: 8GT/sDevice (86:00.1):86:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]Link Width: x16PCI Link Speed: 8GT/sInstallation passed successfullyTo load the new driver, run:/etc/init.d/openibd restart4.Load the new driver by running the following command:user@ubuntugpu2:/tmp/MLNX_OFED_LINUX-4.7-1.0.0.1-ubuntu18.04-x86_64$ sudo /etc/init.d/openibd restartSample output:Unloading HCA driver: [ OK ]Loading HCA driver and Access Layer: [ OK ]Post-installation actionsAfter installation is complete, follow these steps:1.Update IP settings if necessary by updating the /etc/netplan/*.yaml file.a.You can find a way to identify the logical name of the Mellanox Ethernet interface used to update the *.yaml file by running thecommand:user@ubuntu-gpu1:~$ sudo lshw -class networkThis command will list all Ethernet interfaces. The Mellanox Ethernet interface will have listed product: MT27800 Family [ConnectX-5] and vendor: Mellanox Technologies.Sample output:*-network:0description: Ethernet interfaceproduct: MT27800 Family [ConnectX-5]vendor: Mellanox Technologiesphysical id: 0bus info: pci@0000:86:00.0logical name: ens4f0version: 00serial: b8:83:03:58:d3:38width: 64 bitsclock: 33MHzcapabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical autonegotiation configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=4.7-1.0.0 duplex=full firmware=16.24.1000 (HPE0000000009) ip=10.20.100.125 latency=0 link=yesmulticast=yesb.Identify the logical name and use this logical name to edit the network interfaces on the netplan *.yaml file.Sample execution:user@ubuntu-gpu1:/tmp$ cd /etc/netplan/user@ubuntu-gpu1:/etc/netplan$ sudo vi /etc/netplan/01-netcfg.yaml# This file describes the network interfaces available on your system# For more information, see netplan(5).network:version: 2renderer: networkdethernets:eno1:dhcp6: nodhcp4: noaddresses: [X.X.X.X/XX]gateway4: X.X.X.Xnameservers:addresses: [X.X,X.X]ens4f0:dhcp6: nodhcp4: noaddresses: [X.X.X.X/XX]gateway4: X.X.X.Xnameservers:addresses: [X.X,X.X]a.)Reapply netplan if necessaryuser@ubuntu-gpu1:/etc/netplan$ sudo netplan apply2.Confirm activation by issuing the following command:user@ubuntu-gpu1:~$ ibv_devinfoSample output:hca_id: mlx5_1transport: InfiniBand (0)fw_ver: 16.24.1000node_guid: b883:03ff:ff58:d339sys_image_guid: b883:03ff:ff58:d338vendor_id: 0x02c9vendor_part_id: 4119hw_ver: 0x0board_id: HPE0000000009phys_port_cnt: 1Device ports:port: 1state: PORT_DOWN (1)max_mtu: 4096 (5)active_mtu: 1024 (3)sm_lid: 0port_lid: 0port_lmc: 0x00link_layer: Ethernethca_id: mlx5_0transport: InfiniBand (0)fw_ver: 16.24.1000node_guid: b883:03ff:ff58:d338sys_image_guid: b883:03ff:ff58:d338vendor_id: 0x02c9vendor_part_id: 4119hw_ver: 0x0board_id: HPE0000000009phys_port_cnt: 1Device ports:port: 1state: PORT_ACTIVE (4)max_mtu: 4096 (5)active_mtu: 1024 (3)sm_lid: 0port_lid: 0port_lmc: 0x00link_layer: EthernetCheck the firmware version and update if needed. The latest firmware version when this document was published is 16.25.1020.Identify the PCI address by using the following command:user@ubuntu-gpu1:~$ lspci -v | grep MellanoxSample output:86:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]86:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]3.Check the firmware version using the PCI address of the device by issuing the command:~]# mstflint -d < device PCI address> qSample execution:user@ubuntu-gpu1:~$ sudo mstflint -d 86:00.0 q[sudo] password for user:Image type: FS4FW Version: 16.24.1000FW Release Date: 26.11.2018Product Version: 16.24.1000Rom Info: type=UEFI version=14.17.11 cpu=AMD64type=PXE version=3.5.603 cpu=AMD64Description: UID GuidsNumberBase GUID: b88303ffff58d338 8Base MAC: b8830358d338 8Image VSD: N/ADevice VSD: N/APSID: HPE0000000009Security Attributes: secure-fw4.To update the firmware, download the firmware update from:https:///hpsc/swd/public/detail?swItemId=MTX_57d79dca3161495e963d90095e5.Install the firmware by issuing the command:mstflint -d <device number of first device> -i <File path/<FW filename> burnSample execution:user@ubuntu-gpu1:/tmp$ sudo tar -xzf ./fw-ConnectX5-rel-16_25_1020-872726-B21_Ax-UEFI-14.18.19-FlexBoot-3.5.701.tar.gzuser@ubuntu-gpu1:/tmp$ sudo mstflint -d 86:00.0 -i /tmp/fw-ConnectX5-rel-16_25_1020-872726-B21_Ax-UEFI-14.18.19-FlexBoot-3.5.701.signed.bin burnCurrent FW version on flash: 16.24.1000New FW version: 16.25.1020Initializing image partition - OKWriting Boot image component - OK-I- To load new FW run mstfwreset or reboot machine.6.Reboot the server and confirm the firmware update.ubuntu-gpu1:~$ sudo rebootuser@ubuntu-gpu1:~$ sudo mstflint -d 86:00.0 qSample output:[sudo] password for user:Image type: FS4FW Version: 16.25.1020FW Release Date: 30.4.2019Product Version: 16.25.1020Rom Info: type=UEFI version=14.18.19 cpu=AMD64type=PXE version=3.5.701 cpu=AMD64Description: UID GuidsNumberBase GUID: b88303ffff58d338 8Base MAC: b8830358d338 8Image VSD: N/ADevice VSD: N/APSID: HPE0000000009Security Attributes: secure-fwInstallation guideMake the right purchase decision.Contact our presales specialists.Share nowGet updates© Copyright 2020 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change withoutnotice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warrantystatements accompanying such products and services. Nothing herein should be construed as constituting an additionalwarranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. Mellanox and ConnectX are registeredtrademarks of Mellanox Technologies. All third-party marks are property of their respective owners.a00094362ENW, January 2020Check if the document is availablein the language of your choice.ADDITIONAL LINKSHPE and Mellanoxhttps:///oem/hpe/Mellanox OFED documentationhttps:///display/MLNXOFEDv471001HPE firmware downloadhttps:///hpsc/swd/public/detail?swItemId=MTX_57d79dca3161495e963d90095eLEARN MORE AThttps:///v2/GetPDF.aspx/c04154440.pdf。
Mellanox InfiniBand产品指南说明书
BROCHUREInfiniBand Product Guide End-to-end InfiniBand Solutions at HDR / EDR / FDR SpeedsInfiniBand SwitchesFrom Switch-IB®-2 100Gb/s EDR to Mellanox Quantum™ 200Gb/s HDR InfiniBand, the Mellanox family of1RU and modular switches deliver the highest density and performance. Featuring the in-network computing Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)™ technology, Mellanox’s extensive switch portfolio enables compute clusters to operate at any scale, while reducing operational costs and infrastructure complexity. InfiniBand switches also leverage Mellanox’s SHIELD™ (Self-Healing Interconnect Enhancement for Intelligent Datacenters), which increases data center network resiliency by 5000 times, compared to other software-based solution options.InfiniBand AdaptersLeveraging faster speeds and innovative In-Network Computing, Mellanox’s award-winning HDR (200Gb/s), EDR (100Gb/s) and FDR (56Gb/s) InfiniBand Adapters deliver the highest throughput and message rate in theindustry. Providing best-in-class network performance, scale and efficiency, InfiniBand adapters enable extremely low-latency, and advanced application acceleration engines for High-Performance Computing, Machine Learning, Cloud, Storage, Databases and Embedded applications, reducing cost per operation and increasing ROI. Mellanox smart adapters deliver the highest performance and scalable connectivity for Intel, AMD, IBM Power, NVIDIA, Arm and FPGA-based Compute and Storage Platforms.InfiniBand LongHaulMellanox’s MetroX®-2 systems extend the reach of InfiniBand to up to 40 kilometers, enabling native InfiniBand connectivity between remote data centers, remote data center and storage infrastructures, or fordisaster recovery. Delivering up to 100Gb/s data throughout, MetroX-2 enables native RDMA connectivity, high data throughput, advanced routing, and more, across distributed compute or storage platforms. MetroX-2 enables users to easily migrate application jobs from one InfiniBand center to another, or to combine the compute power of multiple remote data centers together for higher overall performance and scalability. InfiniBand Gateway to EthernetMellanox Skyway™ 200 Gigabit HDR InfiniBand to Ethernet gateway appliance enables scalableand efficient connectivity from high performance, low-latency InfiniBand data centers to external Ethernet networks and infrastructures. Mellanox Skyway™ empowers InfiniBand-based high performance andcloud data centers to achieve the lowest interconnect latency, while providing a simple and cost-effective option to connect to remote Ethernet networks.InfiniBand Cables and TransceiversMellanox LinkX® InfiniBand cables and transceivers are designed to maximize the performance of the high-performance InfiniBand networks, to deliver high-bandwidth, low-latency and highly reliable and robust connectivity. To provide superior system performance, Mellanox ensures the highest quality in all LinkX products.InfiniBand Telemetry and Software ManagementMellanox’s comprehensive suite of network telemetry and management software provides an innovative application-centric approach to bridge the gap between servers, applications and fabric elements. Mellanox’s UFM® (Unified Fabric Management) software includes fabric diagnostics, monitoring, alerting, provisioning and advanced features such as congestion monitoring and fabric segmentation and isolation. Users can manage small to extremely large fabrics as a set of inter-related business entities while also performing fabric monitoring and optimizing performance at the application-logical level rather than only at the individual port or device level. InfiniBand Acceleration SoftwareThe Mellanox HPC-X® ScalableHPC Toolkit is a comprehensive MPI and SHMEM/PGAS software suite for high performance computing environments. For scientific research and engineering simulations, the complete Mellanox HPC-X software toolkit, including Mellanox UCX and FCA acceleration engines, provides enhancements that significantly increase the scalability and performance of message communications in the network. HPC-X enables the rapid deployment and delivery of maximum application performance without the complexity and costs of licensedthird-party tools and libraries.Mellanox, Mellanox logo, ConnectX, BlueField, UFM, Switch-IB, Mellanox ScalableHPC, MLNX-OS, LinkX , HPC-X and ConnectX are registered trademarks of Mellanox Technologies, Ltd. BlueFIeld,Mellanox Quantum, NVMe SNAP , SHIELD, Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) are trademarks of Mellanox Technologies, Ltd.All other trademarks are property of their respective owners.350 Oakmead Parkway, Suite 100 Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: 060342BR v1.6。
Mellanox MMS1V00-WM QSFP-DD 400GbE 单模传输器说明书
©2020 Mellanox Technologies. All rights reserved.†For illustration only. Actual products may vary.Mellanox ®MMS1V00-WM transceiver is a single-mode, 4-channel (DR4), QSFP-DD optical transceiver designed for 400 Gigabit Ethernet (GbE) links on up to 500m of single mode fiber.MMS1V00-WM converts 8 input channels of 50Gb/s PAM4 electrical data to 4 channels of 100Gb/s PAM4 optical signals, using a nominal wavelength of 1310nm, for 400Gb/s optical transmission.The transceiver has a standard QSFP-DD connector on the electrical side towards the host system. The optical interface is composed of four optical channels/fibers in each direction, intended for a parallel single mode optical cable via a standard MPO connector.Rigorous production testing ensures the best out-of-the-box installation experience, performance and durability.MMS1V00-WM400GbE QSFP-DD DR4 1310nm Optical TransceiverINTERCONNECT PRODUCT BRIEF†Table 1 - Absolute Maximum RatingsTable 2 - Recommended Operating Conditions and Power Supply Requirements350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: © Copyright 2020. Mellanox Technologies. All rights reserved.Mellanox, Mellanox logo and LinkX are registered trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners.Mellanox 400GbE QSFP-DD DR4 Optical T ransceiverpage 2Warranty InformationMellanox LinkX transceivers include a 1-year limited hardware warranty, which covers parts repair or replacement.60412PB Rev 1.0Table 3 - Part Number and DescriptionMechanical Schematics。
Mellanox Technologies和DataON合作建立微软认证的超融合集群应用程序说明书
SOLUTION BRIEF©2016 Mellanox Technologies. All rights reserved.verged infrastructure with these promises:• Consolidating existing data center silos to streamline operations• Simplifying the IT infrastructure for quick de-ployment and scale-out• Improving agility to accommodate constantly changing business needsDataON™ and Mellanox Empower Next-Gen Hyper-Converged Infrastructurepage 2SOLUTION BRIEF: DataON™ and Mellanox Empower Next-Gen Hyper-Converged Infrastructure 350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: © Copyright 2016. Mellanox Technologies. All rights reserved.Mellanox, Mellanox logo, and ConnectX are registered trademarks of Mellanox Technologies, Ltd. Mellanox NEO, Mellanox Spectrum, and LinkX are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners.15-51693SBRev1.0• Industry Leading Application Performance – The S2D-3110 HCCA with four (4x) cluster nodes is capable of providing over 2.4 million IOPS (running VMFleet) using the latest all flash NVMe™ based SSD technology to scale I/O intensive workloads.• Storage and Network with SMB 3.0 over RDMA – Deliv-ers highest throughput, lowest latency and increased CPU efficiency • Hyper-Converged Scalability – Incremental compute, net-working, and storage resources provide near-linear scalabil-ity. Each HCCA can also be expanded in capacity via 12G SAS JBODs for further storage expansion.。
Mellanox Technologies ROBO 分布式 IT 基础设施说明书
The Nutanix Enterprise Cloud addresses these challenges with its invisible and cloud-native hyperconverged infrastructure. From a converged compute and storage system, the Nutanix enterprise cloud extends one-click simplicity and high availability to remote and branch offices. Streamlined and automated operations and self-healing from infrastructure anomalies, eliminate unnecessary onsite visits and overtime, thus reducing operational and support costs. With the Nutanix solution, enterprise IT staff can deploy and administer ROBO sites as if they were deployed to the public cloud, while maintaining control and security on their own terms.ROBO networking must be simple and transparent as well. Mellanox’s end-to-end Ethernet Storage Fabric™ (ESF) perfectly complements the Nutanix Enterprise Cloud. The Mellanox ESF offers Zero Touch Provisioning (ZTP), guaranteed performance, automated operation, and real-time network visibility at the virtual machine (VM) or container level, for business operations on Day 0/1/2. To provide a transparent, automated experience for application provisioning and mobility, data backup and disaster recovery, Mellanox’s ESF is integrated into the Nutanix enterprise cloud via the management plane. Through REST APIs, Mellanox’s network orchestration and management platform, Mellanox NEO ®, automates network provisioning from Nutanix Prism and eliminates complex and expensive manual configuration for numerous network devices in multiple clouds. Nutanix ROBO in a BoxWith its web-scale efficiency and enterprise-level resilience and security, Nutanix offers hyperconverged clusters for remote and branch offices. Providing options for one-node, two-node, and three-node clusters, the Nutanix solution for ROBO meets various requirements with respect to data protection, high availability, and cost-effectiveness.A three-node cluster is the gold standard for Nutanix ROBO, which provides on-site data protection and tolerates an entire node going down. Apart from that, self-healing with a data rebuild within the cluster eliminates needless trips to remote sites., while a two-node cluster offers reliability for smaller ROBO sites that must keep costs down. Beyond that, a remote witness node is used for data rebuild and automatic upgrades. Moreover, a one-node cluster is a perfect fit for low-availability requirements and strong management for multiple sites. Figure 1. Nutanix ROBO in a Box with Mellanox NetworkingGiven the small number of nodes in the Nutanix cluster and often rigid environmental requirements in space, power and airflow, Mellanox’s half-width top-of-rack (TOR) SN2010 switches are a perfect fit for Nutanix ROBO both in terms of connectivity and cost. Featuring 18 ports of 1/10/25G downlinks and 57 Watts of typical power consumption, two SN2010 switches can be installed side-by-side, along with a 2U Nutanix appliance, to build a ROBO datacenter in a 3U box. The 1G management port on the Nutanix node can be connected to a SN2010 switch port, eliminating the need for a separate management switch.The SN2010 is based on the state-of-art Mellanox Spectrum ® switching ASIC, which provides guaranteed performance for any workload running on the Nutanix cluster, regardless of packet size, network speed, and throughput/latency requirements, making the networking completely transparent. The Mellanox switch provides additional value by allowing Docker containers, such as VPN and DHCP services, to run on the switch, further simplifying manageability and security while reducing costs.Automated Provisioning for Business ContinuityFigure 2. Integrated Network Provisioning for VM MobilityFigure 3. Automated Network Provisioning for Disaster Recovery and Business ContinuityNutanix enterprise cloud streamlines datacenter operations withconsumer–grade management in Prism™, which simplifies applicationmobility and load balancing. It also reduces complex operations suchas disaster recovery to a single click and ensures business continuity ofmission-critical applications.The integration of Nutanix Prism and Mellanox NEO ®, Mellanox’snetwork orchestrator, enables automated network provisioning thatrequires no manual operation. In a CRUD event (i.e., VM or containercreation, migration, and deletion), Mellanox NEO works with Prismthrough RESTful APIs in the background, configuring the virtual local areanetwork (VLAN) for that VM/container on the switch port it’s connectedto. When the VM/container becomes live through Prism, it automaticallycomes online.In the event of a disaster, networking is often the key challenge for implementing business continuity and disaster recovery beyond data replication. The joint Nutanix- Mellanox solution automates network provisioning as part of workload lifecycle management and allows workloads to preserve their IP addresses and gateways when they failover to the remote DR site, enabling uninterrupted business continuity during partial or full failover. These capabilities are delivered through the Mellanox NEO and Prism Central integration forautomation, using Ethernet VPN (EVPN)-based virtual extensible LAN (VXLAN) overlays. This allows the transparent stretching of networks from the ROBO site to the DR site or the main datacenter. Nutanix offers synchronous, asynchronous, and near-synchronous replication options that can be granularly controlled to meet various RPO/RTO goals. In addition, Mellanox NEO provides one-click configuration for mLAG and switch software upgrade at scale.© Copyright 2020. Mellanox Technologies. All rights reserved.Mellanox, Mellanox logo, ConnectX, Mellanox NEO, Mellanox Spectrum, and LinkX are registered trademarks of Mellanox Technologies, Ltd. What Just Happened is a trademark of Mellanox Technologies, Ltd.Nutanix, the Enterprise Cloud Platform, the Nutanix logo and the other Nutanix products, features, and/or programs mentioned herein are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. This docu-ment is provided for informational purposes only and is presented ‘as is’ with no warranties of any kind, whether implied, statutory or otherwise. All other trademarks are property of their respective owners.350 Oakmead Parkway,Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400Fax: Real-time Visibility for AIOpsNutanix offers real-time visibility in the cluster of applications running on the node and associated compute, storage and securitymetrics at the VM/container level. Such visibility is used for remote management, in a cloud-native way, for extensible, intelligent and automated IT Ops – forecast, planning, optimization, and anomaly detection and remediation.Similarly, Mellanox ESF provides real-time visibility into network-related problems,through an event-based, advanced telemetry technology called What Just Happen™(WJH). Mellanox WJH does packet inspection at line rate, accelerated by the switchASIC. In the events of network anomalies, the WJH telemetry agent, running as acontainer on the Mellanox switch, streams out both the packet itself and relatedinformation in JSON or other streaming methods. The telemetry data can be streamedto a database repository or directly to the management software, such as MellanoxNEO, Nutanix Prism, and TIG (Telegraf-InfluxDB-Grafana).While traditional telemetry solutions try to extrapolate root causes of network issuesby analyzing network counters and statistical packet sampling, WJH goes beyond thatby providing actionable details on abnormal network behavior and eliminating the guess work from fast network troubleshooting.ConclusionROBO is common in enterprise IT infrastructures. Deploying and managing ROBO sites efficiently as part of the enterprise cloud is a key imperative for business operations. Nutanix delivers a web-scale, hyperconverged infrastructure solution, and brings the scale, resilience and economic benefits of web-scale architecture to ROBO. Mellanox Ethernet Storage Fabric, with its purpose-built TOR switches in particular, allows a ROBO solution in a box with integrated automation of the Nutanix platform for network provisioning, operation, and troubleshooting. The Nutanix and Mellanox solution brings ROBO into the unified enterprise cloud with efficiency and cost savings throughout the lifecycle of Day 0/1/2 operations.About NutanixNutanix makes infrastructure invisible, elevating IT to focus on the applications and services that power their business. Using Nutanix, customers benefit from predictable performance, linear scalability, and cloud- like infrastructure consumption. A single software fabric unifies your private and public clouds, and delivers one-click simplicity in managing multi-cloud deployments. One OS, one click. Learn more at or visit Twitter @nutanix.About MellanoxMellanox Technologies is a leading supplier of end-to-end Ethernet interconnect solutions and services for enterprise data centers, Web 2.0, cloud, storage, and financial services. More information is available at: 。
迈兰奧克蓝字段参考平台说明书
©2020 Mellanox Technologies. All rights reserved.†For illustration only. Actual products may vary.Today’s network technologies drive OEMs to seek innovative, scalable and cost effective designs for dealing with the exponential growth of data. The Mellanox BlueField Reference Platform provides a multi-purpose fully-programmable hardware environment for evaluation, development and running of software solutions, reducing time-to-market and increasing product development and runtime efficiency.The reference platform delivers all the features of the BlueField Data Processing Unit (DPU) in convenient form factors, making it ideal for a range of software solutions, for the most demanding markets. Features include two 100Gb/s Ethernet or InfiniBand interfaces, a 16-core BlueField processor, up to 512GB of RDIMM DDR4 memory, two PCIe x16 slots, and an NVMe-ready midplane for SSD connectivity.BlueField Platform for Storage AppliancesToday’s fast storage technologies drive storage OEMs to seek innovative, scalable and costeffective designs for their applications. Powered by the BlueField DPU, the BlueField 2U Reference Platform offers a unique combination of on-board capabilities and NVMe-readiness, creating an ideal environment for storage appliance development.Platform Highlights• Leverages the processing power of Arm ® cores for storage applications such as All-FlashArrays using NVMe-oF, Ceph, Lustre, iSCSI/TCP offload, Flash Translation Layer, RAID/Erasure coding, data compression/decompression, and deduplication.• In high-performance storage arrays, BlueField serves as the system’s main CPU, handlingstorage controller tasks and traffic termination.• Provides up to 16 front-mounted 2.5” disk drive bays that are routed to an NVMe-readymidplane within the enclosure. The system can be configured as a storage JBOF with 16 drives using PCIe Gen 3.0 x2, or 8 drives with PCIe Gen 3.0 x4 lanes.BlueField Platform for Machine LearningThe BlueField 2U Reference Platform supports connectivity of up to 2 GPUs via its PCIe x16 Gen 3.0 interface, providing cost effective and integrative solutions for Machine Learning appliances. By utilizing RDMA and RoCE technology, the BlueField network controller data path hardware delivers low latency and high throughput with near-zero CPU cycles.The platform also offers GPUDirect ® RDMA technology, enabling the most efficient data exchange between GPUs and with the Mellanox high speed interconnect, optimizing real-time analytics and data insights.Powerful & Flexible Reference Platform for a Wide Range ofApplications Including Storage, Networking and Machine LearningDATA PROCESSOR PRODUCT BRIEF†NVIDIA ®Mellanox ®BlueField ®Reference Platform© Copyright 2020. Mellanox Technologies. All rights reserved. Mellanox, Mellanox logo, BlueField, BlueOS, ConnectX, ASAP2 - Accelerated Switching and Packet Processing, GPUDirect and Virtual Protocol Interconnect are registered trademarks of Mellanox Technologies, Ltd. PeerDirect is a trademark of Mellanox Technologies. All other trademarks are property of their respective owners.page 2BlueField Reference Platform 350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: 52961PB Rev 1.8SupportFor information about Mellanox support packages, please contact your Mellanox Technologies sales representative or visit our Support Index page .T able 1 - Part Numbers and DescriptionsProductFamily OPNDescriptionBF1200MBE1200A-BN12U BlueField Reference Platform, BlueField E-Series, Crypto disabled. A storage controller platform with option for up to 16 SSDs. (SSDs are not included.)NVMe-Ready MidplaneA modular chassis midplane supports up to eight 2.5” SSDs, which can be duplicated to 16 SSDs. The midplane also supports hot swappable SSD cards, an I 2C switch to enable connectivity of the SMBUS from the platform Baseboard Management Controller (BMC) to each SSD, andan on-board clock generator/buffer.Software SupportThe BlueField Reference Platform comespre-installed with a UEFI-based bootloader and BlueOS, a Linux reference distribution targeted at BlueField-based embedded systems. Based on the Yocto Project Poky distribution, BlueOS is highly customizable for meeting specific Linux distribution requirements through the OpenEmbedded Build System. Yocto producesan SDK with an extremely flexible cross-build environment, ideal for building and running applications seamlessly for the Arm BlueField target system, on any x86 server running any Linux distribution. Mellanox OFED and NVMe-oF support is installed by default. The reference platform also provides a BMC running OpenBMC to manage the entire system. Note: Reference platform customers can run the Linux distribution of their choice.Enclosure Specifications –2U 19”–ATX form factor motherboard –BlueField DPU with 16 Armv8 A72 cores (64-bit)–Two internal x16 PCIe Gen3.0/4.0 expansion connectors–Dual-port ConnectX-5 Virtual Protocol Interconnect ® (VPI) interface• Ethernet: 40/50/100GbE QSFP ports • InfiniBand: FDR/EDR QSFP ports • 10/25Gb/s available with QSA28–Two PCIe risers enabling 2.5” NVMe SSD disk support • 8 x PCIe Gen3 x4 lanes • 16 x PCIe Gen3 x2 lanes –1 x 850W FRU power supply –Integrated BMC–32GB eMMC Flash memory for software–3 x 80mm fan cartridges DRAM DIMM Support–4 sockets for DRAM DIMMs –Up to 512GB total memory –NVDIMM-N Support2U Reference Platform Features1U Reference Platform FeaturesEnclosure Specifications –1U 19”–ATX form factor motherboard –BlueField DPU with 16 Armv8 A72 cores (64-bit)–One internal x16 PCIe Gen3.0 expansion connector–Dual-port ConnectX-5 Virtual Protocol Interconnect ® (VPI) interface• Ethernet: 40/50/100GbE QSFP ports • InfiniBand: FDR/EDR QSFP ports • 10/25Gb/s available with QSA28–1x 400W power supply –Integrated BMC–32GB eMMC Flash memory for software–3 x 80mm fan cartridges DRAM DIMM Support–4 sockets for DRAM DIMMs –Up to 512GB total memory –NVDIMM-N SupportFigure 1. 8 SSD Configuration (2U platform)Figure 2. 16 SSD Configuration (2U platform)†Figure 3: 2U Reference Platform。
Mellanox WinOF for Windows 安装指南说明书
Mellanox TechnologiesMellanox WinOF for WindowsInstallation GuideRev 2.00 GA2© Copyright 2008. Mellanox Technologies, Inc. All Rights Reserved.Mellanox, ConnectX, InfiniBlast, InfiniBridge, InfiniHost, InfiniRISC, InfiniScale, and InfiniPCI are regis-tered trademarks of Mellanox Technologies, Ltd. Virtual Protocol Interconnect is a trademark of Mellanox Technologies, Ltd.Mellanox WinOF for Windows Installation GuideDocument Number: 2956Mellanox Technologies, Inc.350 Oakmead ParkwaySunnyvale, CA 94086U.S.A.Tel: (408) 970-3400Fax: (408) 970-3403Mellanox Technologies LtdP.O. Box 586 Hermon-BuildingYokneam 20692IsraelTel: +972-4-909-7200Fax: +972-4-959-3245Mellanox WinOF for Windows Installation Guide3 1 OverviewThis document describes how to install and test Mellanox WinOF for Windows on a single host machine with Mellanox InfiniBand hardware installed. The document includes the following sections:• This overview•“Web Page and Documentation” (page 3)•“Hardware and Software Requirements” (page 3)•“Identifying Mellanox HCAs on Your Machine” (page 4)•“Downloading Mellanox WinOF” (page 6)•“Installing Mellanox WinOF” (page 6)•“Installation Results” (page 17)•“Assigning IP to IPoIB Adapters After Installation” (page 18)•“Modifying Configuration After Installation” (page 21)•“Updating Firmware After Installation” (page 22)•“Uninstalling Mellanox WinOF” (page 24)2Web Page and DocumentationPlease visit the Mellanox WinOF Web page at /products/MLNX_WinOF.php todownload the package and to reference documentation such as release notes, user manuals, FAQ, trouble-shooting, and archive. After installing Mellanox WinOF (see “Installing Mellanox WinOF” below), you will find release notes and user manuals under “Program Files\Mellanox\MLNX_WinOF\Documenta-tion”.3Hardware and Software Requirements3.1Hardware RequirementsRequired Disk Space for Installation•100MBPlatformsAny computer system with an x86 or x64 CPU architecture, and with a PCI adapter card based on one of the following Mellanox Technologies’ InfiniBand HCA devices:•ConnectX® (firmware: fw-25408 v2.5.000 or later)•InfiniHost® III Ex (firmware: fw-25218 v5.3.000 or later for Mem-Free cards, and fw-25208 v4.8.200 or later for cards with memory)•InfiniHost® III Lx (firmware: fw-25204 v1.2.000 or later)•InfiniHost® (firmware: fw-23108 v3.5.000 or later)Note:For the list of supported architecture platforms, please refer to the MellanoxWinOF for Windows Release Notes file under the “Documentation” folder.Identifying Mellanox HCAs on Your Machine 43.2Software RequirementsInstaller PrivilegesThe installation requires administrator privileges on the target machine.Operating Systems•Windows XP•Windows Server 2003•Windows Server 2008Note:For the list of supported operating system distributions and kernels, please referto the Mellanox WinOF for Windows Release Notes file under the “Documenta-tion” folder.4Identifying Mellanox HCAs on Your MachineStep 1.Check the Device Manager display for PCI devices. If the device driver has not been installed, check under Other Devices.Note:If you cannot find any PCI device, make sure that the HCA card(s) is correctlyinstalled in the PCI slot. If no PCI device is identified after your check, tryinstalling the HCA card into a different PCI slot.Mellanox WinOF for Windows Installation Guide5 Step 2.Select a PCI Device entry, right-click and select Properties to display the PCI Device Properties window.Step 3.Click the Details tab and select Device Instance ID from the Property pull-down menu.Step 4.In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_6340”: VEN isequal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to6340 – this is a valid Mellanox Technologies PCI Device ID.Note:The list of Mellanox Technologies PCI Device IDs can be found in the PCI IDrepository at http://pci-ids.ucw.cz/read/PC/15b3.Step 5.If the PCI device does not have a Mellanox HCA ID, return to Step 2 to check another device.Note:If you cannot find any Mellanox HCA device listed as a PCI device, make surethat the HCA card(s) is correctly installed in the PCI slot. If the HCA deviceremains unidentified, try installing the adapter card into a different PCI slot.Downloading Mellanox WinOF 65Downloading Mellanox WinOFDownload the appropriate MSI to your host from the Mellanox WinOF Web page at/products/MLNX_WinOF.php.The MSI’s name has the format MLNX_WinOF_<arch>_<version>.msi, where arch can be eitherx86 or x64.6Installing Mellanox WinOFThis sections provides instructions for two types of installation:•“Attended Installation” on page 6An attended installation is an installation procedure that requires frequent user intervention.•“Unattended Installation” on page 11An unattended installation is an automated installation procedure that requires no user intervention.•“WDS Installation” on page 12A WDS installation is intended for Windows HPC Server 2008 clusters.6.1Attended InstallationNote:The installation requires administrator privileges on the target machine.For operating systems other than Windows 2008, double click the MSI and follow the GUI instructions to install Mellanox WinOF.For Windows 2008, install the MSI by opening a CMD console (click Start-->Run and enter ‘cmd’) andentering the following command:> msiexec.exe /i MLNX_WinOF_<arch>_<version>.msiThe following is an example of a Mellanox WinOF x64 installation session.Mellanox WinOF for Windows Installation Guide7 Step 1.Click Next in the Welcome screen.Step 2.Select the “accept” radio button and click Next.Installing Mellanox WinOF 8Step 3.Select the destination folder for Mellanox WinOF and click Next.Step 4.Select the type of installation: Typical or Custom.Mellanox WinOF for Windows Installation Guide9 If you select Typical, click Next and advance to the next step.If you select Custom, click Next and you will get the screen below. To install/remove a com-ponent, left-click the component and enable/disable it for installation.To continue, click Next and advance to the next step.Installing Mellanox WinOF 10Step 5.To install the chosen components, click Install.Step 6.In the following window, enable the components you need (if any). To complete the instal-lation, click Finish. (See the figure below.)Note:Even if you do not enable any of the displayed components in this step, you willbe able to enable components after the installation completes – see the fileMLNX_WinOF_README.txt.Mellanox WinOF for Windows Installation Guide116.2Unattended InstallationNote:The installation requires administrator privileges on the target machine.To perform a silent/unattended installation, open a CMD console (click Start->Run and enter ‘cmd’) andenter the following command:> msiexec.exe /i MLNX_WinOF_<arch>_<version>.msi /qn [Parameter]where Parameter is:ADDLOCAL Provides the list of components (separated by com-mas) to install. Available components are: Driver,IPoIB, ND, WSD, SDP, OpenSM, SRP, Tools, Docs, andSDK. You can also provide ADDLOCAL=ALL to installall components.Note: If you do not provide the ADDLOCAL parameter,the script will install the following list ofdefault components: Driver, IPoIB, ND, WSD, OpenSM,Tools, Docs, and SDK.WSDENABLE Enable WSD by adding the parameter WSDENABLE=1. (Theinstallation procedure installs WSD but does notenable it.)Installing Mellanox WinOF 12NDENABLE Enable ND by adding the parameter NDENABLE=1. (Theinstallation procedure installs ND as part of theIPoIB component but does not enable it.)Note:For all command options, enter ‘msiexec.exe /?’.Usage Examples•The following command installs MLNX_WinOF in the default configuration:> msiexec /i MLNX_WinOF_x86_<ver>.msi /qn•The following command installs MLNX_WinOF with all the available components:> msiexec /i MLNX_WinOF_x86_<ver>.msi /qn ADDLOCAL=ALL•The following command installs MLNX_WinOF with the default components plus SRP:> msiexec /i MLNX_WinOF_x86_<ver>.msi /qn \ADDLOCAL=Driver,IPoIB,ND,WSD,OpenSM,SRP,Tools,Docs,SDK•The following command installs MLNX_WinOF in the default configuration and enables WSD:> msiexec /i MLNX_WinOF_x86_<ver>.msi /qn WSDENABLE=16.3WDS InstallationTo perform a WDS installation for a Windows HPC Server 2008 cluster, follow the steps below.Step 1.Extract the package Mellanox_WinOF_x64_<ver>_INF.zip to a directory in the head node.Step 2.On the head node, click start--> All Programs --> Microsoft HPC Pack --> HPC Cluster Manager. Select Configuration in the navigation pane and then select To-do List. Next,Click “Manage drivers” and the following dialog will be displayed.Mellanox WinOF for Windows Installation Guide13 Step 3.Click “Add” and navigate in the Open dialog to the directory chosen in Step 1. Then go to the INF directory.Step 4.Select the listed INF files and click “Open” to add the files.Step 5.Click Close in “Manage drivers” dialog.Step 6.To enable ND perform the following steps. Otherwise, skip to the next step.a.Select Node Templates in the Configuration pane.b.Right click on the desired Node Template and select “edit”. An editor window is displayed (see below).Installing Mellanox WinOF 14c.Click Add Task --> Deployment --> Run OS command.Mellanox WinOF for Windows Installation Guide15d.Locate the new Run OS command listed under the Deployment section in the editor. Next, select in theOptional pane ContinueOnFailure to be True and enter the following text in the Description field:“NetworkDirect registration command”.e.In the Required pane of the editor, enter the following text in the Command field: “ndinstall -i”.f.Click Save. The editor window will close.Installing Mellanox WinOF 16Step 7.Select “Node Management” in the navigation pane of HPC Cluster Manager.Step 8.Right-click the desired node and select “Assign Node Template”. The following dialog will be displayed.Step 9.Select the correct node template and click OK to start MLNX_WinOF installation on the node.Mellanox WinOF for Windows Installation Guide17 7Installation ResultsHardwareDisplaying the Device Manager will show the Mellanox HCA devices, the Mellanox InfiniBand fabric, and an IPoIB (network) device for each InfiniBand port.Software•The MLNX_WinOF package is installed under the directory selected in Step 3 of Section 6.1.•OpenSM is installed as a disabled Windows service. To enable it, enter at the command line:> sc start opensmAssigning IP to IPoIB Adapters After Installation 188Assigning IP to IPoIB Adapters After InstallationBy default, your machine is configured to obtain an automatic IP address via a DHCP server. In some cases, the DHCP server may require the MAC address of the network adapter installed in your machine. To obtain the MAC address, open a CMD console and enter the command ‘ipconfig /all’ ; the MAC address is dis-played as “Physical Address”.To assign a static IP addresses to an IPoIB adapter after installation, perform the following steps:Step 1.Open the Network Connections window. Locate Local Area Connections named Mellanox IPoIB Adapter <#>. Each InfiniBand port has one IPoIB Adapter.Mellanox WinOF for Windows Installation Guide19 Step 2.Right-click a Mellanox Local Area Connection and left-click Properties.Step 3.Select Internet Protocol (TCP/IP) from the scroll list and click Properties.Assigning IP to IPoIB Adapters After Installation 20Step 4.Select the “Use the following IP address:” radio button and enter the desired IP informa-tion. Click OK when you are done.Step 5.Close the Local Area Connection dialog.Step 6.Verify the IP configuration by running ‘ipconfig’ from a CMD console.> ipconfig...Ethernet adapter Local Area Connection 3:Connection-specific DNS Suffix . :IP Address. . . . . . . . . . . . : 11.4.3.204Subnet Mask . . . . . . . . . . . : 255.0.0.0Default Gateway . . . . . . . . . :...Mellanox WinOF for Windows Installation Guide21 9Modifying Configuration After Installation9.1Modifying Mellanox HCA ConfigurationTo modify HCA configuration after installation, perform the following steps:a.Open the Registry editor by clicking Start->Run and entering ‘regedit’.b.In the navigation pane, expand HKEY_LOCAL_MACHINE->SYSTEM->CurrentControlSet->Ser-vices.c.Expand (in the navigation pane) the HCA driver service entry:- ‘mtcha’ for the InfiniHost family- ‘mlx4_hca’ and ‘mlx4_bus’ for the ConnectX familyd.Click the Parameters entry in the expanded driver service entry to display HCA parameters.e.Double click the desired HCA parameter and modify it. Repeat this step for all the parameters youwish to modify.f.Close the Registry editor after completing all modifications.g.Open Device Manager and expand the correct InfiniBand Channel Adapters entry (i.e., the adapterwith modified parameters).h.Right click the expanded HCA entry and left-click Disable. This disables the device.i.Right click the expanded HCA entry and left-click Enable. This re-enables the device.Note:For the changes to take effect, you must disable and re-enable the HCA (steps hand i above).9.2Modifying IPoIB ConfigurationTo modify the IPoIB configuration after installation, perform the following steps:a.Open Device Manager and expand Network Adapters in the device display pane.b.Right-click the Mellanox IPoIB Adapter entry and left-click Properties.c.Click the Advanced tab and modify the desired properties.Note:The IPoIB network interface is automatically restarted once you finish modifyingIPoIB parameters.Note:You need to restart opensm after modifying IPoIB configuration.Updating Firmware After Installation 2210Updating Firmware After InstallationThe following steps describe how to burn new firmware downloaded from Mellanox Technologies’ Webpages under /support/firmware_download.php.Step 1.Install the firmware tools package, MFT for Windows (WinMFT), on your machine. You can download it from /products/management_tools.php. Pleasecheck also the documentation on the same Web page.Step 2.Open a CMD console. (Click Start-->Run and enter ‘cmd’.)Step 3.Start mst.> st start mstSERVICE_NAME: mstTYPE : 1 KERNEL_DRIVERSTATE : 4 RUNNING(STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN))WIN32_EXIT_CODE : 0 (0x0)SERVICE_EXIT_CODE : 0 (0x0)CHECKPOINT : 0x0WAIT_HINT : 0x0PID : 0FLAGS :Step 4.Identify your target InfiniBand device for firmware update.a.Get the list of InfiniBand device names on your machine.> mst statusFound 2 devices:mt25418_pciconf0mt25418_pci_cr0b.Your InfiniBand device is the one with the postfix “_pci_cr0”. In the example listed above, this will bemt25418_pci_cr0. Use the string “mtXXXXX” to identify the device type by checking the Webpage http://pci-ids.ucw.cz/read/PC/15b3. In the example above, mtXXXXX=mt25418, and the deviceis a ConnectX IB.c.Now go to the Mellanox firmware download page at:/support/firmware_download.php.d.Go to the correct firmware download page according to the device type identified in step (b) above.e.Run ‘vstat’ to obtain the PSID of your HCA card. The PSID is a unique identifier for the HCA card.Step ing the PSID obtained in Step 4(e), download the appropriate firmware binary image (*.bin.zip) and unzip it.Step 6.Burn the firmware image using the flint utility (part of your installed WinMFT).Note:Make sure that you burn the correct binary image to your HCA card. Burning thewrong image may cause severe firmware corruption. Please review Step 4 andStep 5 above.> flint -d mt25418_pci_cr0 -image <image>.bin burnMellanox WinOF for Windows Installation Guide23 Note:You may need to run ‘unzip’ on the downloaded firmware image prior to the burnoperation.Step 7.Reboot your machine after the firmware burning is completed.Uninstalling Mellanox WinOF 2411Uninstalling Mellanox WinOFAttended UninstallTo uninstall MLNX_WinOF on a single node, perform one of the following options:a.Click Start->Programs->Mellanox->MLNX_WinOF->Uninstall MLNX_WinOFb.Click Start->Control Panel-> Add Remove Programs-> MLNX_WinOF-> RemoveUnattended UninstallTo uninstall MLNX_WinOF in unattended mode, open a CMD console and perform one of the followingprocedures:a.If the MSI that was used to install MLNX_WinOF is available on your machine, enter the fol-lowing command:> msiexec.exe /x MLNX_WinOF_<arch>_<version>.msi /qn /forcerestartb.Obtain the MLNX_WinOF GUID (product code) by left-clicking Start->Programs->Mellanox->MLNX_WinOF.Now right-click on Uninstall MLNX_WinOF and select Properties, then copy the GUID from the “Tar-get entry. The GUID is the hexadecimal string that appears after ‘/x’. To uninstall the MSI, enter thefollowing command:> msiexec.exe /x <GUID> /qn /forcerestartNote:The ‘/forcerestart’ parameter forces the machine to restart after uninstall-ing the MLNX_WinOF MSI. This is recommended action for a complete unin-stall procedure.Note:For all command options, enter ‘msiexec.exe /?’.。
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
Mellanox OFED for Linux Installation GuideRev 1.5.1Mellanox Technologies350 Oakmead Parkway, Suite 100Sunnyvale, CA 94085U.S.A.Tel: (408) 970-3400Fax: (408) 970-3403Mellanox Technologies, Ltd.PO Box 586 Hermon Building Yokneam 20692Israel Tel: +972-4-909-7200Fax: +972-4-959-3245© Copyright 2010. Mellanox Technologies, Inc. All Rights Reserved.Mellanox®, BridgeX®, ConnectX®, InfiniBlast®, InfiniBridge®, InfiniHost®, InfiniPCI®, InfiniRISC®, InfiniScale®, andVirtual Protocol Interconnect are registered trademarks of Mellanox Technologies, Ltd.CORE-Direct , FabricIT, and PhyX are trademarks of Mellanox Technologies, Ltd.All other marks and names mentioned herein may be trademarks of their respective companies.Document Number: 2914Rev 1.5.1Mellanox Technologies 2NOTE:THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PRO-VIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUS-TOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CAN-NOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY . ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING , BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MER-CHANTABILITY , FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEM-PLARY , OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING , BUT NOT LIMITED TO, PAYMENT FOR PROCURE-MENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY , WHETHER IN CONTRACT, STRICT LIABILITY , OR TORT(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.Mellanox OFED for Linux User’s Manual Rev 1.5.1 InstallationThis chapter describes how to install and test the Mellanox OFED for Linux package on a singlehost machine with Mellanox InfiniBand and/or Ethernet adapter hardware installed. The chapterincludes the following sections:•“Hardware and Software Requirements” (page 3)•“Downloading Mellanox OFED” (page 4)•“Installing Mellanox OFED” (page 4)•“Uninstalling Mellanox OFED” (page 15)Hardware and Software RequirementsHardware RequirementsPlatforms• A server platform with an adapter card based on one of the following Mellanox Technologies’InfiniBand HCA devices:-MT25408 ConnectX®-2 (VPI, IB, EN, FCoE) (firmware: fw-ConnectX2)-MT25408 ConnectX® (VPI, IB, EN, FCoE) (firmware: fw-25408)-MT25208 InfiniHost® III Ex (firmware: fw-25218 for Mem-Free cards, and fw-25208 for cards withmemory)-MT25204 InfiniHost® III Lx (firmware: fw-25204)-MT23108 InfiniHost® (firmware: fw-23108)Note For the list of supported architecture platforms, please refer to the Mellanox OFEDRelease Notes file.Required Disk Space for Installation•400MBSoftware RequirementsOperating System•Linux operating systemNote For the list of supported operating system distributions and kernels, please refer to the Mellanox OFED Release Notes file.Installer Privileges•The installation requires administrator privileges on the target machineMellanox Technologies3Rev 1.5.1Mellanox Technologies 4Downloading Mellanox OFEDStep 1.Verify that the system has a Mellanox network adapter (HCA/NIC) installed by ensuringthat you can see ConnectX or InfiniHost entries in the display.The following example shows a system with an installed Mellanox HCA:host1# lspci -v | grep Mellanox02:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (reva0)Step 2.Download the ISO image to your host.The image’s name has the format MLNX_OFED_LINUX-<ver>-<OS label>.iso. You candownload it from > Products > IB SW/Drivers.Step e the md5sum utility to confirm the file integrity of your ISO image. Run the followingcommand and compare the result to the value provided on the download page.host1$ md5sum MLNX_OFED_LINUX-<ver>-<OS label>.isoInstalling Mellanox OFEDThe installation script, mlnxofedinstall , performs the following:•Discovers the currently installed kernel•Uninstalls any software stacks that are part of the standard operating system distribution oranother vendor's commercial stack•Installs the MLNX_OFED_LINUX binary RPMs (if they are available for the current kernel)•Identifies the currently installed InfiniBand and Ethernet network adapters and automatically 1upgrades the firmwarePre-installation Notes•The installation script removes all previously installed Mellanox OFED packages and re-installsfrom scratch. You will be prompted to acknowledge the deletion of the old packages.Note Pre-existing configuration files will be saved with the extension “.conf.saverpm”.•If you need to install Mellanox OFED on an entire (homogeneous) cluster, a common strategy isto mount the ISO image on one of the cluster nodes and then copy it to a shared file system suchas NFS. To install on all the cluster nodes, use cluster-aware tools (such as pdsh).•If your kernel version does not match with any of the offered pre-built RPMs, you can add yourkernel version by using the “mlnx_add_kernel_support.sh ” script located under the docs/directory.Usage:mlnx_add_kernel_support.sh -i|--iso <mlnx iso>[-t|--tmpdir <local work dir>][-v|--verbose]1.The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option.Mellanox OFED for Linux User’s Manual Rev 1.5.1 ExampleThe following command will create a MLNX_OFED_LINUX ISO image for RedHat 5.4 underthe/tmp directory.MLNX_OFED_LINUX-1.5.1-rhel5.4/docs/mlnx_add_kernel_support.sh -i/mnt/MLNX_OFED_LINUX-1.5.1-rhel5.4.isoAll Mellanox, OEM, OFED, or Distribution IB packages will be removed.Do you want to continue?[y/N]:yRemoving OFED RPMs...Running mkisofs...Created /tmp/MLNX_OFED_LINUX-1.5.1-rhel5.4.isoInstallation ScriptMellanox OFED includes an installation script called mlnxofedinstall. Its usage is describedbelow. You will use it during the installation procedure described in Section , “Installation Proce-dure,” on page 6.Usage./mlnxofedinstall [OPTIONS]Note If no options are provided to the script, then all available RPMs are installed.Options-c|--config <packages config_file>Example of the configuration file can be found under docs -n|--net <network config file>Example of the network configuration file can be foundunder docs-p|--print-available Print available packages for the current platform and cre-ate a corresponding ofed.conf file. The installation scriptexits after creating ofed.conf.--with-fc Install FCoE support — Available on RHEL5.2 ONLY--with-32bit Install 32-bit libraries (default). This is relevant forx86_64 and ppc64 platforms.--without-32bit Skip 32-bit libraries installation--without-ib-bonding Skip ib-bonding RPM installation--without-depcheck Skip Distro's libraries check--without-fw-update Skip firmware update--force-fw-update Force firmware update--force Force installation (without querying the user)--all Install all kernel modules, libibverbs, libibumad, librd-macm, mft, mstflint, diagnostic tools, OpenSM, ib-bonding,MVAPICH, Open MPI, MPI tests, MPI selector, perftest, sdp-netstat and libsdp srptools, rds-tools, static and dynamiclibrariesMellanox Technologies5Rev 1.5.1Mellanox Technologies6--hpc Install all kernel modules, libibverbs, libibumad, librd-macm, mft, mstflint, diagnostic tools, OpenSM , ib-bonding,MVAPICH, Open MPI, MPI tests, MPI selector, dynamic librar-ies--basic Install all kernel modules, libibverbs, libibumad, mft,mstflint, dynamic libraries--msmInstall all kernel modules, libibverbs, libibumad, mft,mstflint, diagnostic tools, OpenSM , ib-bonding, dynamiclibrariesNOTE: With --msm flag, the OpenSM daemon is configured torun upon boot.-v|-vv|-vvvSet verbosity level -q Set quiet - no messages will be printed mlnxofedinstall Return CodesTable 1 lists the mlnxofedinstall script return codes and their meanings.Installation ProcedureStep 1.Login to the installation machine as root.Step 2.Mount the ISO image on your machine host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>.iso /mntNote After mounting the ISO image, /mnt will be a Read Only folder.Step 3.Run the installation scripthost1# /mnt/mlnxofedinstallThis program will install the MLNX_OFED_LINUX package on your machine.Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed.Do you want to continue?[y/N]:yUninstalling the previous version of OFEDTable 1 - mlnxofedinstall Return CodesReturn CodeMeaning 0The Installation ended successfully 1The installation failed 2No firmware was found for the adapter device 3Failed to start the mst driverMellanox OFED for Linux User’s Manual Rev 1.5.1 Starting MLNX_OFED_LINUX-1.5.1-rc6 installation ...Installing kernel-ib RPMPreparing... ########################################### [100%]1:kernel-ib ########################################### [100%]Installing kernel-ib-devel RPMPreparing... ########################################### [100%]1:kernel-ib-devel ########################################### [100%]Installing mft RPMPreparing... ########################################### [100%]1:mft ########################################### [100%]Installing mpi-selector RPMPreparing... ########################################### [100%]1:mpi-selector ########################################### [100%]Install user level RPMs:Preparing... ########################################### [100%]1:libibumad ########################################### [100%]Preparing... ########################################### [100%]1:libibumad ########################################### [100%]Preparing... ########################################### [100%]1:libibmad ########################################### [100%]Preparing... ########################################### [100%]1:libibmad ########################################### [100%]Preparing... ########################################### [100%]1:libibumad-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibumad-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibmad-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibmad-devel ########################################### [100%]Preparing... ########################################### [100%]1:ofed-scripts ########################################### [100%]Preparing... ########################################### [100%]1:libibverbs ########################################### [100%]Preparing... ########################################### [100%]1:libibverbs ########################################### [100%]Preparing... ########################################### [100%]1:libibverbs-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibverbs-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibverbs-devel-static########################################### [100%]Preparing... ########################################### [100%]1:libibverbs-devel-static########################################### [100%]Preparing... ########################################### [100%]1:libibverbs-utils ########################################### [100%]Preparing... ########################################### [100%]1:libmthca ########################################### [100%]Mellanox Technologies7Rev 1.5.1Mellanox Technologies8Preparing... ########################################### [100%]1:libmthca ########################################### [100%]Preparing... ########################################### [100%]1:libmthca-devel-static ########################################### [100%]Preparing... ########################################### [100%]1:libmthca-devel-static ########################################### [100%]Preparing... ########################################### [100%]1:libmlx4 ########################################### [100%]Preparing... ########################################### [100%]1:libmlx4 ########################################### [100%]Preparing... ########################################### [100%]1:libmlx4-devel ########################################### [100%]Preparing... ########################################### [100%]1:libmlx4-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibcm ########################################### [100%]Preparing... ########################################### [100%]1:libibcm ########################################### [100%]Preparing... ########################################### [100%]1:libibcm-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibcm-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibumad-static ########################################### [100%]Preparing... ########################################### [100%]1:libibumad-static ########################################### [100%]Preparing... ########################################### [100%]1:libibmad-static ########################################### [100%]Preparing... ########################################### [100%]1:libibmad-static ########################################### [100%]Preparing... ########################################### [100%]1:ibsim ########################################### [100%]Preparing... ########################################### [100%]1:librdmacm ########################################### [100%]Preparing... ########################################### [100%]1:librdmacm ########################################### [100%]Preparing... ########################################### [100%]1:librdmacm-utils ########################################### [100%]Preparing... ########################################### [100%]1:librdmacm-devel ########################################### [100%]Preparing... ########################################### [100%]1:librdmacm-devel ########################################### [100%]Preparing... ########################################### [100%]1:libsdp ########################################### [100%]Preparing... ########################################### [100%]1:libsdp ########################################### [100%]Preparing... ########################################### [100%]1:libsdp-devel ########################################### [100%]Preparing... ########################################### [100%]1:libsdp-devel ########################################### [100%]Mellanox OFED for Linux User’s Manual Rev 1.5.1Preparing... ########################################### [100%]1:opensm-libs ########################################### [100%]Preparing... ########################################### [100%]1:opensm-libs ########################################### [100%]Preparing... ########################################### [100%]1:opensm ########################################### [100%]opensmd 0:off 1:off 2:off 3:off 4:off 5:off 6:offPreparing... ########################################### [100%]1:opensm-devel ########################################### [100%]Preparing... ########################################### [100%]1:opensm-devel ########################################### [100%]Preparing... ########################################### [100%]1:opensm-static ########################################### [100%]Preparing... ########################################### [100%]1:opensm-static ########################################### [100%]Preparing... ########################################### [100%]1:compat-dapl ########################################### [100%]Preparing... ########################################### [100%]1:compat-dapl ########################################### [100%]Preparing... ########################################### [100%]1:compat-dapl-devel ########################################### [100%]Preparing... ########################################### [100%]1:compat-dapl-devel ########################################### [100%]Preparing... ########################################### [100%]1:dapl ########################################### [100%]Preparing... ########################################### [100%]1:dapl ########################################### [100%]Preparing... ########################################### [100%]1:dapl-devel ########################################### [100%]Preparing... ########################################### [100%]1:dapl-devel ########################################### [100%]Preparing... ########################################### [100%]1:dapl-devel-static ########################################### [100%]Preparing... ########################################### [100%]1:dapl-devel-static ########################################### [100%]Preparing... ########################################### [100%]1:dapl-utils ########################################### [100%]Preparing... ########################################### [100%]1:perftest ########################################### [100%]Preparing... ########################################### [100%]1:mstflint ########################################### [100%]Preparing... ########################################### [100%]1:sdpnetstat ########################################### [100%]Preparing... ########################################### [100%]1:srptools ########################################### [100%]Preparing... ########################################### [100%]1:rds-tools ########################################### [100%]Preparing... ########################################### [100%]1:rnfs-utils ########################################### [100%]Preparing... ########################################### [100%]Mellanox Technologies9Rev 1.5.1Mellanox Technologies 10 1:ibutils ########################################### [100%]Preparing... ########################################### [100%]1:ibutils2 ########################################### [100%]Preparing... ########################################### [100%]1:ibdump ########################################### [100%]Preparing... ########################################### [100%]1:infiniband-diags ########################################### [100%]Preparing... ########################################### [100%]1:qperf ########################################### [100%]Preparing... ########################################### [100%]1:mlnxofed-docs ########################################### [100%]Preparing... ########################################### [100%]1:mvapich_gcc ########################################### [100%]Preparing... ########################################### [100%]1:mvapich_pgi ########################################### [100%]Preparing... ########################################### [100%]1:mvapich_intel ########################################### [100%]Preparing... ########################################### [100%]1:openmpi_gcc ########################################### [100%]Preparing... ########################################### [100%]1:openmpi_pgi ########################################### [100%]Preparing... ########################################### [100%]1:openmpi_intel ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_mvapich_gcc ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_mvapich_pgi ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_mvapich_intel ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_openmpi_gcc ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_openmpi_pgi ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_openmpi_intel ########################################### [100%]Device (15b3:634a):02:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0)Link Width: 8xLink Speed: 2.5Gb/sInstallation finished successfully.Programming HCA firmware for /dev/mst/mt25418_pci_cr0 deviceRunning: mlxburn -d /dev/mst/mt25418_pci_cr0 -fw /tmp/MLNX_OFED_LINUX-1.5.1/MLNX_OFED_LINUX-1.5.1-sles11/firmware/fw-25408/2_7_000/fw-25408-rel.mlx -dev_type 25408 -no-I- Querying device ...-I- Using auto detected configuration file: /tmp/MLNX_OFED_LINUX-1.5.1/MLNX_OFED_LINUX-1.5.1-sles11/firmware/fw-25408/2_7_000/MHGH28-XTC_A4-A7.ini (PSID = MT_04A0140005)-I- Generating image ...Current FW version on flash: 2.6.0New FW version: 2.7.0Burning FW image without signatures - OKRestoring signature - OK-I- Image burn completed successfully.Please reboot your system for the changes to take effect.warning: /etc/infiniband/openib.conf saved as /etc/infiniband/openib.conf.rpmsaveNote In case your machine has the latest firmware, no firmware update will occur and the installation script will print at the end of installation a message similar to the following:...Installation finished successfully.The firmware version 2.7.000 is up to date.Note: To force firmware update use '--force-fw-update' flag.Note In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. Please contact your hardwarevendor for help on firmware updates.Error message:-I- Querying device ...-E- Can't auto detect fw configuration file: ...Step 4.In case the installation script performed firmware updates to your network adapter hard-ware, it will ask you to reboot your machine.Step 5.The script adds the following lines to /etc/security/limits.conf for the userspace components such as MPI:* soft memlock unlimited* hard memlock unlimitedThese settings unlimit the amount of memory that can be pinned by a user space application.If desired, tune the value unlimited to a specific amount of RAM.Step 6.For your machine to be part of the InfiniBand/VPI fabric, a Subnet Manager must be run-ning on one of the fabric nodes. At this point, Mellanox OFED for Linux has alreadyinstalled the OpenSM Subnet Manager on your machine. For details on starting OpenSM,see Chapter 12, “OpenSM – Subnet Manager”.Step 7.(InfiniBand only) Run the hca_self_test.ofed utility to verify whether or not the InfiniBand link is up. The utility also checks for and displays additional information suchas•HCA firmware version•Kernel architecture•Driver version•Number of active HCA ports along with their states•Node GUIDNote For more details on hca_self_test.ofed, see the file hca_self_test.readme under docs/.host1# /usr/bin/hca_self_test.ofed---- Performing InfiniBand HCA Self Test ----Number of HCAs Detected (1)PCI Device Check ....................... PASSKernel Arch ............................ x86_64Host Driver Version .................... MLNX_OFED_LINUX-1.5.1 (OFED-1.5.1-mlnx9) 1.5.1-2.6.9_89.ELlargesmpHost Driver RPM Check .................. PASSHCA Firmware on HCA #0 ................. 2.7.000HCA Firmware Check on HCA #0 ........... PASSHost Driver Initialization ............. PASSNumber of HCA Ports Active 0Port State of Port #0 on HCA #0 ........ INITPort State of Port #0 on HCA #0 ........ DOWNError Counter Check on HCA #0 .......... PASSKernel Syslog Check .................... PASSNode GUID on HCA #0 .................... 00:02:c9:03:00:00:10:e0------------------ DONE ---------------------Note After the installer completes, information about the Mellanox OFED installation such as prefix, kernel version, and installation parameters can be retrieved by running the com-mand /etc/infiniband/info.Installation ResultsSoftware•The OFED and MFT packages are installed under the /usr directory.•The kernel modules are installed under:-InfiniBand subsystem:/lib/modules/`uname -r`/updates/kernel/drivers/infiniband/-mlx4 driver:Under /lib/modules/`uname -r`/updates/kernel/drivers/net/mlx4you will find mlx4_core.ko, mlx4_en.ko, mlx4_ib.ko (and mlx4_fc ifyou ran the installation script with --with-fc)-RDS:/lib/modules/`uname -r`/updates/kernel/net/rds/rds.ko-Bonding module:/lib/modules/`uname -r`/updates/kernel/drivers/net/bonding/bonding.ko •The package kernel-ib-devel include files are placed under /usr/src/ofa_kernel/include/. These include files should be used when building kernel modules that use the stack. (Note that the include files, if needed, are “backported” to your kernel.)•The raw package (un-backported) source files are placed under/usr/src/ofa_kernel-<ver>•The script openibd is installed under /etc/init.d/. This script can be used to load and unload the software stack.•The script connectx_port_config is installed under /sbin. This script can be used to con-figure the ports of ConnectX network adapter cards to Ethernet and/or InfiniBand. For details on this script, please see Section 3.1, “Port Type Management”.•The directory /etc/infiniband is created with the files info and openib.conf and con-nectx.conf. The info script can be used to retrieve Mellanox OFED installation information. The openib.conf file contains the list of modules that are loaded when the openibd script is used. The connectx.conf file saves the ConnectX adapter card’s ports configuration to Ether-net and/or InfiniBand. This file is used at driver start/restart (/etc/init.d/openibd start) .•The file 90-ib.rules is installed under /etc/udev/rules.d/•If OpenSM is installed, the daemon opensmd is installed under /etc/init.d/ and opensm.conf is installed under /etc.•If IPoIB configuration files are included, ifcfg-ib<n> files will be installed under:-/etc/sysconfig/network-scripts/ on a RedHat machine-/etc/sysconfig/network/ on a SuSE machine•The installation process unlimits the amount of memory that can be pinned by a user space application. See Step 5.•Man pages will be installed under /usr/share/man/Firmware•The firmware of existing network adapter devices will be updated if the following two condi-tions are fullfilled:1. You run the installation script in default mode; that is, without the option‘--without-fw-update’.2. The firmware version of the adapter device is older than the firmware version includedwiththe Mellanox OFED ISO imageNote If an adapter’s Flash was originially programmed with an Expansion ROM image, the automatic firmware update will also burn an Expansion ROM image.•In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. Please contact your hardware vendor for help on firmware updates.Error message:。