1A-102_Standardizing SmartNICs for COTS (Horizontally-Integrated) Solutions-VMware.PPTX

1、Standardizing SmartNICs for COTS(Horizontally-Integrated)SolutionsAndrei WarkentinArm Enablement ArchitectVMwareSan Jose,CA April 26-28,2022Whats in a SmartNIC?2 General purpose-compute+accelerators server“sized”Hardware-and software-defined functions(for the purpose of this presentation)San Jose,CA

2、 April 26-28,2022What can you build with a SmartNIC?3WhatHorizontally Integrated?Vertically Integrated?SmartNIC as an accelerator.SmartNIC as a stand-alone server.SmartNIC as the cornerstone of a server.San Jose,CA April 26-28,2022Different classes of solutions4 Horizontally Integrated “PC”A solutio

3、n is formed from generic,interchangeable off-the-shelf parts.All parts are independently upgradeable.Generic(”upstream”)system software.E.g.a Dell server with Windows installed,and a Broadcom SmartNIC running RedHat software.Facilitates upgrades,reuse of existing hardware investment.Vertically Integ

4、rated “iPhone”A solution is co-engineered by a single provider(or a tight cooperation of providers)Custom SKUs,hardware and software.Tight integration,parts are not generic or not field-upgradeable E.g.an AWS“Nitro”server,with AWS SmartNICs,AWS custom server design,(with and without AWS custom hyper

5、visor)Raze and rebuild.San Jose,CA April 26-28,2022SmartNIC as an accelerator.5 Ideally:“Channel cards”in regular or OCP FF Goes into existing unmodified servers.Runs system software stacks from 3rd parties(not SmartNIC vendor)Challenges:Customer expectations around lifecycle.Provisioning SmartNIC O

6、S.Booting SmartNIC OS.Integration with server/server OS.San Jose,CA April 26-28,2022SmartNIC as a microserver.6 Use-cases:Edge compute.Network appliance(CPE).Storage controller(JBOF/NAS)Custom hardware.Custom backplane+card/OCP module.Custom motherboards.Off-the-shelf software.San Jose,CA April 26-2

7、8,2022SmartNIC as a microserver.7 Use-cases:Edge compute.Network appliance(CPE).Storage controller(JBOF/NAS)Custom hardware.Custom backplane+card/OCP module.Custom motherboards.Off-the-shelf software.San Jose,CA April 26-28,2022SmartNIC as a new way of building servers8 Not the focus of todays prese

8、ntation.Offload I/O and control.Clearly a vertically-integrated solution as far as:SmartNIC software,electricals/mechanicals/connectors Rest of server integration.One of the largest opportunities to simplify and secure servers,separating customers from operators.San Jose,CA April 26-28,2022Standards

9、 for SmartNIC as a platform9 Can I run existing system software on it(and get support)Can I make use of the existing software ecosystem?Is it easy to move software from servers to SmartNICs?Can the resulting solution be moved to a different SmartNIC?San Jose,CA April 26-28,2022Embedded vs.industry-s

10、tandardized10EmbeddedIndustry-standardizedNon-standard IP(UARTs,PCIe,storage)Crippling HW erratas okay,can work around in system softwareNeed custom work on OS to work(e.g.BSP,forked kernels,etc)Non-standard bootingDevice Tree for maximum flexibility,tighter integration of hardware/software.Propriet

11、ary security model.Okay for vertically-integrated approaches.Generic OS distributions and kernels work just add ASIC driversUEFI bootingACPI for hardware discovery,specification-compliant interface between OS and hardware.Standard hardware for common IP(ECAM for PCIe,cache coherent DMA,etc)Industry

12、approaches to security(SRTM/DRTM).Mandatory for horizontally-integrated solutions.Virtualization just works.San Jose,CA April 26-28,2022The”industry”way11 Arm SystemReady ES Reduced from the SystemReady SR(actual server spec)Arm Base System Architecture(BSA)hardware Server Base Boot Requirements(SBB

13、R)UEFI ACPI RISC-V OS-A Server Platform UEFI ACPISan Jose,CA April 26-28,2022Using a SmartNIC in a system has an additional set of challenges.12San Jose,CA April 26-28,202213In practice,for an integrated solution Additional vendor specific firmware interfaces for a SmartNIC OS ESX.io examples:BMC au

14、thentication Report SolutionID to SmartNIC OS Report POST/debugging code code to system Signal ESX.io state change Signal fatal error via PCIe Boot watchdog interfaces Mechanism to handle a“crash dump”request from server Custom handling of SmartNIC OS provisioning requests Abstract non-standard PCIe

15、 via DEN0115(if RC hardware doesnt use ECAM)Firmware-first error handling Industry-standard trusted platform support Core,SoC and card-level power management Memory/IO QoSSan Jose,CA April 26-28,2022SmartNIC as an accelerator-expectations14 A plug-in accelerator is not a computer.No expectations of

16、isolation from containing host Doesnt“go down”Surviving server power,rolling reboots.No corruption,no”bricking”,no breaking out the UART cable or removing from server to reflash/upgrade/recover.No third-party infrastructure(PXE servers,management networks,etc)No expectation of persistent state,custo

17、mer-identifiable data.Not tied to a specific server OEM.”channel”cards to plug in to existing infrastructure.No custom connectors.No custom firmware(SmartNIC or server)Not tied to specific server or SmartNIC software stack.Simple provisioning Easily switch/upgradeSan Jose,CA April 26-28,2022SmartNIC

18、 as an accelerator-challenges How are SmartNIC functions presented to the server system?Always present regardless of SmartNIC OS presence/operation?Dynamic/reliant on SmartNIC OS?Does SmartNIC OS have to boot*before*the rest of the server?Can you dynamically reconfigure the exposed PCIe PFs without

19、a full reboot?How”visible”is the SmartNIC OS state to the server system?(e.g.on SmartNIC OS reset)How coupled is the SmartNIC OS driver state with the host driver state?State leakage,component failure Isolating the server system from SmartNIC failuresSan Jose,CA April 26-28,2022SmartNIC as an accele

20、rator-challenges16 Discovery DPU configuration and updates DPU OS Monitoring Provisioning Boot synchronization and status reporting DPU OS configuration and updates Error notification and recovery Power state coordination End of life/data securityChoice of control surfaceBase Management ControllerPr

21、e-Boot/System OSLimited FlexibilityNo BMC or system firmware modsRequires well-behaved hardwareMost controlRequires a BMCRequires changes to BMC firmwareSan Jose,CA April 26-28,2022If BMC controls the SmartNIC How do you avoid custom cabling?PCIe SMBUS PCIe VDM Protocols?NC-SI(over MCTP)PLDM/Redfish

22、 Provisioning SmartNIC OS?Other”stuff”,e.g.serial consoles,etc.BMC+server firmware interactionHard to standardize(worse:even if you align everyone in the future,you cant support existing systems)San Jose,CA April 26-28,2022If we dont rely on a BMC Could use the SmartNIC in any desktop,edge box,etc n

23、ot limited to servers.All communication is strictly“in-band”(over PCIe to host CPU)Configuration and OS provisioning+integration with host OS boot?Remote access Monitoring,querying information/status,signaling Health Provisioned OS status State transition requestsSan Jose,CA April 26-28,2022SmartNIC

24、 as an accelerator-challenges Discovery DPU configuration and updates DPU OS Monitoring Provisioning Boot synchronization and status reporting DPU OS configuration and updates Error notification and recovery Power state coordination End of life/data securityA few thoughts:No need to try and standard

25、ize on interfaces exposed to server OS that can be handled by drivers.Careful engineering on PCIe exposed functions.Its worthwhile to present a common UEFI protocol to enable OSVs to launch a SmartNIC OS(via OpRom).Its tempting to go the BMC route(full custom)but it prevents proliferation.San Jose,CA April 26-28,2022Thanks!20