SNIA-SDC23-Rose-NVMe-oF-Boot-Talk.pdf

1、1|2023 SNIA.All Rights Reserved.Virtual ConferenceSeptember 28-29,2021Booting your OS across NVMe over FabricsNVMe Boot Specification+Boot over NVMe/TCP Reference ImplementationCurtis Ballard,Distinguished Technologist,HPECharles Rose,Senior Principal Engineer,Dell2|2023 SNIA.All Rights Reserved.Age

2、nda NVM Express(NVMe)Boot Specification Overview Standardizing Booting from NVMe and NVMe-oF Namespaces Ecosystem Cooperation:UEFI and DMTF Configuring NVMe-oF Boot(UEFI-Based Example)Reference Implementations&Future Enhancements Q&A3|2023 SNIA.All Rights Reserved.NVM Express,Inc.Overview NVM Expres

3、s is 110+members strong and was created to expose the benefits of non-volatile memory in all types of computing environments NVMe technology delivers high bandwidth,low latency storage and overcomes bottlenecks NVM Express technology includes the below specifications:NVM Express(NVMe)Base Specificat

4、ion NVM Express Boot Specification NVM Express Command Set Specifications NVM Express Transport Specifications NVMe Management Interface(NVMe-MI)Markets enhanced by NVM Express technology include:Artificial Intelligence Composable InfrastructureMachine LearningCloud/Data CenterSSD Controllers Storag

5、e PC/Mobile/IoTHealthcare 0Promoter Group2022-20234|2023 SNIA.All Rights Reserved.*NVMe 2.0 specifications were released on June 3,2021-Refer to nvmexpress.org/developersNVMe Base SpecificationCommand SetSpecificationsTransportSpecificationsNVMe ManagementInterface SpecificationNVMe NVM Command Set

6、SpecificationNVMe Zoned Namespace Command Set SpecificationNVMe over PCIeTransport SpecificationNVMe over RDMATransport SpecificationNVMe Key Value Command Set SpecificationNVMe over TCPTransport SpecificationNVMe Boot SpecificationNVMe 2.0 Family of Specifications5|2023 SNIA.All Rights Reserved.AMD

7、Avery Design SystemsBeijing MemBlaze TechnologyBroadcomDapuStorDell Technologies*FADUFred KnightHewlett Packard EnterpriseHuawei TechnologiesIBMInnoGritInspur Electronic Information Industry Intel*JetIO TechnologyKioxiaLenovoLightBits LabsMarvellMicrochipComputational StorageTask GroupBoard of Direc

8、torsMarketing WorkgroupTechnical WorkgroupManagement Interface Task GroupInterop&Compliance Task GroupFabrics and Multi-DomainTask GroupKey ValueTask GroupZoned NamespacesTask GroupBootTask Group Membership:41 companiesMicron TechnologyMicrosoftNVIDIA*Oracle AmericaPhison ElectronicsPliopsQualcommSa

9、msungScaleFluxSeagate TechnologyShenzhen Unionmemory Information SystemSilicon MotionSolidigmSUSESwissbitTeledyne LeCroyULINK TechnologyUniversity of New HampshireVMWareWestern DigitalYangtze Memory TechnologiesNVMe Boot Task Group*NVMe Boot Task Group Co-Chair6|2023 SNIA.All Rights Reserved.Current

10、ly successful storage networking technologies such as Fibre Channel and iSCSI have standardized solutions that allow attached computer systems to boot from OS images stored on storage nodes.The lack of a standardized capability in NVMe-oF presented a barrier for adoption.This was a missing requireme

11、nt for a networked storage technology.*AFA=All Flash Array Storage SystemWhy Does NVMe Technology Need a Boot Specification7|2023 SNIA.All Rights Reserved.NVMe/TCP boot enabled standardization to leverage past iSCSI lessons and ecosystem enablementiSCSI enabled boot and OS handover through a mechani

12、sm called the“iSCSI Boot Firmware Table”(iBFT)iBFT contains information to be shared between BIOS/pre-boot environments and the OSLeveraging Existing Remote Storage Boot Over EthernetEthernetBoot SWInitiatorTCP/IPiSCSIEthernetTargetiSCSI TargetBlock I/OiSCSIiBFTDisk(LUN)BootloaderOS8|2023 SNIA.All R

13、ights Reserved.Delta between booting from iSCSI and booting from NVMe-oFEthernetBoot SWInitiatorTCP/IPiSCSIEthernetTargetiSCSI TargetBlock I/OiSCSIiBFTDisk(LUN)BootloaderOSAdapterBoot SWNVMe-oF HostFabricNVMe-oFAdapterNVMe-oF DeviceNVMe-oF SubsystemBlock I/ONVMe-oFNBFTNamespaceBootloaderOSNVMe/TCP b

14、oot enabled standardization to leverage past iSCSI lessons and ecosystem enablementiSCSI enabled boot and OS handover through a mechanism called the“iSCSI Boot Firmware Table”(iBFT)iBFT contains information to be shared between BIOS/pre-boot environments and the OSBoot from NVMe/TCP main concepts(bo

15、ot flow and handover mechanism)are similar to booting from iSCSIStandardize Booting from NVMe and NVMe-oF Namespaces9|2023 SNIA.All Rights Reserved.Standardize Booting from NVMe and NVMe-oF Namespaces NVMe/TCP boot enabled standardization to leverage past iSCSI lessons and ecosystem enablement iSCSI

16、 enabled boot and OS handover through a mechanism called the“iSCSI Boot Firmware Table”(iBFT)iBFT contains information to be shared between BIOS/pre-boot environments and the OS Boot from NVMe/TCP main concepts(boot flow and handover mechanism)are similar to booting from iSCSI NVMe needs a similar c

17、onfiguration mechanism,NBFT(NVMe Boot Firmware Table)Delta between booting from iSCSI and booting from NVMe-oFEthernetBoot SWInitiatorTCP/IPiSCSIEthernetTargetiSCSI TargetBlock I/OiSCSIiBFTDisk(LUN)BootloaderOSAdapterBoot SWNVMe-oF HostFabricNVMe-oFAdapterNVMe-oF DeviceNVMe-oF SubsystemBlock I/ONVMe

18、-oFNBFTNamespaceBootloaderOS10|2023 SNIA.All Rights Reserved.NVMe Boot SpecificationPublished on NVMe.org*11/2022Defines constructs&guidelines for booting from NVM Express interfaces over supported transports Version 1.0 defines extensions to the NVMe interface for booting over NVMe/TCP transportNor

19、mative content describes General concepts for NVMe/NVMe-oF bootMechanism for boot device enumeration and configurationhandoff from Pre-OS to OS environments(ACPI tables)Informative content IntroducesBoot stages and flow in a UEFI pre-OS environmentImplementation and adoption guidelines and best-prac

20、ticesNVMe-oF boot configuration in the Pre-boot environmentMechanics for consumption of ACPI tables by the OSOS and fabric transport specifics*https:/nvmexpress.org/specifications/I/O Command Set Specifications(e.g.,NVM,Key Value,Zoned Namespace)Transport Specifications(e.g.,PCIe,RDMA,TCP)NVMe Base

21、SpecificationNVMe Management Interface SpecificationBoot SpecificationStandardize Booting from NVMe and NVMe-oFNamespaces11|2023 SNIA.All Rights Reserved.Collaboration with the following ecosystem and industry partners was key1.UEFI Forum:ACPI Specification(6.5*):Adds ACPI NVMe Boot Firmware Table(N

22、BFT)to ACPI.orgUEFI System Specification(2.10*):Adds device path extension for NVMe-oF boot2.DMTF:Adds standardization for Redfish NVMe-oF secrets registry in the 2021.4 release3.NVMe Boot Spec 1.0 introduces standardization of booting over NVMe and NVMe-oF(starting with Booting over NVMe-TCP)4.Publ

23、ic reference implementation:The code for booting over NVMe-oF is based on open-source frameworks.Ecosystem Cooperation to Enable Standardization*See references slide for publication locationsAdapterBoot SWNVMe-oF HostFabricNVMe-oFAdapterNVMe-oF DeviceNVMe-oF SubsystemBlock I/ONVMe-oFNBFTNamespaceBoo

24、tloaderOSRedfish12|2023 SNIA.All Rights Reserved.Collaboration with the following ecosystem and industry partners was key1.UEFI Forum:ACPI Specification(6.5*):Adds ACPI NVMe Boot Firmware Table(NBFT)to ACPI.orgUEFI System Specification(2.10*):Adds device path extension for NVMe-oF boot2.DMTF:Adds st

25、andardization for Redfish NVMe-oF secrets registry in the 2021.4 release3.NVMe Boot Spec 1.0 introduces standardization of booting over NVMe and NVMe-oF(starting with Booting over NVMe-TCP)4.Public reference implementation:The code for booting over NVMe-oF is based on open-source frameworks.Ecosyste

26、m Cooperation to Enable StandardizationAdapterBoot SWNVMe-oF HostFabricNVMe-oFAdapterNVMe-oF DeviceNVMe-oF SubsystemBlock I/ONVMe-oFNBFTNamespaceBootloaderOSRedfish*See references slide for publication locations13|2023 SNIA.All Rights Reserved.UEFI Collaboration*https:/uefi.org/specs/UEFI/2.10/10_Pr

27、otocols_Device_Path_Protocol.html#nvme-over-fabric-nvme-of-namespace-device-path Added to the ACPI XSDT Signature Table*NVMe over Fabrics Device Path extension to support for NVMe-oF boot from UEFI System Spec*https:/uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html14|2023 SNIA.All Rig

28、hts Reserved.Ecosystem Cooperation to Enable StandardizationAdapterBoot SWNVMe-oF HostFabricNVMe-oFAdapterNVMe-oF DeviceNVMe-oF SubsystemBlock I/ONVMe-oFNBFTNamespaceBootloaderOSRedfish*See references slide for publication locationsCollaboration with the following ecosystem and industry partners was

29、 key1.UEFI Forum:ACPI Specification(ECR into 6.5*):Adds ACPI NVMe Boot Firmware Table(NBFT)to ACPI.orgUEFI System Specification(ECR into 2.10*):Adds device path extension for NVMe-oF boot2.DMTF:Adds standardization for Redfish NVMe-oF secrets registry in the 2021.4 release3.NVMe Boot Spec 1.0 introd

30、uces standardization of booting over NVMe and NVMe-oF(starting with Booting over NVMe-TCP)4.Public reference implementation:The code for booting over NVMe-oF is based on open-source frameworks.15|2023 SNIA.All Rights Reserved.Adds standardization for NVMe-oF secrets registry for RF 2021.4DMTF Collab

31、oration16|2023 SNIA.All Rights Reserved.Collaboration with the following ecosystem and industry partners was key1.UEFI Forum:ACPI Specification(ECR into 6.5*):Adds ACPI NVMe Boot Firmware Table(NBFT)to ACPI.orgUEFI System Specification(ECR into 2.10*):Adds device path extension for NVMe-oF boot2.DMT

32、F:Adds standardization for Redfish NVMe-oF secrets registry in the 2021.4 release3.NVMe Boot Spec 1.0 introduces standardization of booting over NVMe and NVMe-oF(starting with Booting over NVMe-TCP)4.Public reference implementation:The code for booting over NVMe-oF is based on open-source frameworks

33、.Ecosystem Cooperation to Enable StandardizationAdapterBoot SWNVMe-oF HostFabricNVMe-oFAdapterNVMe-oF DeviceNVMe-oF SubsystemBlock I/ONVMe-oFNBFTNamespaceBootloaderOSRedfish*See references slide for publication locations17|2023 SNIA.All Rights Reserved.Information presented to the OS using ACPI XSDT

34、 Table at OS boot provideslocal Pre-OS-OS agnostic configuration communications medium;independent from UEFI,UBOOT,standardized means of passing configuration&connection context from pre-OS Boot environment to an administratively configured OS runtimeElementDescriptionHeaderAn ACPI structure header

35、with some additional NBFT specific info.Control DescriptorIndicates the location of host,HFI,SSNS,security,and discovery descriptors.Host DescriptorHost information.HFI DescriptorAn indexable table of HFI Descriptors,one for each fabric interface on the host.Subsystem Namespace DescriptorAn indexabl

36、e table of SSNS Descriptors.Security DescriptorAn indexable table of Security descriptors.Discovery DescriptorAn indexable table of Discovery Ddescriptors.HFI Transport DescriptorIndicated by an HFI Descriptor,corresponds to a specific transport for a single HFI.SSNS Extended Info DescriptorIndicate

37、d by an SSNS Descriptor if needed.NBFT:Pre-OS to OS Configuration Handoff Mechanismhttps:/nvmexpress.org/specifications/18|2023 SNIA.All Rights Reserved.Public Reference Implementation Based on UEFIReference code*for booting over NVMe-oF is basedon the NVMe Boot Spec 1.0 on open-source frameworksDev

38、eloped by a subset of NVM Express member companies including:Released*under BSD-3-Clause(or other open-source license as required by components)*https:/ SNIA.All Rights Reserved.The seven phases in a UEFI boot sequence*1.Security(SEC)2.Pre-EFI Initialization(PEI)3.Drive Execution Environment(DXE)4.B

39、oot Device Selection(BDS)5.Transient System Load(TSL)6.Runtime(RT)7.After Life(AL)UEFI Boot Phases*Tianocore:EDK2 Build Specification20|2023 SNIA.All Rights Reserved.Boot Attempt configuration is stored in UEFI variables.Administrator configures Pre-OS driver:target subsystem NQNtarget IP addresstar

40、get port#Configuring NVMe-oF Boot(UEFI-based example):Pre-Operating System Boottarget namespacehost NQNsecurity related info21|2023 SNIA.All Rights Reserved.Driver Execution Environment phase:DXE driversupporting NVMe-oF boot is loaded and executed:reads configuration from UEFI variablessets up netw

41、ork(interfaces,routing,)(optionally)retrieves authentication credentials(optionally)performs discovery and authenticationconnects to NVMe subsystems provides namespaces to the UEFI Boot Manager as block devicesstores the configuration in the NBFT:can later be accessed bythe OS as an ACPI tableNew fu

42、nctionalityConfiguring NVMe-oF Boot(UEFI-based example):Pre-Operating System Boot22|2023 SNIA.All Rights Reserved.Boot Device Selection phase:The Namespace can then be selected as final boot device for OS bootExisting functionalityConfiguring NVMe-oF Boot(UEFI-based example):Pre-Operating System Boo

43、t23|2023 SNIA.All Rights Reserved.Transient System Load phase:OS image loaded from boot device UEFI hands over execution to OS specific boot loaderOS Boot Loader continues the OS bootAt this point,the NBFT has been generated,stored inmain memory,and can be accessed by the OS as an ACPI tableExisting

44、 functionalityConfiguring NVMe-oF Boot(UEFI-based example):Pre-Operating System Boot24|2023 SNIA.All Rights Reserved.Configuring NVMe-oF Boot(UEFI-based example):OS Transition to RuntimeRuntime phase:read the configuration from the NBFTset up the network(interfaces,routing,)(optionally)retrieve auth

45、entication credentials(optionally)perform discovery and authenticationconnect to NVMe subsystemsprovide namespaces to other parts of the OS25|2023 SNIA.All Rights Reserved.Configuring NVMe-oF Boot(UEFI-based example):Typical OS Handover and Initialization Normal operating system boot:To persist info

46、 to restore NVMe-oF connections,OS may either:continue using the NBFT Use OS specific mechanism Operating system installation:A user may either:use the NBFT provided host NQN as its own host NQN set a separate host NQN(if NVMe-oF subsystem supports multiple host NQNs)26|2023 SNIA.All Rights Reserved

47、.Reference Implementation of Booting over NVMe/TCPPre-OS time of boot:EDK2 NVMe-oF UEFI Driver for the NVMe/TCP transport ACPI NBFT will be produced by this UEFI implementation prior to OS bootOS Boot and Runtime:Linux reference implementation that:Exposes the NBFT to the user-space Consumes the NBF

48、T contents to connect to configured namespaces Enables common tools(e.g.,dracut,nvme-cli)to use the NBFT27|2023 SNIA.All Rights Reserved.NVMe-oFDriverNVMe/TCPTransport DriverLibSPDK and ShimUEFI TCP StackACPI nBFTUEFI HiiI/ORegisterCryptoPkg PSK EnablementNVMe-oF Security/Authentication LayerUEFI Bl

49、ockIO InterfaceReuseNew WorkSPDKOpenSSL PSKLegendDevelopmentEDK2 Concept ArchitectureEDK2 Reference Architecture as implemented(Not yet published upstream for review)ACPI NBFTConfiguring NVMe-oF Boot(UEFI-based example):Pre-Operating System BootNVMe-oF LibQueueQueue-pair pair managementmanagementCon

50、troller Controller managementmanagementTransport Transport managementmanagementTCP TCP managementmanagementNamespace Namespace managementmanagementUEFI HII28|2023 SNIA.All Rights Reserved.nvmeofcli for EFI:Command Line tool to facilitate basic diagnostics and interoperability with pre-OS reference d

51、river nvmeofcli list command nvmeofcli connect commandPre-Boot Environment Configuration Tool29|2023 SNIA.All Rights Reserved.Pre-Boot Environment HII30|2023 SNIA.All Rights Reserved.OS Handoff Enablement in Reference DesignOS Handoff Enablement in Reference DesignLinux Kernel support for ACPI“NBFT”

52、TableUser-Space Device Connection and Configuration tools consuming Linux sysfsinitrd/dracut changes to support NVMe/TCP:Detects NBFT presenceconnects pertinent networkinguses nvme-cli to connect to NVMe Subsystems/NamespacesNvme-cli two new subcommands:nvme show-nbft for dumping NBFT contentfree te

53、xt/table formatJSON formatnvme connect-nbftconnect to subsystems and namespaces listed in or discovered through the NBFTEverything except network setupGraphic credit Joey Lee,SUSE 31|2023 SNIA.All Rights Reserved.nvme-cli New subcommands:nvme nbft show free-text format32|2023 SNIA.All Rights Reserve

54、d.nvme-cli New subcommands:“nvme nbft show”JSON format33|2023 SNIA.All Rights Reserved.Reference Implementations of Booting over NVMe/TCPProof-of-Concept for NVMe Boot QEMU based PoCs are available for both openSUSE Leap 15.5 and Fedora 37 These examples are useful because the details of early OS br

55、ing-up differ between distributionsPrerequisites An Intel based host platform running a current version of openSUSE or Fedora A connection to the internet and a root privileged account to administer QEMUSetup is simple setup the Host/Hypervisor system then follow the instructions in the PoCs and the

56、 scripts will configure and install the software to run the QEMU based POC automatically.openSUSE and Fedora PoCs are available at:https:/ SNIA.All Rights Reserved.Support for Authentication/TLS Support for DMTF Redfish Secrets Additional OS and installer supportFuture Enhancements:Open Source and E

57、cosystem35|2023 SNIA.All Rights Reserved.Investigate Booting over Additional TransportsBig Namespace Qty Management in Large FleetsMulti-Path Topology ExamplesSupport Device TreeSetting NVMe-oF Boot Entries in OSContributions are welcome!Join the NVMe Consortium and the NVMe Boot Task Grouphttps:/nv

58、mexpress.org/join-nvme/Future Enhancements:NVMe Boot Specification36|2023 SNIA.All Rights Reserved.Transports may require a new ECR to the UEFI System Spec if they do not already have a Device Path Messaging Type supporting themHeader for new HFI Transport Info Descriptor in NBFTBytes 00 05:Mandator

59、y to describe the Header structure for a new Transport Info Descriptor typeThereafter Transport-specific descriptor flags as needed following Figure 13 in the NVMe Boot SpecBytesDescription00Structure ID01Version02HFI Transport Type.03Transport Info Version05:04HFI Descriptor IndexAdding new Transpo

60、rt Support to NVMe Boot Specification37|2023 SNIA.All Rights Reserved.NVM Express:https:/nvmexpress.org/specifications/UEFI 2.10:https:/uefi.org/specs/UEFI/2.10/10_Protocols_Device_Path_Protocol.html ACPI 6.5:https:/uefi.org/specs/ACPI/6.5/05_ACPI_Software_Programming_Model.html Open-Source Software Repos:https:/ software has been pushed upstream.For edk2 use the version off of the Timberland SIG github.For all other software use the latest upstream version.References and Repositories38|2023 SNIA.All Rights Reserved.Questions?Architected for Performance