Novel Hybrid Immersion Liquid Cooling Technique for Data Center.pdf

1、Novel Hybrid Immersion Liquid Cooling Technique for Data CenterMohamad Hnayno Doctor of Engineering in Thermal and EnergyR&D Engineer at OVHcloudNovel Hybrid Immersion Liquid Cooling Technique for Data CenterSERVERCOOLING ENVOVHcloud server cooling via autonomous rack Autonomous rack is a combinatio

2、n of Air and Water coolingCPU and GPU IT equipment rather than CPU and GPU:RAM,MB,hard disks70%of total heat load30%of total heat loadWater cooling using water blocksBuilt-in air cooling fansAir-water heat exchangerServerPipingWater block on CPUAir cooling using air fridge doorsStrategy for a future

3、 designReduce IT operation temperature(RAM,disks)Elimination of fridge doors and CDUsRespect footprint density(U/m2)Fans EliminationUsage of very low environmental impact fluids Immersion liquid cooling technologiesNatural convectionSingle-phaseTo/From Facility waterHeat ExchangerHeat ExchangerTwo-p

4、haseTo/From Facility waterForced convectionSingle-phasePHEXTo/From Facility waterDielectric fluidWatern Servers(n-m)Serversn Serversx(cm)y(cm)x(cm)y(cm)z(cm)z yx(cm)Environmentally friendly nonvolatile fluidsSealed cover not requiredHigh efficiency24U rack(n-m servers)Inflammable(T200C)Volatile flui

5、ds with high environmental impactSealed sophisticated cover is requiredHigh efficiencyTypical 48U rack(n servers)Not inflammableEnvironmentally friendly nonvolatile fluidsSealed cover not requiredLocal pumping systemTypical 48U rack(n servers)Inflammable(T159 C)CondenserOVHclouds Immersion liquid co

6、oling technologyOVHcloud AC/WC systemSingle-phase IC/NCimmersion/liquidtechniqueTo/From Facility waterHeat ExchangerHeat Exchanger(n-m)Serversx(cm)y(cm)European patent EP 4068926A1,April 2023.M.Hnayno,A.Chehade,H.Klaba;OVH.Hybrid immersion cooling system for rack-mounted electronic assemblies.OVHclo

7、uds Immersion liquid cooling technologyHybrid immersion liquid cooling rack CPU and GPU IT equipment rather than CPU and GPU:RAM,MB,hard disks70%of total heat load30%of total heat loadWater cooling using water blocksServerPipingWater block on CPUImmersion coolingOVHclouds Immersion liquid cooling te

8、chnologySealed tank not requiredHigh efficiencyEnvironmentally friendlyAdiabatic coolingeliminationBetter energy recovery PUE and WUEData centerPumping systemDry coolerAdiabatic coolingData centerDry coolerPumping systemHot(up to 65C)Cold(up to 32C)OVHcloud liquid cooling OVHcloud Hybrid CoolingCold

9、(up to 45C)Hot(up to 52C)Tamb 42CSilent DCConceptCold waterWarm waterHot waterPSSDry coolerServers WBs30C45C50CServer chassisCopper serpentine (D=30mm;L=5m)CPUsRAMsNVME disksIC Tank(book)Power supplyOVHclouds hybrid library(48U)Water inlet pipelineWater outlet pipeline1 U serverNetwork switchPower D

10、istribution Unit(PDU)Electrical cabinetBetter footprint ever for immersion cooled racksPoCInputNumerical parametersExperimental parametersScale serverDielectric FluidShell Immersion Cooling Fluid S5XShell Immersion Cooling Fluid S5XMain IT components2 CPUs.16 RAMs.3 NVME disks2 CPUs.16 RAMs.3 NVME d

11、isksWater inlet temperature303K(30C)303K(30C)Water inlet flow rateq=0.55 l/minq=0.55 l/minConfiguration for WC circuit Co-current flow(through the coil first)Co-current flow(through the coil first)IC dielectric fluid behavior-TemperatureExperimentalNumericalLLLUUUMLMULowerMiddleUpperIT behavior13CPU

12、RAMCPU0=73C CPU1=71C RAM=66CExperimental workServers WBsMini WCIC TankTPPTPTTTTTTTDry coolerTTTTTTTTTTCMTemperature sensor Pressure sensorDifferential pressure sensorFlow rate sensorSolenoid valve TPPTServers WBsMini WCIC TankTPPTPTTTTTTTDry coolerTTTTTTTTTTCMTemperature sensor Pressure sensorDiffer

13、ential pressure sensorFlow rate sensorSolenoid valve TPPTNational Instruments data-acquisition systemPower Distribution Unit(PDU)Solenoid valve to control water inlet temperatureIC insulated shelter2U serverSmart differential pressure sensorFlow rate sensorsTC-30TC-35TC-40TC-45Temperature(C)0.350.40

14、0.450.500.55 or+Flow rate(l/min)Experimental workS5XSmart CoolantThermasafe RDensity(kg/m3)806799833Thermal conductivity(W/(m.K)0.140.140.13Specific heat(kJ/(kg.K)2.272.262.20Kinematic viscosity(cSt)9.85.14.6Flash point(C)200159180GWP/ODP000Different fluidsWC circuit selectionServers WBIC TankColdHo

15、tServers WBIC TankColdHotServers WBIC TankColdHotServers WBIC TankColdHotConfiguration 1Configuration 2Configuration 3Configuration 4WB SerpentineCo-current flowWB SerpentineCounter current flowSerpentine WBCounter current flow Serpentine WBCo-current flowExperimental workServers WBIC TankColdHotDif

16、ferent configurations Different servers profilesDifferent fluids1000 W620 W550 W810 WS5XSmart CoolantThermasafe RDifferent fluids505560657075808590951000,350,40,450,50,550,350,40,450,50,550,350,40,450,50,550,350,40,450,50,55IT Equipment avg temperature(C)Flow rate(l/min)S5 X:T CPU(Avg)S5 X:T RAM Avg

17、)SmartCoolant:T CPU(Avg)SmartCoolant:T RAM Avg)ThermaSafe R:T CPU(Avg)ThermaSafe R:T RAM Avg)TC-30 CTC-35 CTC-40 CTC-45 CCPUs temperature limitRAMs temperature limitCooling demandKinematic viscosity(cSt=mPa.s)Water Block(%)Serpentine(%)S5 X9.87525SmartCoolant5.18020ThermaSafe R4.68119404550556065707

18、580Fluid temperature(C)Flow rate(l/min)S5 X _ Lower partS5 X _Upper partSmartCoolant _ Lower partSmartCoolant _ Upper partThermaSafe R _ Lower partThermaSafe R _ Upper partTC-30 CTC-35 CTC-40 CTC-45 CDifferent configurations 405060708090100CPUs Temperature(C)Flow rate(l/min)Configuration 1:T CPU 0Co

19、nfiguration 1:T CPU 1Configuration 2:T CPU 0Configuration 2:T CPU 1Configuration 3:T CPU 0Configuration 3:T CPU 1Configuration 4:T CPU 0Configuration 4:T CPU 1TC-30 CTC-35 CTC-40 CTC-45 CCPUs temperature limit405060708090100RAMs Temperature(C)Flow rate(l/min)Configuration 1Configuration 2Configurati

20、on 3Configuration 4TC-30 CTC-35 CTC-40 CTC-45 CRAMs temperature limitConfiguration 1Configuration 2Configuration 3Configuration 4WB SerpentineCo-current flowWB SerpentineCounter current flowSerpentine WBCounter current flow Serpentine WBCo-current flowServers performance620 WServerFlow rateT-in T-ou

21、tTQRatio Q/Pl/minCCKW%Optimal 30C0.3530.7 54.623.9 585.687%0.4030.7 53.222.5 631.3 94%0.4530.6 51.120.0 630.792%0.5030.4 48.718.0 628.490%0.5530.7 47.116.3 627.5 89%GPU(s)RAMsDisksMB and othersTotalCPU(s)Power(W):30C;0.4l/min30C;0.85l/min99%97%30C;0.55l/min1000W620W810W30C;0.4l/min94%97%550WPPUE and

22、 WUECase study:Roubaix-France 600kW DC1,00661,00531,00420,090,040,000,000,010,020,030,040,050,060,070,080,090,101,00301,00351,00401,00451,00501,00551,00601,00651,0070IFC+ECIFC+5K+ECICWUE(L/kWh)PPUEAverage PPUEWUE0200400600800100012001400160018002000-15-11-7-315913 17 21 25 29 33 37 41Electrical cons

23、umption(kWh)Temperature(C)20.7%36.2%0200400600800100012001400160018002000-15-11-7-315913 17 21 25 29 33 37 41Ej:Electrical consumption(kWh)Temperature(C)ICIFC+ECIFC+5K+EC20.7%36.2%0200400600800100012001400160018002000-15-11-7-315913 17 21 25 29 33 37 41Ej:Electrical consumption(kWh)Temperature(C)ICI

24、FC+ECIFC+5K+EC20.7%36.2%0200400600800100012001400160018002000-15-11-7-315913 17 21 25 29 33 37 41Ej:Electrical consumption(kWh)Temperature(C)ICIFC+ECIFC+5K+EC20.7%36.2%IFC:Indirect Free Cooling+Evaporative Cooling(DC water inlet temperature=27 C)IFC:Indirect Free Cooling+5K Evaporative Cooling(DC wa

25、ter inlet temperature=32 C)IC:Immersion CoolingConclusionCooling capacityUp to 48 kW per rackDC inlet temperatureUp to 45 C20 K,T over the DCPPUE 1.004-1.006WUE tends to zeroHigh footprint density(U/m2)Low OPEXPumping and ventilation systeminside rack not requiredEnergy recoveryFeasible at high temp

26、eraturesLow CAPEXLow,requiring no sealing or sophisticated heat exchangers/pumping circuits Contribution to the OCP communityProviding a comprehensive presentation for the OCP immersion projectContributing to the development of a future OCP white paper Internal Product/Facility Availability beginning of 2025 Where to find additional information(URL links)Scientific Paper:https:/doi.org/10.1016/j.csite.2023.102925Blog OVHcloud:https:/ list:NextThank you!