Design, Development, and Characterization of a Flow Control Device for Dynamic Cooling of Liquid-Cooled Servers

Efficient thermal management of data centers is necessary for the continued safe and reliable operation of the Information Technology Equipment (ITE). Traditionally, air cooling has been widely utilized to dissipate heat loads from the servers using heat sinks and fans. However, low thermal mass and relatively high-power usage effectiveness (PUE) limit its implementation for cooling the latest high-performance central processing units (CPUs) and graphics processing units (GPUs) [1]. Several reports suggest that traditional air-cooled data centers expend approximately 30%–40% of energy for cooling purposes [2]. Recent trends have predicted that while the global data center energy trends are optimistic, the average PUE value still hovers around 1.58, which is mostly attributed to inefficient cooling methods employed [3].

Several methods have been explored to improve inefficiencies in data center cooling [4–6]. Among all these methods, direct liquid cooling (DLC) provides a highly efficient solution to meet the rising cooling demands. This is mainly due to the higher thermal conductivity and thermal mass of water or water-based coolants. A typical liquid-cooled server contains a manifold and cold plate arrangement instead of heat sinks and fans. The manifolds distribute the coolant to each of the servers, which then gets distributed to single or multiple cold plates, mounted on the processors. The earliest of the works in DLC dates to the 1960s but with growth in complementary metal oxide semiconductor (CMOS) technology in the 1990s, the heat loads in the electronic packages were significantly reduced and thus making air cooling yet again a viable technology [7–10]. However, with further developments in the CMOS technology, air cooling is once again reaching its limitations and the focus is again being shifted to liquid cooling [11–13]. Liquid cooling can be deployed in data centers in two main ways. The first one can be a hybrid air and liquid cooling arrangement where the auxiliary components are cooled by air and the primary high-power heat-dissipating components using liquid cooling [14]. An advantage of this method is that it can be directly used in an air-cooled data center setup with minor changes around the compute systems. If the entire server is liquid-cooled, thermal pathways like cold rails for memory cooling are usually designed within the enclosure to ensure adequate cooling for all the components [15]. DLC can alleviate some of the issues related to fan acoustics and airborne contamination in air cooling [16–19]. Also, single-phase DLC cold plates are easier to deploy than two-phase immersion-based and cold plate-based cooling technologies due to well-defined reliability and cooling guidelines without any concerns of using high global warming potential coolants [20–25].

Typically, a constant flow rate is provisioned to the ITE in liquid-cooled data centers irrespective of the operating workloads of the servers. Significant savings in the pumping power can be attained if the flow rates are varied in proportion to the instantaneous server workload. Dynamic cooling has been explored in past for air-cooled data centers where control strategies for automatic computer room air conditioning (CRAC) units are experimentally validated. An improvement of 70% in data center energy performance was achieved by varying the parameters like CRAC fan speeds and electronically actuated vent tiles [26]. Other approaches include real-time monitoring of aisle temperatures and using machine learning algorithms to predict cooling requirements based on server workloads [27–30]. On the package architecture side, thermally aware workload routes, workload migration, and varying server fan speed reduce power consumption [31].

This paper describes the design, development, and validation of a flow control device (FCD) that can be integrated directly with the ITE for dynamic variation of the coolant flow rate at the server level based on the server workload. A control strategy for such an FCD was recently published by Kasukurthy et al. [32,33] using computational fluid dynamics (CFD) where savings in pumping power of 64% were predicted. The present investigation details the concept design, development, and experimental testing of the said FCD. The proposed FCD design contains a V-cut ball valve connected to a microservo motor. The valve position is varied to change the flow rate through the valve by servomotor actuation based on predecided rotational angles. To validate the working of the FCD, variations in flow rates and the pressure drop across the device were analyzed by varying the valve position. The paper is divided into the following main sections: design and development of the FCD, CFD analysis of the final design, finite element analysis (FEA) analysis for valve, and discussion of the results.

The primary motivation behind this work was to develop an FCD that can contribute toward pumping power savings with a simple, cost-effective, and reliable design. Air-cooled data centers use actively controlled dampers to supply the target value of airflow rates to servers. Target delivery of flow rate not only optimizes the coolant transport energy cost but also reduces over-provisioning of coolant, thereby reducing energy expenditure related to heat removal. The purpose of this paper is to propose the design of a flow control device to control the flow of liquid in a cooling loop that can be directly integrated with liquid-cooled servers to save the energy required to pump the coolant. Another motivation behind the design of this FCD was the ease of manufacturing at a large scale and the ability to be manufacture using three-dimensional (3D) printing.

Various valve designs and flow control strategies were studied during the concept design phase. Among the various designs studied, a butterfly valve was opted as the best option for the current flow control application, as shown in Fig. 1. After the valve design was finalized, the entire FCD concept design was first validated using CFD and it was found that variation in the flow through the device was not sufficient as shown in Fig. 2. This variation led us to design a better concept design to obtain the required variation of the flow rate that would suit data center liquid cooling applications. Subsequent design variations on valve design concluded that utilizing ball valves with V-cut would provide sufficient flow control at both high and low flow rates. Traditionally, ball valves have been used for flow on/off purposes and cannot produce precise control overflow. The more recent introduction of the V port in the ball valves has significantly improved the linear response of the flow in ball valves and allow a wider range in terms of flow rate control [34,35]. After analysis and experimentation on the butterfly valve design, it was concluded that the flow rate variation was very small; hence, a new design was chosen which had a ball valve with a V-cut. The final FCD comprises four parts: two symmetrical casing parts, one ball valve with a V cut, and a shaft sealing for a leakproof seal as shown in Fig. 3.

Based on the variations tested during the concept design phase and the different valves tested using CFD, the ball valve with a V-cut was chosen as the final design. The inlet and outlet ports of the device were determined by considering standard pipe diameters, 3/8 inches, and a 10 mm ball valve. The detailed design of FCD was made using SolidWorks and the data file was exported to ansysfluent, a commercial CFD package, for further analysis including the determination of the flow characteristics of the FCD designs. After obtaining the required results, the SolidWorks design file was exported to a slicing software and was manufactured using extrusion 3D printing technology. The advantage of 3D printing is that it has a low manufacturing time compared to conventional manufacturing techniques and the production cost is very low. Also, design changes can be made without considering the tooling costs and extra lead time, and related high costs.

Over the years, CFD has become an important tool to ascertain the design flaws and optimize them further rather than going through intensive experimentation of manufactured models. To visualize the 3D flow inside the FCD and analyze the flow rate variation with changing the angle of the ball valve, momentum, energy, and mass conservation equations were solved using ansysfluent [36]. After preparing the CAD model on SolidWorks, the model was imported to the CFD software where it was first meshed using the ansys Mesh tool and then solved in the CFD solver. The following sections will detail the model setup and solution process in detail.

After importing the geometry to the CFD package, the meshing of only the flow domain was carried out. A good quality mesh is essential for the accuracy of results in CFD modeling. An efficient mesh is composed of a sufficient number of elements near the walls, especially for fluid flows, to resolve the shear effects in the near-wall region. Since the flow variation occurs only in the internal cavity of the device, it was not necessary to include all the components in the meshing and CFD solution. For this purpose, the fluid volume was extracted, and the solid bodies of the design were suppressed. This was repeated for various cases to be simulated in this study, generating four different tetrahedral meshed volumes for four angles studies for flow rate variation. To verify the quality of the mesh, the element skewness and orthogonality were monitored. For a mesh to be of good quality, the value of orthogonality should be closer to 1, and mesh skewness should be close to 0. The minimum element size was set to 0.3 mm and the maximum element size was 0.6 mm. The growth rate was set to the default of 1.2. for inflation, with the number of inflation layers as 10, and the growth rate was set to a default value of 1.2. In all the simulation cases studied, average orthogonality of 0.8 and average skewness of 0.2 were reported. The optimum mesh element count of 739,416 was used based on the mesh independence study as detailed in Table 1. The final mesh characteristics used in the study are summarized in Table 2. A summary of the boundary conditions used for setting up the CFD and FEA simulations domain is shown in Fig. 4.

As the purpose of the CFD simulation was hydraulic verification of the FCD, the CFD simulation was done by deactivating the energy equation. This further deactivates all the thermal inputs possible in the simulation and saves considerable simulation time as well. The pressure drop and flow rate characterizations were carried out at a constant temperature of 22 °C due to which a constant value of the thermo-physical properties was used in the CFD solver. A pressure-based solver algorithm was used to simulate a velocity field through the FCD by a correction in the pressure equation in the continuity equation. Various turbulence models have been implemented in the literature based on specific applications [36]. Literature shows that, for cases with internal flows, $k−ω$ turbulence model is preferable. Standard and shear stress transport (SST) are the two available variants of the $k−ω$ model. It was believed that since flow across the valve in various positions will create large adverse pressure gradients, the SST $k−ω$ model should be used because of its superiority in predicting such flows [37]. This has also been validated in literature where a comparison of various turbulence models was validated for a butterfly valve [38]. This study showed that the SST $k−ω$ model predicted the values of the drag coefficient across the valve with the least error when compared to the standard experimental results. Also, the CFD results become less sensitive to grid size when using this turbulence model. The general governing equations of continuity and momentum used by the solver to are given below in Eqs. (1) and (2)

In Eqs. (3) and (4), $G̃k$ represents the generation of turbulence kinetic energy due to mean velocity gradients and $Gω$ represents the generation of $ω$⁠. $гk$ and $гω$ represent the effective diffusivity of $k$ and $ω$⁠, respectively. $Yk$ and $Yω$ represent the dissipation of $k$ and $ω$ due to turbulence and $Dω$ represents the cross-diffusion term. $Sk$ and $Sω$ are user-defined source terms which were not utilized in the current CFD modeling setup [36]. After choosing the relevant turbulence models, the fluid boundary conditions were chosen based on the anticipated experimental test conditions. The coolant properties of 25% propylene glycol (PG25) are used in the simulation to determine the variation in flow rates and pressure drop with changes in the angle of the valve. A summary of the thermophysical properties of the temperature-dependent properties of the coolant is shown in Table 3. In the boundary conditions, the inlet velocity was calculated using the flow rate for the designed area. The outlet was set as a static pressure outlet with 0 Pa gauge pressure. The walls of the device were treated as adiabatic walls. A steady-state simulation with convergence criteria of 1e-3 for continuity and 1e-6 for 3D velocities, k, and $ω$ values was used.

Here, [K] represents the stiffness matrix of the entire geometry, {x} represents the unknown nodal displacements, and {F} represents the external forces or applied loads.

Experimentation was carried out in a controlled environment. The cooling liquid used for the loop is PG25 and care was taken to avoid contamination to get precise flow rates. A simplified schematic of the experimental setup is shown in Fig. 5. Pressure sensors were attached near the inlet and outlet of the flow control device to determine the pressure drops for various valve angles. The flow rates were obtained using ultrasonic flow sensors, which use ultrasonic frequencies to calculate the flow rates. The loop also consists of a sealed coolant reservoir to hold the cooling liquid. A description of the models and sensor accuracy is shown in Table 5. A 15 W DC pump was used to circulate the fluid through the entire test loop. Variation in flow rates across the FCD was achieved using changing the input voltage to the pump. The pressure and flow sensors were connected to the data acquisition system (DAQ). The DAQ is used for sampling electric signals such as voltage, current, and frequency and convert these electrical signals into numerical values such as temperature and pressure using the software. The flow and pressure sensors were supplied a voltage of 20 V from a different DC power source. The angle of the valve was changed with the help of a servomotor, which is controlled by the Arduino microcontroller. The Arduino was programmed using Arduino IDE in such a way that a change in the resistance value of the potentiometer will change the angles of the valve. Observations were recorded at valve angles of 0, 30, 45, and 90 deg. Data regarding pressure drop from pressure sensors and variation in flow rate with a change in valve angle were also recorded. The final fabricated FCD and its integration in a cooling loop are shown in Fig. 6.

After achieving convergence with an appropriate mesh as discussed earlier with the desired accuracy, the CFD results were analyzed in the form of the pressure drops across the device for different valve angles. This variation in the valve angle was repeated for the three chosen values of flow rates of 0.6, 1.0, and 1.5 lpm. The working of the FCD design was verified from the CFD results where a change in flow rate was obtained at all angles varied in the simulations. Figures 7–9 depict the variation of velocity at valve angles of 0, 75, and 45 deg, respectively. The results obtained from the simulations were compared with experimental data and showed a close agreement in the flow rate variation trend with changing valve angle. As opposed to the first design of the FCD, it was observed that with the closing of the valve, the flow rate decreased across the valve, and the pressure drop increased. The observations obtained for a particular condition had similar values with average error possibilities of 14%. The data in Table 6 summarize flow characteristics when the flow rate is 1 lpm when the valve is fully open at a 0 deg angle.

The stress analysis was done using the Static Structural module in ansysWorkbench. The CFD data of the forces exerted by the fluid flow on the valve were exported to the structural module of the software. This was done to verify the structural integrity (based on material fill ratio in 3D printing) of the device at high lpm that can lead to leakages or the device or the device bursting under pressure. The ambient conditions for the simulation setup were kept at a constant temperature of 22 °C. A fixed support boundary condition was given on device outer walls and the internal pressure force was imported from the CFD simulation for the highest flow rate, 4 lpm, to identify the maximum stresses and deformations. The results of the FEA simulations are shown in Figs. 10–12. The value for maximum deformation, observed near the inlet port, was 7.9 × 10⁻⁴mm with maximum equivalent stress of 0.94 MPa, and the average deformation was 1.13 × 10⁻⁴ with average stress of 0.076 MPa. The average factor of safety for the device was obtained to be 15. The results from FEA analysis verify that the FCD body with current dimensions and material will not fail at pressure conditions at the maximum flow rate of 4 lpm. This, however, needs validation experimentally at different temperatures for prolonged periods and cyclic flow rates (internal pressure) to ascertain the device lifetime and failure rates.

To ascertain the results obtained for the hydraulic performance of the proposed FCD, an experimental study was carried out under the same flow conditions. The pressure drop characteristics and flow rate variation at different valve angles are shown in Figs. 13–15 and summarized in Table 7. The results showed that for a maximum flow rate of 4.0 lpm, a minimum flow rate of 0.2 lpm can be achieved when the FCD is in the fully closed position. It was observed that the maximum change in flow rate occurs between the range of 30–90 deg. The largest variation is obtained for the higher value of flow rates but reduces and becomes rather constant for flow rates less than 0.5 lpm.

A comparison of results from the experimental study and CFD analysis is shown in Figs. 16–18. The comparison of results of flow rates with varying angles of the ball valve shows that a maximum discrepancy of 20% was observed in both the results at a flow rate of 1 lpm. This variation reduces with increasing the angle of the valve or when the valve is fully opened. A large variation in the results at fully closed conditions could be due to unresolved flow disturbances due to the current mesh size. A similar trend is observed in the pressure drop characteristics with varying angles of the valve. This is expected as there is a linearly inverse relationship between pressure drop and flow rate. The maximum error between the pressure drop variation was 16% for at 1 lpm flow rate when the valve is fully closed. The error between the results when the valve was fully opened was within 3% for both pressure drop and flow rate. This is because there could be little to no flow turbulence created by the valve and the CFD code was successfully able to replicate the experimental flow conditions. Based on the plotted graphical data, it was observed that with a change in angle, there is a drop in the flow rate and the pressure drop across the valve increases. The major variation in pressure drops and flow occurs between the angle of 30–90 deg angle. Similar trends were seen in experimental and CFD data when calculated for a starting flow rate of 0.6 lpm, 1 lpm, and 1.5 lpm when the valve is fully open.

Direct liquid cooling using cold plates continues to be the most popular method to overcome the challenges of thermal challenges in high-performance computing systems. While DLC by itself provides an energy-efficient way of dissipating high heat fluxes, further enhancements are possible such as reducing energy consumption related to coolant transport within the data center. Traditional DLC only provides a constant flow rate, typically using a centralized pumping system that delivers a constant flow rate to each server irrespective of the instantaneous workloads of the servers. DLC also provides notable advantages and better equipment reliability of ITE as issues related to contamination and fan acoustics [29–33]. It also provides higher heat flux values than single-phase immersion cooling and overcomes the issues related to material compatibility and vapor pressure in two-phase open bath immersion cooling and two-phase cold plates [34–38].

The FCD design proposed in this study will enable workload-based provisioning of the coolant to each of the servers in a rack. A recent study by the authors on an open compute server retrofitted with cold plates with distributed pumps showed considerable power savings when the flow control at the server level is dynamically varied [39]. The proposed FCD will therefore yield considerable pumping power savings by optimizing the pumping power of the rack-level pumps. Out of the two final designs developed, the FCD design with a V-cut ball valve produced the desired flow rate variation that would be sufficient to cool heat loads in the current high-powered electronic packages. The results from the CFD study and experimental testing show a good agreement in depicting the hydraulic performance data of the FCD. A minimum flow rate of 0.09 lpm and a maximum flow rate of 4 lpm were achieved by varying the angle of the valve inside the FCD cavity. Future work on this subject will investigate FCD implementation at the server and rack level for flow control at the rack level to assess the thermal and flow characteristics along with the maximum achievable power savings. The future work on the FCD will also present an overview of the possible cost savings not only due to reduced pumping power but also provide an estimate of a simple payback period after device deployment. Further studies will also focus on device reliability under operating conditions to assess the device lifetime in the field under transient operating conditions.

• National Science Foundation IUCRC (No. IIP-1738811; Funder ID: 10.13039/100000001).

ASHRAE =

American Society of Heating, Refrigeration and Air-conditioning Engineers

CFD =

computational fluid dynamics

CMOS =

complementary metal oxide semiconductor

CRAC =

computer room air-conditioning

DAQ =

data acquisition unit

DLC =

direct liquid cooling

FCD =

flow control device

FEA =

finite element analysis

ITE =

Information Technology Equipment

PUE =

power usage effectiveness