Datacenter Liquid Cooling Methods

Liquid cooling is defined as the case where liquid must be supplied to an entity for operation. It is important to keep in mind that the this definition do not limit the cooling fluid to water. A variety of liquids could be considered for application, including liquids that could be in a vapor phase in part of the cooling loop.

• Air cooling defines the case where only air must be supplied to an entity for operation.
• Air-cooled rack defines the case where only air must be provided to the rack or cabinet for operation.
• Air-cooled datacom equipment defines the case where only air is provided to the datacom equipment for operation.
• Air-cooled electronics defines the cases where air is provided directly to the electronics for cooling with no other form of heat transfer

When liquids are employed within separate cooling loops that do not communicate thermally, the system is considered to be air cooling. The most obvious illustration covers the chilled-water CRACs that are usually deployed at the periphery of many of today’s data centers. At the other end of the scale, the use of heat pipes or pumped loops inside a computer, wherein the liquid remains inside a closed loop within the server, also qualifies as air-cooled electronics, provided the heat is removed from the internal closed loop via airflow through the electronic equipment chassis.

There are many different implementations of liquid cooling to choose from. Below are several scenarios:

One option uses an air-cooled refrigeration system mounted within the datacom equipment to deliver chilled refrigerant to liquid-cooled cold plates mounted to the processors. For this implementation, the heated air from the liquid-to-air heat exchanger (i.e., condenser) is exhausted directly to the data center environment. From a data center perspective, the rack and electronics are considered to be air-cooled since no liquid lines cross the rack envelope.

A different implementation may use a liquid-to-air heat exchanger mounted above, below, or on the side or rear of the rack. In this case, the heat exchanger removes a substantial portion of the rack’s waste heat from the air that is eventually exhausted to the data center. This implementation does not reduce the volumetric airflow rate needed by the electronics, but it does reduce the temperature of the air that is exhausted back into the data center. This example describes a liquid-cooled rack since liquid lines cross the rack envelope. Figure below.

Yet another implementation uses liquid-cooled cold plates that employ water, dielectrics, or other types of coolants that are chilled by a liquid-to-liquid heat exchanger that rejects the waste heat to the facility water. The waste heat rejection to the facility water can occur via one or more additional liquid loops that eventually terminate at an external cooling tower or chiller plant. This implementation of liquid cooling reduces the amount of waste heat rejected to the facility ambient and also reduces the volumetric airflow rate required by the rack’s electronics. From the data center perspective, this implementation describes liquid-cooled racks and electronics since liquid lines cross the rack envelope and also cross over into the servers themselves. This system is shown in Figure below.

LIQUID COOLING SYSTEMS

Datacom equipment cooling system (DECS)

This system does not extend beyond the IT rack. It is a loop within the rack that is intended to perform heat transfer from the heat-producing components (CPU, memory, power supplies, etc.) to a fluidcooled heat exchanger also contained within the IT rack. Some configurations may eliminate this loop and have the fluid from the coolant distribution unit (CDU) flow directly to the load. This loop may function in single-phase or two-phase heat transfer modes facilitated by heat pipes, thermosyphon, pumped fluids, and/or vaporcompression cycles. Fluids typically used in the datacom equipment include water, ethylene glycol or propylene glycol and water mixture, refrigerants, or dielectrics. At a minimum the datacom equipment cooling system would include a heat collection heat exchanger as well as a heat-of-rejection heat exchanger and may be further enhanced with active components such as compressor/pump, control valves, electronic controls, etc.

Technology cooling system (TCS)

This system would not typically extend beyond the boundaries of the IT space. The exception is a configuration in which the CDU is located outside the data center. It serves as a dedicated loop intended to perform heat transfer from the datacom equipment cooling system into the chilled-water system. This loop is highly recommended, as it is needed to address specific fluid quality issues regarding temperature, purity, and pressure as required by the heat exchangers within the datacom equipment cooling systems. Fluids typically used in the technology cooling loop include water, ethylene glycol or propylene glycol and water mixture, refrigerants, or dielectrics. This loop may also function by singlephase or two-phase heat transfer modes and may facilitate transfer by heat pipes, thermosyphon, pumped fluids, and/or vapor-compression cycles. At a minimum the technology cooling system would include a heat collection heat exchanger (likely integral component of the datacom equipment cooling system), a heat rejection heat exchanger, as well as interconnecting piping. This system may be further enhanced with such active components as compressors/pumps, control valves, electronic controls, filters, hydronic accessories, etc.

Chilled-water system (CHWS)

This system is typically at the facility level and may include a dedicated system for the IT space(s). It primarily consists of the system between the data center chiller(s) and the CDU. The chilled-water system would include the chiller plant, pumps, hydronic accessories, and necessary distribution piping at the facility level. The chiller plant would typically employ a vaporcompression cycle to cool the chilled-water supply temperature (43°F–48°F/6°C–9°C) substantially below indoor ambient temperature (typically 75°F/24°C and up to and beyond 95°F/35°C). The chiller system may offer some level of redundancy for critical components such as chillers, cooling towers, and pumps.

DX equipment can also be used in the chilled-water system. DX equipment provides direct heat dissipation to the atmosphere and is therefore the last loop for that design method. Limitations include distance for the split systems and cost of operation. Generally, in most areas systems become economically breakeven at 400 tons of refrigeration. Larger systems favor non-DX designs unless other circumstances warrant more extensive DX deployment. Smaller thermal ride-through devices can be introduced for individual or special cases within this loop design.

condenser-water system (CWS)

This system consists of the liquid loop between the cooling towers and the data center chiller(s). It is also typically at the facility level and may or may not include a dedicated system for the IT space(s). Condenserwater loops typically fall into one of two fundamental categories: wet-bulb-based or dry-bulb-based system. The wet-bulb-based loops function on an evaporative process, taking advantage of lower wet-bulb temperatures, thereby providing cooler condenser-water temperatures. The dry-bulb-based loops function based upon the difference of condenser-water loop temperature versus ambient dry-bulb temperature. To allow heat transfer with the dry-bulb-based system, the condenser-water loop must be at some temperature substantially above the ambient dry-bulb temperature to allow adequate heat transfer from the condenser-water into the outdoor ambient air. These loops would typically include: outdoor heat rejection device (cooling tower or dry fluid cooler), pumps, expansion tanks, hydronic accessories, and distribution piping.

LIQUID-COOLED RACKS AND CABINETS

A rack or cabinet is considered to be liquid-cooled if liquid must be circulated to and from the rack or cabinet for operation. The following figures illustrate cooling at the rack/cabinet level. The first is a basic air-cooled rack. The remaining figures show other options that utilize liquid cooling or a combination of air cooling and liquid cooling. The figures in this section all show the coolant supply and return lines under the raised floor. Other facility implementations may allow such lines to be routed above the floor or from the ceiling. Coolant supply and return connections for the rack/cabinet can be from the base, side, or top.

Figure 2 shows a combination air-cooled and liquid-cooled rack or cabinet that could receive the chilled working fluid directly from some point within the CHWS or CWS loop. One implementation could have the electronics air-cooled, with the coolant removing a large percentage of the waste heat via a rear door heat exchanger. Another implementation could have the coolant delivered to processor spot coolers (some form of cold plate), with the balance of the electronics being air-cooled.

Figure 3 shows a purely liquid-cooled rack or cabinet. One example of such an implementation may have all the electronics in the rack or cabinet conductioncooled via cold plates. This cooling method could deploy water, refrigerant, or other dielectric coolant as the working fluid. Another implementation may have all the electronics cooled via liquid flow-through (e.g., forced flow boiling), jet impingement, spray cooling, or another method that deploys a dielectric coolant to directly cool the electronics. Yet another implementation would include a totally enclosed rack that uses air as the working fluid and an air-to-liquid heat exchanger.

Figure 4 shows a combination air-cooled and liquid-cooled rack or cabinet with an external CDU. The CDU, as the name implies, conditions the technology cooling system (TCS) or datacom equipment cooling system (DECS) coolant in a variety of manners and circulates it through the TCS or DECS loop to the rack, cabinet, or datacom equipment. This implementation is similar to that of Figure 2, with the exception that there is now a CDU between the facility (CHWS or CWS) level supply of chilled fluid and the rack or cabinet. This implementation allows the CDU to condition the coolant delivered to the rack or cabinet to a temperature above the facility’s dew point.

Figure 5 shows a purely liquid-cooled rack or cabinet implementation. This implementation is similar to that of Figure 3, as well as Figure 4, where an external CDU is included.

Figures 6 and 7 are the final implementations to be discussed in this section. These implementations have a lot in common with the implementations of Figures 4 and 5, respectively. One obvious difference is the fact that the racks or cabinets shown in Figures 6 and 7 now possess dedicated CDUs, i.e., internal CDUs. The CDUs are shown at the bottom of the rack, but other configurations could include them on the side or top of the rack. This implementation provides more flexibility to the datacom center operator in that the racks or cabinets can now condition their coolants to vastly different conditions as a function of the workload or the electronics within. Another benefit is that different coolants (e.g., water, refrigerant, dielectric) can now be deployed in the different racks as a function of workload or electronics type.

Liquid Cooling Guidelines for Datacom Equipment Centers - ASHRAE and cooperation with TC 9.9, Mission Critical Facilities, Technology Spaces, and Electronic Equipment.