The Big Data Chill

Overheating equipment is an archenemy in the big data industry. Producing, storing and analyzing huge amounts of information requires ever faster and denser processing power. The requisite compute cycles — which enable the Internet of Things (IoT) — in turn produce massive quantities of heat that must be removed. In 2013, U.S. data centers alone consumed 86 billion kWh just cooling their servers [1] — the equivalent of powering 7.6 million homes. Worldwide, data center servers demand roughly 1.5 percent of the electricity generated annually.

To help reduce the industry’s massive power demands, many data centers are increasingly using liquid cooling to complement or even replace air cooling systems. The Danish company Asetek is the leading provider of closed-loop liquid cooling systems for data center computing. In response to market demands for both highly dense and highly customized server racks, Asetek is continually innovating to develop cooling systems with constituent building blocks that are configurable for a wide range of data center needs. As part of the company’s overall design process, thermal simulation with ANSYS Icepak computational fluid dynamics software is an important tool for optimizing cooling system components.

LIQUID LOOP COOLING SYSTEM

In an Asetek liquid loop system, the cooling liquid is primarily water. The system directly targets the areas of highest heat flux within a server. These are primarily the CPUs, GPUs and memory modules that can have operating temperature ranges of 70 C to 95 C (158 F to 203 F). The main parts of an individual rack cooling unit are a metal cold plate, a pump, dripless quick connectors, and liquid entry and exit tubes. The cold plate, installed above a targeted area, transfers heat from the chip to the cooling liquid, which is pumped out of the server rack and into the coolant distribution unit for the rack tower. The hot coolant passes through a liquid–liquid heat exchanger where it is cooled by facilities water before re-entering the server rack.

The process of directly cooling hot spots in this manner requires maximizing the heat transfer of the system to cool a particular processor or other module. For Asetek’s engineers, this means optimizing the surface area of the cold plate that can transfer heat to the liquid. The cold plate contains a series of microchannels through which cooling liquid is pumped. The Asetek team uses Icepak to aid in design and configuration of these microchannels to analyze the cooling needs of different chips.

Icepak model of a CPU cold plate with the solid (brown) and fluid (blue) zones. Cooling liquid enters through the tube in the center, passes through the microchannels in the copper cold plate, and exits through the two outlet tubes on the ends.

Contours of temperature on both the copper cold plate and the liquid stream lines showing the fluid heating up over its path through the cooling unit

For Asetek to begin a cold plate design, the chip manufacturer first provides data about the geometry and heat dissipation of the particular card or module that needs to be cooled. Depending on the complexity of the components, Asetek engineers will then either import 3-D CAD geometry or build the card or module geometry directly inside Icepak. The team also builds the cold plate geometry before performing the overall thermal analysis using Icepak. A usual mesh size for the fluid zone ranges from one to three million cells. A simulation of this size range converges in 10 to 30 minutes running on four processors.

The systems are designed to handle the case in which a continuously overclocked chip transfers the maximum possible heat to the cooling liquid, at a maximum liquid supply temperature specified for the servers. Additionally, engineers must create a more compact design to increase the density of processors inside the server racks. Further, the overall thermal impedance of the system must maintain the coolant temperature below a maximum value of 60 C (140 F). Some chip designs will thus require higher coolant flow rates to dissipate more heat.

Typically, the Asetek team will run 30 to 40 simulations for a new cold plate design as part of a parametric study. They will vary such parameters as the width, height and spacing of the microchannel fins, the plate thickness, and the flow rate and inlet temperature of the cooling liquid. Icepak has been shown to be a good tool to do these kinds of parametric studies rather quickly. The entire process, starting from the chip geometry to the final cold plate design, is usually completed within two weeks.

"Thermal analysis with Icepak enables Asetek to continue delivering market-leading cooling solutions to IoT companies."

ANSYS Icepak

ENABLING MATERIAL AND ENERGY SAVINGS

Graphics card, including board components

Graphic card including components and thermal gap pad (green)

Graphics card with liquid-cooled cold plate (dark grey) mounted on top

Pathlines of cooling liquid for a graphics card colored by temperature. Liquid enters at 48 C (dark blue), passes through the microchannels and exits at 60 C (light blue), at which point it can be used to provide heat for a building or hot water.

Temperature contours on the graphics card cold plate. Away from the GPU, the plate does not have microchannels, so the temperature of the plate varies depending on the location of other power-dissipating components, such as GDDR memory modules, field-effect transistors or inductors.

Top view of another configuration with the graphics card on the bottom (grey), copper cold plate (brown), thermal gap pad strips on top (light blue), and the fluid zone (dark blue) with microchannels

End view showing the microchannel pathways (dark blue) inside the part of the cold plate (orange) that is above the GPU

Temperature contours on the cold plate and microchannels

Temperature contours on the graphics card and thermal gap pad strips.

Once a liquid cooling system is implemented in a data center, it can reduce the need for expensive blowers and air-conditioning systems, and cut the overall power needed for cooling by up to 50 percent. Such substantial savings from power consumption enable companies to recover the capital cost of a liquid cooling system within a year. Using the system’s heat exchanger, cooling liquid exiting server racks at 60 C (140 F) can be recycled to provide heat for water and residential and commercial buildings. Reusing heat that would otherwise be vented to the atmosphere can prove beneficial, especially in cold climates. At Tromso University, which is located above the Arctic Circle in Norway, the facilities that house 12,000 students and staff are kept warm year-round by the heat that Asetek liquid loop systems capture from the university’s data center.

At data centers like Tromso and around the world, Asetek’s products are at the forefront of enabling large-scale energy savings. Thermal analysis with Icepak enables Asetek to continue delivering market-leading cooling capabilities to the companies designing IoT solutions that keep billions of people and machines connected to the internet. Going forward, the design team is investigating the use of simulation to optimize the pumps that drive the cooling liquid, and also the air flow through complete server racks to ensure sufficient cooling of all components. All of this is leading the future of big data looking very “chill” indeed.