Designing Supercomputers

By Koichi Yoshimi and Hironori Kawaminami, Fujitsu Limited, Japan, and Norman Chang, Vice President and Senior Product Strategist, Semiconductor Business Unit, ANSYS

One of the greatest challenges in designing fast, data-intensive supercomputers is providing power to and removing heat from hundreds of thousands of computing cores. Fujitsu uses the ANSYS power and thermal toolset to simulate its next-generation 3-D IC semiconductor designs, starting before the systems have been fully defined until final signoff. The result is higher performance and fewer design turns.

Save PDF
supercomp-main

 

Fujitsu SPARC64XIfx package 

Positioned at the forefront of the supercomputing space with 30 years’ experience in the successful development of high-performance systems, Fujitsu cooperates with leading organizations to apply supercomputing to address increasingly complex social, environmental and business challenges. Fujitsu co-developed the K supercomputer with RIKEN, which was first on the TOP500 ranking of supercomputers in 2011.

The K supercomputer again took first place in the 2015 Graph 500 supercomputer ranking, which gauges the ability of supercomputers to handle complex data problems in areas such as cyber-security, medical informatics, data enrichment, social networks, symbolic networks and modeling neuronal circuits in the brain. The same technology used in the K supercomputer is used in Fujitsu’s commercial supercomputer products PRIMEHPC FX10TM and FX100TM.

Fujitsu supercomputers pack an enormous number of compute cores—the FX100 scales to over 100,000 nodes (a node consists of all cores that are connected to a common memory source). But to meet the ever-increasing demand for computing power, Fujitsu needs to pack more processors into a smaller footprint with lower power consumption. Traditionally, Fujitsu uses bleeding-edge semiconductor processes to gain more performance, but the end of semiconductor scaling is on the horizon.

Several promising technologies have the potential to overcome this barrier, including non-Si materials, non-Neumann architecture and 3-D IC structure. Fujitsu is currently studying the use of a 3-D IC structure as a good candidate for achieving the required power, performance and footprint goals. A 3-D IC structure will provide faster cycle time and lower power consumption by improving circuit density and minimizing wire length without requiring further process scaling. But the use of a 3-D IC structure also comes with big challenges, including power integrity, cooling, signal integrity and the most important: cost.

3-D IC DESIGN CHALLENGES

In the past, engineers used system-level thermal analysis to predict a uniform temperature for the entire chip. They were forced to use a high margin of safety to account for the thermal gradients on the real chip, which limited the performance improvements that could be achieved. Lack of knowledge of thermal gradients also made it impossible to accurately determine the resistance (R) and electromigration (EM) limits of individual wires in the chip because R and EM limits are highly dependent on temperature. Thus, engineers could not accurately calculate the IR (voltage) drop and EM in the voltage flowing over each wire.

Typical 3D IC
 
3-D model of typical 3-D IC

IR/EM is a key factor in determining power integrity — the ability to supply power to each complementary metal oxide semiconductor (CMOS) device on the chip. Engineers also had no way to predict the effects of other chip structures, such as through-silicon vias (TSVs) and μBumps, which also have a major impact on power, thermal and signal integrity.

Now, Fujitsu engineers use ANSYS RedHawk to simplify chip design by dividing each layer into rectangular elements comprising a chip thermal model (CTM) that contains information about the temperature dependent power and metal layer density based on the detailed design (if it is available) or a previous design (if it is not). The CTM also contains information about thermal conductivity between layers. The TSV layout is defined in RedHawk-GPS for early power and ground network construction, including TSV placement. This model calculates the power distribution network and temperature profile across each individual chip.

DETERMINING EFFECTS OF NUMBER AND PLACEMENT OF TSVS ON IR/EM

TSV positioning also has a major impact on power and thermal integrity, both of which need to be considered in the early stages of the design process. Fujitsu engineers use RedHawk, which allows them to explore a large design space — possible placements of TSVs, silicon interposer structure, effective power delivery networking, redistribution layer wirings and others — in the early stages of the design process. The packaging details are usually not available at this early stage, but ANSYS Sentinel-TI empowers engineers to generate a package thermal model based on a few parameters, such as the package size, CTMs of each die, position of the μbumps and C4 bumps, etc. Sentinel-TI performs a package-scale power and thermal analysis, and generates power and thermal maps of the entire system. System power and thermal boundary conditions are fed back to RedHawk, and the thermal-aware IR/ EM analysis for each chip is performed again to take the effects of the other chips and package into account.

NSYS RedHawk-GPS thermal model
 
P/G/TSV construction and chip thermal model generation in ANSYS RedHawk-GPS
IR drop
 
IR drop for two different TSV placement cases: (left) TSVs all on right side, (right) TSVs on all four sides. IR drop is substantially lower when TSVs are placed on four sides.

EVALUATING EARLY TSV PLACEMENT

Fujitsu evaluated early TSV placement for a next-generation design for which detailed design information was not readily available. Therefore, the previous generation of the design was used for the evaluation. They assigned power targets to each logic region to account for the higher density of the next-generation design. In one of the studies, two different TSV placements were considered — one solely on the right side of the logic area and the other throughout the logic area. In addition to the analyses described above, an IR drop analysis was performed using total power dissipation to calculate a constant current draw, which was then multiplied by the equivalent resistance of the power distribution network to arrive at the voltage drop. The results showed that IR drop and temperature were much higher on the left side of the chip that had the TSVs only on the right side of the logic area. Then more TSVs were added to the design on four sides, and the corresponding reduction in static IR drop and temperature across the chip were tracked.

The same model was updated throughout the design process and used to investigate effects of changes in chip design, TSV placement, bump placement and packaging design. This allowed engineers to maintain power-, thermal- and signal-integrity not only during the early stages of the design process but all the way through to sign-off. The ability to evaluate thermal-, as well as power- and signal-integrity, made it possible to improve static IR drops by 62 percent and dynamic drops by 15 percent while checking thermal integrity at the same time. This workflow, which ensures that engineers get power-, thermal- and signal-integrity right the first time, reduces expensive design turns.

Temperature simulation of die (top)
emperature simulation of die (bottom)
 
Temperature of top and bottom die in a case in which TSVs are placed on four sides. The bottom die is hotter because the heat sink is on top, but the bottom die is still within the acceptable range.

FOR FUTURE 3-D IC DEVELOPMENT

Going forward, for 3-D ICs it will be even more important to verify the integrity of power, thermal, timing and cost than in the past. ANSYS RedHawk and Sentinel enable Fujitsu engineers to perform many design-space explorations of a 3-D IC from the early design stages. This will significantly shift the design effort to the left, effectively reducing the non-recursive design cost for future 3-D IC development.

click below to start a conversation with ANSYS

Contact Us
Contact Us
Contact