TSUBAME 2.0 Upgraded to TSUBAME 2.5: Aiming Ever Higher

Tsubame, swallow in Japanese, is the symbol of Tokyo Institute of Technology (Tokyo Tech). TSUBAME also stands for Tokyo Tech Supercomputer and Ubiquitously Accessible Mass Storage Environment, which is one of the world's top, large-scale supercomputers. TSUBAME 1.0 began operating in 2006 and was the fastest supercomputer in Japan at that time. In 2010, TSUBAME 2.0 was developed and began operating; its speed measured in petaflops was a first in Japan. (Petaflops is a computer's ability to do one quadrillion floating point operations per second.) TSUBAME 2.0 also achieved world-leading electrical efficiency that earned it a 2010 global rank of No. 2 in the Green500 rankings for supercomputer energy efficiency. TSUBAME was upgraded from 2.0 to 2.5 in the fall of 2013. Specifically, what changed? How is TSUBAME different from K and other supercomputers?

More GPUs dramatically boost the supercomputer's computing power

TSUBAME 2.0, which first began operating in November 2010, was the first petaflop supercomputer in Japan, at 2.4 petaflops theoretical performance. The name TSUBAME is known worldwide because at that time it was ranked fourth in the TOP500 performance rankings based on benchmarks of supercomputer processing speed. The GPU is behind this dramatic performance improvement, both when it was first introduced in 2008 in a supercomputer at the Global Scientific Information and Computing (GSIC) Center at Tokyo Tech and also in the current upgrade.

Basically similar to a PC, a supercomputer is organized into a structure with a chip, an electronic part called a CPU, which is equipped with a CPU core (circuits that process computations). When calculations to process and update data resident on the memory chip are repeated, they produce results of simulations as numeric values and images. In recent years, supercomputers have been developed that are capable of a massive parallel use of these CPUs, which significantly raises their computing power. TSUBAME uses around 17,000 CPU cores. Although each CPU is useful for increasing the speed of word processing among other things, they take up a lot of space and contain many functions that are not often needed by a supercomputer. If these CPUs were laid out side by side, they would fill the area of a gymnasium or in the case of the K supercomputer, exceed 3,000 m2, which is equivalent to the area of a small arena. Furthermore, their power consumption would be at least 10 megawatts, enough to run 10,000 ordinary households. That is too high. So the question was -- what can be done? ....The GPU came to the rescue. It is a multi-core processor with many computation cores built into one package.

The GPU was originally a semiconductor chip used for processing calculations required for 3D graphics displays. Above and beyond its ability to quickly process image data, it is compact with several hundred to several thousand efficient yet small cores, hence the name multi-core processor. When these GPUs are utilized within the central processor of a general-purpose supercomputer, they are notable for their superior energy efficiency and space savings. For these reasons, they were used in the first generation TSUBAME and were then even more widely employed in the 2.0 version described above.

Jump in computing speed performance and energy efficiency

As of June 2013, the Tianhe-2 supercomputer developed in China is said to have the world's fastest computational speed, reaching a theoretical peak performance of around 54 petaflops with double precision computational speed (64 bit length), as noted in the Top500. In Japan, the K supercomputer at the Riken Advanced Institute for Computational Science has a peak theoretical performance of 11.28 petaflops for both single precision (32 bit length) and double precision. Because the TSUBAME 2.5 can compute single precision three times faster than double precision, it can achieve around 17.1 petaflops for its single precision theoretical top performance value, which is faster than K.

Professor Satoshi Matsuoka, GSIC

"Although the TOP500 benchmarks are also important, when practical use is considered, it is not only about computational speed. From now on, it will be important to develop computational algorithms that mix the precision levels, using single precision as the mainstay and using double precision only when necessary. In TSUBAME, we have stressed single precision computational performance so that this capability can be exploited as much as desired." (Professor Satoshi Matsuoka, GSIC)

The designers of TSUBAME stressed not only computational ability, but also energy efficiency when developing the supercomputer. Therefore, in addition to the large-scale deployment of GPUs in the supercomputer as noted above, a state-of-the-art encapsulated liquid cooling system was installed. Electric power efficiency received a large boost from R&D in electric power control technology. TSUBAME 2.0 can compute 30 times faster than TSUBAME 1.0 but with nearly equivalent actual power consumption. The computational ability per unit of electric power consumption is around three times that of an ordinary PC. Additionally, the floor area of the installation was reduced to two thirds the area required by the TSUBAME 1.0. That translates into a space savings the size of two classrooms. TSUBAME 2.5 uses 20 percent less electrical power than the 2.0 despite the fact that its performance is around three times better.

"The most important way to save electric power is to use a lot of multi-core processors. With these processors, the TSUBAME 2.0 uses only one fifth the power it would have used with CPUs, and the TSUBAME 2.5 achieves more than three times the energy savings." (Professor Matsuoka)

Used for numerous and varied purposes, from medicine to earthquake warnings, to social phenomenon analysis

TSUBAME has been well-received since its launch as the "supercomputer for everyone." Even though its main work is research in academia, the door is also wide open for use by industry. The upgraded GPUs in the TSUBAME 2.5 are special because they enable detailed analysis with various simulations achieved through massive computing power.

Cloud-resolving model: 5 km-horizontal resolution on the left,
500 m-horizontal resolution on the right

This meso-scale atmosphere model video of 500 m-horizontal resolution is available on the Aoki Laboratory webpage (Japanese)outer

For example, weather forecasts are essential in our daily life. Although one could say that satellite technology has achieved much, changes in weather conditions cannot be predicted simply by the images that are sent from meteorological satellites. Supercomputers can forecast the weather by dividing atmospheric zones into a 3D grid, then calculating cloud volume. The finer the grid, the more accurate the forecast is. In the past, one grid cube unit was typically 5 km. The research laboratory of Professor Takayuki Aoki, Deputy Director of GSIC and winner of the Special Achievement in Scalability and Time-to-Solution of the ACM Gordon Bell Prize at the SC11 (International Conference for High Performance Computing, Networking, Storage and Analysis), has been able to make finer calculations. Professor Aoki is engaged in joint research with the Japan Meteorological Agency using a 500 m 3D grid for next-generation weather forecasting. He succeeded in doing calculations that enabled tracking cloud movement in more detail than previously possible. In this way, he enabled forecast simulations as detailed as those from satellite photos.

In addition to weather forecasting, there are earthquakes, another natural phenomenon that can have an incredible impact on daily living. TSUBAME is utilized for simulating seismic waves, primarily in the Nankai Trough. In particular, a digital "Earth" resides within the computer, and the regions expected to suffer strong tremors are identified by calculating how seismic waves propagate through the earth. These research findings were very useful for creating a disaster hazard map of Japan.

In the field of health and medicine, the supercomputer is frequently used for the development of pharmaceuticals and for simulating blood flow to organs, among other things. In the pharmaceuticals manufacturing field, TSUBAME is used in joint research with Astellas Pharma Inc. for developing drugs useful in the fight against tropical diseases such as Dengue Fever. For organ simulation, an Italian team performed blood flow simulations for individual patients using captured CT scans of coronary arteries to analyze the potential locations of thrombosis based on heartbeats. In the field of genetics, a team from Osaka University used TSUBAME's analytic capabilities to predict protein structures.

Professor Takayuki Aoki, Deputy Director of GSIC

TSUBAME's supercomputing power is utilized in various fields and industries for analysis of many everyday phenomena such as predicting share prices or traffic congestion, and was used by Toto Ltd. in the development of a low-flow toilet.

"Outsiders often say 'Tokyo Tech raises the bar.' Perhaps they mean that rather than taking a conservative approach, Tokyo Tech is apt to pioneer a new one." (Professor Aoki)

A supercomputer with a presence "much like Mount Fuji's"

TSUBAME's hope for the future is to take the next step and advance to the 3.0 in the latter half of 2015. Beyond that, an exaflop class of supercomputers, with a processing power that can execute a quintillion flops equivalent to a thousand petaflops, is on the horizon. In terms of electric power performance, more energy savings can be obtained with cooling systems, while the use of exhaust heat is also under investigation. In addition, GSIC is working with the world's top research institutes and supercomputer-related corporations to conduct joint research into CPU failures in giant-scale systems, the establishment of new benchmarking methods, and various other research issues involving exaflop supercomputers.

"TSUBAME is like Mount Fuji, standing firm and constantly widening the field of view. By providing opportunities for various people to experience its top-level supercomputing abilities, and with additional levels of development, we hope that new prospects for the future will be opened regardless of whether or not users are inside or outside Tokyo Tech." (Professor Matsuoka)

TSUBAME-KFC Tops Green500 List

TSUBAME-KFC, a prototype system for next-generation supercomputers such as the TSUBAME 3.0, was ranked No. 1 in the world in the November 2013 edition of the Green500 List, demonstrating its leadership in low-power computing. The award was announced on November 20 (US time) at the SC13 - the International Conference for High Performance Computing, Networking, Storage and Analysis - in Denver, Colorado. The Green500 List ranks the top 500 supercomputers in the world by energy efficiency and performance-per-watt. TSUBAME 2.5 was ranked No. 6 on the list.

TSUBAME- KFC, a prototype for advanced cooling and low-power supercomputing, started operating in October 2013. Earlier TSUBAME supercomputers are equipped with water cooling systems, whereas TSUBAME-KFC employs the latest cooling technologies, namely oil-submersion and warm water cooling systems. The compute nodes are submerged in insulated oil, which absorbs any heat. This technology significantly reduces power consumption for cooling purposes.

Published: November 2013