Design and Operation of Energy-Efficient Data Centers
Rasmus Päivärinta
Helsinki University of Technology rasmus.paivarinta@tkk.fi
Abstract
Data centers are facilities containing many server computers.
Their financial and social importance is greater than ever due to growing amount of data and hosted applications used by millions of users. Consecutively, maintaining an infrastruc- ture with such massive computing resources is energy inten- sive. The total amount of energy used by servers around the world is greater than that of Finland. There is pressure to save energy in data centers because of the current needs to cut costs and reduce carbon emissions.
In this paper, propsals for increased energy efficiency in computing facilities are made. The cooling system is analysed and concrete recommendations to save energy are presented. A typical power distribution scheme is explained and its effect to efficiency is studied. In addition, this paper shows that underutilisation causes serious losses and two approaches, namely virtualisation and power-aware request distribution, are introduced as solutions.
KEYWORDS: data center, energy efficiency, virtualisation, blade server
1 Introduction
As a result of several worldwide trends, the energy efficiency of data centers has become an important research topic. The amount of digital information has been growing rapidly and most data is stored and processed in data centers. How- ever, the problem is that these buildings filled with server hardware consume significant amounts of electrical power.
Server electricity use doubled between 2000 and 2005 [9].
At the same time, environmental values have become public and the price of electricity has increased [6]. Advances in energy efficiency lead to substantial cost savings and brand image improvements through environmental friendliness. In some cases, even the availability of a data center can im- prove as a result of better planned cooling solutions. This article gives insight into how data centers can be designed and operated in such a way that less electrical power is re- quired.
The rest of this article is organized as follows, beginning with a brief description of scientific methods applied during the research process in the second chapter. The fundamentals of data centers are summarized in third chapter. The fourth chapter introduces the most important sources of electrical power consumption in a data center environment. Concrete proposals for obtaining energy efficiency are analysed in the
fifth chapter. Chapters six and seven propose further research topics and conclude the article, respectively.
2 Methodology
This paper is based on an analysis of relevant scientific ar- ticles. I began my analysis by reading articles [12] and [9] which discuss the energy efficiency of computing at the macro level. Afterwards, I was convinced that the research question is important, and that the energy efficiency of data centers can have a serious impact both financially and en- vironmentally. The main research question in this paper is how to improve energy efficiency by smart design and oper- ation of computing facilities. The question is addressed very concretely in a 2007 paper by Greenberg et al. [8], and it has been used as a framework for finding the most important top- ics when writing this paper. However, [8] does not discuss anything very deeply and therefore I have analysed topics of cooling, power provisioning, blade servers and virtualisa- tion in greater detail by examining [13], [7], [11] and [1]. A number of other articles have been referenced for additional insight and background information on the discussed topics.
The complete listing can be found in the bibliography.
3 Description of a Data Center
Data centers are dedicated rooms or even whole buildings containing very many computers. These computers are servers which provide applications for companies. Applica- tions are often business and mission critical, meaning that downtime causes economical losses. High availability is achieved by means of redundant network connections and power supplies. All of the electrical applicances installed, such as network and server hardware, add to the total heat generation of a data center. Thus, it is clear that a reliable cooling solution is required to keep the servers up and run- ning constantly.
The smallest data centers are the size of not much larger than a single rack while the biggest spread over thousands of square metres. A rack is a standardised metal frame or en- closure which is 19 inches wide and usually 42 units (U) tall.
Unit refers to the height of 1.75 inches and most commodity servers are 1U or 2U high. In a typical scenario, the server racks are placed on a raised floor area which allows the in- stallation of cabling and flow of cool air under the hardware.
According to Beck [3], on average, the raised floor area ac- counts for half of the total area while the computer equip- ment occupies 25-30% of the raised area.
A number of different ownership structures are common for data center operation, for example corporate, managed or co-located facilities are possible. In all cases because of the proprietary nature of the stored data, computing facilities are often well-secured also from physical threats such as bur- glary or fire. They can be located in old office, warehouse or industrial buildings, or a new building can be purpose-built for data center use. Only a minimal number of employees work in a typical data center. [3]
4 Metrics and Estimation
The measures for data center energy efficiency analysis are not very well established. However, the most used unit is W/m2, which is used with several measures. Computer power density refers to the power consumed by computers divided by the computer room area. In contrast, total com- puter room density refers to the power consumed by com- puters and all supporting infrastructure including power dis- tribution units (PDU), uninterruptible power supplies (UPS), heating, ventilating, and air conditioning (HVAC) and lights divided by the computer room area. [4]
Another useful metric regarding the energy efficiency of a data center is the proportion of computer power consump- tion of total power consumption. Greenberg et al˙found out in their study of 22 data centers that the proportion varied between 0.33 and 0.75. They also state that a value around 0.85 is a realistic goal. Obviously, a higher value is better since it reveals that most power is used in computing instead of inefficient cooling. [8]
Operation of data centers is an energy intensive busi- ness. However, capacity requests from utilities regarding the power needs of new data centers are often overestimated.
According to a research report from the Renewable Energy Policy Project[3], overestimation arises from five factors.
First, the nominal power need of appliances is often used as a basis for calculations eventhough research has shown that computer hardware draws only 20%-35% of its nomi- nal power need. By nominal power need, the value printed on the nameplate of an appliance is meant. The second mis- leading assumption is that all servers are fully utilised all the time. For example, in practice it is often not the case that servers have every memory and hard disk slot filled. Third, racks are usually not fully filled, so estimating power need by viewing each rack as full gives misleading results. The same misleading assumption that resources are fully utilised applies to the physical space such as the raised floor area as well. Rarely is it the case that the whole area is fully populated with racks. The last misleading assumptions con- cern the balance of the system. This means that as a result of above mentioned assumptions, other equipment such as PDUs and computer room air conditioning units (CRAC) will be overestimated as well. In addition, engineers often apply a 10%-20% safety margin for required power. As a result of such overestimation, optimal design solutions are difficult to achieve and energy suppliers struggle to satisfy overesti- mated requirements.
I feel that the most important metric in energy-efficient data center design is the ratio of computing power to elec- trical power. After all, it is the computing power that data
centers are built for and the electrical power that is mini- mized. This ratio is not an easy measure to define precisely.
Rivoire et al. introduce JouleSort benchmark, which nicely captures exactly that and it can be used to compare very dif- ferent kinds of computing systems from mainframes to mo- bile devices [16]. In this context also, the ongoing switch to multicore processors is important. It is known that once chip multithreading (CMT) is utilised by software, the ratio of computing power to electrical power will improve substan- tially. As an example the dual-core AMD Opteron 275 has been found to be 1.8 times faster than the single-core AMD Opteron 248 but the former uses only 7% more power. [2]
5 Recommendations
5.1 Cooling
Computer and network hardware generate excess heat that must be removed. Otherwise, the temperature in a data cen- ter would rise rapidly resulting in unreliable operation of the hardware. Several sources show that cooling infrastructure accounts for 20%-60% of the total power consumption. The percentage is so significant it can be concluded that optimiz- ing the cooling solution is the single most important factor in the energy efficiency of a computing facility. [8] [3] [12]
Air conditioning based solutions still dominate the mar- ket although liquid cooling has some superior characteristics over air cooling [8]. An air cooling solution is comprised of a central plant, air handling units (AHUs) and computer room air conditioning units (CRACs). A CRAC transfers heat from the computer room air to a chilled water loop. The central plant is required to transfer the heat out of the facility. AHUs can be used to affect air flows. [4]
Running air conditioning systems at part load is very in- efficient. Overestimation, as described before, often results in installation of excess cooling capacity. One method that can help size cooling infrastrucure correctly is to use compu- tational fluid dynamics (CFD) modeling as Patel et al. sug- gest. [13] They show that numerical modeling can be used to design airflow and model temperature distribution in a com- puter room. CFD modeling can not address the challenge of changing configurations in server racks. Heat loads can vary dramatically due to phyciscal rearrangements. Vary- ing computational load also has an effect on the temperature distribution. Therefore cooling systems that support varying capacity should be prefered in dynamic environments.
Figure 1: Typical cold aisle - hot aisle layout [8]
One central principle of air conditioning design is that the mixing of incoming cool air and hot air rejected from the equipment should be minimized. A typical solution is to in- stall racks in such a way that cold air inlet sides face each
other. The result is a layout where each aisle between rack rows is either a cold aisle or a hot aisle as shown in Figure 1.
Schmidt et al. [17] present a ratioβ which can be com- puted in order to resolve local inefficiencies in the air flow.
β=
4Tinlet
4Track
, (1)
where4Tinlet is the difference between the rack inlet air temperature and chilled air entry temperature, and4Track
is the difference between rack inlet and outlet air tempera- tures. The denominator used is an average rack value but the nominator value in contrast is a local value. Aβ-value of 0 indicates that cold and hot air flows do not mix at that location. A value greater than one means that there is a self- heating loop so that rack outlet air flows back to rack inlet.
In geographical locations where the outside temperature is less that 13◦C for at least four months in a year, cooling effi- ciency can be considerably improved by taking advantage of the outside environment. It is no coincidence that Google de- cided to launch its newest data center in Finland. Efficiency improvements can be achieved by designing the central plant of a cooling system in such a way that water is circulated in mild outdoor conditions. Another way to capitalize on mild environment is to use air-side economisers which take ad- vantage of the cool outside air in cooling the indoor air. [8]
It should not be forgotten that often the easiest way to save energy is to decrease cooling and let the temperature increase a couple of degrees.
5.2 Power Distribution
Figure 2: Power distribution hierarchy [7]
At this point I will introduce how power is actually pro- visioned to the computers. Power provisioning system is an important topic to understand because clearly all losses here should be minimized in an energy-efficient computing
facility. Figure 2 is a simplified diagram of the power pro- visioning system in a data center with a total capacity of 1 MW. The top part represents how the system is connected to a high voltage feed from an energy provider. A trans- former then converts the main feed down to 480 V. This main feed and a generator are connected to an automatic transfer switch (ATS). The ATS switches the input to the generator in case the main power feed fails. Continuing closer to servers, power is supplied via two independent me- dia which are backed up by uninterruptible power supplies (UPS). Power for racks is distributed via power distribu- tion units (PDU). All PDUs are paired with static transfer switches (STS) which make sure that a functional input feed is chosen. A PDU then transforms the voltage down to 220 V in Europe. Finally, power is provisioned for equipment power supplies in racks. [7]
Generally losses occur at each time voltage is transformed or AC/DC conversions are done. Data center operators should look for greatest possible efficiencies in these appli- cances. Practically for every Watt that is saved in excess heat production, another Watt is saved in cooling. According to Calwell et al. [5] the efficiency of a typical server power sup- ply at part load is 66% on average. By investing in advanced power supplies, the efficiency can be improved to more than 80%.
There are often engine-driven generators which back up the UPS system in case the main input fails. Greenberg et al. [8] notice that generators have constant stand by losses caused by engine heaters. The function of an engine heater is to ensure rapid starting of a generator. As a result of contin- uous stand by, the heater typically uses more energy than the generator will produce during the lifetime of a data center.
This energy loss can be minimized by simply lowering the temperature of the heater to 20◦C. Slightly extended starting times of the generators should be taken into account at the ATS so that UPS batteries are used until the generators have properly warmed up.
5.3 Blade Servers
Blade or dense servers have become increasingly popular in data centers during the latest years. They offer higher density installations, better modularity, easier maintenance and cost savings compared to traditional rack servers. A blade sys- tem is a special type of server which is typically comprised of a 6U enclosure and 8, 10, 12 or 16 blade server modules.
The enclosure provides a chassis, shared power supply units, shared cooling and a signal midplane for the blade modules.
A power supply unit (PSU) is either directly connected to a facility power feed or indirectly to a PDU in the rack. The modules can be server computers, storage servers or inter- connect modules. The interconnect module can be an Ether- net or InfiniBand switch and other modules connect to it via the signal midplane. [11]
I did not find clear proof that blade systems themselves are more energy-efficient than other types of servers. How- ever, there are some factors that make power savings in blade servers easier compared to traditional rack servers. Air flow can be optimized at the system level already by the design- ers of the product. Electric power can be saved if efficient
fans, whose rotational speeds can be adjusted according to the cooling requirement of the blade system, are installed.
Moreover, power distribution in a blade system can be done in an energy-efficient way. PSUs convert AC power from the facility power feeds to a low-voltage DC power that is used by the internal components of a server. Each PSU has a load at which its operation is most energy-efficient. In a blade system, because the PSUs are shared between several servers, PSUs can be turned on and off in such a way that the most energy-efficient load is maintained on the units.
5.4 Virtualisation
There is an ongoing trend towards heavy utilisation of virtu- alisation in data centers [15]. Virtualisation provides easier management of computing resources in terms of consolida- tion. Administrators have traditionally run only one criti- cal service per server because of safety reasons and clear administration. One service per server is often too coarse- grained distribution of workloads leading to underutilisation of computing resources. Using virtualisation, administrators can run several operating system instances on the same hard- ware. This makes possible to run many critical services on a same server entirely isolated from each other. As a result, the utilisation rate of computing resources increases.
How virtualisation relates to energy efficiency in data centers, is that a recent study shows that on average 30%
of servers are idle [10]. Virtualised data centers generally utilise servers more efficiently leading to conclusion that same services are maintained with decreased energy budget.
[15]
5.5 Power-Aware Request Distribution
Current server computers consume more than half of their peak power even when they are idle. One way to tackle this inefficiency would be to shutdown idle servers. Ra- jamani et al. propose a novel approach for shutting down servers to optimise energy-efficieny on their article On Eval- uating Request-Distribution Schemes for Saving Energy in Server Clusters. They discuss whether requests in a server environment could be distributed in such a way that the de- sired service level would be guaranteed while at the same time maximizing the amount of servers turned off. Such ap- proach is called power-aware request distribution (PARD). It can be "characterized as minimizing cluster resource utiliza- tion for a particular workload, while meeting given quality- of-service (QoS) constraints".
Parameters affecting a PARD scheme can be divided into system and workload factors. The most influential system factors are the cluster unit and its capacity, startup delay, shutdown delay, and the possibility of migrating connection between servers. Cluster unit is the smallest unit of comput- ing resource that can be turned on and off independently and is typically a single server. Its capacity means the maximum load it can handle with acceptable QoS. The capacity would typically mean the number of users. In addition, the follow- ing workload factors are important to the scheme: the load profile, the rate of change in load relative to the current load, and the workload unit. The workload unit is the minimal ser-
vice request which can be scheduled to a server, for example a single client connection.
PARD is essentially implemented in a load balancer of a web server farm in the example system presented in the ar- ticle by Rajamani et al. The load balancer computes the re- quired number of servers in a given time and turns off others.
Estimations of the number can take advantage of historical request data but the actual algorithms are outside the scope of this paper. However, a common constraint for PARD im- plemenations is
Nt≥Lt,t+D/C, (2)
whereNt is the number of running servers, Lt,t+D is the maximum number of simultaneous requests between current time and startup delay, and C is the capacity of a single server. In the simplest possible PARD algorithm a constant threshold is added to the current situation in order to be pre- pared for upcoming changes in request activity. The power savings using PARD is heavily dependant on the chosen al- gorithm and on the traffic pattern. Obviously, the request load must be inconstant for turning servers off to pay off.
[14]
6 Further Work
Liquid cooling has not been covered in this article. As power densities increase it will soon be the only appropriate cooling solution because of physically superior characteristics com- pared to air. Already now some efficient liquid cooling so- lutions are in the market but their features should be investi- gated in academic research.
It has been proposed that most of the power provisioning system in a data center would use direct current directly. En- ergy efficiency and suitability of such approach should be analysed. Related to power provisioning, Greenberg et al.
state that computing facilities would be suiteful for on-site power generation. [8]
7 Conclusions
In this paper, I have studied the energy efficiency of data centers. This is an important research topic because the total energy consumption of servers around the world is more than 100 TWh in a year which is more than for example the total power consumption of Finland [9]. Clearly, this amount of energy costs considerably money and its production causes carbon emissions. Even percentually small optimizations in energy efficiency pay back quickly.
Data center design is a complex matter because of the dy- namic nature of hosted resources. However, taking energy efficiency into account early in the design is essential for ob- taining satisfying results. The fourth chapter covered how common and easy it is to overestimate energy and cooling needs. Overestimation results in unefficient and poorly bal- anced system. In addition, it has been recommended that CFD-modeling is used in the design phase to optimize cool- ing.
Studies show that in some data centers more energy is used into cooling than into running computer servers. With sim-
ple analysis, careful planning and known best-practices, the efficiency of cooling can be optimized. The fifth chapter cov- ered the classical hot aisle - cold aisle layout which should be used as a basis for air conditioning. Also, free cooling from mild environment should be utilised, and there is a current trend for building data centers in cold climatic zones such as Finland.
Power provisioning system is critical in obtaining energy efficiency. The number of of voltage conversions and AC/DC transforms should minimized because each one of them causes loss in the system. However, the loss can be reduced by investing in high quality products. Also related to the power provisioning system, it has been recommended that unnecessarily high standby temperature of generator heaters can be scaled down in order to save energy.
Idle servers consume terawatt hours of energy unneces- sarily every year. Two approaches to increase server utili- sation rate were analysed, namely virtualisation and power- aware request distribution. Virtualisation is a technique to run several operating system instances on a single server.
This makes it easier for administrators to fully utilise com- puting resources by safely running several applications on the same server. Power-aware request distribution is a pro- posal to predict load on a cluster and power down excess servers.
References
[1] K. Adams and O. Agesen. A comparison of soft- ware and hardware techniques for x86 virtualization.
In ASPLOS-XII: Proceedings of the 12th international conference on Architectural support for programming languages and operating systems, pages 2–13, New York, NY, USA, 2006. ACM.
[2] L. A. Barroso. The price of performance. Queue, 3(7):48–53, 2005.
[3] F. Beck. Energy smart data centers: Applying energy efficient design and technology to the digital informa- tion sector. Technical report, Renewable Energy Policy Project, Washington, DC, USA, 2001.
[4] M. Blazek, H. Chong, W. Loh, and J. G. Koomey.
Data centers revisited: Assessment of the energy im- pact of retrofits and technology trends in a high-density computing facility. Journal of Infrastructure Systems, 10(3):98–104, 2004.
[5] C. Calwell and A. Mansoor. Ac-dc server power sup- plies: making the leap to higher efficiency. Applied Power Electronics Conference and Exposition, 1:155–
158, 2005.
[6] Eurostat. Electricity prices by type of user.
http://epp.eurostat.ec.europa.eu/
portal/page?_pageid=1996,39140985&_
dad=portal&_schema=PORTAL&screen=
detailref&language=en&product=REF_
TB_energy&root=REF_TB_energy/t_nrg/
t_nrg_price/tsier040, Feb 20, 2008.
[7] X. Fan, W.-D. Weber, and L. A. Barroso. Power pro- visioning for a warehouse-sized computer. In ISCA
’07: Proceedings of the 34th annual international sym- posium on Computer architecture, pages 13–23, New York, NY, USA, 2007. ACM.
[8] S. Greenberg, E. Mills, W. Tschudi, P. Rumsey, and B. Myatt. Best practices for data centers: Results from benchmarking 22 data centers. In Proc. of ACEEE Sum- mer Study on Energy Efficiency in Buildings, 2006.
[9] J. G. Koomey. Estimating total power consumption by servers in the u.s. and the world. Technical report, Lawrence Berkeley National Laboratory, 2007.
[10] D. Kusic, J. O. Kephart, J. E. Hanson, N. Kandasamy, and G. Jiang. Power and performance management of virtualized computing environments via lookahead control. In ICAC ’08: Proceedings of the 2008 Inter- national Conference on Autonomic Computing, pages 3–12, Washington, DC, USA, 2008. IEEE Computer Society.
[11] K. Leigh, P. Ranganathan, and J. Subhlok. General- purpose blade infrastructure for configurable system ar- chitectures. Distrib. Parallel Databases, 22(2-3):197–
198, 2007.
[12] J. D. Mitchell-jackson, D. M. K. Date, J. G. K. Date, K. B. Date, and J. Mitchell-jackson. Energy needs in an internet economy: a closer look at data centers. Tech- nical report, University of California at Berkeley, 2001.
[13] R. D. Patel, R. Sharma, C. E. Bash, and A. Beitel- mal. Thermal considerations in cooling large scale high compute density data centers. In 8th Intersociety Con- ference on Thermal and Thermomechanical Phenom- ena in Electronic Systems, pages 767–776, 2002.
[14] C. Rajamani, K. Lefurgy. On evaluating request- distribution schemes for saving energy in server clus- ters. IEEE International Symposium on Performance Analysis of Systems and Software, pages 111–122, 2003.
[15] P. Ranganathan and N. Jouppi. Enterprise it trends and implications for architecture research. In HPCA ’05:
Proceedings of the 11th International Symposium on High-Performance Computer Architecture, pages 253–
256, Washington, DC, USA, 2005. IEEE Computer So- ciety.
[16] S. Rivoire, M. A. Shah, P. Ranganathan, and C. Kozyrakis. Joulesort: a balanced energy-efficiency benchmark. In SIGMOD ’07: Proceedings of the 2007 ACM SIGMOD international conference on Manage- ment of data, pages 365–376, New York, NY, USA, 2007. ACM.
[17] R. R. Schmidt, E. E. Cruz, and M. K. Iyengar. Chal- lenges of data center thermal management. IBM J. Res.
Dev., 49(4/5):709–723, 2005.
