logo

SCIENTIA SINICA Informationis, Volume 50 , Issue 1 : 1-24(2020) https://doi.org/10.1360/N112018-00293

Research progress and development trend of cross-layer energy efficiency optimization in data centers

More info
  • ReceivedNov 1, 2018
  • AcceptedMar 24, 2019
  • PublishedJan 8, 2020

Abstract


Funded by

国家重点研发计划(2017YFB1010001)

国家自然科学基金(61520206005,61761136014)


References

[1] Delforge P. America's Data Centers Consuming and Wasting Growing Amounts of Energy. NRDC Technical Report. 2014. Google Scholar

[2] Siriwardana J, Jayasekara S, Halgamuge S K. Potential of air-side economizers for data center cooling: A case study for key Australian cities. Appl Energy, 2013, 104: 207-219 CrossRef Google Scholar

[3] Tantar A A, Tantar E. A survey on sustainability in ICT: A computing perspective. In: Proceedings of the Companion Publication of the 2014 Annual Conference on Genetic and Evolutionary Computation, Vancouver, 2014. 1213--1220. Google Scholar

[4] Zhang W W, Wen Y G, Wong Y W. Towards Joint Optimization Over ICT and Cooling Systems in Data Centre: A Survey. IEEE Commun Surv Tutorials, 2016, 18: 1596-1616 CrossRef Google Scholar

[5] Deng W, Liu F M, Jin H. Leveraging renewable energy in cloud computing datacenters: state of the art andfuture research. Chin J Comput, 2013, 36: 582-598 CrossRef Google Scholar

[6] Wan J X, Gui X, Zhang R. Joint Cooling and Server Control in Data Centers: A Cross-Layer Framework for Holistic Energy Minimization. IEEE Syst J, 2018, 12: 2461-2472 CrossRef ADS Google Scholar

[7] Dayarathna M, Wen Y G, Fan R. Data Center Energy Consumption Modeling: A Survey. IEEE Commun Surv Tutorials, 2016, 18: 732-794 CrossRef Google Scholar

[8] Wikipedia contributors. Electrical efficiency. Wikipedia, The Free Encyclopedia, 15 Apr. 2018. Google Scholar

[9] Belady C, Rawson A, Pfleuger J, et al. Green Grid Data Center Power Efficiency Metrics: PUE and DCIE. Technical Report, Green Grid. 2008. Google Scholar

[10] Google. Efficiency: How we do it. Google Technical Report. 2018. Google Scholar

[11] Pelley S, Meisner D, Wenisch T F, et al. Understanding and abstracting total data center power. In: Workshop on Energy-Efficient Design, 2009. Google Scholar

[12] Pakbaznia E, Pedram M. Minimizing data center cooling and server power costs. In: Proceedings of ISLPED, 2009. 145--150. Google Scholar

[13] Ahmad F, Vijaykumar T N. Joint optimization of idle and cooling power in data centers while maintaining response time. In: Proceedings of ASPLOS, 2010. 243--256. Google Scholar

[14] Al-Qawasmeh A M, Pasricha S, Maciejewski A A. Power and Thermal-Aware Workload Allocation in Heterogeneous Data Centers. IEEE Trans Comput, 2015, 64: 477-491 CrossRef Google Scholar

[15] Wan J X, Gui X, Kasahara S. Air Flow Measurement and Management for Improving Cooling and Energy Efficiency in Raised-Floor Data Centers: A Survey. IEEE Access, 2018, 6: 48867-48901 CrossRef Google Scholar

[16] Li S, Le H K, Pham N, et al. Joint optimization of computing and cooling energy: analytic model and a machine room case study. In: Proceedings of ICDCS, 2012. 396--405. Google Scholar

[17] Sharma R K, Bash C E, Patel C D. Balance of Power: Dynamic Thermal Management for Internet Data Centers. IEEE Internet Comput, 2005, 9: 42-49 CrossRef Google Scholar

[18] Moore J D, Chase J S, Ranganathan P, et al. Making scheduling “cool": temperature-aware workload placement in data centers. In: Proceedings of USENIX Annual Technical Conference, 2005. 61--75. Google Scholar

[19] Wang L, Khan S U, Dayal J. Thermal aware workload placement with task-temperature profiles in a data center. J Supercomput, 2012, 61: 780-803 CrossRef Google Scholar

[20] Tang Q H, Gupta S K S, Varsamopoulos G. Energy-Efficient Thermal-Aware Task Scheduling for Homogeneous High-Performance Computing Data Centers: A Cyber-Physical Approach. IEEE Trans Parallel Distrib Syst, 2008, 19: 1458-1472 CrossRef Google Scholar

[21] Beitelmal A H, Patel C D. Thermo-Fluids Provisioning of a High Performance High Density Data Center. Distrib Parallel Databases, 2007, 21: 227-238 CrossRef Google Scholar

[22] Choi J, Kim Y, Sivasubramaniam A. A CFD-Based Tool for Studying Temperature in Rack-Mounted Servers. IEEE Trans Comput, 2008, 57: 1129-1142 CrossRef Google Scholar

[23] Bash C, Forman G. Cool job allocation: measuring the power savings of placing jobs at cooling-efficient locations in the data center. In: Proceedings of USENIX Annual Technical Conference, 2007. 363--368. Google Scholar

[24] Moore J D, Chase J S, Ranganathan P. Weatherman: automated, online and predictive thermal mapping and management for data centers. In: Proceedings of ICAC, 2006. 155--164. Google Scholar

[25] Chen H, Kesavan M, Schwan K, et al. Spatially-aware optimization of energy consumption in consolidated data center systems. In: Proceedings of American Society of Mechanical Engineers Pacific Rim Technical Conference and Exhibition on Packaging and Integration of Electronic and Photonic Systems, 2011. 461--470. Google Scholar

[26] Zhou Y, Taneja S, Dudeja G. Towards thermal-aware Hadoop clusters. Future Generation Comput Syst, 2018, 88: 40-54 CrossRef Google Scholar

[27] Sano K, Shimizu H, Kondo Y. Improving Accuracy of Temperature Distribution and Energy-Saving Technology of Air Conditioners in Data Centers. IEEE Trans Compon Packag Manufact Technol, 2018, 8: 811-817 CrossRef Google Scholar

[28] Jiang Z G, Huang W, You I, et al. Thermal-aware task placement with dynamic thermal model in an established datacenter. In: Proceedings of IMIS, 2014. 1--8. Google Scholar

[29] Cao T, Huang W, He Y, et al. Cooling-aware job scheduling and node allocation for overprovisioned HPC systems. In: Proceedings of IPDPS, 2017. 728--737. Google Scholar

[30] Li L, Liang C L, Liu J, et al. ThermoCast: a cyber-physical forecasting model for datacenters. In: Proceedings of KDD, 2011. 1370--1378. Google Scholar

[31] Wei T S, Wang Y Z, Zhu Q. Deep reinforcement learning for building HVAC control. In: Proceedings of DAC, 2017. 1--6. Google Scholar

[32] Banerjee A, Mukherjee T, Varsamopoulos G. Integrating cooling awareness with thermal aware workload placement for HPC data centers. Sustain Computing-Inf Syst, 2011, 1: 134-150 CrossRef Google Scholar

[33] Yeo S, Lee H H S. Peeling the power onion of data centers. In: Energy Efficient Thermal Management of Data Centers. Boston: Springer, 2012. 137--168. Google Scholar

[34] Manousakis I, Goiri I, Sankar S, et al. CoolProvision: underprovisioning datacenter cooling. In: Proceedings of SoCC, 2015. 356--367. Google Scholar

[35] Garday D, Housley J. Thermal Storage System Provides Emergency Data Center Cooling. White Paper Intel Information Technology, Intel Corporation. 2007. Google Scholar

[36] Wang Y F, Wang X R, Zhang Y W. Leveraging thermal storage to cut the electricity bill for datacenter cooling. In: Proceedings of the 4th Workshop on Power-Aware Computing and Systems, 2011. 8. Google Scholar

[37] Zhang Y W, Wang Y F, Wang X R. TEStore: exploiting thermal and energy storage to cut the electricity bill for datacenter cooling. In: Proceedings of CNSM, 2012. 19--27. Google Scholar

[38] Zheng W L, Ma K, Wang X R. Exploiting thermal energy storage to reduce data center capital and operating expenses. In: Proceedings of HPCA, 2014. 132--141. Google Scholar

[39] Roth K, Zogg R, Brodrick J. Cool thermal energy storage. Ashrae J, 2006, 48: 94--96. Google Scholar

[40] Raghavan A, Luo Y, Chandawalla A, et al. Computational sprinting. In: Proceedings of HPCA, 2012. 249--260. Google Scholar

[41] Raghavan A, Luo Y, Chandawalla A, et al. Computational sprinting on a hardware/software testbed. In: Proceedings of ASPLOS, 2013. 155--166. Google Scholar

[42] Raghavan A, Emurian L, Shao L. Utilizing Dark Silicon to Save Energy with Computational Sprinting. IEEE Micro, 2013, 33: 20-28 CrossRef Google Scholar

[43] Raghavan A, Luo Y, Chandawalla A. Designing for Responsiveness with Computational Sprinting. IEEE Micro, 2013, 33: 8-15 CrossRef Google Scholar

[44] Lei S, Raghavan A, Emurian L, et al. On-chip phase change heat sinks designed for computational sprinting. In: Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), 2014. 29--34. Google Scholar

[45] Skach M, Arora M, Hsu C H, et al. Thermal time shifting: leveraging phase change materials to reduce cooling costs in warehouse-scale computers. In: Proceedings of ISCA, 2015. 439--449. Google Scholar

[46] Skach M, Arora M, Tullsen D M, et al. Virtual Melting Temperature: Managing Server Load to Minimize Cooling Overhead with Phase Change Materials. In: Proceedings of ISCA, 2018. 15--28. Google Scholar

[47] Zhang H N, Shao S Q, Xu H B. Free cooling of data centers: A review. Renew Sustain Energy Rev, 2014, 35: 171-182 CrossRef Google Scholar

[48] Park J. Open Compute Project-Data Center. Facebook Technical Report. 2011. Google Scholar

[49] Facebook. Prineville - Facebook power. 2013. https://fbpuewue.com/prineville. Google Scholar

[50] Miller R. The evolution of Facebook's data center cooling. 2012. http://www.datacenterknowledge.com/archives/20 12/12/04/evolution-of-facebooks-cooling/. Google Scholar

[51] Gebrehiwot B, Aurangabadkar K, Kannan N, et al. CFD analysis of free cooling of modular data centers. In: Proceedings of SEMI-THERM, 2012. 108--111. Google Scholar

[52] Weerts B A, Gallaher D, Weaver R, et al. Green data center cooling: Achieving 90. Google Scholar

[53] Endo H, Kodama H, Fukuda H. Effect of climatic conditions on energy consumption in direct fresh-air container data centers. Sustain Computing-Inf Syst, 2015, 6: 17-25 CrossRef Google Scholar

[54] Zhou R L, Wang Z K, McReynolds A, et al. Optimization and control of cooling microgrids for data centers. In: Proceedings of ITherm, 2012. 338--343. Google Scholar

[55] Kim J, Ruggiero M, Atienza D. Free cooling-aware dynamic power management for green datacenters. In: Proceedings of HPCS, 2012. 140--146. Google Scholar

[56] Kumari N, Shah A, Bash C, et al. Optimizing data center energy efficiency via ambient-aware IT workload scheduling. In: Proceedings of ITherm, 2012. 539--544. Google Scholar

[57] Zapater M, Turk A, Moya J M, et al. Dynamic workload and cooling management in high-efficiency data centers. In: Proceedings of IGSC, 2015. 1--8. Google Scholar

[58] Yeo S, Hossain M M, Huang J, et al. ATAC: ambient temperature-aware capping for power efficient datacenters. In: Proceedings of SoCC, 2014. 1--14. Google Scholar

[59] Pinheiro E, Weber W D, Barroso L A. Failure trends in a large disk drive population. In: Proceedings of FAST, 2007. 17--23. Google Scholar

[60] El-Sayed N, Stefanovici I A, Amvrosiadis G. Temperature management in data centers. SIGMETRICS Perform Eval Rev, 2012, 40: 163 CrossRef Google Scholar

[61] GoiriI, Nguyen T D, Bianchini R. CoolAir: temperature- and variation-aware management for free-cooled datacenters. In: Proceedings of ASPLOS, 2015. 253--265. Google Scholar

[62] Arlitt M, Bash C, Blagodurov S, et al. Towards the design and operation of net-zero energy data centers. In: Proceedings of ITherm, 2012. 552--561. Google Scholar

[63] Liu Z H, Chen Y, Bash C, et al. Renewable and cooling aware workload management for sustainable data centers. In: Proceedings of SIGMETRICS, 2012. 175--186. Google Scholar

[64] Mukherjee T, Banerjee A, Varsamopoulos G. Spatio-temporal thermal-aware job scheduling to minimize energy consumption in virtualized heterogeneous data centers. Comput Networks, 2009, 53: 2888-2904 CrossRef Google Scholar

[65] Buyya R, Ranjan R, Calheiros R N. Modeling and simulation of scalable Cloud computing environments and the CloudSim toolkit: Challenges and opportunities. In: Proceedings of HPCS, 2009. 1--11. Google Scholar

[66] Fan X, Weber W D, Barroso L A. Power provisioning for a warehouse-sized computer. In: Proceedings of ISCA, 2007. 13--23. Google Scholar

[67] Bhattacharya A A, Culler D, Kansal A. The need for speed and stability in data center power capping. Sustain Computing-Inf Syst, 2013, 3: 183-193 CrossRef Google Scholar

[68] Fu X, Wang X R, Lefurgy C. How much power oversubscription is safe and allowed in data centers. In: Proceedings of ICAC, 2011. 21--30. Google Scholar

[69] Kidd T L. Power Management States: P-States, C-States, and Package C-States. Intel development resources, 2014. Google Scholar

[70] Lo D, Cheng L, Govindaraju R, et al. Towards energy proportionality for large-scale latency-critical workloads. In: Proceedings of ISCA, 2014. 301--312. Google Scholar

[71] Popa P. Managing Server Energy Consumption Using IBM PowerExecutive. IBM Techique Report, 2006. Google Scholar

[72] Staff H P. HP Power Capping and Dynamic Power Capping for ProLiant Servers. HP Techique Report TC090303TB, 2009. Google Scholar

[73] Ranganathan P, Leech P, Irwin D E, et al. Ensemble-level Power Management for Dense Blade Servers. In: Proceedings of ISCA, 2006. 66--77. Google Scholar

[74] Pelley S, Meisner D, Zandevakili P, et al. Power routing: dynamic power provisioning in the data center. In: Proceedings of ASPLOS, 2010. 231--242. Google Scholar

[75] Wu Q, Deng Q Y, Ganesh L, et al. Dynamo: Facebook's Data Center-Wide Power Management System. In: Proceedings of ISCA, 2016. 469--480. Google Scholar

[76] Reza Azimi, Masoud Badiei, Xin Zhan, et al. Fast Decentralized Power Capping for Server Clusters. In: Proceedings of HPCA, 2017. 181--192. Google Scholar

[77] Islam M A, Ren X, Ren S L, et al. A market approach for handling power emergencies in multi-tenant data center. In: Proceedings of HPCA, 2016. 432--443. Google Scholar

[78] Sun Q, Ren S, Wu C. Fair Online Power Capping for Emergency Handling in Multi-Tenant Cloud Data Centers. IEEE Trans Cloud Comput, 2017, : 1-1 CrossRef Google Scholar

[79] Govindan S, Sivasubramaniam A, Urgaonkar B. Benefits and limitations of tapping into stored energy for datacenters. In: Proceedings of ISCA, 2011. 341--352. Google Scholar

[80] Wang D, Ren C G, Sivasubramaniam C. Virtualizing power distribution in datacenters. In: Proceedings of ISCA, 2013. 595--606. Google Scholar

[81] Govindan S, Wang D, Sivasubramaniam S, et al. Leveraging stored energy for handling power emergencies in aggressively provisioned datacenters. In: Proceedings of ASPLOS, 2012. 75--86. Google Scholar

[82] Kontorinis V, Zhang L E, Aksanli B, et al. Managing distributed UPS energy for effective power capping in data centers. In: Proceedings of ISCA, 2012. 488--499. Google Scholar

[83] Alanazi S, Dabbagh M, Hamdaoui B. Reducing Data Center Energy Consumption Through Peak Shaving and Locked-In Energy Avoidance. IEEE Trans Green Commun Netw, 2017, 1: 551-562 CrossRef Google Scholar

[84] Zhang Q, Shi W S. UPS-aware workload placement in enterprise data centers. IEEE Computer Magazine, 2015. Google Scholar

[85] Zhang Q, Shi W S. Energy-Efficient Workload Placement in Enterprise Datacenters. Computer, 2016, 49: 46-52 CrossRef Google Scholar

[86] AL-Hazemi F, Peng Y, Youn C H. Dynamic allocation of power delivery paths in consolidated data centers based on adaptive UPS switching. Comput Networks, 2018, 144: 254-270 CrossRef Google Scholar

[87] Abbasi Z, Varsamopoulos G, Gupta S K S. TACOMA: server and workload management in internet data centersconsidering cooling-computing power trade-off and energy proportionality. ACM Trans Archit Code Optim, 2012, 9: 1-37 CrossRef Google Scholar

[88] Antal M, Pintea C, Pintea E, et al. Thermal aware workload consolidation in cloud data centers. In: Proceedings of ICCP, 2016. 377--384. Google Scholar

[89] Fu L J, Wan J X, Yang J. Dynamic thermal and IT resource management strategies for data center energy minimization. J Cloud Comp, 2017, 6: 25 CrossRef Google Scholar

[90] Zheng K Y, Zheng W L, Li L. PowerNetS: Coordinating Data Center Network With Servers and Cooling for Power Optimization. IEEE Trans Netw Serv Manage, 2017, 14: 661-675 CrossRef Google Scholar

[91] Parolini L, Sinopoli B, Krogh B H. A Cyber-Physical Systems Approach to Data Center Modeling and Control for Energy Efficiency. Proc IEEE, 2012, 100: 254-268 CrossRef Google Scholar

[92] Abts D, Marty M R, Wells P M, et al. Energy proportional datacenter networks. In: Proceedings of ISCA, 2010. 338--347. Google Scholar

[93] Wang X, Wang X, Zheng K. Correlation-Aware Traffic Consolidation for Power Optimization of Data Center Networks. IEEE Trans Parallel Distrib Syst, 2016, 27: 992-1006 CrossRef Google Scholar

[94] Yao J, Guan H, Luo J. Adaptive Power Management through Thermal Aware Workload Balancing in Internet Data Centers. IEEE Trans Parallel Distrib Syst, 2015, 26: 2400-2409 CrossRef Google Scholar

[95] Li Y L, Hu H, Wen Y G, et al. Learning-based power prediction for data centre operations via deep neural networks. In: Proceedings of the 5th International Workshop on Energy Efficient Data Centres, Waterloo, 2016. 1--10. Google Scholar

[96] Gao J, Jamidar R. Machine learning applications for data center optimization. Google White Paper. 2014. Google Scholar

[97] Rossi M, Brunelli D. Forecasting data centers power consumption with the Holt-Winters method. In: Workshop of EESMS, 2015. 210--214. Google Scholar

[98] Hsu Y, Matsuda K, Matsuoka M. Self-Aware Workload Forecasting in Data Center Power Prediction. In: Proceedings of CCGrid, 2018. 321--330. Google Scholar

[99] Liu N, Li Z, Xu Z Y, et al. A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning. In: Proceedings of ICDCS, 2017. 372--382. Google Scholar

[100] Berral J L, Goiri I, Nou R, et al. Towards energy-aware scheduling in data centers using machine learning. In: Proceedings of e-Energy, 2010. 215--224. Google Scholar

[101] Li Y L, Wen Y G, Guan K, et al. Transforming cooling optimization for green data center via deep reinforcement learning. 2017,. arXiv Google Scholar

[102] AirPAK. Computational fluid dynamics software by Ansys Inc. http://www.ansys.com/products/airpak/default.asp. Google Scholar

  • Table 1   Classification of cross-layer energy efficiency optimization in data center
    System
    Classification IT Cooling Power distribution
    Cooling optimization via IT scheduling $\surd$
    Power distribution optimization via IT scheduling $\surd$
    Joint optimization of IT and cooling $\surd$ $\surd$
    Joint optimization of IT and power distribution $\surd$ $\surd$
  • Table 2   Comparison of energy efficiency optimization methods of refrigeration system based on IT load scheduling
    Classification Source Objective Constraint Method Platform Result
    12*Heat balance [18] Minimize max cabinet inlet temperature Servercapacity Temperature aware, heuristic Simulation Saving 25% cooling cost
    [20] Minimize max cabinet inlet temperature Server capacity Temperature aware, genetic algorithm, sequential quadratic programming Small data centers Reduce max cabinet inlet temperature by 2$^{\circ}$C$\sim$5$^{\circ}$C, saving20%$\sim$30% cooling cost
    [19] Minimize max cabinet inlet temperature Server capacity,delay Temperature aware, heuristic Simulation Delay increase 11%, reduce average and max temperature by 14.6 F and 4.9 F
    [25] Minimize max cabinet inlet temperature Server capacity Cooling efficiency aware, reinforcement learning Simulation Reduce max cabinet inlet temperature by 2$^{\circ}$C$\sim$3$^{\circ}$C
    [26] Minimize max cabinet inlet temperature Servercapacity Heat aware, heuristic Simulation Reduce max cabinet inlet temperature and cooling cost by 2.5$^{\circ}$C and 15%
    Reduce cooling [32] Minimize chiller plant cost Server capacity, SLA Air supply temperature Simulation Reduce cooling costby 15%, increase throughput perenergy by 6.89%
    cost [24] Minimize thecost of chillerplant and fans Servercapacity Air supply temperature, fan speed Simulation Reduce cooling cost by 13%$\sim$25%
    Peak heat shaving [45] Minimize the peak heat Server capacity, QoS PCM-TTF Single server simulation Reduce peak heat by 12%
    [46] Minimize the peak heat Server capacity, QoS PCM-VMT Simulation Reduce peak heat by 12.8% even when TTF fails
    Use free cooling [55] Minimize cooling cost Server capacity, SLA Use free cool- ing, rolling timedomain control CloudSim simulation [65] Reduce energy cost by 25.7%
    [61] Minimize internal temperature variance Server capacity Internal temperature variance aware Simulation Reduce cooling cost and internal temperature variance
  • Table 3   Comparison of energy efficiency optimization methods of power supply system based on IT load scheduling
    Classification Source Objective Constraint Method Platform Result
    Power capping [73] Reduce power budget Server capacity, PDU capacity, QoS Percentage cut Enterprise data center Reduce power budget by 20%, negligible QoS loss
    [66] Reduce power budget Server capacity, PDU capacity, QoS Percentage cut Google data center Reduce power consumption 23%
    [75] Data center power management atscale Server capacity, PDU capacity Percentage cut Facebook data center Increase QoS and power resource utilization by 13%$\sim$40% and 8%
    [74] Increase power resource utilization Server capacity, PDU capacity Reset power distribution topology Real data center Reduce power budget and data center construction cost by 47% and 32%
    [76] Rapid power management Sever capacity, PDU capacity Distributed optimization Small cluster Reduce setup delayby 72%$\sim$86%, increasethroughput by 16%
    [77] Power manage- ment of multi- tenant data center PDU capacity Supply function bidding Simulation Win-win between tenant and owner
    UPS supply [79] Reduce grid power supply Battery capacity, lifecycle, availability Centralized UPS supply Simulation Reduce power budget by 15%$\sim$45% without user experience degradation
    [81] Reduce QoS degradation Battery capa- city, server capacity Distributed UPS supply Testbed Can solve performance degradation issue in most sases
    [82] Reduce QoS loss, extent battery life Battery capa- city, servercapacity Distributed UPS supply Simulation Shaving peak power by 19.4%, increase power resource utilization by 24%
    [83] Increase power resource utilization Battery capa- city, server capacity Scheduling of load andUPS Simulation Reduce electricity cost by 20%
    [80] Increase power resource utilization Battery capacity, fairness Power allocation betweenapplications Simulation Increase system utilization and application QoS by 50% and 12%$\sim$28%
    UPS scheduling [85] Reduce the loss of UPS transfer Battery capa- city, server capacity Loss awarescheduling Small cluster Reduce power by 5.2%
    [86] Reduce the loss of UPS transfer Battery capa- city, server capacity Loss aware scheduling,UPS sleeping Simulation Reduce loss of UPS transfer and amount by 20%$\sim$40% and 20%
  • Table 4   Comparison of cross layer energy efficiency optimization methods
    Classification Source Objective Constraint Method Platform Result
    Analytic approach [12] Minimize server and cooling energy Server capacity DVFS, server sleeping, airsupply MATLAB Save 13% more energy than [20]
    [13] Minimize server and cooling energy Server capacity Server sleeping, air supply Air-PAK[102] Save 30% energy with delay less than 1.7%
    [14] Maximize server QoS, Minimize server and cooling energy Server capacity P-states, server sleeping, airsupply Simulation Average 17% performanceincrease, 9% energy reduction
    [15] Minimize server and cooling energy Server capacity, QoS DVFS, server sleeping, fanspeed, air supply Simulation Reduce energy by 4%
    [90] Minimize server, network and cooling energy Server capa- city, network bandwidth Server and network sleeping, air supply Hardware testbet Reduce energyby 44.3%, more8.8%$\sim$14.6% reduction with network
    [91] Minimize server and cooling energy Server capacity DVFS, server sleeping, fanspeed, air supply Simulation Reduce energy by 4%
    [16] Minimize server and cooling energy Server capacity, max temperature Server sleeping, fan speed, airsupply Hardware testbed Reduce energy by 5%$\sim$18%
    Prediction approach [96] Minimize PUE Resource capacity 19 types of data center features Google data center Reduce signifi- cant energy