国家自然科学基金(51707102)
国家自然科学基金(61603212)
Appendix 续表A1
[1] Lund H. Large-scale integration of wind power into different energy systems. Energy, 2005, 30: 2402-2412 CrossRef Google Scholar
[2] Soares M.C. Borba B, Szklo A, Schaeffer R. Plug-in hybrid electric vehicles as a way to maximize the integration of variable renewable energy in power systems: The case of wind generation in northeastern Brazil. Energy, 2012, 37: 469-481 CrossRef Google Scholar
[3] Venkat A N, Hiskens I A, Rawlings J B. Distributed MPC Strategies With Application to Power System Automatic Generation Control. IEEE Trans Contr Syst Technol, 2008, 16: 1192-1206 CrossRef Google Scholar
[4] Mallesham G, Mishra S, Jha A N. Maiden application of Ziegler-Nichols method to AGC of distributed generation system. In: Proceedings of IEEE/PES Power Systems Conference and Exposition, Seattle, 2009. 1--7. Google Scholar
[5] Yazdanian M, Mehrizi-Sani A. Distributed Control Techniques in Microgrids. IEEE Trans Smart Grid, 2014, 5: 2901-2909 CrossRef Google Scholar
[6] Busoniu L, Babuska R, De Schutter B. A Comprehensive Survey of Multiagent Reinforcement Learning. IEEE Trans Syst Man Cybern C, 2008, 38: 156-172 CrossRef Google Scholar
[7] Yu T, Zhou B, Chan K W. Stochastic Optimal CPS Relaxed Control Methodology for Interconnected Power Systems Using Q-Learning Method. J Energy Eng, 2011, 137: 116-129 CrossRef Google Scholar
[8] Yu T, Zhou B, Chan K W. Stochastic Optimal Relaxed Automatic Generation Control in Non-Markov Environment Based on Multi-Step Q(lambda) Learning. IEEE Trans Power Syst, 2011, 26: 1272-1282 CrossRef ADS Google Scholar
[9] Yu T, Xi L, Yang B. Multiagent Stochastic Dynamic Game for Smart Generation Control. J Energy Eng, 2016, 142: 04015012 CrossRef Google Scholar
[10] Xi L, Yu T, Yang B. A novel multi-agent decentralized win or learn fast policy hill-climbing with eligibility trace algorithm for smart generation control of interconnected complex power grids. Energy Convers Manage, 2015, 103: 82-93 CrossRef Google Scholar
[11] Foerster J N, Assael Y M, Freitas N D, et al. Learning to communicate to solve riddles with deep distributed decurrent Q-networks. 2016,. arXiv Google Scholar
[12] Banerjee B, Kraemer L. Reinforcement learning with action discovery. In: Proceedings of the Adaptive and Learning Agents Workshop at AAMAS-10, Toronto, 2010. 30--37. Google Scholar
[13] Tan W, Xu Z. Robust analysis and design of load frequency controller for power systems. Electric Power Syst Res, 2009, 79: 846-853 CrossRef Google Scholar
[14] Zhang X, Zheng L, Yu T. Multi-objective optimal carbon emission flow calculation of power grid based on multi-step Q($\lambda~)$ learning algorithm. Automat Electron Power Sys, 2014, 38: 118--123. Google Scholar
[15] Tang Y, Zhang W, Zhang J, et al. Research on control performance standard based control strategy for AGC. Power Sys Te Chno, 2004, 28: 75--79. Google Scholar
[16] Park J, Law K H. A data-driven, cooperative wind farm control to maximize the total power production. Appl Energy, 2016, 165: 151-165 CrossRef Google Scholar
[17] Banerjee B, Peng J. Adaptive policy gradient in multiagent learning. In: Proceedings of the 2nd International Joint Conference on Autonomous Agents and Multiagent Systems. New York: ACM, 2003. 686--692. Google Scholar
[18] Bowling M, Veloso M. Multiagent learning using a variable learning rate. Artificial Intelligence, 2002, 136: 215-250 CrossRef Google Scholar
[19] Yu T, Zhou B, Chan K W. R() imitation learning for automatic generation control of interconnected power grids. Automatica, 2012, 48: 2130-2136 CrossRef Google Scholar
Figure 1
(Color online) The schematic diagram of action discovery
Figure 2
(Color online) Control schematic diagram of SGC system based on DDRQN-AD
Figure 3
(Color online) Active power of wind power, photovoltaic and electricvehicles
Figure 4
(Color online) The two-area micro-grid LFC power system model
Figure 5
(Color online) The pre-learning effect of DDRQN-AD in area-A
Figure 6
(Color online) Control performance of different algorithms under pulsedisturbance. (a) Output pulse of various algorithms; (b) the $\vert~\Delta~f\vert$ and $\vert$ACE$\vert~$ of various algorithms
Figure 7
(Color online) Control performance of different algorithms under random pulsedisturbance. (a) Output results under random pulse disturbance; (b)the $\vert~\Delta~f\vert$ and $\vert~$ACE$\vert~$ ofvarious algorithms
Figure 8
(Color online) The adjusted active power of wind power, photovoltaic andelectric vehicles
Figure 9
(Color online) Effect of DDRQN-AD under white noise disturbance
Figure 10
(Color online) Performance statistics of the three algorithms under whitenoise disturbance
Figure 11
(Color online) Guangdong power grid model
| Case | $\delta$ | $\alpha$ | $\alpha^{-}$ | $\gamma$ |
| Ideal environment | 0.1 | 0.5 | 0.5 | 0.95 |
| Nonideal environment | 0.3 | 0.1 | 0.1 | 0.9 |
基于动作概率分布选择并执行一个探索动作$a^m_t$. |
观察下一时刻的状态$s_{t~+~1}$. |
记录状态观测值$o^m_{t+1}$和内部状态$h^m_t$. |
由式( |
根据式( |
按照式( |
根据式( |
按照式(7)搜索并评估新动作. |
动作集$A(t)$更新为$A(t$+1). |
令$t=t$+1, 返回步骤 |
| Area | Unit type | Unit | $\Delta~P_k^{\rm~max}$ (MW) | $\Delta~P_k^{\rm~min}$ (MW) | $B_{k}$ (kg/kWh) | Unit state (Summer) | Unit state (Winter) |
| Yuebei | Coal-fired units | G1 | 120 | $-120$ | 0.99 | Starting up | Maintenance |
| G2 | 120 | $-$120 | 0.99 | Starting up | Starting up | ||
| G3 | 120 | $-$120 | 0.99 | Cosing down | Starting up | ||
| G4 | 135 | $-$135 | 0.99 | Starting up | Starting up | ||
| G5 | 135 | $-$135 | 0.99 | Starting up | Starting up | ||
| G6 | 300 | $-$300 | 0.99 | Starting up | Starting up | ||
| G7 | 300 | $-$300 | 0.99 | Starting up | Starting up | ||
| G8 | 320 | $-$320 | 0.89 | Starting up | Starting up | ||
| Gas-fired unit | G9 | 188 | $-$188 | 0.5 | Starting up | Starting up | |
| Hydropower unit | G10 | 180 | 0 | 0 | Starting up | 50% capacity | |
| Yuexi | Coal-fired units | G11 | 500 | $-$500 | 0.89 | Starting up | Starting up |
| G12 | 330 | $-$330 | 0.89 | Starting up | Starting up | ||
| G13 | 125 | $-$125 | 0.99 | Starting up | Maintenance | ||
| G14 | 125 | $-$125 | 0.99 | Cosing down | Starting up | ||
| G15 | 150 | $-$150 | 0.99 | Starting up | Starting up | ||
| G16 | 150 | $-$150 | 0.99 | Starting up | Starting up | ||
| G17 | 150 | $-$150 | 0.99 | Starting up | Starting up | ||
| G18 | 220 | $-$220 | 0.99 | Starting up | Starting up | ||
| G19 | 220 | $-$220 | 0.99 | Starting up | Starting up | ||
| G20 | 220 | $-$220 | 0.99 | Starting up | Starting up | ||
| G21 | 660 | $-$660 | 0.87 | Starting up | Starting up | ||
| G22 | 180 | $-$180 | 0.99 | Starting up | Starting up | ||
| G23 | 180 | $-$180 | 0.99 | Starting up | Starting up | ||
| Gas-fired units | G24 | 280 | $-$280 | 0.5 | Starting up | Starting up | |
| G25 | 200 | $-$200 | 0.5 | Starting up | Starting up | ||
| G26 | 200 | $-$200 | 0.5 | Starting up | Starting up | ||
| G27 | 200 | $-$200 | 0.5 | Starting up | Starting up | ||
| Oil-fuel units | G28 | 120 | $-$120 | 0.7 | Starting up | Maintenance | |
| G29 | 120 | $-$120 | 0.7 | Cosing down | Starting up | ||
| Zhusanjiao | Coal-fired units | G30 | 600 | $-$600 | 0.89 | Starting up | Starting up |
| G31 | 100 | $-$100 | 0.99 | Starting up | Maintenance | ||
| G32 | 100 | $-$100 | 0.99 | Starting up | Maintenance | ||
| G33 | 200 | $-$200 | 0.99 | Starting up | Starting up | ||
| G34 | 200 | $-$200 | 0.99 | Starting up | Starting up | ||
| G35 | 200 | $-$200 | 0.99 | Starting up | Starting up | ||
| G36 | 210 | $-$210 | 0.99 | Starting up | Starting up | ||
| G37 | 240 | $-$240 | 0.99 | Starting up | Starting up |
| Area | Unit type | Unit | $\Delta~P_k^{\rm~max}~$ (MW) | $\Delta~P_k^{\rm~min}$ (MW) | $B_{k}$ (kg/kWh) | Unit state (Summer) | Unit state (Winter) |
| G38 | 240 | $-$240 | 0.99 | Starting up | Starting up | ||
| G39 | 280 | $-$280 | 0.99 | Starting up | Starting up | ||
| G40 | 280 | $-$280 | 0.99 | Starting up | Starting up | ||
| G41 | 280 | $-$280 | 0.99 | Starting up | Starting up | ||
| G42 | 250 | $-$250 | 0.99 | Starting up | Starting up | ||
| G43 | 250 | $-$250 | 0.99 | Starting up | Starting up | ||
| G44 | 360 | $-$360 | 0.89 | Starting up | Starting up | ||
| G45 | 360 | $-$360 | 0.89 | Starting up | Starting up | ||
| G46 | 400 | $-$400 | 0.89 | Starting up | Starting up | ||
| G47 | 400 | $-$400 | 0.89 | Starting up | Starting up | ||
| Gas-fired units | G48 | 180 | $-$180 | 0.5 | Starting up | Starting up | |
| G49 | 180 | $-$180 | 0.5 | Starting up | Starting up | ||
| G50 | 180 | $-$180 | 0.5 | Starting up | Starting up | ||
| Oil-fuel units | G51 | 150 | $-$150 | 0.7 | Cosing down | Starting up | |
| G52 | 150 | $-$150 | 0.7 | Cosing down | Starting up | ||
| G53 | 180 | $-$180 | 0.7 | Starting up | Starting up | ||
| G54 | 180 | $-$180 | 0.7 | Starting up | Starting up | ||
| G55 | 180 | $-$180 | 0.7 | Starting up | Starting up | ||
| Hydropower units | G56 | 300 | 0 | 0 | Starting up | 50% capacity | |
| G57 | 300 | 0 | 0 | Starting up | 50% capacity | ||
| G58 | 400 | 0 | 0 | Starting up | 50% capacity | ||
| Yuedong | Coal-fired units | G59 | 100 | $-$100 | 0.99 | Starting up | Maintenance |
| G60 | 196 | $-$196 | 0.99 | Starting up | Starting up | ||
| G61 | 296 | $-$296 | 0.99 | Starting up | Starting up | ||
| G62 | 180 | $-$180 | 0.99 | Cosing down | Starting up | ||
| G63 | 220 | $-$220 | 0.99 | Starting up | Starting up | ||
| G64 | 180 | $-$180 | 0.99 | Cosing down | Starting up | ||
| G65 | 220 | $-$220 | 0.99 | Starting up | Starting up | ||
| G66 | 180 | $-$180 | 0.99 | Starting up | Starting up | ||
| G67 | 100 | $-$100 | 0.99 | Starting up | Maintenance | ||
| G68 | 168 | $-$168 | 0.99 | Cosing down | Starting up | ||
| G69 | 60 | $-$60 | 0.99 | Starting up | Maintenance | ||
| G70 | 210 | $-$210 | 0.99 | Starting up | Starting up | ||
| G71 | 350 | $-$350 | 0.89 | Starting up | Starting up | ||
| G72 | 240 | $-$240 | 0.99 | Starting up | Starting up | ||
| G73 | 240 | $-$240 | 0.99 | Starting up | Starting up | ||
| G74 | 240 | $-$240 | 0.99 | Starting up | Starting up | ||
| G75 | 240 | $-$240 | 0.99 | Starting up | Starting up | ||
| G76 | 200 | $-$200 | 0.99 | Starting up | Starting up |
| Algorithm | Overshoot (% | Steady state error (% | Risetime (s) |
| DDRQN-AD | 7.08 | 0.57 | 138 |
| DDRQN | 7.34 | 5.83 | 202 |
| PDWoLF-PHC($\lambda~)$ | 7.38 | 3.98 | 190 |
| SARSA-AD | 7.38 | 7.24 | 186 |
| DWoLF-PHC($\lambda~)$ | 7.54 | 7.57 | 318 |
| WoLF-PHC | 7.40 | 12.42 | 198 |
| R($\lambda~)$ | 7.36 | 20.84 | 222 |
| Q($\lambda~)$ | 8.24 | 12.28 | 254 |
| Q | 8.13 | 13.69 | 534 |
| Area | Unit type | Unit | $\Delta~P_k^{\rm~max}$ (MW) | $\Delta~P_k^{\rm~min}$ (MW) | $B_{k}$ (kg/kWh) | Unit state (Summer) | Unit state (Winter) |
| G77 | 200 | $-$200 | 0.99 | Starting up | Starting up | ||
| G78 | 220 | $-$220 | 0.99 | Starting up | Starting up | ||
| G79 | 220 | $-$220 | 0.99 | Starting up | Starting up | ||
| G80 | 220 | $-$220 | 0.99 | Starting up | Starting up | ||
| G81 | 350 | $-$350 | 0.89 | Starting up | Starting up | ||
| G82 | 350 | $-$350 | 0.89 | Starting up | Starting up | ||
| Gas-fired units | G83 | 250 | $-$250 | 0.5 | Starting up | Starting up | |
| G84 | 250 | $-$250 | 0.5 | Starting up | Starting up | ||
| G85 | 250 | $-$250 | 0.5 | Starting up | Starting up | ||
| G86 | 250 | $-$250 | 0.5 | Starting up | Starting up | ||
| G87 | 288 | $-$288 | 0.5 | Starting up | Starting up | ||
| G88 | 360 | $-$360 | 0.5 | Starting up | Starting up | ||
| G89 | 100 | $-$100 | 0.5 | Starting up | Maintenance | ||
| Oil-fuel units | G90 | 240 | $-$240 | 0.7 | Starting up | Starting up | |
| G91 | 240 | $-$240 | 0.7 | Starting up | Starting up | ||
| G92 | 120 | $-$120 | 0.7 | Cosing down | Starting up | ||
| Hydropower unit | G93 | 244 | 0 | 0 | Starting up | 50% capacity |
| Area | Algorithm | $\vert~$ACE$\vert~$ (MW) | CPS1 (% | $\vert~\Delta~f\vert~$ (Hz) | CE |
| DDRQN-AD | 4.8465 | 199.9606 | 0.0036 | 637.8061 | |
| DDRQN | 13.2599 | 199.5169 | 0.0066 | 653.1144 | |
| Yuebei | PDWoLF-PHC($\lambda~)$ | 30.8923 | 197.6574 | 0.0096 | 687.4272 |
| DWoLF-PHC($\lambda~)$ | 62.0124 | 194.6082 | 0.0140 | 689.4484 | |
| Q($\lambda~)$ | 82.0249 | 188.5264 | 0.0154 | 694.0980 | |
| DDRQN-AD | 9.3702 | 199.9714 | 0.0063 | 671.2834 | |
| DDRQN | 19.5084 | 198.1847 | 0.0096 | 688.7363 | |
| Yuexi | PDWoLF-PHC($\lambda~)$ | 45.5341 | 195.3937 | 0.0122 | 692.6057 |
| DWoLF-PHC($\lambda~)$ | 72.5696 | 189.6666 | 0.0141 | 693.1040 | |
| Q($\lambda~)$ | 105.0339 | 178.8297 | 0.0155 | 699.8637 | |
| DDRQN-AD | 9.3545 | 199.5875 | 0.0054 | 633.6197 | |
| DDRQN | 18.1722 | 198.8693 | 0.0068 | 652.1616 | |
| Zhusanjiao | PDWoLF-PHC($\lambda~)$ | 45.7089 | 195.0776 | 0.0098 | 683.5096 |
| DWoLF-PHC($\lambda~)$ | 80.8745 | 191.5694 | 0.0142 | 687.8286 | |
| Q($\lambda~)$ | 139.9966 | 173.0605 | 0.0157 | 694.8414 | |
| DDRQN-AD | 2.9283 | 199.8459 | 0.0065 | 635.2505 | |
| DDRQN | 9.7626 | 199.1897 | 0.0069 | 657.7666 | |
| Yuedong | PDWoLF-PHC($\lambda~)$ | 22.7016 | 197.2192 | 0.0100 | 671.7122 |
| DWoLF-PHC($\lambda~)$ | 61.9700 | 194.6535 | 0.0144 | 675.5289 | |
| Q($\lambda~)$ | 102.4672 | 190.5930 | 0.0157 | 698.7767 |
| Area | Algorithm | $\vert~$ACE$\vert~$ (MW) | CPS1 (% | $\vert~\Delta~f\vert~$ (Hz) | CE |
| DDRQN-AD | 4.8690 | 199.8558 | 0.0030 | 704.9624 | |
| DDRQN | 11.5191 | 198.6730 | 0.0050 | 719.7761 | |
| Yuebei | PDWoLF-PHC($\lambda~)$ | 38.7529 | 195.3053 | 0.0101 | 721.6102 |
| DWoLF-PHC($\lambda~)$ | 70.2284 | 194.1612 | 0.0110 | 723.7792 | |
| Q($\lambda~)$ | 135.8834 | 190.3734 | 0.0137 | 737.4928 | |
| DDRQN-AD | 3.8699 | 199.6796 | 0.0029 | 681.6042 | |
| DDRQN | 10.7540 | 198.4822 | 0.0052 | 698.4800 | |
| Yuexi | PDWoLF-PHC($\lambda~)$ | 30.8557 | 194.6218 | 0.0101 | 707.6413 |
| DWoLF-PHC($\lambda~)$ | 71.3868 | 193.3833 | 0.0111 | 717.5449 | |
| Q($\lambda~)$ | 133.4363 | 192.6690 | 0.0138 | 725.5942 | |
| DDRQN-AD | 4.6580 | 199.0981 | 0.0027 | 648.7999 | |
| DDRQN | 15.4619 | 198.3565 | 0.0052 | 670.1390 | |
| Zhusanjiao | PDWoLF-PHC($\lambda~)$ | 32.3917 | 194.2996 | 0.0116 | 672.3745 |
| DWoLF-PHC($\lambda~)$ | 77.1797 | 192.6169 | 0.0112 | 674.3519 | |
| Q($\lambda~)$ | 139.7009 | 179.0764 | 0.0143 | 696.9169 | |
| DDRQN-AD | 4.8149 | 199.9593 | 0.0031 | 644.2032 | |
| DDRQN | 10.5419 | 198.5774 | 0.0056 | 659.5584 | |
| Yuedong | PDWoLF-PHC($\lambda~)$ | 32.3602 | 195.2387 | 0.0099 | 680.9719 |
| DWoLF-PHC($\lambda~)$ | 72.2484 | 193.1077 | 0.0115 | 689.6905 | |
| Q($\lambda~)$ | 147.6071 | 191.5688 | 0.0137 | 702.1414 |
Download PDF
