logo

Journals Books
Advanced Search
Sign in Register

中文

SCIENTIA SINICA Informationis
Download PDF
SCIENTIA SINICA Informationis, Volume 48 , Issue 7 : 767-782(2018) https://doi.org/10.1360/N112018-00001

Research progress on credibility assessment of a complex simulation system

Wei LI , Shenglin LIN , Yuchen ZHOU , Ping MA , Ming YANG *
More info
  • ReceivedMar 30, 2018
  • AcceptedApr 8, 2018
  • PublishedJul 16, 2018
Previous Table of Contents Next
Share
Rating

Abstract


Funded by

国家自然科学基金重大科研仪器研制项目(61627810)


References

[1] Wang Z C. Development and formation of simulation science. J Syst Simul, 2005, 06: 1279--1281. Google Scholar

[2] Yang M, Zhang B, Wang Z C. The analysis of modeling and simulation development direction. J Syst Simul, 2004, 16: 1901--1904. Google Scholar

[3] Robinson S. Simulation: the Practice of Model Development and Use. London: Palgrave Macmillan Press, 2014. 18--21. Google Scholar

[4] Wang Z C, Zhang B, Yang M. Verification, validation and accreditation (VV&A) for simulation system: current status and future. J Syst Simul, 1999, 11: 321--326. Google Scholar

[5] Qian X C. Research on simulation model validation and calibration methods under uncertainty. Dissertation for Ph.D. Degree. Harbin: Harbin Institute of Technology, 2016. Google Scholar

[6] Yang M, Zhang B, Ma P, et al. Five key issues of the development of simulation systems VV&A. J Syst Simul, 2003, 15: 1502--1506. Google Scholar

[7] Sargent R G. Verification and validation of simulation models. J Simul, 2013, 7: 12-24 CrossRef Google Scholar

[8] Distributed Interactive Simulation Committee of the IEEE Computer Society. IEEE Recommended Practice for Distributed Interactive Simulation — Verification, Validation, and Accreditation. Version 1278.4, 1997. Google Scholar

[9] Conway R W, Johnson B M, Maxwell W L. Some problems of digital systems simulation. Manage Sci, 1959, 6: 92-110 CrossRef Google Scholar

[10] Fishman G S, Kiviat P J. The analysis of simulation generated time series. Manage Sci, 1967, 3: 525--557. Google Scholar

[11] Hermann C F. Validation problems in games and simulations with special reference to models of international politics. Syst Res, 1967, 12: 216-231 CrossRef Google Scholar

[12] Schruben L W. Establishing the credibility of simulations. Simulation, 1980, 34: 101-105 CrossRef Google Scholar

[13] Teorey T J. Validation criteria for computer system simulations. Simuletter, 1975, 6: 9--20. Google Scholar

[14] Balci O, Sargent R G. Some examples of simulation model validation using hypothesis testing. In: Proceedings of the 14th Conference on Winter Simulation, San Diego, 1982. 621--629. Google Scholar

[15] Balci O, Sargent R G. Validation of simulation models via simultaneous confidence intervals. Am J Math Manage Sci, 1984, 4: 375--406. Google Scholar

[16] Kheir N A, Holmes W M. On validating simulation models of missile systems. Simulation, 1978, 30: 117-128 CrossRef Google Scholar

[17] Tytula T P. A Method for Validating Missile System Simulation Models. Army Missile R&D Command, Redstone Arsenal, 1978. Google Scholar

[18] Montgomery D C, Conard R G. Comparison of simulation and flight-test data for missile systems. Simulation, 1980, 34: 63-72 CrossRef Google Scholar

[19] Balci O. Validation, verification, and testing techniques throughout the life cycle of a simulation study. Ann Oper Res, 1994, 53: 121-173 CrossRef Google Scholar

[20] Kleijnen J P C. Verification and validation of simulation models. Eur J Oper Res, 1995, 82: 145-162 CrossRef Google Scholar

[21] Defense Modeling and Simulation Office. Verification, Validation and Accreditation Recommended Practice Guide BUILD 1. 1996. Google Scholar

[22] Defense Modeling and Simulation Office. Verification, Validation and Accreditation Recommended Practice Guide BUILD 2. 2000. Google Scholar

[23] Simulation Interoperability Standards Committee of the IEEE Computer Society. IEEE Recommended Practice for Verification, Validation, and Accreditation of A Federation: an Overlay to the High Level Architecture Federation Development and Execution Process. Version 1516.4, 2007. Google Scholar

[24] Simulation Interoperability Standards Organization. Reference for Generic Methodology for Verification and Validation (GM-VV) to Support Acceptance of Models, Simulations and Data. Vol. 1: Introduction and Overview Version 1, 2012. Google Scholar

[25] Simulation Interoperability Standards Organization. Reference for Generic Methodology for Verification and Validation (GM-VV) to Support Acceptance of Models, Simulations and Data. Vol. 2: Implementation Guide Version 2, 2013. Google Scholar

[26] Simulation Interoperability Standards Organization. Reference for Generic Methodology for Verification and Validation (GM-VV) to Support Acceptance of Models, Simulations and Data. Vol. 3: Reference Manual Version 3, 2013. Google Scholar

[27] Mullins J, Ling Y, Mahadevan S. Separation of aleatory and epistemic uncertainty in probabilistic model validation. Reliab Eng Syst Safe, 2016, 147: 49-59 CrossRef Google Scholar

[28] Zhao L F, Lu Z Z, Yun W Y. Validation metric based on Mahalanobis distance for models with multiple correlated responses. Reliab Eng Syst Safe, 2017, 159: 80-89 CrossRef Google Scholar

[29] Kwag S, Gupta A, Dinh N. Probabilistic risk assessment based model validation method using Bayesian network. Reliab Eng Syst Safe, 2018, 169: 380-393 CrossRef Google Scholar

[30] Ao D, Hu Z, Mahadevan S. Design of validation experiments for life prediction models. Reliab Eng Syst Safe, 2017, 165: 22-33 CrossRef Google Scholar

[31] Wu D Q, Lu Z Z, Wang Y P. Model validation and calibration based on component functions of model output. Reliab Eng Syst Safe, 2015, 140: 59-70 CrossRef Google Scholar

[32] Li Y L, Wang X J, Wang C. Non-probabilistic Bayesian update method for model validation. Appl Math Model, 2018, 58: 388-403 CrossRef Google Scholar

[33] Jiang X M, Yuan Y, Mahadevan S. An investigation of Bayesian inference approach to model validation with non-normal data. J Stat Comput Simul, 2013, 83: 1829-1851 CrossRef Google Scholar

[34] Ling Y, Mahadevan S. Quantitative model validation techniques: New insights. Reliab Eng Syst Safe, 2013, 111: 217-231 CrossRef Google Scholar

[35] Sankararaman S, Mahadevan S. Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems. Reliab Eng Syst Safe, 2015, 138: 194-209 CrossRef Google Scholar

[36] Atkinson A D, Hill R R, Pignatiello, Jr. J J. Wavelet ANOVA approach to model validation. Simul Model Pract Theory, 2017, 78: 18-27 CrossRef Google Scholar

[37] Sarin H, Kokkolaras M, Hulbert G, et al. A comprehensive metric for comparing time histories in validation of simulation models with emphasis on vehicle safety applications. In: Proceedings of the ASME 2008 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, New York, 2009. Google Scholar

[38] Poropudas J, Virtanen K. Game theoretic validation of air combat simulation models. In: Proceedings of the 2009 IEEE International Conference on Systems, Man, and Cybernetics, San Antonio, 2009. 3243--3250. Google Scholar

[39] Lemmers A, Roza M, Voogd J, et al. V&V study of an F-16 familiarization training simulator. In: Fall Simulation Interoperability Workshop. USA: SISO, 2013. 98--108. Google Scholar

[40] Dhananjayan A, Seow K T. A formal transparency framework for validation of real-time discrete-event control requirements modeled by timed transition graphs. IEEE Trans Hum Mach Syst, 2015, 45: 350-361 CrossRef Google Scholar

[41] Guerini M, Moneta A. A method for agent-based models validation. J Economic Dyn Control, 2017, 82: 125-141 CrossRef Google Scholar

[42] Jebeile J, Barberousse A. Empirical agreement in model validation.. Stud Hist Philos Sci Part A, 2016, 56: 168-174 CrossRef PubMed Google Scholar

[43] Michopoulos J, Lambrakos S. On the fundamental tautology of validating data-driven models and simulations. In: Proceedings of International Conference on Computational Science, Atlanta, 2005. 738--745. Google Scholar

[44] Szabo C, Teo Y M. On validation of semantic composability in data-driven simulation. In: Proceedings of IEEE Workshop on Principles of Advanced and Distributed Simulation, Atlanta, 2010. 73--80. Google Scholar

[45] Lamperti F. An information theoretic criterion for empirical validation of simulation models. Economet Stat, 2018, 5: 83-106 CrossRef Google Scholar

[46] Guerini M, Moneta A. A method for agent-based models validation. J Economic Dyn Control, 2017, 82: 125-141 CrossRef Google Scholar

[47] Wei H L, Li Z W. Grey relational analysis and its applieation to the validation of computer simulation models for missile systems. Syst Eng Electron, 1997, 2: 55--61. Google Scholar

[48] Li W, Jiao S, Lu L Y, et al. Validation and selection of simulation model based on the feature differences. Act Autom Sin, 2014, 40: 2134--2144. Google Scholar

[49] Li L, Lu Z Z. A new method for model validation with multivariate output. Reliab Eng Syst Safe, 2018, 168: 579--592. Google Scholar

[50] Wu Y J, Wang J M, Yang W G. Approach of credibility evaluation for testing system with small samples. J Beijing Univ Aeronaut Astronaut, 2016, 42: 1911--1917. Google Scholar

[51] Zheng K, Hu J, Zhan Z F, et al. Multivariate responses analysis for model validation of dynamic systems. J Shanghai Jiao Tong Univ, 2015, 49: 191--195. Google Scholar

[52] Wang H Y. Research on multivariate simulation result validation methods under uncertainty. Dissertation for Master Degree. Harbin: Harbin Institute of Technology, 2016. Google Scholar

[53] Min F Y, Yang M, Wang Z C. Knowledge-based method for the validation of complex simulation models. Simul Model Pract Theory, 2010, 18: 500-515 CrossRef Google Scholar

[54] Jiao S, Li W, Yang M. Validation method of simulation model based on triangular fuzzy number. J Cent South Univ (Sci Technol), 2014, 45: 124--131. Google Scholar

[55] Huang Z J, Chen B, Ou Y H. Model checking continuous time Markov process based on timed schedulers. J Guangxi Univ Sci Technol, 2014, 25: 59--63. Google Scholar

[56] Wang S, Wu D H, Qu L, et al. Study on credibility evaluation framework of complex simulation systems. Comput Simul, 2012, 29: 116--122. Google Scholar

[57] Niu S, Lin S L, Li W, et al. Model validation method for discrete event simulation. J Syst Simul, 2017, 29: 1984--1990. Google Scholar

[58] Xia W, Yao Y P, Mu X D. Parallel model checking for discrete event simulation models based on event graphs. J Softw, 2012, 23: 1429-1443 CrossRef Google Scholar

[59] Fang K, Zhou Y C, Zhao E J. Discussion for the factor space of simulation model validation. Syst Eng Electron, 2017, 39: 2592--2602. Google Scholar

[60] Zhao H Y. Analysis on credibility of radar seeker hardware-in-the-loop simulation system. Ship Electron Eng, 2015, 35: 90--92. Google Scholar

[61] Lin S L, Li W, Ma P, et al. A new credibility assessment framework for training simulators. In: Proceedings of the 8th International Conference on Computer Modeling and Simulation (ICCMS), Canberra, 2017. 182--186. Google Scholar

[62] Zhang Z. Research on evaluation method for credibility of simulation system. Dissertation for Ph.D. Degree. Harbin: Harbin Institute of Technology, 2014. Google Scholar

[63] Liu F, Ma P, Yang M, et al. Research on credibility quantification of complex simulation systems. J Harbin Inst Technol. 2007, 39: 1--3. Google Scholar

[64] Fang K, Zhou Y C, Zhao K B. Validation method for simulation models with iteration operation. Syst Eng Electron, 2017, 39: 445--450. Google Scholar

[65] Fang K, Yang M, Wang Z C. The HITVICE VV&A environment. In: Proceedings of the Winter Simulation Conference, Orlando, 2005. 1220--1227. Google Scholar

[66] Ju R S, Yang M, Huang K D, et al. Summary of service oriented modeling and simulation. Syst Eng Electron, 2013, 35: 1539--1546. Google Scholar

[67] Li B H, Chai X D, Hou B C, et al. Networked modeling & simulation platform based on concept of cloud computing — cloud simulation platform. J Syst Simul, 2009, 21: 5292--5299. Google Scholar

  • Click to view Download high-quality image

    Figure 1

    Milestone of foreign simulation credibility assessment

  • Click to view Download high-quality image

    Figure 2

    Index system of credibility assessment for large-scale simulation system

  • Click to view Download high-quality image

    Figure 3

    Index system of credibility assessment for mixed continuous and discrete simulation system

  • Click to view Download high-quality image

    Figure 4

    Index system of credibility assessment for flight training simulator

  • Click to view Download high-quality image

    Figure 5

    Process model of credibility assessment for complex simulation system

  • Click to view Download high-quality image

    Figure 6

    Structure of supporting environment of credibility assessment for complex simulation system

  • Table 1   Classification of methods in assessment method database
    Category of assessment methods Subcategory of assessment methods
    Construction methods of index system Construction methods of tree-based index system
    Construction methods of net-based index system
    Preprocessing methods of data Preprocessing methods of static data
    Preprocessing methods of dynamic data
    Agreement analysis methods Agreement analysis methods of static data
    Agreement analysis methods of dynamic data
    Integration assessment methods Value assessment methods
    Sorting assessment methods