PREDICTING THE TYPE OF AUDITOR OPINION: STATISTICS, MACHINE LEARNING, OR A COMBINATION OF THE TWO?
Abstract
The goal of this study is to overcome the identified methodological limitations of prior studies aimed at predicting the type of auditor opinion and draw definite conclusions on the relative predictive performance of different predictive methods for this particular task. Predictive performance of twelve candidate models from the realms of statistics and machine learning is assessed separately for the two common real-life scenarios: a) when prior information on the client (i.e. types of audit opinion received in the past) is available and can be used in prediction, and b) when such information is not available (e.g. new companies). The results show that, in the first scenario, several methods from both realms achieve comparable predictive performance of around 0.89, as measured by the Area under the curve (AUC). In the second scenario, however, machine learning algorithms, particularly tree-based ones, such as random forest, perform significantly better, achieving the AUC of up to 0.79. Finally, we develop and assess the predictive performance of two hybrid models aimed at combining the strong points of both statistical (i.e. interpretability of results) and machine learning (i.e.handling a large number of predictors and improved accuracy) approaches. The complete procedure is demonstrated in a reproducible manner, using the largest empirical data set ever used in this stream of research, comprising 13,561 pairs of annual financial statements and the corresponding audit reports. The procedures described in this study allow audit and finance professionals around the globe to develop and test predictive models that will aid their procedures of audit planning and risk assessment.
References
Abad, D., Sánchez-Ballesta, J. P., & Yagüe, J. (2017). Audit opinions and information asymmetry in the stock market. Accounting & Finance, 57(2), 565–595. https://doi.org/10.1111/acfi.12175
ASB GAAS Section 315. (2013). GAAS section 315, Understanding the Entity and Its Environment and Assessing the Risks of Material Misstatement. Retrieved from https://www.aicpa.org/Research/Standards/AuditAttest/DownloadableDocuments/AU-C-00315.pdf
Ashbeck, E. L., & Bell, M. L. (2016). Single time point comparisons in longitudinal randomized controlled trials: power and bias in the presence of missing data. BMC Medical Research Methodology, 16(1), 43. https://doi.org/10.1186/s12874-016-0144-0
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. https://doi.org/10.1016/J.JML.2007.12.005
Bartov, E., Gul, F. A., & Tsui, J. S. L. (2000). Discretionary-accruals models and audit qualifications. Journal of Accounting and Economics, 30(3), 421–452. https://doi.org/10.1016/S0165-4101(01)00015-5
Bell, T. B., & Tabor, R. H. (1991). Empirical Analysis of Audit Uncertainty Qualifications. Journal of Accounting Research, 29(2), 350. https://doi.org/10.2307/2491053
Beneish, M. D. (1999). The Detection of Earnings Manipulation. Financial Analysts Journal, 55(5), 24–36. https://doi.org/10.2469/faj.v55.n5.2296
Bergmeir, C., & Benitez, J. M. (2012). Neural Networks in R Using the Stuttgart Neural Network Simulator: RSNNS. Journal of Statistical Software, 46(7), 1–26. Retrieved from http://www.jstatsoft.org/v46/i07/
Blandón, J. G., & Bosch, J. M. A. (2013). Audit firm tenure and qualified opinions: New evidence from Spain. Revista de Contabilidad, 16(2), 118-125. https://doi.org/10.1016/j.rcsar.2013.02.001
Bürkner, P.-C. (2017). brms: An R Package for Bayesian Multilevel Models Using Stan. Journal of Statistical Software, 80(1), 1–28. https://doi.org/10.18637/jss.v080.i01
Caramanis, C., & Spathis, C. (2006). Auditee and audit firm characteristics as determinants of audit qualifications.Managerial Auditing Journal, 21(9), 905-920. https://doi.org/10.1108/02686900610705000
Chen, T., He, T., Benesty, M., Khotilovich, V., & Tang, Y. (2017). xgboost: Extreme Gradient Boosting. Retrieved from https://cran.r-project.org/package=xgboost
Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
DeAngelo, L. E. (1986). Accounting numbers as market valuation substitutes: A study of management buyouts of public stockholders. Accounting Review, 61(3), 400–420. Retrieved from/han/GoogleScholar/www.jstor.org/stable/10.2307/247149
Dechow, P., Ge, W., & Schrand, C. (2010). Understanding earnings quality: A review of the proxies, their determinants and their consequences. Journal of Accounting and Economics, 50(2–3), 344–401. https://doi.org/10.1016/j.jacceco.2010.09.001
Dechow, P. M., Ge, W., Larson, C. R., & Sloan, R. G. (2011). Predicting Material Accounting Misstatements. Contemporary Accounting Research, 28(1), 17–82. https://doi.org/10.1111/j.1911-3846.2010.01041.x
Dechow, P. M., Sloan, R. G., & Sweeney, A. P. (1995). Detecting Earnings Management. The Accounting Review, 70(2), 193–225. https://doi.org/10.2307/248303
DeFond, M. L., & Jiambalvo, J. (1994). Debt covenant violation and manipulation of accruals. Journal of Accounting and Economics, 17(1–2), 145–176. https://doi.org/10.1016/0165-4101(94)90008-6
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics, 44(3), 837. https://doi.org/10.2307/2531595
Demler, O. V, Pencina, M. J., & D’Agostino, R. B. (2012). Misuse of DeLong test to compare AUCs for nested models. Statistics in Medicine, 31(23), 2577–2587. https://doi.org/10.1002/sim.5328
Deng, H. (2013). Guided Random Forest in the RRF Package. ArXiv, 1–2. Retrieved from http://arxiv.org/abs/1306.0237
Deng, H. (2014). Package ‘ inTrees .’ Retrieved from https://cran.r-project.org/package=inTrees
Deng, H., & Runger, G. (2012). Feature selection via regularized trees. In 2012 Annual International Joint Conference on Neural Networks, IJCNN 2012. Brisbane, Australia. https://doi.org/10.1109/IJCNN.2012.6252640
Deng, H., & Runger, G. (2013). Gene selection with guided regularized random forest. Pattern Recognition, 46(12), 3483–3489.
Dhaliwal, D. S., Liu, Q., Xie, H., & Zhang, J. (2014). Negative Press Coverage, Litigation Risk, and Audit Opinions in China. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2381696
Dopuch, N., Holthausen, R., & Leftwich, R. (1987). Predicting audit qualifications with financial and market variables. The Accounting Review, 62(3), 431–454.
Doumpos, M., Gaganis, C., & Pasiouras, F. (2005). Explaining qualifications in audit reports using a support vector machine methodology. Intelligent Systems in Accounting, Finance and Management, 13(4), 197–215. https://doi.org/10.1002/isaf.268
Fernández-Gámez, M. A., García-Lagos, F., & Sánchez-Serrano, J. R. (2016). Integrating corporate governance and financial variables for the identification of qualified audit opinions with neural networks. Neural Computing and Applications, 27(5), 1427–1444. https://doi.org/10.1007/s00521-015-1944-6
Francis, J. R., & Krishnan, J. J. (1999). Accounting Accruals and Auditor Reporting Conservatism. Contemporary Accounting Research, 16(1), 135–165. https://doi.org/10.1111/j.1911-3846.1999.tb00577.x
Gaganis, C., & Pasiouras, F. (2006). Auditing models for the detection of qualified audit opinions in the UK public services sector. International Journal of Accounting, Auditing and Performance Evaluation, 3(4),471. https://doi.org/10.1504/IJAAPE.2006.011207
Gaganis, C., Pasiouras, F., & Doumpos, M. (2007). Probabilistic neural networks for the identification of qualified audit opinions. Expert Systems with Applications, 32(1), 114–124. https://doi.org/10.1016/j.eswa.2005.11.003
Gaganis, C., Pasiouras, F., Spathis, C., & Zopounidis, C. (2007). A comparison of nearest neighbours, discriminant and logit models for auditing decisions. Intelligent Systems in Accounting, Finance and Management, 15(1–2), 23–40. https://doi.org/10.1002/isaf.283
Gassen, J., & Skaife, H. A. (2009). Can Audit Reforms Affect the Information Role of Audits? Evidence from the German Market. Contemporary Accounting Research, 26(3), 867–898. https://doi.org/10.1506/car.26.3.10
Gibbons, R. D., Hedeker, D., & DuToit, S. (2010). Advances in Analysis of Longitudinal Data. Annual Review of Clinical Psychology, 6(1), 79–107. https://doi.org/10.1146/annurev.clinpsy.032408.153550
Glancy, F. H., & Yadav, S. B. (2011). A computational model for financial reporting fraud detection. Decision Support Systems, 50(3), 595–601. https://doi.org/10.1016/j.dss.2010.08.010
Healy, P. M. (1985). The effect of bonus schemes on accounting decisions. Journal of Accounting and Economics, 7(1), 85–107.
Humpherys, S. L., Moffitt, K. C., Burns, M. B., Burgoon, J. K., & Felix, W. F. (2011). Identification of fraudulent financial statements using linguistic credibility analysis. Decision Support Systems, 50(3), 585–594. https://doi.org/10.1016/j.dss.2010.08.009
IAASB ISA 315. (2013). ISA 315, Identifying and Assessing the Risks of Material Misstatement through Understanding the Entity and Its Environment. Retrieved from https://www.iaasb.org/system/files/meetings/files/20130415-IAASB-Agenda_Item_5-D_Disclosures - ISA 315 %28Revised%29 for reference ONLY.pdf
IAASB ISA 570. (2013). ISA 570, Going Concern. Retrieved from http://www.ifac.org/system/files/downloads/a031-2010-iaasb-handbook-isa-570.pdf
Jones, J. J. (1991). Earnings Management During Import Relief Investigations. Journal of Accounting Research, 29(2), 193–228. https://doi.org/10.2307/2491047
Jones, K. L., Krishnan, G. V., & Melendrez, K. D. (2008). Do Models of Discretionary Accruals Detect Actual Cases of Fraudulent and Restated Earnings? An Empirical Analysis. Contemporary Accounting Research, 25(2), 499–531. https://doi.org/10.1506/car.25.2.8
Karatzoglou, A., Smola, A., Hornik, K., & Zeileis, A. (2004). kernlab - An S4 Package for Kernel Methods in R. Journal of Statistical Software, 11(9), 1–20. Retrieved from http://www.jstatsoft.org/v11/i09/
Kinney, W. R., & McDaniel, L. S. (1989). Characteristics of firms correcting previously reported quarterly earnings. Journal of Accounting and Economics, 11(1), 71–93. https://doi.org/10.1016/0165-4101(89)90014-1
Kirkos, E., Spathis, C., Nanopoulos, A., & Manolopoulos, Y. (2007). Identifying Qualified Auditors’ Opinions: A Data Mining Approach. Journal of Emerging Technologies in Accounting, 4, 183–197.
Krishnan, J., & Krishnan, J. (1996). The Role of Economic Trade-Offs in the Audit Opinion Decision: An Empirical Analysis. Journal of Accounting, Auditing & Finance, 11(4), 565–586. https://doi.org/10.1177/0148558X9601100403
Krishnan, J., Krishnan, J., & Stephens, R. G. (1996). The Simultaneous Relation Between Auditor Switching andAudit Opinion: An Empirical Analysis. Accounting & Business Research (Wolters Kluwer UK), 26(3), 224–236.
Kuhn, M. (2017). caret: Classification and Regression Training. Retrieved from https://cran.r-project.org/package=caret
Kuhn, M., & Ross, Q. (2017). C50: C5.0 Decision Trees and Rule-Based Models. Retrieved from https://cran.rproject.org/package=C50
Laitinen, E. K., & Laitinen, T. (1998). Qualified audit reports in Finland: evidence from large companies. European Accounting Review, 7(4), 639–653. https://doi.org/10.1080/096381898336231
Liaw, A., & Wiener, M. (2002). Classification and Regression by randomForest. R News, 2(3), 18–22. Retrieved from http://cran.r-project.org/doc/Rnews/
Maggina, A., & Tsaklanganos, A. A. (2011). Predicting audit opinions evidence from the athens stock exchange. Journal of Applied Business Research, 27(4), 53–68.
Monroe, G. S., & Teh, S. T. (2009). Predicting uncertainty audit qualifications in Australia using publicly available information. Accounting & Finance, 33(2), 79–106. https://doi.org/10.1111/j.1467-629X.1993.tb00200.x
Mutchler, J. F., & Hopwood, W. (1997). The Influence of Contrary Information and Mitigating Factors on Audit Opinion Decisions on Bankrupt Companies. Journal of Accounting Research, 35(2), 295–310. https://doi.org/10.2307/2491367
Ngai, E. W. T., Hu, Y., Wong, Y. H., Chen, Y., & Sun, X. (2011). The application of data mining techniques in financial fraud detection: A classification framework and an academic review of literature. Decision Support Systems, 50(3), 559-569. https://doi.org/10.1016/j.dss.2010.08.006
Pedersen, A. B., Mikkelsen, E. M., Cronin-Fenton, D., Kristensen, N. R., Pham, T. M., Pedersen, L., & Petersen, I.(2017). Missing data and multiple imputation in clinical epidemiological research. Clinical Epidemiology, 9, 157-166. https://doi.org/10.2147/CLEP.S129785
Perols, J. (2011). Financial Statement Fraud Detection: An Analysis of Statistical and Machine Learning Algorithms. AUDITING: A Journal of Practice & Theory, 30(2), 19–50. https://doi.org/10.2308/ajpt-50009
Perols, J. L., Bowen, R. M., Zimmermann, C., & Samba, B. (2017). Finding needles in a haystack: Using data analytics to improve fraud prediction. In Accounting Review (Vol. 92, pp. 221–245). https://doi.org/10.2308/accr-51562
Pourheydari, O., Nezamabadi-Pour, H., & Aazami, Z. (2012). Identifying qualified audit opinions by artificial neural networks. African Journal of Business Management, 6(44), 11077–11087. https://doi.org/10.5897/AJBM12.855
R Core Team. (2017). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.r-project.org
Ridgeway, G. (2017). gbm: Generalized Boosted Regression Models. Retrieved from https://cran.r-project.org/package=gbm
Ruiz-Barbadillo, E., Gómez-Aguilar, N., De Fuentes-Barberá, C., & García-Benau, M. A. (2004). Audit quality and the going-concern decision-making process: Spanish evidence. European Accounting Review, 13(4), 597–620. https://doi.org/10.1080/0963818042000216820
Saif, S. M., Sarikhani, M., & Ebrahimi, F. (2012). Finding rules for audit opinions prediction through data mining methods. European Online Journal of Natural and Social Sciences, 1(2), 28–36.
Saif, S. M., Sarikhani, M., & Ebrahimi, F. (2013). An Expert System with Neural Network and Decision Tree for Predicting Audit Opinions. IAES International Journal of Artificial Intelligence (IJ-AI), 2(4), 151–158. Retrieved from http://iaesjournal.com/online/index.php/IJAI/article/view/3950
Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. https://doi.org/10.1037/1082-989X.7.2.147
Spathis, C., Doumpos, M., & Zopounidis, C. (2003). Using client performance measures to identify pre-engagement factors associated with qualified audit reports in Greece. The International Journal of Accounting, 38(3), 267–284. https://doi.org/10.1016/S0020-7063(03)00047-5
Stice, J. D. (1991). Using Financial and Market Information to Identify Pre-Engagement Factors Associated with Lawsuits against Auditors. The Accounting Review, 66(3), 516–533.
Venables, W. N., & Ripley, B. D. (2002). Modern Applied Statistics with S (4th ed.). New York: Springer. Retrieved from http://www.stats.ox.ac.uk/pub/MASS4
Yasar, A., Yakut, E., & Gutnu, M. M. (2015). Predicting Qualified Audit Opinions Using Financial Ratios : Evidence from the Istanbul Stock Exchange. International Journal of Business and Social Science, 6(8), 57–67.
Yeh, C.-C., Chi, D.-J., & Lin, Y.-R. (2014). Going-concern prediction using hybrid random forests and rough set approach. Information Sciences, 254, 98-110. https://doi.org/10.1016/j.ins.2013.07.011
Zdolšek, D., Jagrič, T., & Odar, M. (2015). Identification of auditor’s report qualifications: An empirical analysis for Slovenia. Economic Research-Ekonomska Istrazivanja, 28(1), 994–1005. https://doi.org/10.1080/1331677X.2015.1101960
Zhou, W., & Kapoor, G. (2011). Detecting evolutionary financial statement fraud. Decision Support Systems, 50(3), 570–575. https://doi.org/10.1016/j.dss.2010.08.007
