PREDICTING THE TYPE OF AUDITOR OPINION: STATISTICS, MACHINE LEARNING, OR A COMBINATION OF THE TWO?

  • Nemanja Stanišić Singidunum University
  • Tijana Radojević Singidunum University
  • Nenad Stanić Singidunum University
Keywords: ensembles, guided regularized random forest, random forest, generalized linear mixed models, financial reports, auditor opinion,

Abstract


The goal of this study is to overcome the identified methodological limitations of prior studies aimed at predicting the type of auditor opinion and draw definite conclusions on the relative predictive performance of different predictive methods for this particular task. Predictive performance of twelve candidate models from the realms of statistics and machine learning is assessed separately for the two common real-life scenarios: a) when prior information on the client (i.e. types of audit opinion received in the past) is available and can be used in prediction, and b) when such information is not available (e.g. new companies). The results show that, in the first scenario, several methods from both realms achieve comparable predictive performance of around 0.89, as measured by the Area under the curve (AUC). In the second scenario, however, machine learning algorithms, particularly tree-based ones, such as random forest, perform significantly better, achieving the AUC of up to 0.79. Finally, we develop and assess the predictive performance of two hybrid models aimed at combining the strong points of both statistical (i.e. interpretability of results) and machine learning (i.e.handling a large number of predictors and improved accuracy) approaches. The complete procedure is demonstrated in a reproducible manner, using the largest empirical data set ever used in this stream of research, comprising 13,561 pairs of annual financial statements and the corresponding audit reports. The procedures described in this study allow audit and finance professionals around the globe to develop and test predictive models that will aid their procedures of audit planning and risk assessment.

References

Abad, D., Sánchez-Ballesta, J. P., & Yagüe, J. (2017). Audit opinions and information asymmetry in the stock market. Accounting & Finance, 57(2), 565–595. https://doi.org/10.1111/acfi.12175

ASB GAAS Section 315. (2013). GAAS section 315, Understanding the Entity and Its Environment and Assessing the Risks of Material Misstatement. Retrieved from https://www.aicpa.org/Research/Standards/AuditAttest/DownloadableDocuments/AU-C-00315.pdf

Ashbeck, E. L., & Bell, M. L. (2016). Single time point comparisons in longitudinal randomized controlled trials: power and bias in the presence of missing data. BMC Medical Research Methodology, 16(1), 43. https://doi.org/10.1186/s12874-016-0144-0

Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. https://doi.org/10.1016/J.JML.2007.12.005

Bartov, E., Gul, F. A., & Tsui, J. S. L. (2000). Discretionary-accruals models and audit qualifications. Journal of Accounting and Economics, 30(3), 421–452. https://doi.org/10.1016/S0165-4101(01)00015-5

Bell, T. B., & Tabor, R. H. (1991). Empirical Analysis of Audit Uncertainty Qualifications. Journal of Accounting Research, 29(2), 350. https://doi.org/10.2307/2491053

Beneish, M. D. (1999). The Detection of Earnings Manipulation. Financial Analysts Journal, 55(5), 24–36. https://doi.org/10.2469/faj.v55.n5.2296

Bergmeir, C., & Benitez, J. M. (2012). Neural Networks in R Using the Stuttgart Neural Network Simulator: RSNNS. Journal of Statistical Software, 46(7), 1–26. Retrieved from http://www.jstatsoft.org/v46/i07/

Blandón, J. G., & Bosch, J. M. A. (2013). Audit firm tenure and qualified opinions: New evidence from Spain. Revista de Contabilidad, 16(2), 118-125. https://doi.org/10.1016/j.rcsar.2013.02.001

Bürkner, P.-C. (2017). brms: An R Package for Bayesian Multilevel Models Using Stan. Journal of Statistical Software, 80(1), 1–28. https://doi.org/10.18637/jss.v080.i01

Caramanis, C., & Spathis, C. (2006). Auditee and audit firm characteristics as determinants of audit qualifications.Managerial Auditing Journal, 21(9), 905-920. https://doi.org/10.1108/02686900610705000

Chen, T., He, T., Benesty, M., Khotilovich, V., & Tang, Y. (2017). xgboost: Extreme Gradient Boosting. Retrieved from https://cran.r-project.org/package=xgboost

Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104

DeAngelo, L. E. (1986). Accounting numbers as market valuation substitutes: A study of management buyouts of public stockholders. Accounting Review, 61(3), 400–420. Retrieved from/han/GoogleScholar/www.jstor.org/stable/10.2307/247149

Dechow, P., Ge, W., & Schrand, C. (2010). Understanding earnings quality: A review of the proxies, their determinants and their consequences. Journal of Accounting and Economics, 50(2–3), 344–401. https://doi.org/10.1016/j.jacceco.2010.09.001

Dechow, P. M., Ge, W., Larson, C. R., & Sloan, R. G. (2011). Predicting Material Accounting Misstatements. Contemporary Accounting Research, 28(1), 17–82. https://doi.org/10.1111/j.1911-3846.2010.01041.x

Dechow, P. M., Sloan, R. G., & Sweeney, A. P. (1995). Detecting Earnings Management. The Accounting Review, 70(2), 193–225. https://doi.org/10.2307/248303

DeFond, M. L., & Jiambalvo, J. (1994). Debt covenant violation and manipulation of accruals. Journal of Accounting and Economics, 17(1–2), 145–176. https://doi.org/10.1016/0165-4101(94)90008-6

DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics, 44(3), 837. https://doi.org/10.2307/2531595

Demler, O. V, Pencina, M. J., & D’Agostino, R. B. (2012). Misuse of DeLong test to compare AUCs for nested models. Statistics in Medicine, 31(23), 2577–2587. https://doi.org/10.1002/sim.5328

Deng, H. (2013). Guided Random Forest in the RRF Package. ArXiv, 1–2. Retrieved from http://arxiv.org/abs/1306.0237

Deng, H. (2014). Package ‘ inTrees .’ Retrieved from https://cran.r-project.org/package=inTrees

Deng, H., & Runger, G. (2012). Feature selection via regularized trees. In 2012 Annual International Joint Conference on Neural Networks, IJCNN 2012. Brisbane, Australia. https://doi.org/10.1109/IJCNN.2012.6252640

Deng, H., & Runger, G. (2013). Gene selection with guided regularized random forest. Pattern Recognition, 46(12), 3483–3489.

Dhaliwal, D. S., Liu, Q., Xie, H., & Zhang, J. (2014). Negative Press Coverage, Litigation Risk, and Audit Opinions in China. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2381696

Dopuch, N., Holthausen, R., & Leftwich, R. (1987). Predicting audit qualifications with financial and market variables. The Accounting Review, 62(3), 431–454.

Doumpos, M., Gaganis, C., & Pasiouras, F. (2005). Explaining qualifications in audit reports using a support vector machine methodology. Intelligent Systems in Accounting, Finance and Management, 13(4), 197–215. https://doi.org/10.1002/isaf.268

Fernández-Gámez, M. A., García-Lagos, F., & Sánchez-Serrano, J. R. (2016). Integrating corporate governance and financial variables for the identification of qualified audit opinions with neural networks. Neural Computing and Applications, 27(5), 1427–1444. https://doi.org/10.1007/s00521-015-1944-6

Francis, J. R., & Krishnan, J. J. (1999). Accounting Accruals and Auditor Reporting Conservatism. Contemporary Accounting Research, 16(1), 135–165. https://doi.org/10.1111/j.1911-3846.1999.tb00577.x

Gaganis, C., & Pasiouras, F. (2006). Auditing models for the detection of qualified audit opinions in the UK public services sector. International Journal of Accounting, Auditing and Performance Evaluation, 3(4),471. https://doi.org/10.1504/IJAAPE.2006.011207

Gaganis, C., Pasiouras, F., & Doumpos, M. (2007). Probabilistic neural networks for the identification of qualified audit opinions. Expert Systems with Applications, 32(1), 114–124. https://doi.org/10.1016/j.eswa.2005.11.003

Gaganis, C., Pasiouras, F., Spathis, C., & Zopounidis, C. (2007). A comparison of nearest neighbours, discriminant and logit models for auditing decisions. Intelligent Systems in Accounting, Finance and Management, 15(1–2), 23–40. https://doi.org/10.1002/isaf.283

Gassen, J., & Skaife, H. A. (2009). Can Audit Reforms Affect the Information Role of Audits? Evidence from the German Market. Contemporary Accounting Research, 26(3), 867–898. https://doi.org/10.1506/car.26.3.10

Gibbons, R. D., Hedeker, D., & DuToit, S. (2010). Advances in Analysis of Longitudinal Data. Annual Review of Clinical Psychology, 6(1), 79–107. https://doi.org/10.1146/annurev.clinpsy.032408.153550

Glancy, F. H., & Yadav, S. B. (2011). A computational model for financial reporting fraud detection. Decision Support Systems, 50(3), 595–601. https://doi.org/10.1016/j.dss.2010.08.010

Healy, P. M. (1985). The effect of bonus schemes on accounting decisions. Journal of Accounting and Economics, 7(1), 85–107.

Humpherys, S. L., Moffitt, K. C., Burns, M. B., Burgoon, J. K., & Felix, W. F. (2011). Identification of fraudulent financial statements using linguistic credibility analysis. Decision Support Systems, 50(3), 585–594. https://doi.org/10.1016/j.dss.2010.08.009

IAASB ISA 315. (2013). ISA 315, Identifying and Assessing the Risks of Material Misstatement through Understanding the Entity and Its Environment. Retrieved from https://www.iaasb.org/system/files/meetings/files/20130415-IAASB-Agenda_Item_5-D_Disclosures - ISA 315 %28Revised%29 for reference ONLY.pdf

IAASB ISA 570. (2013). ISA 570, Going Concern. Retrieved from http://www.ifac.org/system/files/downloads/a031-2010-iaasb-handbook-isa-570.pdf

Jones, J. J. (1991). Earnings Management During Import Relief Investigations. Journal of Accounting Research, 29(2), 193–228. https://doi.org/10.2307/2491047

Jones, K. L., Krishnan, G. V., & Melendrez, K. D. (2008). Do Models of Discretionary Accruals Detect Actual Cases of Fraudulent and Restated Earnings? An Empirical Analysis. Contemporary Accounting Research, 25(2), 499–531. https://doi.org/10.1506/car.25.2.8

Karatzoglou, A., Smola, A., Hornik, K., & Zeileis, A. (2004). kernlab - An S4 Package for Kernel Methods in R. Journal of Statistical Software, 11(9), 1–20. Retrieved from http://www.jstatsoft.org/v11/i09/

Kinney, W. R., & McDaniel, L. S. (1989). Characteristics of firms correcting previously reported quarterly earnings. Journal of Accounting and Economics, 11(1), 71–93. https://doi.org/10.1016/0165-4101(89)90014-1

Kirkos, E., Spathis, C., Nanopoulos, A., & Manolopoulos, Y. (2007). Identifying Qualified Auditors’ Opinions: A Data Mining Approach. Journal of Emerging Technologies in Accounting, 4, 183–197.

Krishnan, J., & Krishnan, J. (1996). The Role of Economic Trade-Offs in the Audit Opinion Decision: An Empirical Analysis. Journal of Accounting, Auditing & Finance, 11(4), 565–586. https://doi.org/10.1177/0148558X9601100403

Krishnan, J., Krishnan, J., & Stephens, R. G. (1996). The Simultaneous Relation Between Auditor Switching andAudit Opinion: An Empirical Analysis. Accounting & Business Research (Wolters Kluwer UK), 26(3), 224–236.

Kuhn, M. (2017). caret: Classification and Regression Training. Retrieved from https://cran.r-project.org/package=caret

Kuhn, M., & Ross, Q. (2017). C50: C5.0 Decision Trees and Rule-Based Models. Retrieved from https://cran.rproject.org/package=C50

Laitinen, E. K., & Laitinen, T. (1998). Qualified audit reports in Finland: evidence from large companies. European Accounting Review, 7(4), 639–653. https://doi.org/10.1080/096381898336231

Liaw, A., & Wiener, M. (2002). Classification and Regression by randomForest. R News, 2(3), 18–22. Retrieved from http://cran.r-project.org/doc/Rnews/

Maggina, A., & Tsaklanganos, A. A. (2011). Predicting audit opinions evidence from the athens stock exchange. Journal of Applied Business Research, 27(4), 53–68.

Monroe, G. S., & Teh, S. T. (2009). Predicting uncertainty audit qualifications in Australia using publicly available information. Accounting & Finance, 33(2), 79–106. https://doi.org/10.1111/j.1467-629X.1993.tb00200.x

Mutchler, J. F., & Hopwood, W. (1997). The Influence of Contrary Information and Mitigating Factors on Audit Opinion Decisions on Bankrupt Companies. Journal of Accounting Research, 35(2), 295–310. https://doi.org/10.2307/2491367

Ngai, E. W. T., Hu, Y., Wong, Y. H., Chen, Y., & Sun, X. (2011). The application of data mining techniques in financial fraud detection: A classification framework and an academic review of literature. Decision Support Systems, 50(3), 559-569. https://doi.org/10.1016/j.dss.2010.08.006

Pedersen, A. B., Mikkelsen, E. M., Cronin-Fenton, D., Kristensen, N. R., Pham, T. M., Pedersen, L., & Petersen, I.(2017). Missing data and multiple imputation in clinical epidemiological research. Clinical Epidemiology, 9, 157-166. https://doi.org/10.2147/CLEP.S129785

Perols, J. (2011). Financial Statement Fraud Detection: An Analysis of Statistical and Machine Learning Algorithms. AUDITING: A Journal of Practice & Theory, 30(2), 19–50. https://doi.org/10.2308/ajpt-50009

Perols, J. L., Bowen, R. M., Zimmermann, C., & Samba, B. (2017). Finding needles in a haystack: Using data analytics to improve fraud prediction. In Accounting Review (Vol. 92, pp. 221–245). https://doi.org/10.2308/accr-51562

Pourheydari, O., Nezamabadi-Pour, H., & Aazami, Z. (2012). Identifying qualified audit opinions by artificial neural networks. African Journal of Business Management, 6(44), 11077–11087. https://doi.org/10.5897/AJBM12.855

R Core Team. (2017). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.r-project.org

Ridgeway, G. (2017). gbm: Generalized Boosted Regression Models. Retrieved from https://cran.r-project.org/package=gbm

Ruiz-Barbadillo, E., Gómez-Aguilar, N., De Fuentes-Barberá, C., & García-Benau, M. A. (2004). Audit quality and the going-concern decision-making process: Spanish evidence. European Accounting Review, 13(4), 597–620. https://doi.org/10.1080/0963818042000216820

Saif, S. M., Sarikhani, M., & Ebrahimi, F. (2012). Finding rules for audit opinions prediction through data mining methods. European Online Journal of Natural and Social Sciences, 1(2), 28–36.

Saif, S. M., Sarikhani, M., & Ebrahimi, F. (2013). An Expert System with Neural Network and Decision Tree for Predicting Audit Opinions. IAES International Journal of Artificial Intelligence (IJ-AI), 2(4), 151–158. Retrieved from http://iaesjournal.com/online/index.php/IJAI/article/view/3950

Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. https://doi.org/10.1037/1082-989X.7.2.147

Spathis, C., Doumpos, M., & Zopounidis, C. (2003). Using client performance measures to identify pre-engagement factors associated with qualified audit reports in Greece. The International Journal of Accounting, 38(3), 267–284. https://doi.org/10.1016/S0020-7063(03)00047-5

Stice, J. D. (1991). Using Financial and Market Information to Identify Pre-Engagement Factors Associated with Lawsuits against Auditors. The Accounting Review, 66(3), 516–533.

Venables, W. N., & Ripley, B. D. (2002). Modern Applied Statistics with S (4th ed.). New York: Springer. Retrieved from http://www.stats.ox.ac.uk/pub/MASS4

Yasar, A., Yakut, E., & Gutnu, M. M. (2015). Predicting Qualified Audit Opinions Using Financial Ratios : Evidence from the Istanbul Stock Exchange. International Journal of Business and Social Science, 6(8), 57–67.

Yeh, C.-C., Chi, D.-J., & Lin, Y.-R. (2014). Going-concern prediction using hybrid random forests and rough set approach. Information Sciences, 254, 98-110. https://doi.org/10.1016/j.ins.2013.07.011

Zdolšek, D., Jagrič, T., & Odar, M. (2015). Identification of auditor’s report qualifications: An empirical analysis for Slovenia. Economic Research-Ekonomska Istrazivanja, 28(1), 994–1005. https://doi.org/10.1080/1331677X.2015.1101960

Zhou, W., & Kapoor, G. (2011). Detecting evolutionary financial statement fraud. Decision Support Systems, 50(3), 570–575. https://doi.org/10.1016/j.dss.2010.08.007

Published
2019/10/15
Section
Original Scientific Paper