• Nemanja Stanišić Singidunum University
  • Tijana Radojević Singidunum University
  • Nenad Stanić Singidunum University
Keywords: ensembles, guided regularized random forest, random forest, generalized linear mixed models, financial reports, auditor opinion,


The goal of this study is to overcome the identified methodological limitations of prior studies aimed at predicting the type of auditor opinion and draw definite conclusions on the relative predictive performance of different predictive methods for this particular task. Predictive performance of twelve candidate models from the realms of statistics and machine learning is assessed separately for the two common real-life scenarios: a) when prior information on the client (i.e. types of audit opinion received in the past) is available and can be used in prediction, and b) when such information is not available (e.g. new companies). The results show that, in the first scenario, several methods from both realms achieve comparable predictive performance of around 0.89, as measured by the Area under the curve (AUC). In the second scenario, however, machine learning algorithms, particularly tree-based ones, such as random forest, perform significantly better, achieving the AUC of up to 0.79. Finally, we develop and assess the predictive performance of two hybrid models aimed at combining the strong points of both statistical (i.e. interpretability of results) and machine learning (i.e.handling a large number of predictors and improved accuracy) approaches. The complete procedure is demonstrated in a reproducible manner, using the largest empirical data set ever used in this stream of research, comprising 13,561 pairs of annual financial statements and the corresponding audit reports. The procedures described in this study allow audit and finance professionals around the globe to develop and test predictive models that will aid their procedures of audit planning and risk assessment.


Abad, D., Sánchez-Ballesta, J. P., & Yagüe, J. (2017). Audit opinions and information asymmetry in the stock market. Accounting & Finance, 57(2), 565–595.

ASB GAAS Section 315. (2013). GAAS section 315, Understanding the Entity and Its Environment and Assessing the Risks of Material Misstatement. Retrieved from

Ashbeck, E. L., & Bell, M. L. (2016). Single time point comparisons in longitudinal randomized controlled trials: power and bias in the presence of missing data. BMC Medical Research Methodology, 16(1), 43.

Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412.

Bartov, E., Gul, F. A., & Tsui, J. S. L. (2000). Discretionary-accruals models and audit qualifications. Journal of Accounting and Economics, 30(3), 421–452.

Bell, T. B., & Tabor, R. H. (1991). Empirical Analysis of Audit Uncertainty Qualifications. Journal of Accounting Research, 29(2), 350.

Beneish, M. D. (1999). The Detection of Earnings Manipulation. Financial Analysts Journal, 55(5), 24–36.

Bergmeir, C., & Benitez, J. M. (2012). Neural Networks in R Using the Stuttgart Neural Network Simulator: RSNNS. Journal of Statistical Software, 46(7), 1–26. Retrieved from

Blandón, J. G., & Bosch, J. M. A. (2013). Audit firm tenure and qualified opinions: New evidence from Spain. Revista de Contabilidad, 16(2), 118-125.

Bürkner, P.-C. (2017). brms: An R Package for Bayesian Multilevel Models Using Stan. Journal of Statistical Software, 80(1), 1–28.

Caramanis, C., & Spathis, C. (2006). Auditee and audit firm characteristics as determinants of audit qualifications.Managerial Auditing Journal, 21(9), 905-920.

Chen, T., He, T., Benesty, M., Khotilovich, V., & Tang, Y. (2017). xgboost: Extreme Gradient Boosting. Retrieved from

Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46.

DeAngelo, L. E. (1986). Accounting numbers as market valuation substitutes: A study of management buyouts of public stockholders. Accounting Review, 61(3), 400–420. Retrieved from/han/GoogleScholar/

Dechow, P., Ge, W., & Schrand, C. (2010). Understanding earnings quality: A review of the proxies, their determinants and their consequences. Journal of Accounting and Economics, 50(2–3), 344–401.

Dechow, P. M., Ge, W., Larson, C. R., & Sloan, R. G. (2011). Predicting Material Accounting Misstatements. Contemporary Accounting Research, 28(1), 17–82.

Dechow, P. M., Sloan, R. G., & Sweeney, A. P. (1995). Detecting Earnings Management. The Accounting Review, 70(2), 193–225.

DeFond, M. L., & Jiambalvo, J. (1994). Debt covenant violation and manipulation of accruals. Journal of Accounting and Economics, 17(1–2), 145–176.

DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics, 44(3), 837.

Demler, O. V, Pencina, M. J., & D’Agostino, R. B. (2012). Misuse of DeLong test to compare AUCs for nested models. Statistics in Medicine, 31(23), 2577–2587.

Deng, H. (2013). Guided Random Forest in the RRF Package. ArXiv, 1–2. Retrieved from

Deng, H. (2014). Package ‘ inTrees .’ Retrieved from

Deng, H., & Runger, G. (2012). Feature selection via regularized trees. In 2012 Annual International Joint Conference on Neural Networks, IJCNN 2012. Brisbane, Australia.

Deng, H., & Runger, G. (2013). Gene selection with guided regularized random forest. Pattern Recognition, 46(12), 3483–3489.

Dhaliwal, D. S., Liu, Q., Xie, H., & Zhang, J. (2014). Negative Press Coverage, Litigation Risk, and Audit Opinions in China. SSRN Electronic Journal.

Dopuch, N., Holthausen, R., & Leftwich, R. (1987). Predicting audit qualifications with financial and market variables. The Accounting Review, 62(3), 431–454.

Doumpos, M., Gaganis, C., & Pasiouras, F. (2005). Explaining qualifications in audit reports using a support vector machine methodology. Intelligent Systems in Accounting, Finance and Management, 13(4), 197–215.

Fernández-Gámez, M. A., García-Lagos, F., & Sánchez-Serrano, J. R. (2016). Integrating corporate governance and financial variables for the identification of qualified audit opinions with neural networks. Neural Computing and Applications, 27(5), 1427–1444.

Francis, J. R., & Krishnan, J. J. (1999). Accounting Accruals and Auditor Reporting Conservatism. Contemporary Accounting Research, 16(1), 135–165.

Gaganis, C., & Pasiouras, F. (2006). Auditing models for the detection of qualified audit opinions in the UK public services sector. International Journal of Accounting, Auditing and Performance Evaluation, 3(4),471.

Gaganis, C., Pasiouras, F., & Doumpos, M. (2007). Probabilistic neural networks for the identification of qualified audit opinions. Expert Systems with Applications, 32(1), 114–124.

Gaganis, C., Pasiouras, F., Spathis, C., & Zopounidis, C. (2007). A comparison of nearest neighbours, discriminant and logit models for auditing decisions. Intelligent Systems in Accounting, Finance and Management, 15(1–2), 23–40.

Gassen, J., & Skaife, H. A. (2009). Can Audit Reforms Affect the Information Role of Audits? Evidence from the German Market. Contemporary Accounting Research, 26(3), 867–898.

Gibbons, R. D., Hedeker, D., & DuToit, S. (2010). Advances in Analysis of Longitudinal Data. Annual Review of Clinical Psychology, 6(1), 79–107.

Glancy, F. H., & Yadav, S. B. (2011). A computational model for financial reporting fraud detection. Decision Support Systems, 50(3), 595–601.

Healy, P. M. (1985). The effect of bonus schemes on accounting decisions. Journal of Accounting and Economics, 7(1), 85–107.

Humpherys, S. L., Moffitt, K. C., Burns, M. B., Burgoon, J. K., & Felix, W. F. (2011). Identification of fraudulent financial statements using linguistic credibility analysis. Decision Support Systems, 50(3), 585–594.

IAASB ISA 315. (2013). ISA 315, Identifying and Assessing the Risks of Material Misstatement through Understanding the Entity and Its Environment. Retrieved from - ISA 315 %28Revised%29 for reference ONLY.pdf

IAASB ISA 570. (2013). ISA 570, Going Concern. Retrieved from

Jones, J. J. (1991). Earnings Management During Import Relief Investigations. Journal of Accounting Research, 29(2), 193–228.

Jones, K. L., Krishnan, G. V., & Melendrez, K. D. (2008). Do Models of Discretionary Accruals Detect Actual Cases of Fraudulent and Restated Earnings? An Empirical Analysis. Contemporary Accounting Research, 25(2), 499–531.

Karatzoglou, A., Smola, A., Hornik, K., & Zeileis, A. (2004). kernlab - An S4 Package for Kernel Methods in R. Journal of Statistical Software, 11(9), 1–20. Retrieved from

Kinney, W. R., & McDaniel, L. S. (1989). Characteristics of firms correcting previously reported quarterly earnings. Journal of Accounting and Economics, 11(1), 71–93.

Kirkos, E., Spathis, C., Nanopoulos, A., & Manolopoulos, Y. (2007). Identifying Qualified Auditors’ Opinions: A Data Mining Approach. Journal of Emerging Technologies in Accounting, 4, 183–197.

Krishnan, J., & Krishnan, J. (1996). The Role of Economic Trade-Offs in the Audit Opinion Decision: An Empirical Analysis. Journal of Accounting, Auditing & Finance, 11(4), 565–586.

Krishnan, J., Krishnan, J., & Stephens, R. G. (1996). The Simultaneous Relation Between Auditor Switching andAudit Opinion: An Empirical Analysis. Accounting & Business Research (Wolters Kluwer UK), 26(3), 224–236.

Kuhn, M. (2017). caret: Classification and Regression Training. Retrieved from

Kuhn, M., & Ross, Q. (2017). C50: C5.0 Decision Trees and Rule-Based Models. Retrieved from

Laitinen, E. K., & Laitinen, T. (1998). Qualified audit reports in Finland: evidence from large companies. European Accounting Review, 7(4), 639–653.

Liaw, A., & Wiener, M. (2002). Classification and Regression by randomForest. R News, 2(3), 18–22. Retrieved from

Maggina, A., & Tsaklanganos, A. A. (2011). Predicting audit opinions evidence from the athens stock exchange. Journal of Applied Business Research, 27(4), 53–68.

Monroe, G. S., & Teh, S. T. (2009). Predicting uncertainty audit qualifications in Australia using publicly available information. Accounting & Finance, 33(2), 79–106.

Mutchler, J. F., & Hopwood, W. (1997). The Influence of Contrary Information and Mitigating Factors on Audit Opinion Decisions on Bankrupt Companies. Journal of Accounting Research, 35(2), 295–310.

Ngai, E. W. T., Hu, Y., Wong, Y. H., Chen, Y., & Sun, X. (2011). The application of data mining techniques in financial fraud detection: A classification framework and an academic review of literature. Decision Support Systems, 50(3), 559-569.

Pedersen, A. B., Mikkelsen, E. M., Cronin-Fenton, D., Kristensen, N. R., Pham, T. M., Pedersen, L., & Petersen, I.(2017). Missing data and multiple imputation in clinical epidemiological research. Clinical Epidemiology, 9, 157-166.

Perols, J. (2011). Financial Statement Fraud Detection: An Analysis of Statistical and Machine Learning Algorithms. AUDITING: A Journal of Practice & Theory, 30(2), 19–50.

Perols, J. L., Bowen, R. M., Zimmermann, C., & Samba, B. (2017). Finding needles in a haystack: Using data analytics to improve fraud prediction. In Accounting Review (Vol. 92, pp. 221–245).

Pourheydari, O., Nezamabadi-Pour, H., & Aazami, Z. (2012). Identifying qualified audit opinions by artificial neural networks. African Journal of Business Management, 6(44), 11077–11087.

R Core Team. (2017). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from

Ridgeway, G. (2017). gbm: Generalized Boosted Regression Models. Retrieved from

Ruiz-Barbadillo, E., Gómez-Aguilar, N., De Fuentes-Barberá, C., & García-Benau, M. A. (2004). Audit quality and the going-concern decision-making process: Spanish evidence. European Accounting Review, 13(4), 597–620.

Saif, S. M., Sarikhani, M., & Ebrahimi, F. (2012). Finding rules for audit opinions prediction through data mining methods. European Online Journal of Natural and Social Sciences, 1(2), 28–36.

Saif, S. M., Sarikhani, M., & Ebrahimi, F. (2013). An Expert System with Neural Network and Decision Tree for Predicting Audit Opinions. IAES International Journal of Artificial Intelligence (IJ-AI), 2(4), 151–158. Retrieved from

Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177.

Spathis, C., Doumpos, M., & Zopounidis, C. (2003). Using client performance measures to identify pre-engagement factors associated with qualified audit reports in Greece. The International Journal of Accounting, 38(3), 267–284.

Stice, J. D. (1991). Using Financial and Market Information to Identify Pre-Engagement Factors Associated with Lawsuits against Auditors. The Accounting Review, 66(3), 516–533.

Venables, W. N., & Ripley, B. D. (2002). Modern Applied Statistics with S (4th ed.). New York: Springer. Retrieved from

Yasar, A., Yakut, E., & Gutnu, M. M. (2015). Predicting Qualified Audit Opinions Using Financial Ratios : Evidence from the Istanbul Stock Exchange. International Journal of Business and Social Science, 6(8), 57–67.

Yeh, C.-C., Chi, D.-J., & Lin, Y.-R. (2014). Going-concern prediction using hybrid random forests and rough set approach. Information Sciences, 254, 98-110.

Zdolšek, D., Jagrič, T., & Odar, M. (2015). Identification of auditor’s report qualifications: An empirical analysis for Slovenia. Economic Research-Ekonomska Istrazivanja, 28(1), 994–1005.

Zhou, W., & Kapoor, G. (2011). Detecting evolutionary financial statement fraud. Decision Support Systems, 50(3), 570–575.

Original Scientific Paper