Saturated linear unit as an universal symmetric activation function for deep learning

  • Maja Lutovac Banduka aRT-RK LLC (former Department of RT-RK Institute, Computer Based Systems), 2v Dunavska, 11158 Belgrade, Serbia
  • Vladimir Milićević Faculty of Mechanical and Civil Engineering, University of Kragujevac, Dositejeva 19, 36000 Kraljevo, Serbia
  • Igor Franc Faculty of Mechanical and Civil Engineering, University of Kragujevac, Dositejeva 19, 36000 Kraljevo, Serbia
  • Nemanja Zdravković Faculty of Information Technology, Belgrade Metropolitan University, Tadeuša Košćuška 63, 11158 Belgrade, Serbia
  • Nikola Dimitrijević cFaculty of Information Technology, Belgrade Metropolitan University, Tadeuša Košćuška 63, 11158 Belgrade, Serbia
DOI: https://doi.org/10.5937/sjm21-57891
Keywords: artificial neural networks, closed-form solutions, classification decision, feature extraction, machine learning algorithms, deep learning

Abstract



There is a number of symmetric activation functions used in artificial neural networks for deep learning. In this paper, we propose a universal activation function based on the Saturated Linear Unit (SaLU) that can be used instead of any known symmetric activation function. It is not necessary for classification tasks to have an exact calculation of the probability of detected classes. The classification decision is made based on the highest probability for the input values. We propose, as a proof of concept, that the two most commonly used hyperbolic tangent and algebraic sigmoid activation functions can be effectively replaced by SaLU by choosing a single parameter. Moreover, the theoretical step function can also be replaced by SaLU for a wider transition range. All derivations use symbolic processing. Also shown is a visualization of the range of inputs that result in a suitable classification. This can help scientists and programmers design complex machine learning algorithms and understand how deep learning algorithms work.

References

Khan, S., Rahmani, H., Shah, S., & Bennamoun, M. (2018). A Guide to Convolutional Neural Networks for Computer Vision, Switzerland, Springer Nature, 2018, 54-56.

Bernard, E. (2022). Introduction to Machine Learning, Champaign, IL, USA: Wolfram Media

Wolfram, S. (2023). An Elementary Introduction to the Wolfram Language, 3rd ed., Champaign, IL, USA: Wolfram Media

Lutovac, M.D., & Tosic, D. (2014). Schematic Solver, Mathematica application package, symbolic signal processing, software implementation, mouse driven interactive drawing tool, https://library.wolfram.com/infocenter/TechNotes/4814/

Milićević, V., Franc, I., Lutovac Banduka, M., Zdravković, N., & Dimitrijević, N. (2025). Symbolic analysis of classical neural networks for deep learning, International Journal for Quality Research, 19 (1), 85-100.

Lutovac Banduka, M., & Lutovac, M. (2024). Multiplierless neural networks for deep learning, in Proc. 13th Mediterranean Conference on Embedded Computing, Budva, Montenegro, Jun. 11-14, 2024, 262-265.

Zhao, F., & Huang, S. L. (2022). On the universally optimal activation function for a class of residual neural networks, AppliedMath, 2 (4), 574-584.

Kamienny, P.A., d’Ascoli, S., Lample, G., & Charton, F. (2022). End-to-end symbolic regression with transformers,” presented at the 36th Conference on Neural Information Processing Systems (NeurIPS 2022), New Orleans, USA, Nov. 28th - Dec. 9th 2022, Adv. Neural Inf. Proc. Syst., 35, 10269-10281.

Goodfellow, I., Bengio, Y., & Courville, A. (2016), Deep Learning, Cambridge, MA, USA: MIT Press.

Setzu, M., Guidotti, R., Monreale, A., Turini, F., Pedreschi, D., & Giannotti, F. (2021), GLocalX - from local to global explanations of black box AI models, Artificial Intelligence, 103457, 294, 1-15

Wiratsin, O., & Ragkhitwetsagul, C. (2025). Effectiveness of Explainable Artificial Intelligence (XAI) Techniques for Improving Human Trust in Machine Learning Models: A Systematic Literature Review, IEEE Access, 13, 121326-121350, 2025, doi: 10.1109/ACCESS.2025.3575022

Lăzăroiu, G., Andronie, M., Iatagan, M., Geamanu, M., Stefanescu, R., & Diamarescu, I. (2022). Review deep learning-assisted smart process planning, robotic wireless sensor networks, and geospatial big data management algorithms in the internet of manufacturing things, ISPRS International Journal of Geo-Information, 11 (5), 277.

Dubey, S.R., Singh S.K., & Chauduri, B.B. (2021). Activation functions in deep learning: A comprehensive survey and benchmark, arXiv preprint arXiv:2109.14545.

Stursa, D. & Dolezel P. (2019). Comparison of ReLU and linear saturated activation functions in neural network for universal approximation, 22nd International Conference on Process Control (PC19), 11-14 June 2019, Strbske Pleso, Slovakia

Wang, J.H., Cao, J., Li, W., Yu, P., & Huang, K. (2019). A novel parallel accelerated CRPF algorithm," Applied Intelligence, 50 (3), 849-859.

Chen, J., & Pan. Z. (2023). Saturated Non-Monotonic Activation Functions, arXiv preprint arXiv:2305.07537.

Lau, M.M., & Hann Lim, K. (2018). Review of Adaptive Activation Function in Deep Neural Network," 2018 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES). 03-06 December 2018, Sarawak, Malaysia

Yang, X., Gao, Z., & Zhang, J. (2020). Event-driven H∞ control with critic learning for nonlinear systems, Neural Networks, 132, 30-42.

Lee, Y., Park, C., & Kang, S. (2023). Deep Embedded Clustering Framework for Mixed Data, IEEE Access, 11, 33-40.

Zhang, N., Zhao, J., Ma. L., Kong, H., & Li, H. (2022). Tool Wear Monitoring Based on Transfer Learning and Improved Deep Residual Network, IEEE Access, 10, 119546-119557.

Ren, Y.M., Alhajeri, M.S., Luo, J., & Chen, S. (2022). A tutorial review of neural network modeling approaches for model predictive control, Computers & Chemical Engineering, 165 (3), 107956.

Published
2026/06/09
Issue
Vol. 21 No. 1 (2026): Serbian Journal of Management
Section
Original Scientific Paper

The Author wishes to submit the Work to SJM for publication. To enable SJM to publish the Work and to give effect to the parties’ intention set forth herein, they have agreed to cede the first right to publication and republication in the SJM Journal.

 

Cession

 

The Author hereby cedes to SJM, who accepts the cession, to the copyright in and to the paper.

 

The purpose of the cession is to enable SJM to publish the Work, as first publisher world-wide, and for republication in the SJM Journal, and to grant the right to others to publish the Work world-wide, for so long as such copyright subsists;

 

SJM shall be entitled to edit the work before publication, as it deems fit, subject to the Authors approval

 

The Author warrants to SJM that:

 

- the Author is the owner of the copyright in the Work, whether as author or as reassigned from the Author’s employee and that the Author is entitled to cede the copyright to SJM;

 

- the paper (or any of its part) is not submitted or accepted for publication in any other Journal;

 

- the Work is an original work created by the Author;

 

- the Author has not transferred, ceded, or assigned the copyright, or any part thereof, to any third party; or granted any third party a licence or other right to the copyright, which may affect or detract from the rights granted to SJM in terms of this agreement.

 

The Author hereby indemnifies the SJM as a body and its individual members, to the fullest extent permitted in law, against all or any claims which may arise consequent to the warranties set forth.

 

No monetary consideration shall be payable by SJM to the Author for the cession, but SJM shall clearly identify the Author as having produced the Work and ensure that due recognition is given to the Author in any publication of the Work.

 

Should SJM, in its sole discretion, elect not to publish the Work within 1 year after the date of this agreement, the cession shall lapse and be of no further effect. In such event the copyright shall revert to the Author and SJM shall not publish the Work, or any part thereof, without the Author’s prior written consent.