FIXED-BUDGET APPROXIMATION OF THE INVERSE KERNEL MATRIX FOR IDENTIFICATION OF NONLINEAR DYNAMIC PROCESSES
Abstract
The paper considers the identification of nonlinear dynamic processes using kernel algorithms. Kernel algorithms rely on a nonlinear transformation of the input data points into a high-dimensional space, that allow solving nonlinear problems through construction of kernelized counterparts of linear methods by replacing the inner products with kernels. One key issue of the kernel algorithms is high complexity of the inverse kernel matrix calculation. Nowadays, there are basically two approaches to this problem. The first one is based on using reduced training data sample instead of full one. In case of kernel methods, this approach could cause model misspecification, since kernel methods are directly based on training data. The second one is based on the reduced-rank approximations of the kernel matrix. Major limitation of this approach is that the rank of the approximation is either unknown until approximation is done or it is predefined by user, both of which are not so efficient. In this paper, we propose a new regularized kernel least squares algorithm based on the fixed-budget approximation of the kernel matrix. Proposed algorithm allows regulating computational burden of the identification algorithm and obtaining the least approximation error. We have shown some simulations results illustrating the efficiency of the proposed algorithm compared to other algorithms. Application of the proposed algorithm is considered on the identification problem of the input and output pressure of the pump station.
References
Liu, Q., Chen, W., Hu, H., Zhu, Q., Xie Z. (2020). An optimal NARX Neural Network Identification Model for a Magnetorheological Damper With Force-Distortion Behavior. Frontiers in Materials. DOI: 10.3389/fmats.2020.00010
Tavoosi, J., Mohammadzadeh, A., Jermsittiparsert, K. (2021). A review on type-2 fuzzy neural networks for system identification. Soft Computing, vol. 25, 7197-7212, DOI: 10.1007/s00500-021-05686-5
Li, J., Ding, F. (2021). Identification methods of nonlinear systems based on the kernel functions. Nonlinear Dynamics, vol. 104, 2537-2552, DOI: 10.1007/s11071-021-06417-z
Ning, H., Qing, G., Tian, T., Jing, X. (2019). Online Identification of Nonlinear Stochastic Spatiotemporal System With Multiplicative Noise by Robust Optimal Control-Based Kernel Learning Methods. IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 2, p. 389-404, DOI: 10.1109/TNNLS.2018.2843883
Zhang, T., Wang, S., Huang, X., Jia, L. (2020). Kernel Recursive Least Squares Algorithm Based on the Nyström Method With k-Means Sampling. IEEE Signal Processing Letters, vol. 27, p. 361-365, DOI: 10.1109/LSP.2020.2972164
Mazzoleni, M., Scandella, M., Formentin, S., Previdi, F. (2020). Enhanced kernels for nonparametric identification of a class of nonlinear systems. European control Conference (ECC), p. 540-545, DOI: 10.23919/ECC51009.2020.9143785
Blanken, L., Oomen, T. (2020). Kernel-based identification of non-causal systems with application to inverse model control. Automatica, vol. 114, p. 108830, DOI: 10.1016/j.automatica.2020.108830
Huh, M. (2015). Kernel-Trick Regression and Classification. Communications for Statistical Applications and Methods, vol. 22, no. 2, 201-207, DOI: 10.5351/CSAM.2015.22.2.201
Rojo-Álvarez J.L., Martínez-Ramón M., Muñoz-Marí J., Camps-Valls G. (2018). Kernel Functions and Reproducing Kernel Hilbert Spaces. Digital Signal Processing with Kernel Methods, IEEE, p. 165-207, DOI: 10.1002/9781118705810.ch4
Dey, A.U., Harit, G., Hafez, A.H.A. (2018). Greedy Gaussian Process Regression Applied to Object Categorization and Regression. Proceeding of the 11th Indian Conference. In proceeding of the 11th Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP 2018), no. 51, p. 1-8, DOI: 10.1145/3293353.3293404
Wenzel, T., Santin, G., Haasdonk, B. (2021). A novel class of stabilized greedy kernel approximation algorithms: Convergence, stability and uniform point distribution. Journal of Approximation Theory, vol. 262, 105508.
Harbrecht, H., Jakeman, J.D., Zaspel, P. (2021). Cholesky-Based Experimental Design for Gaussian Process and Kernel-Based Emulation and Calibration. Communications in Computational Physics, vol. 29, no. 4, p. 1152-1185, DOI: 10.4208/cicp.OA-2020-0060
Zhang, H., Jiang, H., Wang, S. (2020). Kernel Least Mean Square Based on the Sparse Nyström Method. 2020 IEEE International Symposium on Circuits and Systems (ISCAS), p. 1-5, DOI: 10.1109/ISCAS45731.2020.9181116
Lei, D., Tang, J., Li, Z., Wu, Y. (2019). Using Low-Rank Approximations to Speed Up Kernel Logistic Regression Algorithm. In IEEE Access, vol. 7, p. 84242-84252, DOI: 10.1109/ACCESS.2019.2924542
Niu, W., Xia, K., Zu, B., Bai, J. (2017). Efficient Multiple Kernel Learning Algorithms Using Low-Rank Representation. Computational Intelligence and Neuroscience, vol. 2017, 3678487, DOI: 10.1155/2017/3678487
He, L., Zhang, H. (2018). Kernel K-Means sampling for Nyström Approximation. IEEE Transactions on Image Processing, p. 2108-2120, DOI: 10.1109/TIP.2018.2796860
Li, M., Bi, W., Kwok, J., Lu, B. (2015). Large-scale Nyström kernel matrix approximation using randomized SVD. IEEE Transactions on Neural Networks and Learning Systems, vol. 26, no. 1, p. 152-164, DOI: 10.1109/TNNLS.2014.2359798
Harbrecht, H.,Peters, M., Schneider, R. (2012). On the low-rank approximation by the pivoted Cholesky decomposition. Applied Numerical Mathematics, vol. 62, no. 4, 428-440, DOI: 10.1016/j.apnum.2011.10.001
Seth, S., Príncipe, J. C. (2009). On speeding up computation in information theoretic learning. 2009 International Joint Conference on Neural Networks, p. 2883-2887, DOI: 10.1109/IJCNN.2009
Saunders, C., Gammerman, A., Vovk, V. (1998). Ridge regression learning algorithm in dual variables. Proceedings of the 15th International Conference on. Machine Learning (ICML), p. 515-521.
Kocijan, J. (2016). Modeling and Control of Dynamic Systems Using Gaussian Process Models. Advances in Industrial Control, Springer, Switzerland, DOI: 10.1007/978-3-319-21021-6
Rasmussen, C. E., Williams, C. K. I. (2006). Gaussian processes for machine learning. The MIT Press, Cambridge, Massachusetts.
Golub, G. H., Van Loan, Ch. F. (1996). Matrix computations, 3 edition. The Johns Hopkins University Press, Baltimore and London, 3 edition.
Wang, R., Li, Y. (2018). On the Numerical Rank of Radial Basis Function Kernels in High Dimensions. SIAM Journal on Matrix Analysis and Applications, vol. 39, no. 4, 1810-1835, DOI: 10.1137/17M1135803
Press, W. H., Teukolsky, S. A., Vetterling, W. T., Flannery, B. P. (1992). Numerical recipes in C. Cambridge University Press, Second edition.
Narendra, K. S., Parthasarathy, K. (1990). Identification and control of dynamical systems using neural networks. IEEE Transactions of Neural Networks, vol. 1, no. 1, 4-27.DOI: 10.1109/72.80202
