Stochastic Influence on Levenberg-Marquardt Method for Nonlinear Least Squares Problems

Gerend Christopher; Janson Naiborhu

doi:10.22342/jims.v32i2.1960

Gerend Christopher ⁽¹⁾ , Janson Naiborhu ⁽²⁾

(1) Department of Mathematics, Institut Teknologi Bandung, Indonesia,

(2) Department of Mathematics, Institut Teknologi Bandung, Indonesia

https://doi.org/10.22342/jims.v32i2.1960

Issue
Vol. 32 No. 2 (2026): JUNE

Submitted
March 3, 2025

Accepted
November 29, 2025

Published
April 23, 2026

Keywords:

nonlinear least squares, stochastic Levenberg-Marquardt, cluster, accuracy, K-means, computational efficiency

XML PDF

Abstract

In this digital era, a lot of data can be collected to be extracted and used for decision making through mathematical models in solving cases in certain domains. This paper will focus on solving nonlinear least squares problems in building an optimal model, namely one that has good accuracy and is efficient. In practice, the model is built using numerical methods. The main objective of this study is to investigate the effect of stochasticity in numerical methods that utilize gradients, namely Levenberg-Marquardt, on accuracy and computational efficiency. In addition, several numerical results from several variants of Stochastic Levenberg-Marquardt sampling and data taken in clusters with K-Means will be compared. The results of this paper are Stochastic Levenberg-Marquardt with 100, 200, 300, 400, and 500 data sample sizes outperformed classical Levenberg-Marquardt since they have very low relative errors to Levenberg-Marquardt and are faster in computational time than Levenberg-Marquardt. Therefore, when solving nonlinear least squares problems with large data sets, the Levenberg-Marquardt method requires only a small number of samples, which suggests that using the Stochastic Levenberg-Marquardt approach is advantageous.

Full text article

Generated from XML file

References

D. Kapadiya, C. Shekhawat, and P. Sharma, “A study on largescale applications of big data in modern era,” in Proceedings of the 5th International Conference on Information Management & Machine Intelligence, ACM, 2023. https://doi.org/10.1145/3647444.3647880.

E. Horvitz and T. Mitchell, “From data to knowledge to action: A global enabler for the 21st century,” 2020. https://doi.org/10.48550/ARXIV.2008.00045.

J. M. Hokanson, “Projected nonlinear least squares for exponential fitting,” SIAM Journal on Scientific Computing, vol. 39, no. 6, pp. A3107–A3128, 2017. https://doi.org/10.1137/16M1084067.

G. Grisetti, T. Guadagnino, I. Aloise, M. Colosi, B. Della Corte, and D. Schlegel, “Least squares optimization: From theory to practice,” Robotics, vol. 9, no. 3, p. 51, 2020. https://doi.org/10.3390/robotics9030051.

Y.-X. Yuan, “Recent advances in numerical methods for nonlinear equations and nonlinear least squares,” Numerical Algebra, Control & Optimization, vol. 1, no. 1, pp. 15–34, 2011. https://doi.org/10.3934/naco.2011.1.15.

J. Nocedal and S. J. Wright, Numerical Optimization. Springer, 2006. https://doi.org/10.1007/978-0-387-40065-5.

H. Robbins and S. Monro, “A stochastic approximation method,” The Annals of Mathematical Statistics, vol. 22, no. 3, pp. 400–407, 1951. https://doi.org/10.1214/aoms/1177729586.

J. Kiefer and J. Wolfowitz, “Stochastic estimation of the maximum of a regression function,” The Annals of Mathematical Statistics, vol. 23, no. 3, pp. 462–466, 1952. https://doi.org/10.1214/aoms/1177729392.

E. H. Bergou, Y. Diouane, V. Kungurtsev, and C. W. Royer, “A stochastic levenberg–marquardt method using random models with complexity results,” SIAM/ASA Journal on Uncertainty Quantification, vol. 10, no. 1, pp. 507–536, 2022. https://doi.org/10.1137/20M1366253.

R. S. V. Kumar, M. D. Alsulami, I. E. Sarris, G. Sowmya, and F. Gamaoun, “Stochastic levenberg–marquardt neural network implementation for analyzing the convective heat transfer in a wavy fin,” Mathematics, vol. 11, no. 10, p. 2401, 2023. https://doi.org/10.3390/math11102401.

D. E. Budil, S. Lee, S. Saxena, and J. H. Freed, “Nonlinear-least-squares analysis of slow motion epr spectra in one and two dimensions using a modified levenberg–marquardt algorithm,” Journal of Magnetic Resonance Series A, vol. 120, no. 2, pp. 155–189, 1996. https://doi.org/10.1006/jmra.1996.0113.

M. Lampton, “Damping–undamping strategies for the levenberg–marquardt nonlinear least-squares method,” Computers in Physics, vol. 11, no. 1, pp. 110–115, 1997. https://doi.org/10.1063/1.168600.

Z. Yan, S. Zhong, L. Lin, and Z. Cui, “Adaptive levenberg–marquardt algorithm: A new optimization strategy for levenberg–marquardt neural networks,” Mathematics, vol. 9, no. 17, p. 2176, 2021. https://doi.org/10.3390/math9172176.

Y. Hong et al., “Stochastic levenberg–marquardt for solving optimization problems on hardware accelerators,” in IEEE Conference Proceedings, IEEE, 2020. https://repository.kaust.edu.sa/handle/10754/666072.

K. Madsen, H. B. Nielsen, and O. Tingleff, Methods for Non-Linear Least Squares Problems. 2 ed., 2004.

S. Ruder, “An overview of gradient descent optimization algorithms,” 2016. https://doi.org/10.48550/ARXIV.1609.04747.

Y. Tian, Y. Zhang, and H. Zhang, “Recent advances in stochastic gradient descent in deep learning,” Mathematics, vol. 11, no. 3, p. 682, 2023. https://doi.org/10.3390/math11030682.

G. Garrigos and R. M. Gower, “Handbook of convergence theorems for (stochastic) gradient methods,” 2023. https://doi.org/10.48550/ARXIV.2301.11235.

W.-Y. Shao and J.-Y. Fan, “Global convergence of a stochastic levenberg–marquardt algorithm based on trust region,” Journal of the Operations Research Society of China, 2024. https://doi.org/10.1007/s40305-023-00529-6.

J. B. MacQueen, “Some methods for classification and analysis of multivariate observations,” in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 281–297, 1967. http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s5_v1_article-17.pdf.

S. Na, L. Xumin, and G. Yong, “Research on k-means clustering algorithm: An improved k-means clustering algorithm,” in 2010 Third International Symposium on Intelligent Information Technology and Security Informatics, pp. 63–67, IEEE, 2010. https://doi.org/10.1109/IITSI.2010.74.

D. Arthur and S. Vassilvitskii, “k-means++: The advantages of careful seeding,” in Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 1027–1035, 2007. https://doi.org/10.5555/1283383.1283494.

X. Wan, “Influence of feature scaling on convergence of gradient iterative algorithm,” Journal of Physics: Conference Series, vol. 1213, no. 3, p. 032021, 2019. https://doi.org/10.1088/1742-6596/1213/3/032021.

S. C. Nayak, B. B. Misra, and H. S. Behera, “Impact of data normalization on stock index forecasting,” International Journal of Computer Information Systems and Industrial Management Applications, vol. 6, pp. 357–369, 2014.

Authors

Gerend Christopher

Department of Mathematics, Institut Teknologi Bandung

Janson Naiborhu

Department of Mathematics, Institut Teknologi Bandung

https://orcid.org/0000-0003-1789-7187

janson@itb.ac.id (Primary Contact)

Christopher, G., & Naiborhu, J. (2026). Stochastic Influence on Levenberg-Marquardt Method for Nonlinear Least Squares Problems. Journal of the Indonesian Mathematical Society, 32(2), 1960. https://doi.org/10.22342/jims.v32i2.1960

Download Citation

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Downloads

Download data is not yet available.

1. INTRODUCTION

Advancements in mathematics, science, and technology has driven humans to digital and information era. This modern era leads to the exponential growth of creating, collecting, and utilizing data as stated in Kapadiya et al.[1]and Horvitz and Mitchell[2]. As data collection and processing have advanced, the need for efective methods to extract meaningful insights from this data has become increasingly apparent. The development of mathematical models that capable of handling complex dataset is a necessity as well. Hokanson[3]stated that among the various approaches to model data, nonlinear least squares problems have emerged as a critical area of focus, particularly for their applications in data fitting and parameter estimation across diverse fields such as engineering, economics, and machine learning.

Nonlinear Least Squares problems involve finding the best fit curve to a set of data points by minimizing the sum of the squares of the residuals, which usually called the objective function of this problem. This task for finding optimal model that has high accuracy or lower error and has computational eficiency led to the development of numerous of numerical methods [4,5]. In Nocedal[6]and Yuan[5], Gradient-based numerical methods is one of those methods for solving nonlinear least squares problems.

With the growth of data created, recent advances have explored the integration of stochastic elements into these numerical methods to potentially enhance their performance and its applications [7,8,9,10]. One specific method, Stochastic Gradient Descent that introduce randomness into the optimization process can influence both accuracy and computational eficiency as in Robbins[7]and Kiefer[8]. The impact of such stochastic methods on gradient-based numerical methods remains a topic of ongoing research.

The Levenberg-Marquardt algorithm has been extensively studied for solving nonlinear least squares problems. Early applications include the analysis of EPR spectra (1996)[11]and the development of damping and undamping strategies (1997)[12]. More recent eforts have focused on improving the accuracy and computational eficiency of the algorithm. For example, Yan et al. proposed AdaLM (Adaptive Levenberg-Marquardt), which adapts the step size to enhance performance[13]. Building on this line of work, Hong et al. (2020) introduced a stochastic version of the Levenberg-Marquardt algorithm by using randomly selected subsets of data.[14]. In contrast, our approach introduces a new variation of stochastic sampling: at each iteration, a diferent random subset of the data is selected. Furthermore, this study incorporates K-Means clustering, using cluster centroids as representative data points, which diferentiates our method from previous work.

The main purpose of this research is to investigate the influence of stochastic in numerical methods that utilize gradients, especially for Levenberg-Marquardt method, on accuracy and computational eficiency. In this study, numerical result with data taken in clusters will be compared to Stochastic Levenberg-Marquardt.

2. PROBLEM STATEMENT AND METHODS

2.1. Least Squares Problem.

When we have data \(( x _ { i } , y _ { i } ) , \ i = 1 , 2 , \ldots , m\), the relationship of \(x _ { i }\) and \(y _ { i }\) can be written in equation (1)

where \(\mathbf { p } = [ p _ { 1 } , p _ { 2 } , \ldots , p _ { n } ] ^ { \mathrm { T } }\) is a vector consist of model parameters, \(M ( \mathbf { p } , x _ { i } )\) is a fitting model, and \(e _ { i }\) is measurement error on the data ordinate.

For any choice of \(\mathbf { p }\), residuals can be computed by

The least squares problem is to minimize the sum of the squared residuals by obtaining \(\mathbf { p } ^ { * }\) as the minimizer. Madsen[15]define least squares problem in Definition2.1

Definition 2.1.[Least Squares Problem] Find a local minimizer\(\mathbf { p } ^ { * } ~ f o r\)

where \(f _ { i } : \mathbb { R } ^ { n } \mapsto \mathbb { R } , \ i = 1 , 2 , \ldots ,m\) are given functions, and \(m \geq n\).

As an objective function, \(F ( \mathbf { p } )\) can also be called as loss function or cost function.

2.2. Gradient Descent.

Gradient Descent is an iterative optimization numerical method that uses first order derivative approach with the aim of minimizing the objective function. In Ruder[16], Gradient Descent is a way to minimize objective function \(F ( \mathbf { p } _ { k } ; \zeta )\) parameterized by its model parameters \(\mathbf { p } _ { k } \in \mathbb { R } ^ { n }\) by updating the value of param eters with opposite direction of gradient of objective function. In Tian et al.[17], updating the value of parameters \(\mathbf { p }\) using Gradient Descent can be written as in equation(4)

where \(k\) denoted as iteration of the simulation. Symbol \(\tau _ { k }\) called as learning rate which usually constant greater than zero that regulate updating steps using gradient. Symbol \(\zeta : ( x _ { i } , y _ { i } ) , \ i = 1 , 2 , . . . ,

.....

Read Counter : 218 Download : 444

Share Now

Stochastic Influence on Levenberg-Marquardt Method for Nonlinear Least Squares Problems

Abstract

Full text article

References

Authors

Address

Contact Info

Article Sidebar

Abstract

Full text article

References

Authors

Article Details

Address

Contact Info