Strategy/projects/files/splitting/split-sgd/cover_letter_jcam.md
+

cover_letter_jcam

Cover Letter — JCAM Submission

Paper: “Stochastic gradient algorithms from ODE splitting perspective”
Authors: Daniil Merkulov, Ivan Oseledets
Journal: Journal of Computational and Applied Mathematics (Elsevier)

Dear Editors of the Journal of Computational and Applied Mathematics,

We submit for your consideration the manuscript entitled “Stochastic gradient algorithms from ODE splitting perspective” by Daniil Merkulov and Ivan Oseledets (Skolkovo Institute of Science and Technology, Moscow, Russia).

Summary of Contributions

This paper presents a unifying framework for stochastic gradient methods through the lens of operator splitting for gradient-flow ODEs. The key contributions are:

  1. SGD as operator splitting: We prove that one epoch of SGD is precisely a first-order Lie–Trotter splitting of the gradient-flow ODE $\dot{\theta} = -\nabla f(\theta)$. This reinterpretation connects the rich literature on geometric numerical integration to the analysis of machine learning optimizers.

  2. Global splitting error bound: We derive a new upper bound on the global splitting error for the Lie–Trotter scheme, providing theoretical grounding for the empirical robustness observations.

  3. Kaczmarz limit: We establish analytically that the Kaczmarz method is the exact limiting case of unit-batch SGD for linear least-squares, unifying two apparently distinct algorithms.

  4. Splitting optimization: Replacing the Euler step with a higher-order local ODE solver yields a “splitting optimization” scheme that is markedly more robust to the learning-rate choice, demonstrated across linear least squares, logistic regression, and softmax regression benchmarks.

  5. LASSO bias analysis: For composite objectives ($\ell_2 + \ell_1$), we show that the Lie–Trotter scheme (ISTA) converges with zero asymptotic bias, while the Strang splitting (Strang-ISTA) converges to a biased fixed point with excess objective scaling as $C_D h^2 \lambda^2$. We derive a lower bound $C_D \geq \|A\,\mathrm{sign}(\theta^*)\|^2/8$ and confirm the $h^2$-scaling and component-wise fixed-point shift formula numerically.

Significance

The splitting perspective provides a principled way to derive and analyze new optimization algorithms by importing tools from numerical ODEs. The paper bridges computational mathematics and machine learning, which aligns with JCAM’s scope. Our results yield both theoretical insights (bias analysis of proximal methods) and practical improvements (robustness to hyperparameter tuning).

Originality

This manuscript has not been published elsewhere and is not under consideration at any other journal. A shorter preliminary version appeared as a conference abstract; the present submission contains substantially extended theory (Sections 4–7), new experiments, and the complete LASSO bias analysis (Sections 5–6), which are entirely new.

Suggested Reviewers

  • Prof. Peter Richtárik (King Abdullah Univ. of Sci. and Tech.) — expert in randomized optimization, SGD theory.
  • Prof. Lorenzo Rosasco (Università di Genova / MIT) — expert in numerical methods for machine learning.
  • Prof. Defeng Sun (Hong Kong Polytechnic Univ.) — expert in structured optimization and splitting methods.
  • Prof. Dirk Lorenz (TU Braunschweig) — expert in proximal algorithms, ISTA/FISTA theory.

Corresponding Author

Daniil Merkulov
Center for Computational and Data-Intensive Science and Engineering
Skolkovo Institute of Science and Technology
Bolshoy Boulevard 30, bld. 1, Moscow, Russia, 121205
Email: daniil.merkulov@skolkovotech.ru

We hope that the manuscript will be found suitable for publication in JCAM and look forward to the reviewers’ feedback.

Sincerely,
Daniil Merkulov
(on behalf of all authors)

Prepared by Feanor agent, 2026-03-17. Status: draft, awaiting Oselodets approval of the manuscript before submission.

Choose icon