Tag: adam
-
ADAM Optimization with Adaptive Batch Selection
ADAM Optimization with Adaptive Batch Selection arXiv:2512.06795v1 Announce Type: new Abstract: Adam is a widely used optimizer in neural network training due to its adaptive learning rate. However, because different data samples influence model updates to varying degrees, treating them equally can lead to inefficient convergence. To address this, a prior work proposed adapting the…
-
Understanding and Improving the Shampoo Optimizer via Kullback-Leibler Minimization
Understanding and Improving the Shampoo Optimizer via Kullback-Leibler Minimization arXiv:2509.03378v1 Announce Type: new Abstract: As an adaptive method, Shampoo employs a structured second-moment estimation, and its effectiveness has attracted growing attention. Prior work has primarily analyzed its estimation scheme through the Frobenius norm. Motivated by the natural connection between the second moment and a covariance…
-
Convergence of Adam in Deep ReLU Networks via Directional Complexity and Kakeya Bounds
Convergence of Adam in Deep ReLU Networks via Directional Complexity and Kakeya Bounds arXiv:2505.15013v1 Announce Type: new Abstract: First-order adaptive optimization methods like Adam are the default choices for training modern deep neural networks. Despite their empirical success, the theoretical understanding of these methods in non-smooth settings, particularly in Deep ReLU networks, remains limited. ReLU…