mailitics

Category: aimldsaimlds

  • Where to Go After Data Science: Unconventional / Weird Exits?

    Where to Go After Data Science: Unconventional / Weird Exits? Data science careers often feel like they funnel into the same few paths—FAANG, ML/AI engineering, or analytics leadership—but people actually branch into wildly unexpected directions. I’m curious about those off-the-beaten-path exits: roles in unexpected industries, analytics-adjacent pivots, international moves, or entirely new ventures. Would love…

  • Meta’s top AI researchers thinks LLMs are a dead end. Do many people here feel the same way from a technical perspective?

    Meta’s top AI researchers thinks LLMs are a dead end. Do many people here feel the same way from a technical perspective? submitted by /u/sext-scientist [link] [comments] /u/sext-scientist Go to original source

  • Tech Is Shrinking… and Growing? The 2026 Job Market Plot Twist.

    Tech Is Shrinking… and Growing? The 2026 Job Market Plot Twist. do you agree with the article that the ‘shrinking’ side is only for the short-term? what’s your own outlook? submitted by /u/nullstillstands [link] [comments] /u/nullstillstands Go to original source

  • UPenn mse-ds or berkeley mids?

    UPenn mse-ds or berkeley mids? I have been very fortunate to get into both programs, but I’m having a hard time deciding between the two. I applied to these two programs half a year ago when I was a new grad struggling to land a job. It was my last resort. But after 1k applications,…

  • I Built an IOS App in 3 Days with Literally No Prior Swift Knowledge

    I Built an IOS App in 3 Days with Literally No Prior Swift Knowledge What I learned about vibe coding, AI tools, and getting started as a solopreneur The post I Built an IOS App in 3 Days with Literally No Prior Swift Knowledge appeared first on Towards Data Science. Soner Yıldırım Go to original…

  • Stop Worrying about AGI: The Immediate Danger is Reduced General Intelligence (RGI)

    Stop Worrying about AGI: The Immediate Danger is Reduced General Intelligence (RGI) Let’s make conscious and deliberate choices when we use AI. The post Stop Worrying about AGI: The Immediate Danger is Reduced General Intelligence (RGI) appeared first on Towards Data Science. Shashank Vatedka Go to original source

  • How to Automate Workflows with AI

    How to Automate Workflows with AI Learn how to take a manual process and optimize it using AI The post How to Automate Workflows with AI appeared first on Towards Data Science. Eivind Kjosbakken Go to original source

  • I Measured Neural Network Training Every 5 Steps for 10,000 Iterations

    I Measured Neural Network Training Every 5 Steps for 10,000 Iterations Image by Pixabay.com The post I Measured Neural Network Training Every 5 Steps for 10,000 Iterations appeared first on Towards Data Science. Javier Marin Go to original source

  • “The success of an AI product depends on how intuitively users can interact with its capabilities”

    “The success of an AI product depends on how intuitively users can interact with its capabilities” Janna Lipenkova on AI strategy, AI products, and how domain knowledge can change the entire shape of an AI solution. The post “The success of an AI product depends on how intuitively users can interact with its capabilities” appeared…

  • How to Crack Machine Learning System-Design Interviews

    How to Crack Machine Learning System-Design Interviews A comprehensive guide into Meta, Apple, Reddit, Amazon, Google, and Snap ML design interviews The post How to Crack Machine Learning System-Design Interviews appeared first on Towards Data Science. Aliaksei Mikhailiuk Go to original source

  • Music, Lyrics, and Agentic AI: Building a Smart Song Explainer using Python and OpenAI

    Music, Lyrics, and Agentic AI: Building a Smart Song Explainer using Python and OpenAI This is how to build an AI-powered Song Explainer using Python and OpenAI The post Music, Lyrics, and Agentic AI: Building a Smart Song Explainer using Python and OpenAI appeared first on Towards Data Science. Piero Paialunga Go to original source

  • Critical Mistakes Companies Make When Integrating AI/ML into Their Processes

    Critical Mistakes Companies Make When Integrating AI/ML into Their Processes What I’ve learned leading AI teams across industries The post Critical Mistakes Companies Make When Integrating AI/ML into Their Processes appeared first on Towards Data Science. Andrey Chubin Go to original source

  • Siegel Neural Networks

    Siegel Neural Networks arXiv:2511.09577v1 Announce Type: new Abstract: Riemannian symmetric spaces (RSS) such as hyperbolic spaces and symmetric positive definite (SPD) manifolds have become popular spaces for representation learning. In this paper, we propose a novel approach for building discriminative neural networks on Siegel spaces, a family of RSS that is largely unexplored in machine…

  • Generalized infinite dimensional Alpha-Procrustes based geometries

    Generalized infinite dimensional Alpha-Procrustes based geometries arXiv:2511.09801v1 Announce Type: new Abstract: This work extends the recently introduced Alpha-Procrustes family of Riemannian metrics for symmetric positive definite (SPD) matrices by incorporating generalized versions of the Bures-Wasserstein (GBW), Log-Euclidean, and Wasserstein distances. While the Alpha-Procrustes framework has unified many classical metrics in both finite- and infinite- dimensional…

  • Masked Mineral Modeling: Continent-Scale Mineral Prospecting via Geospatial Infilling

    Masked Mineral Modeling: Continent-Scale Mineral Prospecting via Geospatial Infilling arXiv:2511.09722v1 Announce Type: new Abstract: Minerals play a critical role in the advanced energy technologies necessary for decarbonization, but characterizing mineral deposits hidden underground remains costly and challenging. Inspired by recent progress in generative modeling, we develop a learning method which infers the locations of minerals…

  • Theory and computation for structured variational inference

    Theory and computation for structured variational inference arXiv:2511.09897v1 Announce Type: new Abstract: Structured variational inference constitutes a core methodology in modern statistical applications. Unlike mean-field variational inference, the approximate posterior is assumed to have interdependent structure. We consider the natural setting of star-structured variational inference, where a root variable impacts all the other ones. We…

  • Operator Models for Continuous-Time Offline Reinforcement Learning

    Operator Models for Continuous-Time Offline Reinforcement Learning arXiv:2511.10383v1 Announce Type: new Abstract: Continuous-time stochastic processes underlie many natural and engineered systems. In healthcare, autonomous driving, and industrial control, direct interaction with the environment is often unsafe or impractical, motivating offline reinforcement learning from historical data. However, there is limited statistical understanding of the approximation errors…

  • LLMs Are Randomized Algorithms

    LLMs Are Randomized Algorithms A surprising connection between the newest AI models and a 50-year old academic field The post LLMs Are Randomized Algorithms appeared first on Towards Data Science. Udayan Kanade Go to original source

  • Robotics with Python: Q-Learning vs Actor-Critic vs Evolutionary Algorithms

    Robotics with Python: Q-Learning vs Actor-Critic vs Evolutionary Algorithms Build a Custom 3D Environment for your RL Robot The post Robotics with Python: Q-Learning vs Actor-Critic vs Evolutionary Algorithms appeared first on Towards Data Science. Mauro Di Pietro Go to original source

  • Organizing Code, Experiments, and Research for Kaggle Competitions

    Organizing Code, Experiments, and Research for Kaggle Competitions Lessons and tips learned while earning a Kaggle Competition Medal The post Organizing Code, Experiments, and Research for Kaggle Competitions appeared first on Towards Data Science. Ibrahim Habib Go to original source

  • Spearman Correlation Coefficient for When Pearson Isn’t Enough

    Spearman Correlation Coefficient for When Pearson Isn’t Enough Not all relationships are linear, and that is where Spearman comes in. The post Spearman Correlation Coefficient for When Pearson Isn’t Enough appeared first on Towards Data Science. Nikhil Dasari Go to original source

  • Optimal Control of the Future via Prospective Foraging

    Optimal Control of the Future via Prospective Foraging arXiv:2511.08717v1 Announce Type: new Abstract: Optimal control of the future is the next frontier for AI. Current approaches to this problem are typically rooted in either reinforcement learning or online learning. While powerful, these frameworks for learning are mathematically distinct from Probably Approximately Correct (PAC) learning, which…

  • The Probably Approximately Correct Learning Model in Computational Learning Theory

    The Probably Approximately Correct Learning Model in Computational Learning Theory arXiv:2511.08791v1 Announce Type: new Abstract: This survey paper gives an overview of various known results on learning classes of Boolean functions in Valiant’s Probably Approximately Correct (PAC) learning model and its commonly studied variants. Rocco A. Servedio Go to original source

  • Effects of label noise on the classification of outlier observations

    Effects of label noise on the classification of outlier observations arXiv:2511.08808v1 Announce Type: new Abstract: This study investigates the impact of adding noise to the training set classes in classification tasks using the BCOPS algorithm (Balanced and Conformal Optimized Prediction Sets), proposed by Guan & Tibshirani (2022). The BCOPS algorithm is an application of conformal…

  • Robust Sampling for Active Statistical Inference

    Robust Sampling for Active Statistical Inference arXiv:2511.08991v1 Announce Type: new Abstract: Active statistical inference is a new method for inference with AI-assisted data collection. Given a budget on the number of labeled data points that can be collected and assuming access to an AI predictive model, the basic idea is to improve estimation accuracy by…

  • Convergence and Stability Analysis of Self-Consuming Generative Models with Heterogeneous Human Curation

    Convergence and Stability Analysis of Self-Consuming Generative Models with Heterogeneous Human Curation arXiv:2511.09002v1 Announce Type: new Abstract: Self-consuming generative models have received significant attention over the last few years. In this paper, we study a self-consuming generative model with heterogeneous preferences that is a generalization of the model in Ferbach et al. (2024). The model…

  • Deploy Your AI Assistant to Monitor and Debug n8n Workflows Using Claude and MCP

    Deploy Your AI Assistant to Monitor and Debug n8n Workflows Using Claude and MCP Use Claude AI to monitor, analyse, and troubleshoot your n8n automation workflows through natural conversation. The post Deploy Your AI Assistant to Monitor and Debug n8n Workflows Using Claude and MCP appeared first on Towards Data Science. Samir Saci Go to…

  • The Ultimate Guide to Power BI Aggregations

    The Ultimate Guide to Power BI Aggregations Aggregations are one of the most powerful features in Power BI — learn how to leverage this feature to improve the performance of your Power BI solution The post The Ultimate Guide to Power BI Aggregations appeared first on Towards Data Science. Nikola Ilic Go to original source

  • How to Evaluate Retrieval Quality in RAG Pipelines (Part 3): DCG@k and NDCG@k

    How to Evaluate Retrieval Quality in RAG Pipelines (Part 3): DCG@k and NDCG@k The third and final part for evaluating the retrieval quality of your RAG pipeline with graded measures The post How to Evaluate Retrieval Quality in RAG Pipelines (Part 3): DCG@k and NDCG@k appeared first on Towards Data Science. Maria Mouschoutzi Go to…

  • Feature Detection, Part 2: Laplace & Gaussian Operators

    Feature Detection, Part 2: Laplace & Gaussian Operators Laplace meets Gaussian — the story of two operators in edge detection The post Feature Detection, Part 2: Laplace & Gaussian Operators appeared first on Towards Data Science. Vyacheslav Efimov Go to original source

  • Tractable Instances of Bilinear Maximization: Implementing LinUCB on Ellipsoids

    Tractable Instances of Bilinear Maximization: Implementing LinUCB on Ellipsoids arXiv:2511.07504v1 Announce Type: new Abstract: We consider the maximization of $x^top theta$ over $(x,theta) in mathcal{X} times Theta$, with $mathcal{X} subset mathbb{R}^d$ convex and $Theta subset mathbb{R}^d$ an ellipsoid. This problem is fundamental in linear bandits, as the learner must solve it at every time step…

  • Robust Experimental Design via Generalised Bayesian Inference

    Robust Experimental Design via Generalised Bayesian Inference arXiv:2511.07671v1 Announce Type: new Abstract: Bayesian optimal experimental design is a principled framework for conducting experiments that leverages Bayesian inference to quantify how much information one can expect to gain from selecting a certain design. However, accurate Bayesian inference relies on the assumption that one’s statistical model of…

  • Infinite-Dimensional Operator/Block Kaczmarz Algorithms: Regret Bounds and $lambda$-Effectiveness

    Infinite-Dimensional Operator/Block Kaczmarz Algorithms: Regret Bounds and $lambda$-Effectiveness arXiv:2511.07604v1 Announce Type: new Abstract: We present a variety of projection-based linear regression algorithms with a focus on modern machine-learning models and their algorithmic performance. We study the role of the relaxation parameter in generalized Kaczmarz algorithms and establish a priori regret bounds with explicit $lambda$-dependence to…

  • Distributionally Robust Online Markov Game with Linear Function Approximation

    Distributionally Robust Online Markov Game with Linear Function Approximation arXiv:2511.07831v1 Announce Type: new Abstract: The sim-to-real gap, where agents trained in a simulator face significant performance degradation during testing, is a fundamental challenge in reinforcement learning. Extansive works adopt the framework of distributionally robust RL, to learn a policy that acts robustly under worst case…

  • PrAda-GAN: A Private Adaptive Generative Adversarial Network with Bayes Network Structure

    PrAda-GAN: A Private Adaptive Generative Adversarial Network with Bayes Network Structure arXiv:2511.07997v1 Announce Type: new Abstract: We revisit the problem of generating synthetic data under differential privacy. To address the core limitations of marginal-based methods, we propose the Private Adaptive Generative Adversarial Network with Bayes Network Structure (PrAda-GAN), which integrates the strengths of both GAN-based…

  • Do You Really Need GraphRAG? A Practitioner’s Guide Beyond the Hype

    Do You Really Need GraphRAG? A Practitioner’s Guide Beyond the Hype A perspective on GraphRAG design best practices, challenges and learnings The post Do You Really Need GraphRAG? A Practitioner’s Guide Beyond the Hype appeared first on Towards Data Science. Partha Sarkar Go to original source

  • The Three Ages of Data Science: When to Use Traditional Machine Learning, Deep Learning, or an LLM (Explained with One Example)

    The Three Ages of Data Science: When to Use Traditional Machine Learning, Deep Learning, or an LLM (Explained with One Example) A practical use case to describe how the data scientist job changed across three generations of machine learning The post The Three Ages of Data Science: When to Use Traditional Machine Learning, Deep Learning,…

  • How to Build Agents with GPT-5

    How to Build Agents with GPT-5 Learn how to use GPT-5 as a powerful AI Agent on your data. The post How to Build Agents with GPT-5 appeared first on Towards Data Science. Eivind Kjosbakken Go to original source

  • AI Hype: Don’t Overestimate the Impact of AI

    AI Hype: Don’t Overestimate the Impact of AI Targeting moonshots instead of trolleys The post AI Hype: Don’t Overestimate the Impact of AI appeared first on Towards Data Science. Pascal Janetzky Go to original source

  • Benchmarking of Clustering Validity Measures Revisited

    Benchmarking of Clustering Validity Measures Revisited arXiv:2511.05983v1 Announce Type: new Abstract: Validation plays a crucial role in the clustering process. Many different internal validity indexes exist for the purpose of determining the best clustering solution(s) from a given collection of candidates, e.g., as produced by different algorithms or different algorithm hyper-parameters. In this study, we…

  • Sparsity via Hyperpriors: A Theoretical and Algorithmic Study under Empirical Bayes Framework

    Sparsity via Hyperpriors: A Theoretical and Algorithmic Study under Empirical Bayes Framework arXiv:2511.06235v1 Announce Type: new Abstract: This paper presents a comprehensive analysis of hyperparameter estimation within the empirical Bayes framework (EBF) for sparse learning. By studying the influence of hyperpriors on the solution of EBF, we establish a theoretical connection between the choice of…

  • Functional Adjoint Sampler: Scalable Sampling on Infinite Dimensional Spaces

    Functional Adjoint Sampler: Scalable Sampling on Infinite Dimensional Spaces arXiv:2511.06239v1 Announce Type: new Abstract: Learning-based methods for sampling from the Gibbs distribution in finite-dimensional spaces have progressed quickly, yet theory and algorithmic design for infinite-dimensional function spaces remain limited. This gap persists despite their strong potential for sampling the paths of conditional diffusion processes, enabling…

  • Fast Riemannian-manifold Hamiltonian Monte Carlo for hierarchical Gaussian-process models

    Fast Riemannian-manifold Hamiltonian Monte Carlo for hierarchical Gaussian-process models arXiv:2511.06407v1 Announce Type: new Abstract: Hierarchical Bayesian models based on Gaussian processes are considered useful for describing complex nonlinear statistical dependencies among variables in real-world data. However, effective Monte Carlo algorithms for inference with these models have not yet been established, except for several simple cases.…

  • Non-Negative Stiefel Approximating Flow: Orthogonalish Matrix Optimization for Interpretable Embeddings

    Non-Negative Stiefel Approximating Flow: Orthogonalish Matrix Optimization for Interpretable Embeddings arXiv:2511.06425v1 Announce Type: new Abstract: Interpretable representation learning is a central challenge in modern machine learning, particularly in high-dimensional settings such as neuroimaging, genomics, and text analysis. Current methods often struggle to balance the competing demands of interpretability and model flexibility, limiting their effectiveness in…

  • Make Python Up to 150× Faster with C

    Make Python Up to 150× Faster with C A practical guide to offloading performance-critical code to C without abandoning Python. The post Make Python Up to 150× Faster with C appeared first on Towards Data Science. Thomas Reid Go to original source

  • Why Storytelling With Data Matters for Business and Data Analysts

    Why Storytelling With Data Matters for Business and Data Analysts Data is driving the future of business and here’s how you can be prepared for that future The post Why Storytelling With Data Matters for Business and Data Analysts appeared first on Towards Data Science. Rashi Desai Go to original source

  • Does More Data Always Yield Better Performance?

    Does More Data Always Yield Better Performance? Exploring and challenging the conventional wisdom of “more data → better performance” by experimenting with the interactions between sample size, attribute set, and model complexity. The post Does More Data Always Yield Better Performance? appeared first on Towards Data Science. Mohannad Elhamod Go to original source

  • Data Culture Is the Symptom, Not the Solution

    Data Culture Is the Symptom, Not the Solution The hidden reason your data investments fail The post Data Culture Is the Symptom, Not the Solution appeared first on Towards Data Science. Jens Linden Go to original source

  • Prototype Selection Using Topological Data Analysis

    Prototype Selection Using Topological Data Analysis arXiv:2511.04873v1 Announce Type: new Abstract: Recently, there has been an explosion in statistical learning literature to represent data using topological principles to capture structure and relationships. We propose a topological data analysis (TDA)-based framework, named Topological Prototype Selector (TPS), for selecting representative subsets (prototypes) from large datasets. We demonstrate…

  • Estimating Bidirectional Causal Effects with Large Scale Online Kernel Learning

    Estimating Bidirectional Causal Effects with Large Scale Online Kernel Learning arXiv:2511.05050v1 Announce Type: new Abstract: In this study, a scalable online kernel learning framework is proposed for estimating bidirectional causal effects in systems characterized by mutual dependence and heteroskedasticity. Traditional causal inference often focuses on unidirectional effects, overlooking the common bidirectional relationships in real-world phenomena.…

  • A New Framework for Convex Clustering in Kernel Spaces: Finite Sample Bounds, Consistency and Performance Insights

    A New Framework for Convex Clustering in Kernel Spaces: Finite Sample Bounds, Consistency and Performance Insights arXiv:2511.05159v1 Announce Type: new Abstract: Convex clustering is a well-regarded clustering method, resembling the similar centroid-based approach of Lloyd’s $k$-means, without requiring a predefined cluster count. It starts with each data point as its centroid and iteratively merges them.…

  • QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design

    QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design arXiv:2410.07961v2 Announce Type: cross Abstract: Quantum computing is an emerging field recognized for the significant speedup it offers over classical computing through quantum algorithms. However, designing and implementing quantum algorithms pose challenges due to the complex nature of quantum mechanics and the necessity for precise control…

  • Self-adaptive weighting and sampling for physics-informed neural networks

    Self-adaptive weighting and sampling for physics-informed neural networks arXiv:2511.05452v1 Announce Type: new Abstract: Physics-informed deep learning has emerged as a promising framework for solving partial differential equations (PDEs). Nevertheless, training these models on complex problems remains challenging, often leading to limited accuracy and efficiency. In this work, we introduce a hybrid adaptive sampling and weighting…

  • Weekly Entering & Transitioning – Thread 10 Nov, 2025 – 17 Nov, 2025

    Weekly Entering & Transitioning – Thread 10 Nov, 2025 – 17 Nov, 2025 Welcome to this week’s entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include: Learning resources (e.g. books, tutorials, videos) Traditional education (e.g. schools, degrees, electives) Alternative education (e.g.…

  • How to Decide Between Regression and Time Series Models for “Forecasting”?

    How to Decide Between Regression and Time Series Models for “Forecasting”? Hi everyone, I’m trying to understand intuitively when it makes sense to use a time series model like SARIMAX versus a simpler approach like linear regression, especially in cases of weak autocorrelation. For example, in wind power generation forecasting, energy output mainly depends on…

  • LLMs vs DSLMs — has anyone shown significant improvements when applying this in companies?

    LLMs vs DSLMs — has anyone shown significant improvements when applying this in companies? I’ve been hearing a lot about DSLMs. We’ve stuck with the larger LLMs like GPT. Has anyone seen significant improvements with the DSLMs instead? https://devnavigator.com/2025/11/07/the-lifecycle-of-a-domain-specific-language-model/ submitted by /u/WarChampion90 [link] [comments] /u/WarChampion90 Go to original source

  • Questions about ARIMA modelling

    Questions about ARIMA modelling I am facing weird issue trying to model my NET_DEMAND. I have done unit roots tests and noticed that two levels of differencing is required and 1 level of seasonal differencing is required. But after that when I am trying to plot the ACF and PACF plots I am not seeing…

  • Free Learning Paths for Data Analysts, Data Scientists, and Data Engineers – Using 100% Open Resources

    Free Learning Paths for Data Analysts, Data Scientists, and Data Engineers – Using 100% Open Resources Hey, I’m Ryan, and I’ve created https://www.datasciencehive.com/learning-paths A platform offering free, structured learning paths for data enthusiasts and professionals alike. The current paths cover: • Data Analyst: Learn essential skills like SQL, data visualization, and predictive modeling. • Data…

  • LLM-Powered Time-Series Analysis

    LLM-Powered Time-Series Analysis Part 2: Prompts for Advanced Model Development The post LLM-Powered Time-Series Analysis appeared first on Towards Data Science. Sara Nobrega Go to original source

  • How to Build Your Own Agentic AI System Using CrewAI

    How to Build Your Own Agentic AI System Using CrewAI This article demonstrates how to develop your own Agentic AI system using CrewAI framework. By orchestrating specialized agents with distinct roles and tools, we implement a multi-agent team that is capable of generating optimized content for different social media platforms. The post How to Build…

  • Power Analysis in Marketing: A Hands-On Introduction

    Power Analysis in Marketing: A Hands-On Introduction Part 1: What is statistical power and how do we compute it? The post Power Analysis in Marketing: A Hands-On Introduction appeared first on Towards Data Science. Sam Arrington Go to original source

  • Evaluating Synthetic Data — The Million Dollar Question

    Evaluating Synthetic Data — The Million Dollar Question Learn how to evaluate synthetic data quality using the Maximum Similarity Test — a simple, quantitative approach for assessing fidelity, utility, and privacy in synthetic datasets. The post Evaluating Synthetic Data — The Million Dollar Question appeared first on Towards Data Science. Andrew Skabar Go to original…

  • Beyond Numbers: How to Humanize Your Data & Analysis

    Beyond Numbers: How to Humanize Your Data & Analysis The scintillating grid optical illusion is a perfect metaphor for how raw data can mislead us, causing us to see false trends. To escape the “data-rich, action-poor” paradox, organizations should need data humanization. This approach focuses on turning abstract metrics (the what) into clear, actionable stories…

  • How to Use GPT-5 Effectively

    How to Use GPT-5 Effectively Learn about GPT-5’s features and settings, and how to optimally apply them to your use case The post How to Use GPT-5 Effectively appeared first on Towards Data Science. Eivind Kjosbakken Go to original source

  • Friction on Demand: A Generative Framework for the Inverse Design of Metainterfaces

    Friction on Demand: A Generative Framework for the Inverse Design of Metainterfaces arXiv:2511.03735v1 Announce Type: new Abstract: Designing frictional interfaces to exhibit prescribed macroscopic behavior is a challenging inverse problem, made difficult by the non-uniqueness of solutions and the computational cost of contact simulations. Traditional approaches rely on heuristic search over low-dimensional parameterizations, which limits…

  • Bifidelity Karhunen-Lo`eve Expansion Surrogate with Active Learning for Random Fields

    Bifidelity Karhunen-Lo`eve Expansion Surrogate with Active Learning for Random Fields arXiv:2511.03756v1 Announce Type: new Abstract: We present a bifidelity Karhunen-Lo`eve expansion (KLE) surrogate model for field-valued quantities of interest (QoIs) under uncertain inputs. The approach combines the spectral efficiency of the KLE with polynomial chaos expansions (PCEs) to preserve an explicit mapping between input uncertainties…

  • Learning Paths for Dynamic Measure Transport: A Control Perspective

    Learning Paths for Dynamic Measure Transport: A Control Perspective arXiv:2511.03797v1 Announce Type: new Abstract: We bring a control perspective to the problem of identifying paths of measures for sampling via dynamic measure transport (DMT). We highlight the fact that commonly used paths may be poor choices for DMT and connect existing methods for learning alternate…

  • A general technique for approximating high-dimensional empirical kernel matrices

    A general technique for approximating high-dimensional empirical kernel matrices arXiv:2511.03892v1 Announce Type: new Abstract: We present simple, user-friendly bounds for the expected operator norm of a random kernel matrix under general conditions on the kernel function $k(cdot,cdot)$. Our approach uses decoupling results for U-statistics and the non-commutative Khintchine inequality to obtain upper and lower bounds…

  • High-dimensional limit theorems for SGD: Momentum and Adaptive Step-sizes

    High-dimensional limit theorems for SGD: Momentum and Adaptive Step-sizes arXiv:2511.03952v1 Announce Type: new Abstract: We develop a high-dimensional scaling limit for Stochastic Gradient Descent with Polyak Momentum (SGD-M) and adaptive step-sizes. This provides a framework to rigourously compare online SGD with some of its popular variants. We show that the scaling limits of SGD-M coincide…

  • TDS Newsletter: The Theory and Practice of Using AI Effectively

    TDS Newsletter: The Theory and Practice of Using AI Effectively When we encounter a new technology — say, LLM applications — some of us tend to jump right in, sleeves rolled up, impatient to start tinkering. Others prefer a more cautious approach: reading a few relevant research papers, or browsing through a bunch of blog posts, with…

  • Expected Value Analysis in AI Product Management

    Expected Value Analysis in AI Product Management An introduction to key concepts and practical applications The post Expected Value Analysis in AI Product Management appeared first on Towards Data Science. Chinmay Kakatkar Go to original source

  • The Reinforcement Learning Handbook: A Guide to Foundational Questions

    The Reinforcement Learning Handbook: A Guide to Foundational Questions Simplifying all the concepts required to master reinforcement learning The post The Reinforcement Learning Handbook: A Guide to Foundational Questions appeared first on Towards Data Science. Avishek Biswas Go to original source

  • Multi-Agent SQL Assistant, Part 2: Building a RAG Manager

    Multi-Agent SQL Assistant, Part 2: Building a RAG Manager A hands-on guide to comparing multiple RAG strategies — Keyword, FAISS, and Chroma The post Multi-Agent SQL Assistant, Part 2: Building a RAG Manager appeared first on Towards Data Science. Alle Sravani Go to original source

  • Scalable Single-Cell Gene Expression Generation with Latent Diffusion Models

    Scalable Single-Cell Gene Expression Generation with Latent Diffusion Models arXiv:2511.02986v1 Announce Type: new Abstract: Computational modeling of single-cell gene expression is crucial for understanding cellular processes, but generating realistic expression profiles remains a major challenge. This difficulty arises from the count nature of gene expression data and complex latent dependencies among genes. Existing generative models…

  • Unifying Information-Theoretic and Pair-Counting Clustering Similarity

    Unifying Information-Theoretic and Pair-Counting Clustering Similarity arXiv:2511.03000v1 Announce Type: new Abstract: Comparing clusterings is central to evaluating unsupervised models, yet the many existing similarity measures can produce widely divergent, sometimes contradictory, evaluations. Clustering similarity measures are typically organized into two principal families, pair-counting and information-theoretic, reflecting whether they quantify agreement through element pairs or aggregate…

  • Precise asymptotic analysis of Sobolev training for random feature models

    Precise asymptotic analysis of Sobolev training for random feature models arXiv:2511.03050v1 Announce Type: new Abstract: Gradient information is widely useful and available in applications, and is therefore natural to include in the training of neural networks. Yet little is known theoretically about the impact of Sobolev training — regression with both function and gradient data…

  • Provable Separations between Memorization and Generalization in Diffusion Models

    Provable Separations between Memorization and Generalization in Diffusion Models arXiv:2511.03202v1 Announce Type: new Abstract: Diffusion models have achieved remarkable success across diverse domains, but they remain vulnerable to memorization — reproducing training data rather than generating novel outputs. This not only limits their creative potential but also raises concerns about privacy and safety. While empirical…

  • Provable Accelerated Bayesian Optimization with Knowledge Transfer

    Provable Accelerated Bayesian Optimization with Knowledge Transfer arXiv:2511.03125v1 Announce Type: new Abstract: We study how Bayesian optimization (BO) can be accelerated on a target task with historical knowledge transferred from related source tasks. Existing works on BO with knowledge transfer either do not have theoretical guarantees or achieve the same regret as BO in the…

  • We Didn’t Invent Attention — We Just Rediscovered It

    We Didn’t Invent Attention — We Just Rediscovered It How selective amplification emerged across evolution, chemistry, and AI through convergent mathematical solutions The post We Didn’t Invent Attention — We Just Rediscovered It appeared first on Towards Data Science. Javier Marin Go to original source

  • AI Papers to Read in 2025

    AI Papers to Read in 2025 Reading suggestions to keep you up-to-date with the latest and classic breakthroughs in AI and Data Science. The post AI Papers to Read in 2025 appeared first on Towards Data Science. Ygor Serpa Go to original source

  • How to Evaluate Retrieval Quality in RAG Pipelines (part 2): Mean Reciprocal Rank (MRR) and Average Precision (AP)

    How to Evaluate Retrieval Quality in RAG Pipelines (part 2): Mean Reciprocal Rank (MRR) and Average Precision (AP) Evaluating the retrieval quality of your RAG pipeline with binary, order-aware measures The post How to Evaluate Retrieval Quality in RAG Pipelines (part 2): Mean Reciprocal Rank (MRR) and Average Precision (AP) appeared first on Towards Data…

  • Why Nonparametric Models Deserve a Second Look

    Why Nonparametric Models Deserve a Second Look Discover how nonparametric conditional distributions unify regression, classification, and synthetic data generation—without assuming functional forms. The post Why Nonparametric Models Deserve a Second Look appeared first on Towards Data Science. Andrew Skabar Go to original source

  • Data-driven Learning of Interaction Laws in Multispecies Particle Systems with Gaussian Processes: Convergence Theory and Applications

    Data-driven Learning of Interaction Laws in Multispecies Particle Systems with Gaussian Processes: Convergence Theory and Applications arXiv:2511.02053v1 Announce Type: new Abstract: We develop a Gaussian process framework for learning interaction kernels in multi-species interacting particle systems from trajectory data. Such systems provide a canonical setting for multiscale modeling, where simple microscopic interaction rules generate complex…

  • DoFlow: Causal Generative Flows for Interventional and Counterfactual Time-Series Prediction

    DoFlow: Causal Generative Flows for Interventional and Counterfactual Time-Series Prediction arXiv:2511.02137v1 Announce Type: new Abstract: Time-series forecasting increasingly demands not only accurate observational predictions but also causal forecasting under interventional and counterfactual queries in multivariate systems. We present DoFlow, a flow based generative model defined over a causal DAG that delivers coherent observational and interventional…

  • Limit Theorems for Stochastic Gradient Descent in High-Dimensional Single-Layer Networks

    Limit Theorems for Stochastic Gradient Descent in High-Dimensional Single-Layer Networks arXiv:2511.02258v1 Announce Type: new Abstract: This paper studies the high-dimensional scaling limits of online stochastic gradient descent (SGD) for single-layer networks. Building on the seminal work of Saad and Solla, which analyzed the deterministic (ballistic) scaling limits of SGD corresponding to the gradient flow of…

  • An Adaptive Sampling Framework for Detecting Localized Concept Drift under Label Scarcity

    An Adaptive Sampling Framework for Detecting Localized Concept Drift under Label Scarcity arXiv:2511.02452v1 Announce Type: new Abstract: Concept drift and label scarcity are two critical challenges limiting the robustness of predictive models in dynamic industrial environments. Existing drift detection methods often assume global shifts and rely on dense supervision, making them ill-suited for regression tasks…

  • A new class of Markov random fields enabling lightweight sampling

    A new class of Markov random fields enabling lightweight sampling arXiv:2511.02373v1 Announce Type: new Abstract: This work addresses the problem of efficient sampling of Markov random fields (MRF). The sampling of Potts or Ising MRF is most often based on Gibbs sampling, and is thus computationally expensive. We consider in this work how to circumvent…

  • NumPy for Absolute Beginners: A Project-Based Approach to Data Analysis

    NumPy for Absolute Beginners: A Project-Based Approach to Data Analysis Build a high-performance sensor data pipeline from scratch and unlock the true speed of Python’s scientific computing core The post NumPy for Absolute Beginners: A Project-Based Approach to Data Analysis appeared first on Towards Data Science. Ibrahim Salami Go to original source

  • What Building My First Dashboard Taught Me About Data Storytelling

    What Building My First Dashboard Taught Me About Data Storytelling Why clarity beats complexity when turning data into stories people actually understand The post What Building My First Dashboard Taught Me About Data Storytelling appeared first on Towards Data Science. Benjamin Nweke Go to original source

  • What to Do When Your Credit Risk Model Works Today, but Breaks Six Months Later

    What to Do When Your Credit Risk Model Works Today, but Breaks Six Months Later Here’s why it happens — and how to fix it The post What to Do When Your Credit Risk Model Works Today, but Breaks Six Months Later appeared first on Towards Data Science. Javier Marin Go to original source

  • Train a Humanoid Robot with AI and Python

    Train a Humanoid Robot with AI and Python 3D simulations and Reinforcement Learning with MuJoCo and Gym The post Train a Humanoid Robot with AI and Python appeared first on Towards Data Science. Mauro Di Pietro Go to original source

  • Gradient Boosted Mixed Models: Flexible Joint Estimation of Mean and Variance Components for Clustered Data

    Gradient Boosted Mixed Models: Flexible Joint Estimation of Mean and Variance Components for Clustered Data arXiv:2511.00217v1 Announce Type: new Abstract: Linear mixed models are widely used for clustered data, but their reliance on parametric forms limits flexibility in complex and high-dimensional settings. In contrast, gradient boosting methods achieve high predictive accuracy through nonparametric estimation, but…

  • A Streaming Sparse Cholesky Method for Derivative-Informed Gaussian Process Surrogates Within Digital Twin Applications

    A Streaming Sparse Cholesky Method for Derivative-Informed Gaussian Process Surrogates Within Digital Twin Applications arXiv:2511.00366v1 Announce Type: new Abstract: Digital twins are developed to model the behavior of a specific physical asset (or twin), and they can consist of high-fidelity physics-based models or surrogates. A highly accurate surrogate is often preferred over multi-physics models as…

  • Accuracy estimation of neural networks by extreme value theory

    Accuracy estimation of neural networks by extreme value theory arXiv:2511.00490v1 Announce Type: new Abstract: Neural networks are able to approximate any continuous function on a compact set. However, it is not obvious how to quantify the error of the neural network, i.e., the remaining bias between the function and the neural network. Here, we propose…

  • Perturbations in the Orthogonal Complement Subspace for Efficient Out-of-Distribution Detection

    Perturbations in the Orthogonal Complement Subspace for Efficient Out-of-Distribution Detection arXiv:2511.00849v1 Announce Type: new Abstract: Out-of-distribution (OOD) detection is essential for deploying deep learning models in open-world environments. Existing approaches, such as energy-based scoring and gradient-projection methods, typically rely on high-dimensional representations to separate in-distribution (ID) and OOD samples. We introduce P-OCS (Perturbations in the…

  • SOCRATES: Simulation Optimization with Correlated Replicas and Adaptive Trajectory Evaluations

    SOCRATES: Simulation Optimization with Correlated Replicas and Adaptive Trajectory Evaluations arXiv:2511.00685v1 Announce Type: new Abstract: The field of simulation optimization (SO) encompasses various methods developed to optimize complex, expensive-to-sample stochastic systems. Established methods include, but are not limited to, ranking-and-selection for finite alternatives and surrogate-based methods for continuous domains, with broad applications in engineering and…

  • It Doesn’t Need to Be a Chatbot

    It Doesn’t Need to Be a Chatbot A more organic, incremental approach to integrating AI into existing products The post It Doesn’t Need to Be a Chatbot appeared first on Towards Data Science. Dr. Janna Lipenkova Go to original source

  • How to Apply Vision Language Models to Long Documents

    How to Apply Vision Language Models to Long Documents Learn how to apply powerful VLMs for long context document understanding tasks The post How to Apply Vision Language Models to Long Documents appeared first on Towards Data Science. Eivind Kjosbakken Go to original source

  • Does AI Need to Be Conscious to Care?

    Does AI Need to Be Conscious to Care? Towards new forms of artificial moral agency The post Does AI Need to Be Conscious to Care? appeared first on Towards Data Science. Javier Marin Go to original source

  • Building a Multimodal RAG That Responds with Text, Images, and Tables from Sources

    Building a Multimodal RAG That Responds with Text, Images, and Tables from Sources Why do few chatbots return figures from source documents in their responses? The post Building a Multimodal RAG That Responds with Text, Images, and Tables from Sources appeared first on Towards Data Science. Partha Sarkar Go to original source