mailitics

Category: aimldsaimlds

  • Integrating Random Effects in Variational Autoencoders for Dimensionality Reduction of Correlated Data

    Integrating Random Effects in Variational Autoencoders for Dimensionality Reduction of Correlated Data arXiv:2412.16899v1 Announce Type: new Abstract: Variational Autoencoders (VAE) are widely used for dimensionality reduction of large-scale tabular and image datasets, under the assumption of independence between data observations. In practice, however, datasets are often correlated, with typical sources of correlation including spatial, temporal…

  • Gradient-Based Non-Linear Inverse Learning

    Gradient-Based Non-Linear Inverse Learning arXiv:2412.16794v1 Announce Type: new Abstract: We study statistical inverse learning in the context of nonlinear inverse problems under random design. Specifically, we address a class of nonlinear problems by employing gradient descent (GD) and stochastic gradient descent (SGD) with mini-batching, both using constant step sizes. Our analysis derives convergence rates for…

  • Learning from Summarized Data: Gaussian Process Regression with Sample Quasi-Likelihood

    Learning from Summarized Data: Gaussian Process Regression with Sample Quasi-Likelihood arXiv:2412.17455v1 Announce Type: new Abstract: Gaussian process regression is a powerful Bayesian nonlinear regression method. Recent research has enabled the capture of many types of observations using non-Gaussian likelihoods. To deal with various tasks in spatial modeling, we benefit from this development. Difficulties still arise…

  • Classifier-free guidance for LLMs performance enhancing

    Classifier-free guidance for LLMs performance enhancing Classifier-Free Guidance for LLMs Performance Enhancing Check and improve classifier-free guidance for text generation large language models. While participating in NeurIPS 2024 Competitions track I was awarded the second prize in the LLM Privacy challenge. The solution I had used classifier-free guidance (CFG). I noticed that with high CFG guidance…

  • How to Tackle an Optimization Problem with Constraint Programming

    How to Tackle an Optimization Problem with Constraint Programming Case study: the travelling salesman problem TLDR Constraint Programming is a technique of choice for solving a Constraint Satisfaction Problem. In this article, we will see that it is also well suited to small to medium optimization problems. Using the well-known travelling salesman problem (TSP) as an…

  • How to Clean Your Data for Your Real-Life Data Science Projects

    How to Clean Your Data for Your Real-Life Data Science Projects How I treat missing values—with a quick Python Guide Continue reading on Towards Data Science » Mythili Krishnan Go to original source

  • Adapted Prediction Intervals by Means of Conformal Predictions and a Custom Non-Conformity Score

    Adapted Prediction Intervals by Means of Conformal Predictions and a Custom Non-Conformity Score How confident should I be in a machine learning model’s prediction for a new data point? Could I get a range of likely values? Image by author When working on a supervised task, machine learning models can be used to predict the outcome for…

  • Why Sets Are So Useful in Programming

    Why Sets Are So Useful in Programming And how you can use them to boost your code performance A set is a simple structure defined as a collection of distinct elements. Sets are most commonly seen in fields like mathematics or logic, but they’re also useful in programming for writing efficient code. In this article,…

  • Enhancing Masked Time-Series Modeling via Dropping Patches

    Enhancing Masked Time-Series Modeling via Dropping Patches arXiv:2412.15315v1 Announce Type: new Abstract: This paper explores how to enhance existing masked time-series modeling by randomly dropping sub-sequence level patches of time series. On this basis, a simple yet effective method named DropPatch is proposed, which has two remarkable advantages: 1) It improves the pre-training efficiency by…

  • Deep learning joint extremes of metocean variables using the SPAR model

    Deep learning joint extremes of metocean variables using the SPAR model arXiv:2412.15808v1 Announce Type: new Abstract: This paper presents a novel deep learning framework for estimating multivariate joint extremes of metocean variables, based on the Semi-Parametric Angular-Radial (SPAR) model. When considered in polar coordinates, the problem of modelling multivariate extremes is transformed to one of…

  • Using matrix-product states for time-series machine learning

    Using matrix-product states for time-series machine learning arXiv:2412.15826v1 Announce Type: new Abstract: Matrix-product states (MPS) have proven to be a versatile ansatz for modeling quantum many-body physics. For many applications, and particularly in one-dimension, they capture relevant quantum correlations in many-body wavefunctions while remaining tractable to store and manipulate on a classical computer. This has…

  • On Robust Cross Domain Alignment

    On Robust Cross Domain Alignment arXiv:2412.15861v1 Announce Type: new Abstract: The Gromov-Wasserstein (GW) distance is an effective measure of alignment between distributions supported on distinct ambient spaces. Calculating essentially the mutual departure from isometry, it has found vast usage in domain translation and network analysis. It has long been shown to be vulnerable to contamination…

  • Learning sparsity-promoting regularizers for linear inverse problems

    Learning sparsity-promoting regularizers for linear inverse problems arXiv:2412.16031v1 Announce Type: new Abstract: This paper introduces a novel approach to learning sparsity-promoting regularizers for solving linear inverse problems. We develop a bilevel optimization framework to select an optimal synthesis operator, denoted as $B$, which regularizes the inverse problem while promoting sparsity in the solution. The method…

  • You Get a Dataset and Need to Find a “Good” Model Quickly (in Hours or Days), what’s your strategy?

    You Get a Dataset and Need to Find a “Good” Model Quickly (in Hours or Days), what’s your strategy? Typical Scenario: Your friend gives you a dataset and challenges you to beat their model’s performance. They don’t tell you what they did, but they provide a single CSV file and the performance metric to optimize.…

  • Weekly Entering & Transitioning – Thread 23 Dec, 2024 – 30 Dec, 2024

    Weekly Entering & Transitioning – Thread 23 Dec, 2024 – 30 Dec, 2024 Welcome to this week’s entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include: Learning resources (e.g. books, tutorials, videos) Traditional education (e.g. schools, degrees, electives) Alternative education (e.g.…

  • tHe wINdoWs mL EcOsYteM

    tHe wINdoWs mL EcOsYteM submitted by /u/Hire_Ryan_Today [link] [comments] /u/Hire_Ryan_Today Go to original source

  • Statisticians, Scripts, and Chaos: My Journey Back to the 90s

    Statisticians, Scripts, and Chaos: My Journey Back to the 90s We often hear a lot about how data science teams can lack statistical expertise and how this can lead to flawed analyses or misinterpretation of results. It’s a valid concern, and the dangers are real. But let me tell you, there’s another side of the…

  • ML pipeline questions

    ML pipeline questions I am building an application that processes videos and that needs to run many tasks (some need to be sequentially and some in parallel). Think audio extraction, ASR, diarization, translation, video classification, etc… Note that this is in supposed to be run online, i.e. this is supposed to be used in a…

  • An Agentic Approach to Reducing LLM Hallucinations

    An Agentic Approach to Reducing LLM Hallucinations Simple techniques to alleviate LLM hallucinations using LangGraph Photo by Greg Rakozy on Unsplash If you’ve worked with LLMs, you know they can sometimes hallucinate. This means they generate text that’s either nonsensical or contradicts the input data. It’s a common issue that can hurts the reliability of LLM-powered…

  • Creating a WhatsApp AI Agent with GPT-4o

    Creating a WhatsApp AI Agent with GPT-4o Created with DALL-E How to use the Meta API to build your own LLM-powered Whatsapp chatbot A game-changer in the field of AI and business management is the integration of AI agents with widely used communication tools. Think of having a familiar chat interface with real-time data requests, updates, and…

  • How (and Where) ML Beginners Can Find Papers

    How (and Where) ML Beginners Can Find Papers From conferences to surveys Continue reading on Towards Data Science » Pascal Janetzky Go to original source

  • What Every Aspiring Machine Learning Engineer Must Know to Succeed

    What Every Aspiring Machine Learning Engineer Must Know to Succeed Your Guide to Avoiding Critical Errors with Machine Learning in Production Continue reading on Towards Data Science » Claudia Ng Go to original source

  • Propensity-Score Matching Is the Bedrock of Causal Inference

    Propensity-Score Matching Is the Bedrock of Causal Inference And how to get started with it using Python Continue reading on Towards Data Science » Ari Joury, PhD Go to original source

  • Should you switch from VSCode to Cursor?

    Should you switch from VSCode to Cursor? My experience using VSCode (GitHub Copilot) and Cursor (Claude 3.5 Sonnet) as a Data Scientist. Continue reading on Towards Data Science » Marc Matterson Go to original source

  • Evaluation-Driven Development for agentic applications using PydanticAI

    Evaluation-Driven Development for agentic applications using PydanticAI An open-source, model-agnostic agentic framework that supports dependency injection Ideally, you can evaluate agentic applications even as you are developing them, instead of evaluation being an afterthought. For this to work, though, you need to be able to mock both internal and external dependencies of the agent you…

  • Conditional Variational Autoencoders for Text to Image Generation

    Conditional Variational Autoencoders for Text to Image Generation Investigating an early generative architecture and applying it to image generation from text input Recently I was tasked with text-to-image synthesis using a conditional variational autoencoder (CVAE). Being one of the earlier generative structures, it has its limitations but is easily implementable. This article will cover CVAEs at…

  • The 80/20 problem of generative AI — a UX research insight

    The 80/20 problem of generative AI — a UX research insight Image by author The 80/20 problem of generative AI — a UX research insight When an LLM solves a task 80% correctly, that often only amounts to 20% of the user value. The Pareto principle says if you solve a problem 20% through, you get 80% of the value. The opposite…

  • Top 3 Strategies to Search Your Data

    Top 3 Strategies to Search Your Data Strategies from traditional index seek to AI based semantic search that every software engineer should know! Continue reading on Towards Data Science » Shawn Shi Go to original source

  • A New Approach to AI Safety: Layer Enhanced Classification (LEC)

    A New Approach to AI Safety: Layer Enhanced Classification (LEC) LEC surpasses best in class models, like GPT-4o, by combining the efficiency of a ML classifier with the language understanding of an LLM Imagine sitting in a boardroom, discussing the most transformative technology of our time — artificial intelligence — and realizing we’re riding a rocket with no reliable safety…

  • Semantically Compress Text to Save On LLM Costs

    Semantically Compress Text to Save On LLM Costs LLMs are great… if they can fit all of your data Photo by Christopher Burns on Unsplash Originally published at https://blog.developer.bazaarvoice.com on October 28, 2024. Introduction Large language models are fantastic tools for unstructured text, but what if your text doesn’t fit in the context window? Bazaarvoice faced exactly this…

  • When Averages Lie: Moving Beyond Single-Point Predictions

    When Averages Lie: Moving Beyond Single-Point Predictions The Case for Predicting Full Probability Distributions in Decision-Making Some people like hot coffee, some people like iced coffee, but no one likes lukewarm coffee. Yet, a simple model trained on coffee temperatures might predict that the next coffee served should be… lukewarm. This illustrates a fundamental problem…

  • Ranking Basics: Pointwise, Pairwise, Listwise

    Ranking Basics: Pointwise, Pairwise, Listwise Because thy neighbour matters Image taken from unsplash.com First, let’s talk about where ranking comes into play. Ranking is a big deal in e-commerce and search applications — essentially, any scenario where you need to organize documents based on a query. It’s a little different from classic classification or regression problems. For…

  • Understanding Deduplication Methods: Ways to Preserve the Integrity of Your Data

    Understanding Deduplication Methods: Ways to Preserve the Integrity of Your Data Increasing growth and data complexities have made data deduplication even more relevant Data duplication is still a problem for many organisations. Although data processing and storage systems have developed rapidly along with technological advances, the complexity of the data produced is also increasing. Moreover, with…

  • Statistical Undersampling with Mutual Information and Support Points

    Statistical Undersampling with Mutual Information and Support Points arXiv:2412.14527v1 Announce Type: new Abstract: Class imbalance and distributional differences in large datasets present significant challenges for classification tasks machine learning, often leading to biased models and poor predictive performance for minority classes. This work introduces two novel undersampling approaches: mutual information-based stratified simple random sampling and…

  • On the Robustness of Spectral Algorithms for Semirandom Stochastic Block Models

    On the Robustness of Spectral Algorithms for Semirandom Stochastic Block Models arXiv:2412.14315v1 Announce Type: new Abstract: In a graph bisection problem, we are given a graph $G$ with two equally-sized unlabeled communities, and the goal is to recover the vertices in these communities. A popular heuristic, known as spectral clustering, is to output an estimated…

  • From Point to probabilistic gradient boosting for claim frequency and severity prediction

    From Point to probabilistic gradient boosting for claim frequency and severity prediction arXiv:2412.14916v1 Announce Type: new Abstract: Gradient boosting for decision tree algorithms are increasingly used in actuarial applications as they show superior predictive performance over traditional generalized linear models. Many improvements and sophistications to the first gradient boosting machine algorithm exist. We present in…

  • FedSTaS: Client Stratification and Client Level Sampling for Efficient Federated Learning

    FedSTaS: Client Stratification and Client Level Sampling for Efficient Federated Learning arXiv:2412.14226v1 Announce Type: cross Abstract: Federated learning (FL) is a machine learning methodology that involves the collaborative training of a global model across multiple decentralized clients in a privacy-preserving way. Several FL methods are introduced to tackle communication inefficiencies but do not address how…

  • Projected gradient methods for nonconvex and stochastic optimization: new complexities and auto-conditioned stepsizes

    Projected gradient methods for nonconvex and stochastic optimization: new complexities and auto-conditioned stepsizes arXiv:2412.14291v1 Announce Type: cross Abstract: We present a novel class of projected gradient (PG) methods for minimizing a smooth but not necessarily convex function over a convex compact set. We first provide a novel analysis of the “vanilla” PG method, achieving the…

  • How to Stand Out as a Junior Data Scientist

    How to Stand Out as a Junior Data Scientist 7 things you can do to show your skills even if you have no experience at all Continue reading on Towards Data Science » Idit Cohen Go to original source

  • From Prototype to Production: Enhancing LLM Accuracy

    From Prototype to Production: Enhancing LLM Accuracy Implementing evaluation frameworks to optimize accuracy in real-world applications Image created by DALL-E 3 Building a prototype for an LLM application is surprisingly straightforward. You can often create a functional first version within just a few hours. This initial prototype will likely provide results that look legitimate and be…

  • Synthetic Control Sample for Before and After A/B Test

    Synthetic Control Sample for Before and After A/B Test Learn a simple way to use linear regression to create a synthetic control sample for your A/B test Continue reading on Towards Data Science » Gustavo R Santos Go to original source

  • Transform Customer Feedback into Actionable Insights with CrewAI and Streamlit

    Transform Customer Feedback into Actionable Insights with CrewAI and Streamlit Build an AI-powered app to analyze unstructured feedback, generate insightful reports, and create interactive visualizations Continue reading on Towards Data Science » Alan Jones Go to original source

  • 2024 Highlights: The AI and Data Science Articles That Made a Splash

    2024 Highlights: The AI and Data Science Articles That Made a Splash Feeling inspired to write your first TDS post before the end of 2024? We’re always open to contributions from new authors. And just like that, 2024 is (almost) in the books. It was a year of exciting transitions — both for the TDS team and, in…

  • Time-Reversible Bridges of Data with Machine Learning

    Time-Reversible Bridges of Data with Machine Learning arXiv:2412.13665v1 Announce Type: new Abstract: The analysis of dynamical systems is a fundamental tool in the natural sciences and engineering. It is used to understand the evolution of systems as large as entire galaxies and as small as individual molecules. With predefined conditions on the evolution of dy-namical…

  • jinns: a JAX Library for Physics-Informed Neural Networks

    jinns: a JAX Library for Physics-Informed Neural Networks arXiv:2412.14132v1 Announce Type: new Abstract: jinns is an open-source Python library for physics-informed neural networks, built to tackle both forward and inverse problems, as well as meta-model learning. Rooted in the JAX ecosystem, it provides a versatile framework for efficiently prototyping real-problems, while easily allowing extensions to…

  • Preconditioned Subspace Langevin Monte Carlo

    Preconditioned Subspace Langevin Monte Carlo arXiv:2412.13928v1 Announce Type: new Abstract: We develop a new efficient method for high-dimensional sampling called Subspace Langevin Monte Carlo. The primary application of these methods is to efficiently implement Preconditioned Langevin Monte Carlo. To demonstrate the usefulness of this new method, we extend ideas from subspace descent methods in Euclidean…

  • Adaptive Nonparametric Perturbations of Parametric Bayesian Models

    Adaptive Nonparametric Perturbations of Parametric Bayesian Models arXiv:2412.10683v2 Announce Type: cross Abstract: Parametric Bayesian modeling offers a powerful and flexible toolbox for scientific data analysis. Yet the model, however detailed, may still be wrong, and this can make inferences untrustworthy. In this paper we study nonparametrically perturbed parametric (NPP) Bayesian models, in which a parametric…

  • Deep Learning for Hydroelectric Optimization: Generating Long-Term River Discharge Scenarios with Ensemble Forecasts from Global Circulation Models

    Deep Learning for Hydroelectric Optimization: Generating Long-Term River Discharge Scenarios with Ensemble Forecasts from Global Circulation Models arXiv:2412.12234v1 Announce Type: cross Abstract: Hydroelectric power generation is a critical component of the global energy matrix, particularly in countries like Brazil, where it represents the majority of the energy supply. However, its strong dependence on river discharges,…

  • Introduction to TensorFlow’s Functional API

    Introduction to TensorFlow’s Functional API Learn what the Functional API is, and how to build complex keras models using it Continue reading on Towards Data Science » Javier Martínez Ojeda Go to original source

  • Classifier-Free Guidance in LLMs Safety — NeurIPS 2024 Challenge Experience

    Classifier-Free Guidance in LLMs Safety — NeurIPS 2024 Challenge Experience Classifier-Free Guidance in LLMs Safety — NeurIPS 2024 Challenge Experience This article briefly describes NeurIPS 2024 LLM-PC submission that was awarded the second prize — the approach to effective LLM unlearning without any retaining dataset. This is achieved through the formulation of the unlearning task as an alignment problem with the…

  • The Algorithm That Made Google Google

    The Algorithm That Made Google Google How PageRank transformed how we searched the internet, and why it’s still playing an important role in LLMs with Graph RAG. Continue reading on Towards Data Science » Cristian Leo Go to original source

  • 100 Years of (eXplainable) AI

    100 Years of (eXplainable) AI Reflecting on advances and challenges in deep learning and explainability in the ever-evolving era of LLMs and AI governance Image by author Background Imagine you are navigating a self-driving car, relying entirely on its onboard computer to make split-second decisions. It detects objects, identifies pedestrians, and even can anticipate behavior of…

  • Navigating Soft Actor-Critic Reinforcement Learning

    Navigating Soft Actor-Critic Reinforcement Learning Understanding the theory and implementation of SAC RL in the context of Bioengineering Image generated by the author using ChatGPT-4o Introduction The research domain of Reinforcement Learning (RL) has evolved greatly over the past years. The use of deep reinforcement learning methods such as Proximal Policy Optimisation (PPO) (Schulman, 2017)…

  • How to Choose a Threshold for an Evaluation Metric for Large Language Models

    How to Choose a Threshold for an Evaluation Metric for Large Language Models arXiv:2412.12148v1 Announce Type: new Abstract: To ensure and monitor large language models (LLMs) reliably, various evaluation metrics have been proposed in the literature. However, there is little research on prescribing a methodology to identify a robust threshold on these metrics even though…

  • Adversarially robust generalization theory via Jacobian regularization for deep neural networks

    Adversarially robust generalization theory via Jacobian regularization for deep neural networks arXiv:2412.12449v1 Announce Type: new Abstract: Powerful deep neural networks are vulnerable to adversarial attacks. To obtain adversarially robust models, researchers have separately developed adversarial training and Jacobian regularization techniques. There are abundant theoretical and empirical studies for adversarial training, but theoretical foundations for Jacobian…

  • BOIDS: High-dimensional Bayesian Optimization via Incumbent-guided Direction Lines and Subspace Embeddings

    BOIDS: High-dimensional Bayesian Optimization via Incumbent-guided Direction Lines and Subspace Embeddings arXiv:2412.12918v1 Announce Type: new Abstract: When it comes to expensive black-box optimization problems, Bayesian Optimization (BO) is a well-known and powerful solution. Many real-world applications involve a large number of dimensions, hence scaling BO to high dimension is of much interest. However, state-of-the-art high-dimensional…

  • Sequential Harmful Shift Detection Without Labels

    Sequential Harmful Shift Detection Without Labels arXiv:2412.12910v1 Announce Type: new Abstract: We introduce a novel approach for detecting distribution shifts that negatively impact the performance of machine learning models in continuous production environments, which requires no access to ground truth data labels. It builds upon the work of Podkopaev and Ramdas [2022], who address scenarios…

  • On Model Extrapolation in Marginal Shapley Values

    On Model Extrapolation in Marginal Shapley Values arXiv:2412.13158v1 Announce Type: new Abstract: As the use of complex machine learning models continues to grow, so does the need for reliable explainability methods. One of the most popular methods for model explainability is based on Shapley values. There are two most commonly used approaches to calculating Shapley…

  • The Anatomy of an Autonomous Agent

    The Anatomy of an Autonomous Agent A blueprint for autonomous agents in an Agentic Mesh ecosystem. Continue reading on Towards Data Science » Eric Broda Go to original source

  • Will Your Christmas Be White? Ask An AI Weather Model!

    Will Your Christmas Be White? Ask An AI Weather Model! Learn how to visualize AI weather and create your own forecast for the holidays Continue reading on Towards Data Science » Caroline Arnold Go to original source

  • 2024 in Review: What I Got Right, Where I Was Wrong, and Bolder Predictions for 2025

    2024 in Review: What I Got Right, Where I Was Wrong, and Bolder Predictions for 2025 What I got right (and wrong) about trends in 2024 and daring to make bolder predictions for the year ahead AI Buzzword and Trend Bingo (Image by the author) In 2023, building AI-powered applications felt full of promise, but the challenges…

  • The Invisible Bug That Broke My Automation: How OCR Changed The Game

    The Invisible Bug That Broke My Automation: How OCR Changed The Game The evolution of AI in test automation: from locators to generative AI (Part 3) Continue reading on Towards Data Science » Abdelkader HASSINE Go to original source

  • Four Career-Savers Data Scientists Should Incorporate into Their Work

    Four Career-Savers Data Scientists Should Incorporate into Their Work You might damage your data science career progress without even realising it — but avoiding that fate isn’t too difficult Continue reading on Towards Data Science » Egor Howell Go to original source

  • Generative Modeling with Diffusion

    Generative Modeling with Diffusion arXiv:2412.10948v1 Announce Type: new Abstract: We introduce the diffusion model as a method to generate new samples. Generative models have been recently adopted for tasks such as art generation (Stable Diffusion, Dall-E) and text generation (ChatGPT). Diffusion models in particular apply noise to sample data and then “reverse” this noising process…

  • Representation learning of dynamic networks

    Representation learning of dynamic networks arXiv:2412.11065v1 Announce Type: new Abstract: This study presents a novel representation learning model tailored for dynamic networks, which describes the continuously evolving relationships among individuals within a population. The problem is encapsulated in the dimension reduction topic of functional data analysis. With dynamic networks represented as matrix-valued functions, our objective…

  • Deep Learning-based Approaches for State Space Models: A Selective Review

    Deep Learning-based Approaches for State Space Models: A Selective Review arXiv:2412.11211v1 Announce Type: new Abstract: State-space models (SSMs) offer a powerful framework for dynamical system analysis, wherein the temporal dynamics of the system are assumed to be captured through the evolution of the latent states, which govern the values of the observations. This paper provides…

  • datadriftR: An R Package for Concept Drift Detection in Predictive Models

    datadriftR: An R Package for Concept Drift Detection in Predictive Models arXiv:2412.11308v1 Announce Type: new Abstract: Predictive models often face performance degradation due to evolving data distributions, a phenomenon known as data drift. Among its forms, concept drift, where the relationship between explanatory variables and the response variable changes, is particularly challenging to detect and…

  • Prediction-Enhanced Monte Carlo: A Machine Learning View on Control Variate

    Prediction-Enhanced Monte Carlo: A Machine Learning View on Control Variate arXiv:2412.11257v1 Announce Type: new Abstract: Despite being an essential tool across engineering and finance, Monte Carlo simulation can be computationally intensive, especially in large-scale, path-dependent problems that hinder straightforward parallelization. A natural alternative is to replace simulation with machine learning or surrogate prediction, though this…

  • Four Signs It’s Time to Leave Your Data Science Job

    Four Signs It’s Time to Leave Your Data Science Job Four tell-tale signs that you should look for another job Continue reading on Towards Data Science » Egor Howell Go to original source

  • A Case for Bagging and Boosting as Data Scientists’ Best Friends

    A Case for Bagging and Boosting as Data Scientists’ Best Friends Leveraging wisdom of the crowd in ML models. Continue reading on Towards Data Science » Farzad Nobar Go to original source

  • The Good, the Bad, An Ugly Memory for a Neural Network

    The Good, the Bad, An Ugly Memory for a Neural Network Memory can play tricks, to learn best it is not always good to memorize Continue reading on Towards Data Science » Salvatore Raieli Go to original source

  • USGS DEM Files: How to Load, Merge, and Crop with Python

    USGS DEM Files: How to Load, Merge, and Crop with Python A quick guide to prepping digital elevation data Continue reading on Towards Data Science » Lee Vaughan Go to original source

  • Structured LLM Output Using Ollama

    Structured LLM Output Using Ollama Control your model responses effectively Continue reading on Towards Data Science » Thomas Reid Go to original source

  • Langevin Monte Carlo Beyond Lipschitz Gradient Continuity

    Langevin Monte Carlo Beyond Lipschitz Gradient Continuity arXiv:2412.09698v1 Announce Type: new Abstract: We present a significant advancement in the field of Langevin Monte Carlo (LMC) methods by introducing the Inexact Proximal Langevin Algorithm (IPLA). This novel algorithm broadens the scope of problems that LMC can effectively address while maintaining controlled computational costs. IPLA extends LMC’s…

  • Investigating the Impact of Balancing, Filtering, and Complexity on Predictive Multiplicity: A Data-Centric Perspective

    Investigating the Impact of Balancing, Filtering, and Complexity on Predictive Multiplicity: A Data-Centric Perspective arXiv:2412.09712v1 Announce Type: new Abstract: The Rashomon effect presents a significant challenge in model selection. It occurs when multiple models achieve similar performance on a dataset but produce different predictions, resulting in predictive multiplicity. This is especially problematic in high-stakes environments,…

  • A Statistical Analysis for Supervised Deep Learning with Exponential Families for Intrinsically Low-dimensional Data

    A Statistical Analysis for Supervised Deep Learning with Exponential Families for Intrinsically Low-dimensional Data arXiv:2412.09779v1 Announce Type: new Abstract: Recent advances have revealed that the rate of convergence of the expected test error in deep supervised learning decays as a function of the intrinsic dimension and not the dimension $d$ of the input space. Existing…

  • DQA: An Efficient Method for Deep Quantization of Deep Neural Network Activations

    DQA: An Efficient Method for Deep Quantization of Deep Neural Network Activations arXiv:2412.09687v1 Announce Type: cross Abstract: Quantization of Deep Neural Network (DNN) activations is a commonly used technique to reduce compute and memory demands during DNN inference, which can be particularly beneficial on resource-constrained devices. To achieve high accuracy, existing methods for quantizing activations…

  • Matrix Completion via Residual Spectral Matching

    Matrix Completion via Residual Spectral Matching arXiv:2412.10005v1 Announce Type: new Abstract: Noisy matrix completion has attracted significant attention due to its applications in recommendation systems, signal processing and image restoration. Most existing works rely on (weighted) least squares methods under various low-rank constraints. However, minimizing the sum of squared residuals is not always efficient, as…

  • Weekly Entering & Transitioning – Thread 16 Dec, 2024 – 23 Dec, 2024

    Weekly Entering & Transitioning – Thread 16 Dec, 2024 – 23 Dec, 2024 Welcome to this week’s entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include: Learning resources (e.g. books, tutorials, videos) Traditional education (e.g. schools, degrees, electives) Alternative education (e.g.…

  • What projects are you working on and what is the benefit of your efforts?

    What projects are you working on and what is the benefit of your efforts? I would really like to hear what you guys are working on, challenges you’re facing and how your project is helping your company. Let’s hear it. submitted by /u/Firm-Message-2971 [link] [comments] /u/Firm-Message-2971 Go to original source

  • Data science is a luxury for almost all companies

    Data science is a luxury for almost all companies Let’s face it, most of the data science project you work on only deliver small incremental improvements. Emphasis on the word “most”, l don’t mean all data science projects. Increments of 3% – 7% are very common for data science projects. I believe it’s mostly useful…

  • Capital One Power Day for Data Scientist

    Capital One Power Day for Data Scientist Hi all, I have an upcoming Capital One Power Day interview for a Data Scientist role, and I was hoping to get some insights from those who have recently gone through the process. The day consists of 4 rounds: Stats Role Play Analyst Case Technical Interview Job Fit…

  • What’s the point of testing machine learning model knowledge during interviews for non-research data science roles?

    What’s the point of testing machine learning model knowledge during interviews for non-research data science roles? I always make an effort to learn how a model works and how it differs from other similar models whenever I encounter a new model. So it felt natural to me that these topics were brought up in interviews.…

  • Credit Card Fraud Detection with Different Sampling Techniques

    Credit Card Fraud Detection with Different Sampling Techniques How to deal with imbalanced data Photo by Bermix Studio on Unsplash Credit card fraud detection is a plague that all financial institutions are at risk with. In general fraud detection is very challenging because fraudsters are coming up with new and innovative ways of detecting fraud, so…

  • API Design of X (Twitter) Home Timeline

    API Design of X (Twitter) Home Timeline How X (Twitter) Designed Its Home Timeline API: Lessons to Learn A closer look at X’s API: fetching data, linking entities, and solving under-fetching. When designing a system’s API, software engineers often evaluate various approaches, such as REST vs RPC vs GraphQL, or hybrid models, to determine the best…

  • Bayes’ Theorem: Understanding business outcomes with evidence

    Bayes’ Theorem: Understanding business outcomes with evidence A practical introduction to Bayes’ Theorem: Probability for Data Science Series (2) Continue reading on Towards Data Science » Sunghyun Ahn Go to original source

  • Data Valuation — A Concise Overview

    Data Valuation — A Concise Overview Understanding the Value of your Data: Challenges, Methods, and Applications ChatGPT and similar LLMs were trained on insane amounts of data. OpenAI and Co. scraped the internet, collecting books, articles, and social media posts to train their models. It’s easy to imagine that some of the texts (like scientific or news…

  • A Design Researcher’s Guide to Publishing

    A Design Researcher’s Guide to Publishing A Guide to Publishing Human-Computer Interaction (HCI) and Design Research Papers Turn ‘publish or perish’ into ‘learn, write, and share’ When I first started my PhD three years ago, I was very new to the world of academia and the process of publishing in journals and conferences. Coming from Computer Engineering,…

  • How Have Data Science Interviews Changed Over 4 Years?

    How Have Data Science Interviews Changed Over 4 Years? An aggregated look on the differences between then & now: 2020 vs 2024 — some big frustrations and positive learnings. Continue reading on Towards Data Science » Matt Przybyla Go to original source

  • Master Machine Learning: 4 Classification Models Made Simple

    Master Machine Learning: 4 Classification Models Made Simple A Beginner’s Guide to Building Models in 15 Practical Steps Continue reading on Towards Data Science » Leo Anello Go to original source

  • Is Complex Writing Nothing But Formulas?

    Is Complex Writing Nothing But Formulas? Text analytics hints at how volumes of writing get created In the broadest of strokes, Natural Language Processing transforms language into constructs that can be usefully manipulated. Since deep-learning embeddings have proven so powerful, they’ve also become the default: pick a model, embed your data, pick a metric, do some…

  • Agentic AI: Building Autonomous Systems from Scratch

    Agentic AI: Building Autonomous Systems from Scratch A Step-by-Step Guide to Creating Multi-Agent Frameworks in the Age of Generative AI Continue reading on Towards Data Science » Luís Roque Go to original source

  • How I’d Learn AI in 2025 (If I Knew Nothing)

    How I’d Learn AI in 2025 (If I Knew Nothing) A 5-step roadmap for today’s landscape Today, more people than ever are trying to learn AI. Although there are countless free learning resources online, navigating this rapidly evolving landscape can be overwhelming (especially as a beginner). In this article, I discuss how I’d approach learning…

  • Efficient Large Dimensional Self-Organising Maps with PyTorch

    Efficient Large Dimensional Self-Organising Maps with PyTorch Because it’s fun to self-organise Continue reading on Towards Data Science » Mathieu d’Aquin Go to original source

  • Addressing the Butterfly Effect: Data Assimilation Using Ensemble Kalman Filter

    Addressing the Butterfly Effect: Data Assimilation Using Ensemble Kalman Filter Learn how to implement the Ensemble Kalman Filter for data assimilation, with mathematical details step-by-step code. Continue reading on Towards Data Science » Wencong Yang, PhD Go to original source

  • Transcriptions dataset

    Transcriptions dataset We teamed up with Miska Knapek to transcribe our 170 episodes into full written text — resulting in 1,539,957 spoken words overall, including 61 mentions of weather, 923 mentions of maps and 48 mentions of AI. Check out a little tour of the data, browse and search episodes using our brand new archive page, or, for the technically inclined, check out data and code…

  • GeoConformal prediction: a model-agnostic framework of measuring the uncertainty of spatial prediction

    GeoConformal prediction: a model-agnostic framework of measuring the uncertainty of spatial prediction arXiv:2412.08661v1 Announce Type: new Abstract: Spatial prediction is a fundamental task in geography. In recent years, with advances in geospatial artificial intelligence (GeoAI), numerous models have been developed to improve the accuracy of geographic variable predictions. Beyond achieving higher accuracy, it is equally…

  • On the Precise Asymptotics and Refined Regret of the Variance-Aware UCB Algorithm

    On the Precise Asymptotics and Refined Regret of the Variance-Aware UCB Algorithm arXiv:2412.08843v1 Announce Type: new Abstract: In this paper, we study the behavior of the Upper Confidence Bound-Variance (UCB-V) algorithm for Multi-Armed Bandit (MAB) problems, a variant of the canonical Upper Confidence Bound (UCB) algorithm that incorporates variance estimates into its decision-making process. More…

  • $(epsilon, delta)$-Differentially Private Partial Least Squares Regression

    $(epsilon, delta)$-Differentially Private Partial Least Squares Regression arXiv:2412.09164v1 Announce Type: new Abstract: As data-privacy requirements are becoming increasingly stringent and statistical models based on sensitive data are being deployed and used more routinely, protecting data-privacy becomes pivotal. Partial Least Squares (PLS) regression is the premier tool for building such models in analytical chemistry, yet it…

  • Belted and Ensembled Neural Network for Linear and Nonlinear Sufficient Dimension Reduction

    Belted and Ensembled Neural Network for Linear and Nonlinear Sufficient Dimension Reduction arXiv:2412.08961v1 Announce Type: new Abstract: We introduce a unified, flexible, and easy-to-implement framework of sufficient dimension reduction that can accommodate both linear and nonlinear dimension reduction, and both the conditional distribution and the conditional mean as the targets of estimation. This unified framework…