Tag: pipelines
-
LLMs for data pipelines without losing control (API → DuckDB in ~10 mins)
LLMs for data pipelines without losing control (API → DuckDB in ~10 mins) Hey folks, I’ve been doing data engineering long enough to believe that “real” pipelines meant writing every parser by hand, dealing with pagination myself, and debugging nested JSON until it finally stopped exploding. I’ve also been pretty skeptical of the “just prompt…
-
Unlocking the Untapped Potential of Retrieval-Augmented Generation (RAG) Pipelines
Unlocking the Untapped Potential of Retrieval-Augmented Generation (RAG) Pipelines Essential Metrics and Methods to Enhance Performance Across Retrieval, Generation, and End-to-End Pipelines Continue reading on Towards Data Science » Saleh Alkhalifa Go to original source
-
ML pipeline questions
ML pipeline questions I am building an application that processes videos and that needs to run many tasks (some need to be sequentially and some in parallel). Think audio extraction, ASR, diarization, translation, video classification, etc… Note that this is in supposed to be run online, i.e. this is supposed to be used in a…