Advice on processing ~1M jobs\/month with LLaMA for cost savings
\n \t
\n
<\/BR>
\n
\n \t
\n
<\/BR><\/p>\n
I’m using GPT-4o-mini to process ~1 million jobs\/month. It’s doing things like deduplication, classification, title normalization, and enrichment.<\/p>\n
This setup is fast and easy, but the cost is starting to hurt. I’m considering distilling this pipeline into an open-source LLM, like LLaMA 3 or Mistral, to reduce inference costs, most likely self-hosted on GPU on Google Coud. <\/p>\n
Questions:<\/p>\n
* Has anyone done a similar migration? What were your real-world cost savings (e.g., from GPT-4o to self-hosted LLaMA\/Mistral)<\/p>\n
* Any recommended distillation workflows? I’d be fine using GPT-4o to fine-tune an open model on our own tasks.<\/p>\n
* Are there best practices for reducing inference costs even further (e.g., batching, quantization, routing tasks through smaller models first)?<\/p>\n
* Is anyone running LLM inference on consumer GPUs for light-to-medium workloads successfully?<\/p>\n
Right now, our GPT-4o-mini usage is costing me thousands\/month (I’m paying for it out of pocket, no investors). Would love to hear what\u2019s worked for others!<\/p>\n<\/p><\/div>\n
submitted by \/u\/hamed_n <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span>\n<\/div>\n