mailitics

Tag: datasets

  • A memory effecient TF-IDF project in Python to vectorize datasets large than RAM

    A memory effecient TF-IDF project in Python to vectorize datasets large than RAM Re-designed at C++ level, this library can easily process datasets around 100GB and beyond on as small as a 4GB memory It does have its constraints but the outputs are comparable to sklearn’s output fasttfidf submitted by /u/mrnerdy59 [link] [comments] /u/mrnerdy59 Go…

  • Has anyone tried training models on raw discussions instead of curated datasets?

    Has anyone tried training models on raw discussions instead of curated datasets? I’ve always followed the usual advice when training models, like clean the data, normalize everything, remove noise, structure it nicely Recently I tried something different. Instead of polished datasets, I fed models long, messy discussion threads, real conversations, people arguing, correcting themselves, misunderstanding…

  • Collaborative Prediction: To Join or To Disjoin Datasets

    Collaborative Prediction: To Join or To Disjoin Datasets arXiv:2506.11271v1 Announce Type: new Abstract: With the recent rise of generative Artificial Intelligence (AI), the need of selecting high-quality dataset to improve machine learning models has garnered increasing attention. However, some part of this topic remains underexplored, even for simple prediction models. In this work, we study…

  • Pandas Can’t Handle This: How ArcticDB Powers Massive Datasets

    Pandas Can’t Handle This: How ArcticDB Powers Massive Datasets Python has grown to dominate data science, and its package Pandas has become the go-to tool for data analysis. It is great for tabular data and supports data files of up to 1GB if you have a large RAM. Within these size limits, it is also…

  • Harmonizing and Pooling Datasets for Health Research in R

    Harmonizing and Pooling Datasets for Health Research in R R code to extract data from unique datasets and combine them in one harmonized dataset ready for seamless analysis Continue reading on Towards Data Science » Rodrigo M Carrillo Larco, MD, PhD Go to original source