How would you structure a project (data frame) to scrape and track listing changes over time?
\n \t
\n
<\/BR>
\n
\n \t
\n
<\/BR><\/p>\n
I\u2019m working on a project where I want to scrape data daily (e.g., real estate listings from a site like RentFaster or Zillow) and track how each listing changes over time. I want to be able to answer questions like:<\/p>\n
When did a listing first appear? How long did it stay up? What changed (e.g., price, description, status)? What\u2019s new today vs yesterday?<\/p>\n
My rough mental model is: 1. Scrape today\u2019s data into a CSV or database. 2. Compare with previous days to find new\/removed\/updated listings. 3. Over time, build a longitudinal dataset with per-listing history (kind of like slow-changing dimensions in data warehousing).<\/p>\n
I\u2019m curious how others would structure this kind of project:<\/p>\n
How would you handle ID tracking if listings don\u2019t always have persistent IDs? Would you use a single master table with change logs? Or snapshot tables per day? How would you set up comparisons (diffing rows, hashing)? Any Python or DB tools you\u2019d recommend for managing this type of historical tracking?<\/p>\n
I\u2019m open to best practices, war stories, or just seeing how others have solved this kind of problem. Thanks!<\/p>\n<\/p><\/div>\n
submitted by \/u\/Proof_Wrap_2150 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span>\n<\/div>\n