{"id":1814,"date":"2026-07-10T08:08:05","date_gmt":"2026-07-10T13:38:05","guid":{"rendered":"https:\/\/www.propellum.com\/blog\/?p=1814"},"modified":"2026-07-10T08:28:00","modified_gmt":"2026-07-10T13:58:00","slug":"real-time-job-feed-update-frequency","status":"publish","type":"post","link":"https:\/\/www.propellum.com\/blog\/real-time-job-feed-update-frequency\/","title":{"rendered":"How Fast Should a Job Feed Actually Update? And Why Getting It Wrong Is Costing Your Job Board."},"content":{"rendered":"\n

At some point, most job board operators ask their data provider the same question: how often does the feed actually update?<\/p>\n\n\n\n

The answer, when one comes back at all, is usually “daily,” “multiple times per day,” or some variant of “real-time.” None of those phrases tells you anything useful. Daily could mean once at 2 am. Multiple times could mean three. Real-time, in the job data industry, has come to mean whatever the provider needs it to mean to close the conversation.<\/p>\n\n\n\n

So the gap between what your feed is described as and what it actually does stays invisible, until a job seeker emails to say she applied for a role that closed four days ago, or a team member checks an employer’s career page and finds a listing your board is still showing was quietly removed last Tuesday.<\/p>\n\n\n\n

That is when the question changes. Not “how often does the feed update?” but “how fast should it actually update?” and what happens to your board when it doesn’t.<\/p>\n\n\n\n

The answer is not the same for every board. Getting it wrong produces two failure modes that look opposite but cost the same: too slow fills your board with dead listings, too fast triggers rate-limit blocks that stop listings arriving at all.<\/p>\n\n\n\n

Here is a practical framework for finding the right number, and four metrics that tell you whether you have.<\/p>\n\n\n\n

Start by Classifying Your Sources Into Three Tiers<\/strong><\/h2>\n\n\n\n

Not all sources behave the same way. A large employer posting<\/a> dozens of roles daily needs to be checked far more frequently than a small business that updates its careers page twice a month. Applying the same crawl frequency across all sources either wastes crawl budget on low-activity sources or underserves high-activity ones.<\/p>\n\n\n\n

High-volume sources<\/strong> post frequently, often daily. Large enterprise employers, staffing agencies, and prolific job boards fall here. These sources warrant continuous or near-continuous monitoring. At a 24-hour batch cycle, a new listing from a high-volume source can sit undetected for up to 12 hours on average. For a board competing on freshness, that gap is the margin a faster competitor exploits.<\/p>\n\n\n\n

Mid-volume sources<\/strong> post between 50 and 200 listings per month. Hourly or bi-hourly checks are appropriate here. More frequent than that, and you risk triggering rate-limit responses before the source has meaningfully changed. Less frequent and you accumulate freshness gaps that compound across a large source list.<\/p>\n\n\n\n

Low-volume sources<\/strong> post fewer than 50 listings per month. A 6 to 12 hour refresh cycle produces no meaningful freshness loss for these sources and keeps crawl infrastructure focused where it matters.<\/p>\n\n\n\n

What Too Slow Actually Costs<\/strong><\/h2>\n\n\n\n

At a 24-hour batch cycle, average detection latency per new listing is 12 hours. That is manageable for low-volume sources. For high-volume sources, it is the gap where a job seeker searches, finds nothing, and goes elsewhere.<\/p>\n\n\n\n

The expiry side is where the real cost compounds. At a 5% daily expiry rate across a 100,000 listing index, a 24-hour batch cycle means 5,000 dead listings visible to job seekers at any given moment. Each one is an application submitted to a role that no longer exists. Each one is a job seeker who does not come back.<\/p>\n\n\n\n

What Too Fast Actually Costs<\/strong><\/h2>\n\n\n\n

Crawling a high-volume source every two minutes produces rate-limit violations within hours. A blocked source does not slow down: it stops delivering listings entirely, which is a worse outcome than a slow refresh cycle. Overly aggressive crawling also generates duplicate records before deduplication runs, inflating listing counts and sending job seekers in circles. At a 5% duplicate rate across 100,000 listings, 5,000 listings are returning the same role under different entries.<\/p>\n\n\n\n

The right refresh cycle is not the fastest one you can run. It is the fastest one that the source can sustain without blocking.<\/p>\n\n\n\n

Four Metrics That Tell You Whether Your Feed Is Actually Working<\/strong><\/h2>\n\n\n\n

These four numbers give an accurate picture of feed health regardless of what your vendor’s marketing materials claim.<\/p>\n\n\n\n

Listing freshness rate.<\/strong> Target 85% or more of listings verified live within the last 48 hours. Below that threshold, a meaningful share of your index is unverified and potentially stale.<\/p>\n\n\n\n

Detection latency.<\/strong> Target under 30 minutes for high-volume sources. This measures the time between a new listing going live on the source and appearing on your board.<\/p>\n\n\n\n

Expiry lag.<\/strong> Target under 24 hours from role removal on the source to removal from your board. Longer than that and dead listings are accumulating faster than they are being cleared.<\/p>\n\n\n\n

Duplicate rate.<\/strong> Target under 3% across the full index. Above that, search quality degrades and job seekers start noticing the same role appearing multiple times.<\/p>\n\n\n\n

Running the Audit<\/strong><\/h2>\n\n\n\n

The most useful first step is measuring where these four numbers currently sit against your existing feed setup. Most boards have not measured them explicitly, which means the degradation has been invisible until a job seeker or employer raises it.<\/p>\n\n\n\n

Propellum<\/a> works with job boards to run exactly this audit against their current configuration, identifying which source tier is causing the most freshness degradation and what the right refresh cycle per source type looks like in practice. The starting point is a sample feed from your target sources, delivered within 24 hours, so you can see what your four metrics look like against a production-grade continuous pipeline before committing to anything.<\/p>\n\n\n\n

Self-Audit Checklist<\/strong><\/h2>\n\n\n\n

Before your next vendor conversation, run these four checks against your current feed:<\/p>\n\n\n\n

What percentage of your listings have been verified live within the last 48 hours? Target: 85% or above.<\/p>\n\n\n\n

What is the average time between a new listing appearing on a source and appearing on your board? Target: under 30 minutes for high-volume sources.<\/p>\n\n\n\n

How long does it take a closed role to disappear from your board after the source removes it? Target: under 24 hours.<\/p>\n\n\n\n

What percentage of your index are duplicates of the same listing from different sources? Target: under 3%.<\/p>\n\n\n\n

If any of these numbers are unknown, that is the first finding.<\/p>\n\n\n\n

Start with a free test feed. See what your freshness metrics look like on a real pipeline \u2192<\/strong><\/a><\/p>\n\n\n\n

Frequently Asked Questions<\/strong><\/h2>\n\n\n\n
How often should a job feed refresh?<\/strong><\/strong>

It depends on the source tier. High-volume sources warrant continuous or near-continuous monitoring. Mid-volume sources need hourly or bi-hourly checks. Low-volume sources can sustain a 6 to 12 hour refresh cycle without meaningful freshness loss. Applying the same rate across all tiers wastes crawl budget or creates freshness gaps where they matter most.<\/p> <\/div>

What is a real-time job feed?<\/strong><\/strong>

A job feed that reflects changes at the source within minutes rather than hours, through continuous crawling of source pages rather than scheduled batch pulls. The term is used loosely across the industry, so the practical test is measuring detection latency directly: how long between a listing going live at the source and appearing on your board.<\/p> <\/div>

What is the ideal job feed update frequency?<\/strong><\/strong>

For high-volume sources, as close to continuous as the source will allow without triggering rate-limit responses. For mid-volume sources, hourly. For low-volume sources, every 6 to 12 hours. The right frequency is the fastest sustainable cycle per source tier, not a single universal rate applied across the full index.<\/p> <\/div>

How does job feed speed affect job board performance?<\/strong><\/strong>

In two directions. Too slow and listings arrive late, expire without being removed, and job seekers encounter dead applications. Too fast and rate-limit violations block sources entirely, which is worse than a slow cycle. Feed speed affects listing count accuracy, search freshness, candidate trust, and the board’s competitive position against faster alternatives.<\/p> <\/div> <\/div>\n\n\n