fix: ensure navigation sidebar serves fresh data after course publish

[wgu-taylor-payne]

Description

After a course publish in Studio, the navigation sidebar (CourseNavigationBlocksView) can serve stale block structure data for up to 1 hour. This happens because the sidebar cache key previously used course_version (which changes immediately on publish), causing a cache miss during the ~30s window before the async block structure rebuild task completes. The view would read the still-stale block structure, cache that stale result under the new key for 1 hour, and serve stale data indefinitely.

Fix: Replace course_version with block_structure_version in the navigation sidebar cache key. This version is derived from BlockStructureModel.data_version and only changes when the block structure rebuild task actually completes. Additionally, get_block_structure_version() never populates the cache — only the writer (_update_block_structure_version, called after a successful rebuild) sets the cached value. This eliminates a race condition where concurrent readers could permanently cache a stale version.

Impacted user roles: Learner (sees correct course outline sooner after a publish), Course Author (changes reflected faster).

Fixes openedx/wg-build-test-release#600

How it works

1. update_course_in_cache rebuilds the block structure, then calls _update_block_structure_version() which reads the new data_version from BlockStructureModel and sets it in cache with timeout=None
2. get_block_structure_version() reads from cache; on a miss it falls back to the database without caching the result — the cache is populated exclusively by the writer after a successful rebuild
3. CourseNavigationBlocksView uses block_structure_version in its cache key instead of course_version
4. The cache key only changes when data is actually ready — no more caching stale data for 1 hour

Why not synchronous rebuild on the request path?

The initial approach called update_collected_if_needed() on the GET path. As noted in review, block structure collection for large courses can exceed 30s and risks a stampede under concurrent load. The version-based approach avoids any expensive work on the request path.

Performance impact

• Every request (cache hit): One additional cache.get() call to read block_structure_version.
• Every request (cache miss, during rebuild window): One database call - BlockStructureModel.objects.get() by primary key
• Cache miss on sidebar (version bumped): Same as before — get_course_outline_block_tree() runs and result is cached for 1 hour
• During 30s rebuild window: Old sidebar cache entries continue to serve (consistent pre-publish data, same version key)
• No stampede risk: Version only bumps once the rebuild is complete

Deploy considerations

BlockStructureModel.data_version stores the same value as course.course_version (set from root_block.course_version at rebuild time). Since the sidebar cache key substitutes one for the other, existing cached sidebar entries remain warm — the key resolves to the same string.

The block_structure_version cache key itself won't exist until the first post-deploy publish per course. Until then, get_block_structure_version() falls through to a single PK database lookup on BlockStructureModel.

Supporting information

This issue was discovered testing an internal Open edX instance. After deleting a unit in Studio and refreshing the course in the learning MFE within a few seconds, the deleted unit remained visible in the sidebar for up to 1 hour.

The root cause involves three components:

• 30s Celery task delay for block structure rebuild (configured here)
• Navigation cache key using eagerly-changing course_version
• 1-hour cache TTL on the stale response

Testing instructions

Automated:

tutor dev run lms pytest \
  lms/djangoapps/course_home_api/outline/tests/test_view.py::SidebarBlocksTestViews::test_navigation_does_not_cache_stale_data_after_publish \
  --ds=lms.envs.test --no-migrations -x

Manual (requires async Celery):

1. Open a course in the learning MFE, note a unit in the sidebar
2. In Studio, delete that unit (auto-publishes)
3. Immediately refresh the course in the MFE
4. Wait 30+ seconds, refresh again
5. Verify the deleted unit is no longer in the sidebar

Without the fix: unit remains visible for up to 1 hour after step 2.
With the fix: unit disappears after step 4 (once the rebuild task completes).

Deadline

None

Other information

• No database migrations
• No new dependencies
• The block_structure_version cache key is set with timeout=None (no expiry) by the writer. Readers never populate the cache, eliminating the race condition entirely.
• If the cache is flushed, readers gracefully fall back to the database until the next rebuild populates the cache.

AI Usage

Used Kiro to aid in root cause discovery, evaluate multiple fix approaches (synchronous rebuild, TTL-based self-healing, writer-only cache ownership), debug the race condition via log analysis, and draft this PR description.

[openedx-webhooks]

Thanks for the pull request, @wgu-taylor-payne!

This repository is currently maintained by @openedx/wg-maintenance-openedx-platform.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[ormsbee]

I don't think this is operationally feasible for large courses and high traffic. Let's talk more about other possible mitigations.

[ormsbee]

Going through the collection phase of a large course can be extremely expensive, which is why it's done asynchronously in celery tasks or management commands (it can often exceed the 30s timeout that many sites use for giving up on web worker requests). Placing it in the GET here also risks causing a stampede if it is a popular course that many concurrent users are trying to access, as parallel workers try to recompute the same collection phase data.

[wgu-taylor-payne]

Thank you, I appreciate this feedback. I'll look into another way of preventing the stale cache.

[wgu-taylor-payne]

@ormsbee I've pushed a new approach where instead of caching on the course version number, we cache on a block structure version which is updated after the block structure has been updated. We keep this block structure version in the cache for each course. I've updated the PR description with more details on this. Any thoughts on this approach?

[ormsbee]

For instance, I think a course's navigation being incorrect for a minute after a deletion is a bad, but not necessarily release-blocking bug (FYI @crathbun428 and @jmakowski1123, who can weigh in here). If the wrong navigation is getting cached for an hour, then maybe that's the part that we should focus on for this fix.

[jmakowski1123]

I agree with Dave, I would not classify a 30-60sec cache as a blocker. But an hour is a bigger problem.

[ormsbee]

This is going to be '' when this is first deployed to a site, right (e.g. right after a version upgrade, looking at already-published courses). Why use this instead of the data_version field that's on the BlockStructureModel?

[wgu-taylor-payne]

Thanks for pointing that out; I agree that it would be a better value to use for the cache keys. I've updated my code to use it.