Skip to content

fix(scheduler): stop retaining retrieval payloads across mem_update b… - #2402

Open
larryluozhang wants to merge 1 commit into
MemTensor:mainfrom
larryluozhang:upstream-pr/scheduler-mem-leak
Open

larryluozhang wants to merge 1 commit into
MemTensor:mainfrom
larryluozhang:upstream-pr/scheduler-mem-leak

Conversation

@larryluozhang

Copy link
Copy Markdown

Problem

Each mem_update batch retrieves candidate memories carrying full embeddings
(~33KB/item as Python float lists) plus long source payloads. These result
lists stay referenced by the long-lived scheduler thread frames, so they
accumulate indefinitely.

Production evidence (tracemalloc + objgraph, RSS 4.98GB snapshot):

1832.4MB  n=900   memory_update_handler.py:101 (process_session_turn results)
1547.0MB  n=191   (same chain)

~3.4GB retained across ~900 batches → RSS grows ~660MB/h → watchdog restarts
(10-90/day in our deployment).

Fix

  • Strip metadata.embedding from candidates immediately after retrieval —
    vectors are retrieval artifacts; the working-memory replacement decision
    only uses text.
  • del batch references + gc.collect() after replace_working_memory,
    because scheduler threads are long-lived and locals otherwise linger.

No behavior change to outputs; purely memory hygiene.

…atches

Added garbage collection to free memory and prevent leaks in long-lived scheduler threads. Removed unnecessary embeddings from search results to optimize memory usage.
@Memtensor-AI Memtensor-AI added area:scheduler 调度模块 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Sep 22, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2402
Task: 73f8fc2583d8d6f5
Base: main
Head: upstream-pr/scheduler-mem-leak

🔍 OpenCodeReview found 3 issue(s) in this PR.


1. src/memos/mem_scheduler/task_schedule_modules/handlers/memory_update_handler.py (L126-L129)

Bug (critical): cur_working_memory is used after being del-eted, causing a NameError at runtime.

After del cur_working_memory, new_candidates on line 126, the code on lines 137, 151, and 152 references cur_working_memory again (to build old_memory_texts and pass it to logger.info). This will raise a NameError: name 'cur_working_memory' is not defined every time this path is executed.

Suggestion: Move the del statement to after the logger.info call that still needs cur_working_memory, or capture the values needed for logging before deleting.

# Build log strings before dropping references
old_memory_texts = "\n- " + "\n- ".join(
    [f"{one.id}: {one.memory}" for one in cur_working_memory]
)
old_memory_count = len(cur_working_memory)

# Drop batch references explicitly
del cur_working_memory, new_candidates
import gc
gc.collect()

logger.info(
    "...",
    old_memory_count,
    old_memory_texts,
    ...
)

2. src/memos/mem_scheduler/task_schedule_modules/handlers/memory_update_handler.py (L127-L129)

Style/Maintainability: import gc is placed inside the function body, mid-logic.

import gc should be at the top of the module alongside other imports, not embedded inline between del and gc.collect(). While Python caches repeated imports, placing it here reduces readability and is against standard conventions (PEP 8). Move it to the module-level import section.


3. src/memos/mem_scheduler/task_schedule_modules/handlers/memory_update_handler.py (L286-L291)

Bug/correctness: Silently swallowing exceptions in the embedding-strip loop hides unexpected errors.

The bare except Exception: pass discards all exceptions without logging. If _r.metadata raises an unexpected error (e.g., a property getter that raises, or a read-only attribute), the failure will be invisible. At minimum, log the exception at DEBUG or WARNING level so production incidents can be diagnosed.

for _r in results:
    try:
        if getattr(_r.metadata, "embedding", None):
            _r.metadata.embedding = None
    except Exception as exc:
        logger.debug("Failed to clear embedding on result %s: %s", getattr(_r, 'id', _r), exc)

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

✅ Automated Test Results: PASSED

All tests passed (2/2 executed). memos_python_core/changed-python-source: 2/2. Duration: 8s [advisory, non-gating] AI-generated tests on branch test/auto-gen-73f8fc2583d8d6f5-20260922090738: 47/48 passed, 1 failed — these do NOT affect the PR verdict; review the branch manually.

Branch: upstream-pr/scheduler-mem-leak

@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scheduler 调度模块 status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants