Contents

Jev-Mem is an open-source memory framework that separates organizing and retrieving AI agent memories from reasoning about answers. The paper’s LoCoMo evaluation reports results that outperform the compared systems in both answer quality using GPT-4o-mini and retrieval speed.

Key Points#

  • A lightweight System One manages and retrieves memories, while the System Two language model produces the final answer.
  • It preserves observations and their sources, retrieving information through four types of relationships: semantic, temporal, causal, and entity.
  • The LoCoMo evaluation reports an answer quality score of 0.777, memory construction in 158 seconds, and an average response time of 0.93 seconds.
  • The offline demo requires no API keys, but verifying the paper’s performance requires real data and models.

What the Tool Does#

As conversation histories grow, finding relevant information becomes more costly. Jev-Mem does more than store observations: it controls which information to connect, where to search for a given question, and when to stop searching.

Memory is represented as four graphs: semantic, temporal, causal, and entity. For example, the system is designed to answer “when did it happen?” by following temporal relationships and “why did it happen?” by following causal relationships. It also preserves the original observations, timestamps, and sources.

How It Works and Retrieves Information#

When a new observation arrives, Jev determines its memory type and relationships to existing information. Conventional code handles links that can be determined programmatically, such as timestamp ordering. During retrieval, vector and keyword searches gather candidates, then the system selects and explores graphs suited to the question.

During retrieval, it also assesses whether the evidence found is sufficient. It stops when there is enough, explores further when needed, and passes the selected evidence to System Two to generate an answer. Configuration and execution traces let you inspect where it searched, how much it explored, and why it stopped.

Examining the Paper’s Results#

The LoCoMo evaluation presented in the paper uses GPT-4o-mini as the answer model. Its overall score was 0.777, compared with MAGMA’s 0.700. Memory construction took 158 seconds, less than the 1,044 seconds required by Nemori, the fastest comparison system. Average response time was 0.93 seconds, below MAGMA’s 1.47 seconds.

By question category, Jev-Mem achieved the highest score in four of five categories. MAGMA performed better in the temporal category. These measurements were reported in the paper under matched dataset, model, and scoring conditions. Your own use case may not show the same differences.

Try It Offline First#

Python 3.11 or later is required. After cloning the repository, create a virtual environment, install the dependencies, and launch the demo.

snippet.txt
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m jev_mem.demo

This demo uses synthetic data and deterministic mock processing. It requires neither API keys nor model downloads and lets you inspect the memory and retrieval flow and decision traces, but it does not measure answer quality. On Windows, use .venv\Scripts\Activate.ps1 to activate the virtual environment.

Considerations When Using Real Data#

To use real models, set TYPESAFE_API_KEY and OPENAI_API_KEY in the .env file at the repository root. Jev makes memory decisions, while an OpenAI-compatible model handles answer generation and evaluation. The default embedding method, minilm, downloads a model on first use.

To run decision-making locally, you can also use Laya for System One. Install it with pip install '.[laya]'; Laya’s decisions require no TypeSafe API key. However, answer generation and evaluation still use the configured model provider, and model weights are downloaded on first use. The paper’s reported results use Jev; it does not include benchmark results for Laya.

During execution, text is sent to the configured providers, and usage charges may apply. Before using real data, review how secret keys, conversation data, and generated caches are handled.

Next Steps#

First, run the offline demo from GitHub and inspect the decision traces to see how memories are added and retrieved. Then, if you want to try your own data, check where it will be sent and what it will cost before configuring .env.

Source: https://github.com/libingzheren/Jev-Mem

Contents